Clinicians’ perspectives on the design of early warning systems presenting AI-based multiple outputs for clinical deterioration: A qualitative analysis

Clinicians’ perspectives on the design of early warning systems presenting AI-based multiple outputs for clinical deterioration: A qualitative analysis | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Clinicians’ perspectives on the design of early warning systems presenting AI-based multiple outputs for clinical deterioration: A qualitative analysis Monica Noselli, Anton Van Der Vegt, Maxime Cordeil, Victoria Campbell, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8071546/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background AI-enabled clinical decision support systems (CDSS) can help detect patient deterioration, but their value depends on outputs that fit workflows, support reasoning, and avoid alert fatigue. Objective. Little is known about clinician perspectives on multi-output Early Warning Systems (MO-EWS), which provide simultaneous predictions (e.g., overall deterioration, organ- or diagnosis-specific). This study explores clinician views on future MO-EWS and offers user-centred recommendations to improve design, usability, and adoption. Methods Reflexive thematic analysis was used to examine clinicians’ perceptions of existing early warning systems (EWS) and envisioned AI-based tools for predicting clinical deterioration. Between January and June 2025, 22 clinicians from Australian hospitals with experience in managing clinical deterioration were interviewed using a semi-structured questionnaire. Resulting themes were mapped to system components (user interface [UI], AI model, clinical workflow) to inform practical design recommendations which were then assessed for their feasibility. Results Participants found existing EWS distracting and sometimes unsafe. They envisioned AI-based systems that provide multiple, contextualised outputs (e.g., causes of deterioration, organ dysfunction, clinical severity) as layered, adaptable tools to support reasoning, prioritisation, and communication. While many UI improvements are readily achievable, model development is more complex, and workflow integration will depend on infrastructure, policies, and clinician preferences. Conclusion Clinicians perceive potential benefits in an AI-enabled system that provides more insightful information when predicting clinical deterioration in individual patients, but various design and implementation challenges remain. Medical Informatics Software Engineering Artificial Intelligence and Machine Learning Artificial Intelligence Clinical decision support system Human Factors Qualitative analysis Figures Figure 1 Figure 2 1. Introduction As artificial intelligence (AI) continues to be integrated into clinical decision support systems (CDSS) [ 1 ], there is growing interest in how AI-generated predictions can assist clinicians to identify patient deterioration thereby enhancing decision-making, facilitating early intervention and supporting resource allocation [ 2 ]. While risk prediction algorithms are getting more accurate [ 3 ], their impact on clinical decision making and care outcomes depends on the actionable content of the CDSS outputs and how these are presented to clinicians [ 4 ]. Poorly designed CDSS user interfaces can lead to poor utility, misinterpretation, alert fatigue and non-adoption of the system [ 5 ]. Previous AI-based CDSS (AI-CDSS) user interface research focused mostly on single condition or single alert scores (e.g., sepsis or general deterioration only) [ 6 ] [ 7 ] [ 8 ] [ 9 ] [ 10 ]. These studies identified important design factors. For example, Blythe et al [ 8 ] explored how early warning scores (EWS) influence clinicians' decision-making, highlighting the importance of models that facilitate but not replace critical thinking, provide transparency regarding included and excluded variables, and support decision-making processes. Owoyemi et al. [ 9 ] examined user experiences with the Epic Sepsis System (ESS) and identified challenges with workflow fit and usability. A second study of the ESS [ 10 ] proposed that supporting clinicians in hypothesis generation and information gathering was more important than providing a final diagnosis or decision. AI-CDSS user-interface investigators have also found that successful implementation of AI-based tools requires understanding clinicians’ cognitive needs and framing AI as complementary support [ 11 ]. In all these investigations, a singular AI-based CDSS tool was considered. However, as more AI-based predictive models are researched and developed, it is likely that future AI-CDSS deterioration systems will encompass multiple predictive models working in concert. Currently, there is very limited research and understanding of how multiple concurrent AI-based models should present their outputs within a single AI-CDSS to support clinician decision-making. The purpose of this study is to investigate clinicians’ views on how multiple AI-based predictions could and should be presented within a deterioration focused AI-CDSS. The purpose of this investigation is to better understand (i) the issues that clinicians have with their current deterioration early warning tools that they would like to see resolved in future solutions; and (ii) clinician preferences with respect to the scope and design of future multi-prediction AI-based deterioration CDSS. This study applies a clinician-centric lens to identify actionable insights for system developers, co-development teams and researchers seeking to build multiple AI-based prediction CDSS that are both more effective with high levels of clinical adoption. 2. Methods 1.1. Participants and recruitment Between January and June 2025, clinicians experienced in managing deterioration were recruited via email containing a flyer and participant information sheet. These were distributed through professional networks including two clinical research groups (AI-KEYS: Artificial Intelligence – Keeping You Safer, and Queensland Health Sepsis group). Contacted participants were invited to forward the call for participation email to extend recruitment reach. Recruitment was initially aimed at ward-based staff but expanded to include clinicians from Intensive Care Unit (ICU), Emergency Department (ED), and Post-Anaesthesia Care Unit (PACU) in order to capture a broader range of perspectives across settings where early warning systems are used. Participation was voluntary with consent. Recruitment was guided by the principle of informational power [ 12 ], given the narrow and clearly defined study aim, the participants’ high level of clinical expertise, and the anticipated richness of dialogue, a relatively modest sample (22 participants) was considered sufficient. Recruitment concluded once the available networks had been fully approached and no additional eligible participants could be identified. No prior relationship existed between the researcher and participants. Ethical approval was obtained from the [name deleted to maintain the integrity of the review process]. 1.2. Data collection Researcher [name deleted to maintain the integrity of the review process], a female PhD student, conducted 45-minute one-on-one (single sessions only, without follow-up) Microsoft Teams interviews, using a semi-structured questionnaire, constructed de novo with all authors, that consistently captured: (a) participants’ clinical roles and settings; (b) challenges in using current EWT for identifying clinical deterioration; and (c) perceptions of a novel MO-EWS. Two pilot interviews with two clinician co-authors helped refine clarity, relevance and suitability of the questions. Participants gave verbal consent to have interviews recorded for analysis using Microsoft Teams. Participants did not comment or correct the transcripts, they were invited to provide feedback on the findings, however, no responses were received. 1.3. Analysis The study was conducted in three phases (Fig. I). First, following Braun and Clarke’s approach to using Reflexive Thematic Analysis (RTA) (see Box) [ 16 , 17 ]. Author [name deleted to maintain the integrity of the review process] engaged iteratively with transcripts, moving recursively between coding, theme development, review, and writing. Using NVivo 14 software [ 15 ], [name deleted to maintain the integrity of the review process] coded semantic and latent content, initially distinguishing 'actual' and 'envisioned' tools. However, since participants often discussed envisioned tool through the lens of their experience with existing tools, this overlap shaped the final coding. While some codes aligned with frameworks like technology acceptance model (TAM) [ 16 ], these limited reflexive analysis, so different themes were developed. Second, the final themes were then mapped to system components comprising (a) UI: what clinicians see and interact with; (b) AI algorithms that analyse data, make predictions, or generate alerts; (c) Workflow: the sequence of tasks and processes clinicians undertake when using the system in practice. Third, this mapping was used to derive practical design recommendations which were then assessed for their feasibility by consensus of all authors who comprised the main author [name deleted to maintain the integrity of the review process], two senior clinicians and system analysts [name deleted to maintain the integrity of the review process], a senior data scientist [name deleted to maintain the integrity of the review process] and a senior social scientist [name deleted to maintain the integrity of the review process]. In contrast to prior studies. we did not employ or demonstrate any tool prototypes to avoid cueing or biasing effects on clinicians in exploring these issues. This study is reported according to the COREQ (COnsolidated criteria for REporting Qualitative research) Checklist [ 17 ]. Figure I Study goals and processes 3. Results 2.1. Participants Twenty-two clinicians participated in interviews, comprising 11 nurses and 11 doctors, working in different departments across hospitals (ED = 8, ICU = 6, Ward = 5, PACU = 2, ICU + ED = 1) throughout Australia and all having a minimum 4 years of postgraduate experience. Participants mentioned various currently used clinical deterioration alert tools, including Queensland-Adult Deterioration Detection System (Q-ADDS) and Between the Flags (BTF) [ 18 ], and downstream systems that report these alerts (Compass). They also worked with different electronic medical record (EMR) systems, such as Sunrise, Oracle Cerner, and the Cerner module FirstNet used in EDs. 2.2. Themes The themes derived from interviews (Fig. II) are discussed in more detail in the following sections. Consistency between data and findings was ensured and illustrative participant quotes, identified using numbers and letters in parentheses in the following text, are provided in the supplementary material (Online Resource 1). Figure II Themes and subthemes derived from interviews on clinicians’ perceptions and expectations 2.2.1. Theme 1 - It cried wolf too many times and sometime not at all Participants described how frequent, redundant, and poorly calibrated alerts (Table S1: Q-1 to Q-4) lead to alert burden and fatigue and eventual rejection of the alerting system through desensitisation, including when clinical intuition contradicts algorithmic outputs (Q-11). The effective use of EWTs is shaped by a complex interplay of data quality, system design, clinical context and human judgment. As P10-D cautions, even the most sophisticated AI model is “only as good as the data you feed into it”, which can be subjective (Q-5), inconsistent across settings (Q-6), have long time gaps between nursing observations (Q-7), and contain omissions or entry errors (Q-8, Q-9). Among design factors, rigid criteria embedded into systems can introduce bias and cognitive error (Q-10). Threshold-based triggers can miss deteriorations happening below the cut-off point (Q-11) and patient-specific thresholds and delta calculations, not incorporated into EWTs, also influence decisions (Q-12, Q-13). Human factors such as both over-reliant and overly cautious response to outputs can lead to missed deterioration in patients requiring urgent review (Q-14). Vital signs automatically plotted by the computer may cause clinicians to forego the cognitive act of processing data that makes explicit their understanding of the trends (Q-15). We grouped both false positives (Q-1/10) and false negatives (Q-11/15) under this theme. Although opposite in form, participants perceived them as part of the same reliability problem: a system that either alarms unnecessarily or fails to alarm when needed ultimately reduces clinicians’ trust in its outputs 2.2.2. Theme 2 - Hunting for answers in a fractured system Poor UI layouts, fragmented workflows, or excessive alerts increase cognitive load and slow responses. Participants reported frustration from information being often dispersed across multiple pages or tabs, obscuring temporal trends and forcing them to mentally integrate disparate data sources (Table S2. Q-16). While some graphical displays offer trend visualization, inability to cross-reference trends directly with related clinical documentation disrupts workflow. Accessibility is insufficient if information is not presented in ways that align with clinicians’ mental models (Q-17). Icon-heavy dashboards and copy-pasted documentation bring noise, information overload and reduced visibility of critical clinical indicators (Q-18). Even with interactive functions, information may be updated in different pages or tabs, diminishing their utility for dynamic patient monitoring (Q-19). Clinicians disliked systems that are not easily searchable, wanting functionality that retrieves pertinent information with a minimal number of steps (Q-20). Moreover, multiple, non-integrated platforms and data sources, combined with limited ability to extract actionable meaning from disjointed or cluttered interfaces, all foster redundancy, inefficiency, and cognitive strain, diverting time from patient care (Q-21, Q-22). 2.2.3. Theme 3 - Ghost in the Interface This theme captures clinicians’ vision of AI as an intelligent collaborator delivering actionable, context-aware outputs aligned with reasoning modes, workflow, and expertise. Interfaces should be layered, drillable, and adaptive, with alerts providing timely, user-sensitive decision support integrated seamlessly into workflows. Sub-Theme 1 – Navigating Complexity: Clinician perceptions of single vs multi-output predictions This theme highlights a tension between potential benefits and risks of MO-EWS. In general, diagnosis-specific outputs are well regarded as they mimic the reasoning process of differential diagnosis (Table S3, Q-23), allowing clinicians to pursue their own hypotheses (Q-24). Participants see value in multiple outputs that provide richer detail and help identify diagnoses that may be overlooked or prioritise investigations. AI augments, not replaces, clinician reasoning, offering a proactive, exploratory approach to patient assessment (Q-25). On the other hand, a general deterioration score is often sufficient to initiate investigation into underlying causes (Q-26), and where offering specific causative conditions could lead clinicians to prematurely close on certain diagnoses without considering alternative explanations (Q-27). Multiple, simultaneous outputs could also overwhelm clinicians during busy shifts, further impairing situational awareness and timely responses (Q-28). Overall, clinicians desired a layered or multi-functional system with drill-down interactions (Q-29 and Q-30). Sub-theme 2 - From chasing problems to catching them early Clinicians envision a tool that prompts action before deterioration becomes clinically overt (Q-31). A novel, proactive MO-EWS should surface relevant, context-aware information before the alert triggers, enabling clinicians to efficiently initiate preventive actions (Q-32). One clinician actively searches the data for early signs of deterioration prior to receiving alerts, supported by hospital resources (staff, hardware), centralised alert system design, and timely communication with front-line nurses (Q-33). Proactive intervention could potentially free up time in caring for more patients (Q-34), as well as potentially saving lives when clinicians’ gestalt sees patients as worsening prior to triggering of alerts (Q-35). Observations and trends should remain visible in the UI as a first indicator of possible deterioration (Q-36). Sub-theme 3 – It is only useful if I can act on it Clinicians emphasised reactive alert systems should be action-oriented, regardless of the output, whether a deterioration score, or diagnosis-specific, or organ-specific outputs. For example, one participant pointed to actionable alerts tied to severity, expressing willingness to receive prompts suggesting appropriate actions based on the level of urgency (Q-37). Others gave practical examples of outputs tied to lab results (Q-38), not just showing the raw result (e.g. microurine analysis) but also giving diagnosis-specific guidance or suggested next step (Q-30.1). The system not only considers sepsis as a cause, for example, but also identifies a missing action (no fluids charted) and suggests next actions. Prioritising alerts according to type and severity of organ failure (Q-31.1) aids in ranking the urgency of response and suggesting remedial actions. Other clinicians considered time to act as necessary (Q-32.1), presenting both the required response and the urgency to act. Participants noted specific diagnoses with a pre-determined response within established protocols reduced cognitive burden, while general deterioration was more ambiguous as to cause and appropriate response. They also stressed the need for role-appropriate recommendations, particularly for junior clinicians (Q-33.1) but also for specific staff groups (Q-34.1) and different levels of expertise (Q-35.1). Also important was the desire for on-demand, flexible querying, describing a model that monitors and notifies using an interactive approach according to pre-specified rules (Q-36.1). Sub-theme 4 – More Than a Number: Supporting Interpretation of Deterioration Alerts Clinicians wanted to understand the alerts given by AI models (Q-37.1), wanting clinically meaningful AI explanations. Simply reporting a probability of deterioration was insufficient, as clinicians often trust their own judgement unless information is provided that complements or challenges their reasoning (Q-38.1). While some clinicians valued patient-specific details (Q-39), others considered cohort-level outputs as providing more useful insights and context (Q-40). Preferences varied between those seeking simplicity (e.g., clear trends) (Q-41), and those wanting more in-depth, feature-based information (Q-42). Metrics of model confidence in its predictions were considered important (Q-43) for gauging reliability, and senior clinicians in particular desired supporting evidence about model performance from validation studies (Q-44). Finally, explanations were seen as necessary for both real-time decision support and training and adoption (Q-45). Sub-theme 5 – Suggesting priority of actions but not mandating them Ranking patient risk can help separate high-risk from low-risk patients, thereby lessening alert fatigue and failure to recognise critical deterioration (Q-46). High-risk patients with life threatening conditions (Q-47), for which established treatments are available, receive priority (Q-48). However, the system should not mandate particular responses because different alerts may have similar levels of criticality (Q-49), and clinicians must decide which alert warrants immediate action (Q-50). Sub-theme 6 - Responsive systems for real-world clinical complexity Alert systems should facilitate model refinement and improved accuracy through continuous clinician feedback (Q-51), particularly when clinicians disagree with system outputs (Q-52 - Q-54). Any new AI-based tool should include adaptability to clinicians’ intuitive, experience-based gestalt (Q-55), rather than relying solely on quantitative data from EMRs (Q-56) with their inherent technical and infrastructure limitations (Q-57). Clinicians expressed contrary positions regarding personal customisation of the system (Q-58, Q-59), depending on their roles, desiring critical patient deterioration data calibrated to the level of expertise, while also optimising communication among different staff (Q-60), again highlighting the importance of adaptative UIs. Sub-theme 7 – Single Watchtower vs Local Patrols In a centralised system, alerts are directed to a dedicated team responsible for monitoring and escalation. In a distributed system, alerts go directly to clinicians caring for individual patients, with escalation managed locally. Clinicians were undecided about which system to adopt. Some nurses envisioned a centralised overview of patient alerts (Q-61), while some doctors preferred a mixed approach (Q-62). Doctors valued seeing outputs specific to their patients which facilitated response coordination, streamlined workflows and time savings (Q-63). One doctor described a mixed system (Q-64) in which both nurses and doctors shared access to the same interface and could independently update alert status (e.g., from active to deactivated). A nurse (Q-65) cautioned that alerts sent only to the clinician on duty, who had to interpret prediction outputs, could create a single point of failure. In contrast, a mixed approach allows alerts to be managed collaboratively by a team with junior and more senior clinicians, an important safeguard given the “fear of speaking up” often experienced by junior clinicians. The method for delivering an alert can vary (Q-66) to a device carried by clinicians (Q-67) as one option, especially one with a dedicated UI (Q-68). However, staff will not actively interact with a device throughout all working hours, as duties vary, so the device would have to be shared or monitored by others. This requires the new system to support individual logins and logouts for each clinician, so that urgent calls can be managed more efficiently (Q-69). One nurse noted that having alerts delivered at the bedside using an audible sound made her more likely to respond (Q-70). Sub-theme 8 - More than just accurate: a system that works for us! This sub-theme covers UI features raised by participants not covered in previous sub-themes. Alerts should be visible but not distracting (Q-71), understandable (Q-72), convey information clearly (Q-73), consider the patient’s clinical status (Q-74), be pertinent to the user (Q-75), and be amenable to customisation (Q-76). Moreover, each alert should include a ‘status’ indicator (e.g., ‘active’, ‘acknowledged’) to help clinicians track and manage them effectively (Q-77). In ward settings, multiple modalities (e.g., text, sounds, visual cues) (Q-78) could help nurses prioritise urgent cases and provide reassurance to patients, when they hear or see these alerts, that their own, less urgent needs will be addressed (Q-79). Participants noted clinical deterioration tools can serve as a safety net (Q-80), particularly for junior staff (Q-81), while maintaining clinician control to select issues of most concern (Q-82). The need for rigorous evaluation of system utility (Q-83) in building trust and supporting adoption (Q-84) was emphasised, with evidence of effectiveness preceding implementation (Q-85) in reducing resistance to adoption. 2.2.4. Theme 4 - The Ghost in the Machine MO-EWS brings a fundamental tension between AI’s intended role as a clinical aid versus becoming an intrusive or unreliable agent. For some participants, MO-EWS risk fragmenting attention and creating cognitive overload (Q-86), alert fatigue (Q-87), desensitisation and eventual dismissal of potentially critical outputs (Q-88). Overreliance on system outputs with little or no critical appraisal (Q-89) (i.e. automation bias), could inadvertently erode critical thinking skills, particularly among junior staff (Q-90). How clinicians with different expertise levels respond to AI recommendations raises concerns in that system performance is only as effective in guiding care as its user’s ability to consider and apply its outputs appropriately. This is more so if outdated or incomplete input data also risks generating flawed recommendations (Q-91). Other concerns relate to how AI outputs might unintentionally distort priorities in care delivery, whether due to technical design limitations (Q-92), constraints in the wider healthcare system (Q-93), or the AI’s inability to dynamically adapt to evolving clinical exigencies (Q-94). 2.2.5. Theme 5 – Ghost in the loop: The Role of User Training in Shaping the Machine Successful adoption of AI in clinical settings depends on comprehensive, well-timed staff training (Q-95), involvement of clinical champions or ‘super users’ in tool design and training (Q-96), attention to generational differences in AI familiarity, from paper-based tools to emerging generative AI systems (Q-97), and positioning AI as an always-available support tool for less experienced clinicians dealing with urgent situations (Q-98). Training should occur prior to AI system implementation and be reinforced through in-service education, ensuring staff understand the system’s purpose and benefits (Q-99). 2.2.6. Theme 6 - Beyond the ghost: the ecosystem matters Beyond the AI algorithm, factors such as digital infrastructure, organisational policies, communication channels, and social dynamics critically influence the success of an AI system. Changed workflows due to system implementation must be standardised and accepted by clinicians, making this a clinical governance issue (Q-100, Q-105). In addition, a new clinical deterioration tool should be integrated into the current EMR platform in avoiding time consuming dispersal of outputs across different systems (Q-101) and ensuring easy and quick clinician access to information (Q-102). The AI system should aim to ameliorate communication breakdown between different staff (Q-103), and between staff and patients and their families (Q-104), while cognisant of constraints in the wider healthcare system (Q-106). 3. Mapping of themes to system components and deriving recommendations Mapping these clinician-identified themes to system components of UI, AI model, and workflow [ 3 ] allowed derivation of recommendations for designing MO-EWS, summarised in Table 1 (acknowledging most of them can be useful for single output systems). In Table 2 , we propose a categorisation of the feasibility of these different design options for each of the three system components based on consensus following discussion among all authors. Table I Recommendations according to theme. Abbreviations: UI, User Interface, AI, Artificial Intelligence UI AI Workflow It cried wolf too many times but sometimes not at all Reduce alert noise through intuitive visualization of trends, patient-specific thresholds, and contextual cues that support clinician judgment or multiple AI-based prediction outputs. Use adaptive thresholds, delta changes, precision medicine, and calibrated predictions to minimise false positives and better capture subtle deterioration. Deliver timely, role-appropriate alerts while preserving opportunities for hands-on scrutiny of patient data. Hunting for answers in a fractured system Integrate vitals, labs, documentation, and multiple predictions into a clean, role-adapted dashboard with trends and quick search functions related to the multiple prediction outputs Enable cross-referencing of predictions with clinical data, provide search and filtering functions, and align predictions with clinicians’ mental models to support rapid interpretation. Embed predictions seamlessly into core EMR workflows, timed and tailored to clinicians’ roles and responsibilities. Navigating complexity: single vs multi-predictions Design layered, drill-down interfaces that allow clinicians to explore multiple condition-specific alerts without overwhelming the main view; support proactive exploration while minimising cognitive overload. Provide multiple, detailed AI-based predictions that support differential diagnosis and augment clinician reasoning without priming towards specific conclusions; allow clinicians to filter, prioritise, or dismiss alerts as needed. Integrate alert presentation into workflow in ways that prevent simultaneous alert overload, ensuring clinicians can respond effectively during busy shifts and maintain situational awareness. From chasing the problems to catching them early Keep observations and trends visible in the interface to allow early recognition of deterioration; use dashboards that highlight subtle changes before alerts trigger. Provide proactive, context-aware predictions to identify early signs of deterioration and enable preventive actions. Support clinicians in initiating preventive care before deterioration occurs; integrate proactive alerts with centralised communication and hospital resources to improve efficiency and patient safety. It is only useful if I can act of it Present alerts in actionable formats with clear severity ranking, prioritization, and role-specific guidance; allow on-demand querying and interactive exploration of patient data. Generate predictions with actionable recommendations, urgency indicators, and clinical guidance tailored to existing protocols and staff expertise. Integrate alerts into workflow to ensure timely responses, support role-appropriate actions (especially for juniors), and highlight priorities to prevent missed or delayed interventions. More than a number: supporting interpretation of deterioration alerts Display AI explanations in a clear, interpretable way, offering both simple trends and deeper feature-based insights; indicate model confidence and provide supporting evidence to enhance trust. Provide meaningful, patient-specific and cohort-level explanations that complement clinician reasoning; include metrics of prediction reliability and links to validation studies. Integrate explanations into the clinical workflow to support real-time decisions, training, and adoption without slowing urgent actions. Suggesting priority of actions but not mandating them Display alerts with clear prioritization cues for high-risk patients without forcing actions; use visual hierarchy to highlight urgency. Prioritise high-risk predictions based on severity and likelihood of life-threatening conditions while allowing clinician discretion. Support clinician decision-making by filtering and ranking patients according to risk, avoiding alert fatigue, and keeping clinicians in control of prioritization and actions. Responsive systems for real clinical complexity Implement adaptive, role-specific user interfaces that allow clinicians to personalise alert displays and visualizations; enable intuitive interaction to provide feedback and additional data. Incorporate clinician gestalt and experience-based judgment into AI predictions; allow AI models to learn from continuous clinician feedback to refine accuracy and relevance. Maintain clinician control over choice of response; support communication among staff and integrate feedback loops to improve system performance and coordination in patient care. Single watchtower vs local patrol Design interfaces that support both centralised and distributed alert viewing; provide login/logout flexibility and device-agnostic delivery; use auditory or visual cues to increase responsiveness. Ensure AI alerts can be routed to multiple recipients, track acknowledgement of receipt, and support collaborative management; adapt alert prioritisation based on real-time context and user roles. Enable collaborative alert management across teams to prevent single points of failure; integrate mixed centralised/distributed approaches for efficiency, coordination, and safeguarding junior clinicians. More than just accurate: system that works for us! Design alerts that are visible but non-distracting, clear, understandable, and use multiple modalities (text, sound, visuals); include status indicators (active, acknowledged, dismissed); allow user customisation. Prioritise alerts based on patient status and context; ensure relevance to the clinician receiving it; support safety-net functionality for new or junior staff. Integrate alerts into clinical workflow as supportive tools rather than mandates; provide evidence of effectiveness to build trust and facilitate adoption; allow clinicians to select which alerts to act on while maintaining oversight of critical cases. Theme 4 – The ghost in the machine Design interfaces that provide alert visibility and prioritisation but avoid overwhelming the user. AI should provide interpretable, context-aware predictions, indicate confidence levels, and dynamically adapt to evolving patient status while mitigating alarm fatigue. Workflow should preserve clinician critical thinking; system should support appropriate verification of AI predictions, prevent desensitisation, and account for varying expertise levels and real-world clinical constraints. Theme 5 - Ghost in the loop: the role of user training in shaping the machine Interface should be intuitive and user-friendly to support learning; include guided tutorials, contextual help, and clear visualizations to reinforce understanding. AI should act as an on-demand expert, offering real-time guidance, explanations, and actionable insights to bridge experience gaps. Workflow should incorporate structured training programs, engage clinical champions, and provide ongoing reinforcement; consider generational differences in AI familiarity and ensure alignment with clinical reasoning. Theme 6 – Beyond the ghost: the ecosystem matters The interface should integrate information from multiple sources into a single, coherent view, reducing fragmentation and enabling quick access. AI can support communication by highlighting relevant alerts or updates to appropriate staff, while considering system constraints. Standardised workflows, policies, and governance should be established before system design; integration into existing systems is crucial to streamline care and avoid inefficiencies. Feasibility took into consideration aspects of UI, AI, and workflow that are technically moderately or highly feasible versus those constrained by hospital infrastructure and organisational factors. These recommendations serve as a checklist for co-design and prototyping sessions, ensuring tools are grounded in clinician experience while effectively leveraging AI capabilities. Table 2 Assessment of feasibility of different design options for system components. Abbreviation: UI, User Interface, AI, Artificial Intelligence. UI AI Workflow High feasibility : Layered dashboards, status indicators, hierarchical structures, trend visualizations, search/filter functions, multi-modal alerts (text, sound, visual). These are mostly standard UI/UX practices and achievable with modern software development. Moderate feasibility : Fully adaptive and role-specific interfaces, drill-down views for multiple AI predictions, integrating explanations and model confidence dynamically. Requires significant development effort and close collaboration with clinicians. Low feasibility : Real-time integration of multiple AI prediction outputs while avoiding cognitive overload; presenting all relevant patient data without clutter for proactive monitoring. Technically complex, and financially challenging, especially for legacy/proprietary EMR systems. High feasibility : Generating multiple condition-specific outputs, providing patient-specific and cohort-level insights, integrating confidence scores, supporting prioritization based on severity. Current ML/AI methods can achieve this. Moderate feasibility : Adaptive AI that learns continuously from clinician feedback, accounts for clinician gestalt, and dynamically updates predictions in real time. Conceptually possible but requires extensive validation and regulatory approval. Low feasibility : AI that perfectly mimics clinician reasoning, dynamically balances multiple alerts to prevent fatigue, and customises predictions per staff experience level. This is still cutting-edge and challenging to deploy safely. High feasibility : Delivering alerts in centralised or distributed systems, supporting collaborative alert management, providing actionable recommendations and role-appropriate guidance, integrating alerts into existing workflows. Moderate feasibility : Seamless proactive monitoring before alerts trigger, integrating multi-source data into a single coherent workflow view, supporting on-demand querying. Requires workflow redesign and clinician input. Low feasibility : Personalising alert prioritisation across staff while maintaining safety and efficiency, dynamically adapting workflows in real time based on evolving patient states. Complex due to human, policy, and infrastructural constraints. 4. Discussion This study explored clinicians’ experiences with current and envisioned clinical deterioration tools, revealing a disconnect between the intended CDSS function and its real-world implementation [ 2 , 8 , 12 , 18 , 19 ]. A MO-EWS can help bridge this gap by offering context-sensitive, adaptive outputs rather than a single fixed deterioration score. By learning patient-specific baselines and trajectories, such systems can support nuanced decision-making while reducing false alarms. When paired with transparent UIs, co-designed workflows, and customisable levels of detail, they can minimise alert fatigue, avoid tool overreliance, and preserve critical thinking [ 20 ]. Continuous feedback loops [ 23 , 24 ] further ensure predictions evolve with additional quantitative data and clinician input. 4.1. Single vs. Multiple AI-based Prediction Outputs AI predictions with multiple versus single outputs are not inherently better. Senior clinicians (P01-D, 30y; P04-D, 37y) disagreed in their choice of either, showing experience shapes views on usefulness, risk, and cognitive load. Accordingly, MO-EWS should adopt a customisable interface supporting both single and multiple outputs as desired by clinicians. This can be achieved through progressive disclosure (providing a key prediction by default, with the option to expand into multiple outputs), adaptive presentation (tailoring detail to patient risk level or clinician role), and co-design with diverse clinical groups in ensuring alignment with varying workflows, cognitive needs, and decision-making styles. 4.2. Proactive systems Doctors desire proactive systems, not just reactive alerts, that monitor trends, respond to queries, and disclose context-aware information before alerts trigger, supporting clinicians to act preventively. Displaying patient-specific trends and trajectories across multiple conditions and offering layered outputs beginning with a synthesised summary but allowing drill-down into detailed predictions when required, can all help manage informational complexity without adding cognitive burden [ 11 ]. Such systems can foster human–AI teaming [ 8 ], whereby AI is most effective when it reflects how clinicians think, not just delivers facts they need [ 23 ]. The “layered” interface metaphor aligns with iterative reasoning and differential diagnosis, especially when coupled with explanations that make clinical sense [ 8 ]. 4.3. Situational awareness or tailored user experience Combining centralised and distributed methods of alert delivery can enhance team communication and decision-making, user-centred personalisation of output presentation tailored to cognitive style, role, expertise, or setting [ 10 ] may further support workflow [ 11 ] but could create information silos and undermine multidisciplinary collaboration and shared situational awareness. A more standardised approach would support consistency, coordination, and safety in acute care. This tension reveals a design dilemma, reflecting differing assumptions about hierarchy, role boundaries, and knowledge parity. Effective CDSS design should try to accommodate both perspectives with role-aware and modifiable interfaces, adapted to user preferences where possible. 4.4. Authority or safety net Some clinicians value AI as a safety net or second opinion, while others resist systems that dictate decisions or priorities and threaten clinician autonomy and accountability [ 24 ]. Safety concerns around clinicians overriding alerts highlights the need to balance automation with human judgment (“human-in-the-loop AI”) and take account of medico-legal liability if patients are harmed by inaccurate or misinterpreted AI systems. MO-EWS should offer context-aware recommendations while preserving clinicians’ autonomy to interpret, adapt, and act. 4.5. Reliability, integration and interpretability System reliability and integration are critical since out-of-date data and fragmented systems can delay or distort decision-making. Cognitive ergonomics and system design aligned with how clinicians think and act under pressure are required. Clear, timely and easily interpretable predictions delivered on the appropriate platform [ 25 ] are also essential for trust and adoption [ 8 ]. Failing to prioritise cognitive usability over computational precision can lead to alert fatigue, overload, and mistrust [ 22 ]. Regarding feasibility, most UI and AI features are technically achievable with existing software development and machine learning methods. Major bottlenecks lie in workflow integration within hospital settings. High-level UI/AI capabilities risk being undermined if alerts, data, and predictions cannot be seamlessly integrated into clinical workflows. Grading of feasibility depends not only on technical UI/AI advances but also on the level of hospital digital infrastructure, organisational policies, and clinician preferences. Legacy EMRs and proprietary systems, associated with poor interoperability and organisational inertia, make MO-EWS implementation particularly challenging. 5. Study implications Clinician adoption of EWS rests on meeting diverse, often subjective expectations [ 26 ]. Even well-designed tools may fail if expectations are unrealistic and not addressed through iterative, user-centred design and evaluation. Preserving autonomy and shared mental models requires careful design trade-offs, adaptive UIs, and frameworks centred on clinical reasoning needs rather than accuracy alone. Designers may underestimate workflow disruption and alert burden; clinician–system dynamics should be understood, not treated as a straightforward technical problem to fix. This study proposes an opportunity for clinicians to compare perspectives, invites developers to collaborate with medical staff and organisations, and encourages researchers to use the identified themes as an investigative checklist that ensures EWS accuracy is not prioritised at the expense of usability and workflow integration. 6. Limitations This study is limited by reliance on self-reported experiences from volunteer clinicians in specific hospitals with particular EMR systems, which brings selection bias and may not generalise to other contexts. The 22 participants constrained subgroup exploration, limiting transferability of themes, although this is not a methodological weakness in RTA. Negative experiences may be overrepresented, introducing potential bias against beneficial redesign. Finally, conducting RTA from a single positional perspective may limit interpretive diversity, although clinician authors [name deleted to maintain the integrity of the review process] with expertise in deterioration reviewed the analyses. 5. Conclusion This study of clinician perspectives on current EWTs and future MO-EWS highlights persistent challenges in optimising CDSS. Effective solutions require real-time monitoring, responsive queries, user-centred modelling, and calibrated predictions, without violating cognitive usability principles[ 19 ]. More AI does not guarantee better support; interfaces must fit clinicians’ mental models and workflows so the system, not the user, carries the burden of sense-making. Even small design–practice mismatches risk unsafe decisions, missed detections, or delayed responses. References Tun HM, Rahman HA, Naing L, Malik OA (Dec. 2024) Trust in AI-Based Clinical Decision Support Systems Among Healthcare Workers: A Systematic Review (Preprint). J Med Internet Res. 10.2196/69678 Yuan S, Yang Z, Li J, Wu C, Liu S (June 2025) AI-Powered early warning systems for clinical deterioration significantly improve patient outcomes: a meta-analysis. BMC Med Inf Decis Mak 25(1):203. 10.1186/s12911-025-03048-x Van Der Vegt AH et al (Jan. 2024) Systematic review and longitudinal analysis of implementing Artificial Intelligence to predict clinical deterioration in adult hospitals: what is known and what remains uncertain. J Am Med Inf Assoc 31(2):509–524. 10.1093/jamia/ocad220 Noselli M et al (2025) Presenting Artificial Intelligence Predictions Based on Electronic Medical Records to Clinicians in Hospitals: A Systematic Review, IEEE J. Biomed. Health Inform. , vol. 29, no. 10, pp. 7619–7632, Oct. 10.1109/JBHI.2025.3572494 Jayawardena T, Baysari M, Bamgboje-Ayodele A (2025 Apr) Interface design features of clinical decision support systems for real-time detection of deterioration: A scoping review. Int J Med Informatics 22:105946 Verma AA (Oct. 2024) Toward the Rigorous Evaluation of Early Warning Scores. JAMA Netw Open 7(10):e2438966. 10.1001/jamanetworkopen.2024.38966 Cánovas-Segura B, Morales A, Juarez JM, Campos M (July 2023) Meaningful time-related aspects of alerts in Clinical Decision Support Systems. A unified framework. J Biomed Inf 143:104397. 10.1016/j.jbi.2023.104397 Blythe R et al (Sept. 2024) Clinician perspectives and recommendations regarding design of clinical prediction models for deteriorating patients in acute care. BMC Med Inf Decis Mak 24(1):241. 10.1186/s12911-024-02647-4 Owoyemi A, Okpara E, Salwei M, Boyd A (Oct. 2024) End user experience of a widely used artificial intelligence based sepsis system. JAMIA Open 7(4):ooae096. 10.1093/jamiaopen/ooae096 Zhang S et al (2024) Rethinking Human-AI Collaboration in Complex Medical Decision Making: A Case Study in Sepsis Diagnosis, in Proceedings of the CHI Conference on Human Factors in Computing Systems , May pp. 1–18. 10.1145/3613904.3642343 Butler JM et al (2025) ‘Be Really Careful about That’: Clinicians’ Perceptions of an Intelligence Augmentation Tool for In-Hospital Deterioration Detection, Appl. Clin. Inform. , vol. 16, no. 02, pp. 377–392, Mar. 10.1055/a-2505-7743 Malterud K, Siersma VD, Guassora AD (2016) Sample Size in Qualitative Interview Studies: Guided by Information Power, Qual. Health Res. , vol. 26, no. 13, pp. 1753–1760, Nov. 10.1177/1049732315617444 Braun V, Clarke V (2024) Supporting best practice in reflexive thematic analysis reporting in Palliative Medicine : A review of published research and introduction to the Reflexive Thematic Analysis Reporting Guidelines (RTARG), Palliat. Med. , vol. 38, no. 6, pp. 608–616, June 10.1177/02692163241234800 Byrne D (June 2022) A worked example of Braun and Clarke’s approach to reflexive thematic analysis. Qual Quant 56(3):1391–1412. 10.1007/s11135-021-01182-y Azeem M, Salfi NA, USAGE OF NVIVO SOFTWARE FOR QUALITATIVE DATA ANALYSIS (2012), vol. 2, no. 1 Ammenwerth E (2019) Technology Acceptance Models in ealth nformatics: TAM and UTAUT. Stud Health Technol Inf 263:64–71. 10.3233/SHTI190111 Tong A, Sainsbury P, Craig J (2007) Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups, Int. J. Qual. Health Care , vol. 19, no. 6, pp. 349–357, Sept. 10.1093/intqhc/mzm042 Campbell V et al (Aug. 2020) Predicting clinical deterioration with Q-ADDS compared to NEWS, Between the Flags, and eCART track and trigger tools. Resuscitation 153:28–34. 10.1016/j.resuscitation.2020.05.027 Cho H et al (May 2022) Assessing the Usability of a Clinical Decision Support System: Heuristic Evaluation. JMIR Hum Factors 9:e31758. no. 210.2196/31758 Abdulnour R-EE, Gin B, Boscardin CK (2025) Educational Strategies for Clinical Supervision of Artificial Intelligence Use, N. Engl. J. Med. , vol. 393, no. 8, pp. 786–797, Aug. 10.1056/NEJMra2503232 Weigand L, Viers T, Tipton E (2025) Integration of Rapid Response Teams and Early Warning Systems to Reduce Cardiac Arrests and Intensive Care Unit Readmissions, Crit. Care Nurse , vol. 45, no. 4, pp. 49–56, Aug. 10.4037/ccn2025131 Gandhi TK et al (July 2023) How can artificial intelligence decrease cognitive and work burden for front line practitioners? JAMIA Open 6(3):ooad079. 10.1093/jamiaopen/ooad079 Wang L et al (June 2023) Human-centered design and evaluation of AI-empowered clinical decision support systems: a systemtic review. Front Comput Sci 5:1187299. 10.3389/fcomp.2023.1187299 Elgin CY, Elgin C (Dec. 2024) Ethical implications of AI-driven clinical decision support systems on healthcare resource allocation: a qualitative study of healthcare professionals’ perspectives. BMC Med Ethics 25(1):148. 10.1186/s12910-024-01151-8 Sirajuddin AM, Osheroff JA, Sittig DF, Chuo J, Velasco F, Collins DA (2012) Implementation Pearls from a New Guidebook on Improving Medication Use and Outcomes with Clinical Decision Support: Raita E, Oulasvirta A (July 2011) Too good to be bad: Favorable product expectations boost subjective usability ratings. Interact Comput 23(4):363–371. 10.1016/j.intcom.2011.04.002 Box Box is available in the Supplementary Files section. Additional Declarations The authors declare no competing interests. Supplementary Files RTACOREQChecklist.pdf SupplementaryTableofParticipantsQuotesAnonymised.docx box1.png Box. Overview of Reflexive Thematic Analysis Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8071546","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":542315672,"identity":"57967117-e0fd-4ad1-9af1-9b474551c04f","order_by":0,"name":"Monica Noselli","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAzElEQVRIiWNgGAWjYPACCxk29gYQg5loLRI8bDwHSNXCIJFApBbdBt6DD39USPDwSb4xvMFQYZ3YwH7GAK8WswN8ycY8Z4AOk84xtmA4k57YwJNDSAuPmTRjG1iLmQRj2+HEBgYitEj+/AfUInkGqOUfUAv/G8JaJHgbgFokgAzGBqAWCUK2HOYxNuY5BgrktGKLhGPpxm0SzwrwazneY/jwR42NnHz74Y03PtRYy/bzJ2/AqwUlIsBRw4ZfPRqQIEn1KBgFo2AUjBgAABsHN3DFLpBPAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0009-0003-5756-9214","institution":"The University of Queensland","correspondingAuthor":true,"prefix":"","firstName":"Monica","middleName":"","lastName":"Noselli","suffix":""},{"id":542315673,"identity":"f52caef5-5d72-4c0c-8e1a-c4ffb93f5c37","order_by":1,"name":"Anton Van Der Vegt","email":"","orcid":"https://orcid.org/0000-0001-5642-5188","institution":"The University of Queensland","correspondingAuthor":false,"prefix":"","firstName":"Anton","middleName":"Van Der","lastName":"Vegt","suffix":""},{"id":542315674,"identity":"7ae90250-aa10-4351-badb-8db3e8eb39df","order_by":2,"name":"Maxime Cordeil","email":"","orcid":"https://orcid.org/0000-0002-9732-4874","institution":"The University of Queensland","correspondingAuthor":false,"prefix":"","firstName":"Maxime","middleName":"","lastName":"Cordeil","suffix":""},{"id":542315675,"identity":"85bbe9fe-0bdd-4291-8934-a4385698f0f1","order_by":3,"name":"Victoria Campbell","email":"","orcid":"https://orcid.org/0000-0001-9846-9410","institution":"Clinical Excellence Queensland","correspondingAuthor":false,"prefix":"","firstName":"Victoria","middleName":"","lastName":"Campbell","suffix":""},{"id":542315676,"identity":"26c885a8-0176-4028-92df-3a3eb1af2a59","order_by":4,"name":"Ian A. Scott","email":"","orcid":"https://orcid.org/0000-0002-7596-0837","institution":"4.\tMetro South Digital Health and Informatics, Princess Alexandra Hospital","correspondingAuthor":false,"prefix":"","firstName":"Ian","middleName":"A.","lastName":"Scott","suffix":""}],"badges":[],"createdAt":"2025-11-10 00:35:22","currentVersionCode":1,"declarations":{"humanSubjects":true,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":true,"humanSubjectConsent":true,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-8071546/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8071546/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":95670817,"identity":"b3beab97-0279-4ced-b723-75ed2a1a3e2c","added_by":"auto","created_at":"2025-11-11 17:22:45","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":322503,"visible":true,"origin":"","legend":"","description":"","filename":"RTAFinalText.docx","url":"https://assets-eu.researchsquare.com/files/rs-8071546/v1/571ae5ab3f411ca99f90cb3a.docx"},{"id":95670812,"identity":"73cb7f6e-3f1e-4e63-ae68-255ef2319a81","added_by":"auto","created_at":"2025-11-11 17:22:45","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":342,"visible":true,"origin":"","legend":"","description":"","filename":"rs8071546.json","url":"https://assets-eu.researchsquare.com/files/rs-8071546/v1/4aeaacfad99378d3789fae9d.json"},{"id":95670814,"identity":"c8cecb8d-364e-492f-bb12-b9bcf1f3cd5d","added_by":"auto","created_at":"2025-11-11 17:22:45","extension":"xml","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":93018,"visible":true,"origin":"","legend":"","description":"","filename":"rs80715460enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-8071546/v1/17605e91a0cf1c791c354c85.xml"},{"id":95797279,"identity":"5e58edc2-4a20-4e62-af41-d1d657e5898c","added_by":"auto","created_at":"2025-11-13 08:02:57","extension":"png","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":79216,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-8071546/v1/51df31673160b998a5a7be68.png"},{"id":95670818,"identity":"b8da7193-0b09-4216-b8bf-b7b3b7655023","added_by":"auto","created_at":"2025-11-11 17:22:45","extension":"png","order_by":5,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":117757,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-8071546/v1/e8735185400cf51f197f5b39.png"},{"id":95670822,"identity":"a8f78b49-998b-4840-88be-9bf79a99e4aa","added_by":"auto","created_at":"2025-11-11 17:22:45","extension":"jpeg","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1454085,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage3.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-8071546/v1/24aa7382f83db8ee2c788550.jpeg"},{"id":95670821,"identity":"9b069de6-74ae-4eee-b4ac-47427d4cbe1a","added_by":"auto","created_at":"2025-11-11 17:22:45","extension":"png","order_by":7,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":33226,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-8071546/v1/3373286513888d4ba9e2fe95.png"},{"id":95670823,"identity":"9a25a734-e5b4-4df7-8a37-0164710e4a67","added_by":"auto","created_at":"2025-11-11 17:22:45","extension":"png","order_by":8,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":41620,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-8071546/v1/ecda3d29757d1e8689332523.png"},{"id":95670826,"identity":"30e6357a-94a1-4d19-92fd-958b6a1c390d","added_by":"auto","created_at":"2025-11-11 17:22:45","extension":"png","order_by":9,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":276162,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-8071546/v1/9a6a20d8f4d77ed9716f0620.png"},{"id":95670827,"identity":"9f8bf7f9-f368-4e7b-abaa-cdef8785d0cc","added_by":"auto","created_at":"2025-11-11 17:22:46","extension":"xml","order_by":10,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":90136,"visible":true,"origin":"","legend":"","description":"","filename":"rs80715460structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-8071546/v1/10941348cf8cc934dbf821f8.xml"},{"id":95799211,"identity":"8cf77f26-200e-4eff-bedc-3ff9a1fb803e","added_by":"auto","created_at":"2025-11-13 08:19:03","extension":"html","order_by":11,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":100569,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-8071546/v1/82770f6e7da9ade3ba587aee.html"},{"id":95670811,"identity":"a807306d-6876-4417-921e-454079601f98","added_by":"auto","created_at":"2025-11-11 17:22:45","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":112945,"visible":true,"origin":"","legend":"\u003cp\u003eStudy goals and processes\u003c/p\u003e","description":"","filename":"image.png","url":"https://assets-eu.researchsquare.com/files/rs-8071546/v1/b1476b3a1ca3dace967ef2f7.png"},{"id":95799798,"identity":"c046af57-c812-4157-a205-62b2756a5773","added_by":"auto","created_at":"2025-11-13 08:20:50","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":125888,"visible":true,"origin":"","legend":"\u003cp\u003eThemes and subthemes derived from interviews on clinicians’ perceptions and expectations\u003c/p\u003e","description":"","filename":"image.png","url":"https://assets-eu.researchsquare.com/files/rs-8071546/v1/aa62e45432b3f52133d27aea.png"},{"id":95804654,"identity":"2e7f5647-fcfe-4ab2-8a4e-b661da9b1443","added_by":"auto","created_at":"2025-11-13 08:38:54","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1306163,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8071546/v1/c763e72e-c856-4927-aeb5-2d403a365341.pdf"},{"id":95797284,"identity":"5cb33252-add3-4cac-9dab-34123485d91e","added_by":"auto","created_at":"2025-11-13 08:03:14","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":372010,"visible":true,"origin":"","legend":"","description":"","filename":"RTACOREQChecklist.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8071546/v1/8854cae72e06cd6a8bf12944.pdf"},{"id":95799791,"identity":"aeb832f3-cf8f-44c9-a386-55b9b99e1d46","added_by":"auto","created_at":"2025-11-13 08:20:48","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":39935,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryTableofParticipantsQuotesAnonymised.docx","url":"https://assets-eu.researchsquare.com/files/rs-8071546/v1/114d05ad95429ad24501b9a5.docx"},{"id":95798628,"identity":"4f96857c-58e8-4f5a-b4ae-b7b1f93a6702","added_by":"auto","created_at":"2025-11-13 08:17:21","extension":"png","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":143346,"visible":true,"origin":"","legend":"\u003cp\u003eBox. Overview of Reflexive Thematic Analysis\u003c/p\u003e","description":"","filename":"box1.png","url":"https://assets-eu.researchsquare.com/files/rs-8071546/v1/c505806b054a31f25afedb15.png"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003eClinicians’ perspectives on the design of early warning systems presenting AI-based multiple outputs for clinical deterioration: A qualitative analysis\u003c/p\u003e","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eAs artificial intelligence (AI) continues to be integrated into clinical decision support systems (CDSS) [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e], there is growing interest in how AI-generated predictions can assist clinicians to identify patient deterioration thereby enhancing decision-making, facilitating early intervention and supporting resource allocation [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. While risk prediction algorithms are getting more accurate [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e], their impact on clinical decision making and care outcomes depends on the actionable content of the CDSS outputs and how these are presented to clinicians [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. Poorly designed CDSS user interfaces can lead to poor utility, misinterpretation, alert fatigue and non-adoption of the system [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e].\u003c/p\u003e\u003cp\u003ePrevious AI-based CDSS (AI-CDSS) user interface research focused mostly on single condition or single alert scores (e.g., sepsis or general deterioration only) [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e] [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e] [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e] [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e] [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. These studies identified important design factors. For example, Blythe et al [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e] explored how early warning scores (EWS) influence clinicians' decision-making, highlighting the importance of models that facilitate but not replace critical thinking, provide transparency regarding included and excluded variables, and support decision-making processes. Owoyemi et al. [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e] examined user experiences with the Epic Sepsis System (ESS) and identified challenges with workflow fit and usability. A second study of the ESS [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e] proposed that supporting clinicians in hypothesis generation and information gathering was more important than providing a final diagnosis or decision. AI-CDSS user-interface investigators have also found that successful implementation of AI-based tools requires understanding clinicians\u0026rsquo; cognitive needs and framing AI as complementary support [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. In all these investigations, a singular AI-based CDSS tool was considered.\u003c/p\u003e\u003cp\u003eHowever, as more AI-based predictive models are researched and developed, it is likely that future AI-CDSS deterioration systems will encompass multiple predictive models working in concert. Currently, there is very limited research and understanding of how multiple concurrent AI-based models should present their outputs within a single AI-CDSS to support clinician decision-making.\u003c/p\u003e\u003cp\u003eThe purpose of this study is to investigate clinicians\u0026rsquo; views on how multiple AI-based predictions could and should be presented within a deterioration focused AI-CDSS. The purpose of this investigation is to better understand (i) the issues that clinicians have with their current deterioration early warning tools that they would like to see resolved in future solutions; and (ii) clinician preferences with respect to the scope and design of future multi-prediction AI-based deterioration CDSS. This study applies a clinician-centric lens to identify actionable insights for system developers, co-development teams and researchers seeking to build multiple AI-based prediction CDSS that are both more effective with high levels of clinical adoption.\u003c/p\u003e"},{"header":"2. Methods","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e\u003ch2\u003e1.1. Participants and recruitment\u003c/h2\u003e\u003cp\u003eBetween January and June 2025, clinicians experienced in managing deterioration were recruited via email containing a flyer and participant information sheet. These were distributed through professional networks including two clinical research groups (AI-KEYS: Artificial Intelligence \u0026ndash; Keeping You Safer, and Queensland Health Sepsis group). Contacted participants were invited to forward the call for participation email to extend recruitment reach. Recruitment was initially aimed at ward-based staff but expanded to include clinicians from Intensive Care Unit (ICU), Emergency Department (ED), and Post-Anaesthesia Care Unit (PACU) in order to capture a broader range of perspectives across settings where early warning systems are used.\u003c/p\u003e\u003cp\u003eParticipation was voluntary with consent. Recruitment was guided by the principle of informational power [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e], given the narrow and clearly defined study aim, the participants\u0026rsquo; high level of clinical expertise, and the anticipated richness of dialogue, a relatively modest sample (22 participants) was considered sufficient. Recruitment concluded once the available networks had been fully approached and no additional eligible participants could be identified. No prior relationship existed between the researcher and participants. Ethical approval was obtained from the [name deleted to maintain the integrity of the review process].\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec4\" class=\"Section2\"\u003e\u003ch2\u003e1.2. Data collection\u003c/h2\u003e\u003cp\u003eResearcher [name deleted to maintain the integrity of the review process], a female PhD student, conducted 45-minute one-on-one (single sessions only, without follow-up) Microsoft Teams interviews, using a semi-structured questionnaire, constructed de novo with all authors, that consistently captured: (a) participants\u0026rsquo; clinical roles and settings; (b) challenges in using current EWT for identifying clinical deterioration; and (c) perceptions of a novel MO-EWS. Two pilot interviews with two clinician co-authors helped refine clarity, relevance and suitability of the questions. Participants gave verbal consent to have interviews recorded for analysis using Microsoft Teams. Participants did not comment or correct the transcripts, they were invited to provide feedback on the findings, however, no responses were received.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec5\" class=\"Section2\"\u003e\u003ch2\u003e1.3. Analysis\u003c/h2\u003e\u003cp\u003eThe study was conducted in three phases (Fig. I). First, following Braun and Clarke\u0026rsquo;s approach to using Reflexive Thematic Analysis (RTA) (see Box) [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. Author [name deleted to maintain the integrity of the review process] engaged iteratively with transcripts, moving recursively between coding, theme development, review, and writing. Using NVivo 14 software [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e], [name deleted to maintain the integrity of the review process] coded semantic and latent content, initially distinguishing 'actual' and 'envisioned' tools. However, since participants often discussed envisioned tool through the lens of their experience with existing tools, this overlap shaped the final coding. While some codes aligned with frameworks like technology acceptance model (TAM) [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e], these limited reflexive analysis, so different themes were developed. Second, the final themes were then mapped to system components comprising (a) UI: what clinicians see and interact with; (b) AI algorithms that analyse data, make predictions, or generate alerts; (c) Workflow: the sequence of tasks and processes clinicians undertake when using the system in practice. Third, this mapping was used to derive practical design recommendations which were then assessed for their feasibility by consensus of all authors who comprised the main author [name deleted to maintain the integrity of the review process], two senior clinicians and system analysts [name deleted to maintain the integrity of the review process], a senior data scientist [name deleted to maintain the integrity of the review process] and a senior social scientist [name deleted to maintain the integrity of the review process]. In contrast to prior studies. we did not employ or demonstrate any tool prototypes to avoid cueing or biasing effects on clinicians in exploring these issues.\u003c/p\u003e\u003cp\u003eThis study is reported according to the COREQ (COnsolidated criteria for REporting Qualitative research) Checklist [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e].\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003e\u003cem\u003eFigure I Study goals and processes\u003c/em\u003e\u003c/p\u003e\u003c/div\u003e"},{"header":"3. Results","content":"\u003cdiv id=\"Sec7\" class=\"Section2\"\u003e\u003ch2\u003e2.1. Participants\u003c/h2\u003e\u003cp\u003eTwenty-two clinicians participated in interviews, comprising 11 nurses and 11 doctors, working in different departments across hospitals (ED\u0026thinsp;=\u0026thinsp;8, ICU\u0026thinsp;=\u0026thinsp;6, Ward\u0026thinsp;=\u0026thinsp;5, PACU\u0026thinsp;=\u0026thinsp;2, ICU\u0026thinsp;+\u0026thinsp;ED\u0026thinsp;=\u0026thinsp;1) throughout Australia and all having a minimum 4 years of postgraduate experience.\u003c/p\u003e\u003cp\u003eParticipants mentioned various currently used clinical deterioration alert tools, including Queensland-Adult Deterioration Detection System (Q-ADDS) and Between the Flags (BTF) [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e], and downstream systems that report these alerts (Compass). They also worked with different electronic medical record (EMR) systems, such as Sunrise, Oracle Cerner, and the Cerner module FirstNet used in EDs.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec8\" class=\"Section2\"\u003e\u003ch2\u003e2.2. Themes\u003c/h2\u003e\u003cp\u003eThe themes derived from interviews (Fig. II) are discussed in more detail in the following sections. Consistency between data and findings was ensured and illustrative participant quotes, identified using numbers and letters in parentheses in the following text, are provided in the supplementary material (Online Resource 1).\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003e\u003cem\u003eFigure II Themes and subthemes derived from interviews on clinicians\u0026rsquo; perceptions and expectations\u003c/em\u003e\u003c/p\u003e\u003cdiv id=\"Sec9\" class=\"Section3\"\u003e\u003ch2\u003e2.2.1. Theme 1 - It cried wolf too many times and sometime not at all\u003c/h2\u003e\u003cp\u003eParticipants described how frequent, redundant, and poorly calibrated alerts (Table S1: Q-1 to Q-4) lead to alert burden and fatigue and eventual rejection of the alerting system through desensitisation, including when clinical intuition contradicts algorithmic outputs (Q-11).\u003c/p\u003e\u003cp\u003eThe effective use of EWTs is shaped by a complex interplay of data quality, system design, clinical context and human judgment. As P10-D cautions, even the most sophisticated AI model is \u0026ldquo;only as good as the data you feed into it\u0026rdquo;, which can be subjective (Q-5), inconsistent across settings (Q-6), have long time gaps between nursing observations (Q-7), and contain omissions or entry errors (Q-8, Q-9). Among design factors, rigid criteria embedded into systems can introduce bias and cognitive error (Q-10). Threshold-based triggers can miss deteriorations happening below the cut-off point (Q-11) and patient-specific thresholds and delta calculations, not incorporated into EWTs, also influence decisions (Q-12, Q-13). Human factors such as both over-reliant and overly cautious response to outputs can lead to missed deterioration in patients requiring urgent review (Q-14). Vital signs automatically plotted by the computer may cause clinicians to forego the cognitive act of processing data that makes explicit their understanding of the trends (Q-15). We grouped both false positives (Q-1/10) and false negatives (Q-11/15) under this theme. Although opposite in form, participants perceived them as part of the same reliability problem: a system that either alarms unnecessarily or fails to alarm when needed ultimately reduces clinicians\u0026rsquo; trust in its outputs\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec10\" class=\"Section3\"\u003e\u003ch2\u003e2.2.2. Theme 2 - Hunting for answers in a fractured system\u003c/h2\u003e\u003cp\u003ePoor UI layouts, fragmented workflows, or excessive alerts increase cognitive load and slow responses. Participants reported frustration from information being often dispersed across multiple pages or tabs, obscuring temporal trends and forcing them to mentally integrate disparate data sources (Table S2. Q-16). While some graphical displays offer trend visualization, inability to cross-reference trends directly with related clinical documentation disrupts workflow. Accessibility is insufficient if information is not presented in ways that align with clinicians\u0026rsquo; mental models (Q-17). Icon-heavy dashboards and copy-pasted documentation bring noise, information overload and reduced visibility of critical clinical indicators (Q-18).\u003c/p\u003e\u003cp\u003eEven with interactive functions, information may be updated in different pages or tabs, diminishing their utility for dynamic patient monitoring (Q-19). Clinicians disliked systems that are not easily searchable, wanting functionality that retrieves pertinent information with a minimal number of steps (Q-20).\u003c/p\u003e\u003cp\u003eMoreover, multiple, non-integrated platforms and data sources, combined with limited ability to extract actionable meaning from disjointed or cluttered interfaces, all foster redundancy, inefficiency, and cognitive strain, diverting time from patient care (Q-21, Q-22).\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec11\" class=\"Section3\"\u003e\u003ch2\u003e2.2.3. Theme 3 - Ghost in the Interface\u003c/h2\u003e\u003cp\u003eThis theme captures clinicians\u0026rsquo; vision of AI as an intelligent collaborator delivering actionable, context-aware outputs aligned with reasoning modes, workflow, and expertise. Interfaces should be layered, drillable, and adaptive, with alerts providing timely, user-sensitive decision support integrated seamlessly into workflows.\u003c/p\u003e\u003cp\u003e\u003cem\u003eSub-Theme 1 \u0026ndash; Navigating Complexity: Clinician perceptions of single vs multi-output predictions\u003c/em\u003e\u003c/p\u003e\u003cp\u003eThis theme highlights a tension between potential benefits and risks of MO-EWS.\u003c/p\u003e\u003cp\u003eIn general, diagnosis-specific outputs are well regarded as they mimic the reasoning process of differential diagnosis (Table S3, Q-23), allowing clinicians to pursue their own hypotheses (Q-24). Participants see value in multiple outputs that provide richer detail and help identify diagnoses that may be overlooked or prioritise investigations. AI augments, not replaces, clinician reasoning, offering a proactive, exploratory approach to patient assessment (Q-25).\u003c/p\u003e\u003cp\u003eOn the other hand, a general deterioration score is often sufficient to initiate investigation into underlying causes (Q-26), and where offering specific causative conditions could lead clinicians to prematurely close on certain diagnoses without considering alternative explanations (Q-27). Multiple, simultaneous outputs could also overwhelm clinicians during busy shifts, further impairing situational awareness and timely responses (Q-28).\u003c/p\u003e\u003cp\u003eOverall, clinicians desired a layered or multi-functional system with drill-down interactions (Q-29 and Q-30).\u003c/p\u003e\u003cp\u003e\u003cem\u003eSub-theme 2 - From chasing problems to catching them early\u003c/em\u003e\u003c/p\u003e\u003cp\u003eClinicians envision a tool that prompts action before deterioration becomes clinically overt (Q-31). A novel, proactive MO-EWS should surface relevant, context-aware information before the alert triggers, enabling clinicians to efficiently initiate preventive actions (Q-32). One clinician actively searches the data for early signs of deterioration prior to receiving alerts, supported by hospital resources (staff, hardware), centralised alert system design, and timely communication with front-line nurses (Q-33). Proactive intervention could potentially free up time in caring for more patients (Q-34), as well as potentially saving lives when clinicians\u0026rsquo; gestalt sees patients as worsening prior to triggering of alerts (Q-35). Observations and trends should remain visible in the UI as a first indicator of possible deterioration (Q-36).\u003c/p\u003e\u003cp\u003e\u003cem\u003eSub-theme 3 \u0026ndash; It is only useful if I can act on it\u003c/em\u003e\u003c/p\u003e\u003cp\u003eClinicians emphasised reactive alert systems should be action-oriented, regardless of the output, whether a deterioration score, or diagnosis-specific, or organ-specific outputs. For example, one participant pointed to actionable alerts tied to severity, expressing willingness to receive prompts suggesting appropriate actions based on the level of urgency (Q-37). Others gave practical examples of outputs tied to lab results (Q-38), not just showing the raw result (e.g. microurine analysis) but also giving diagnosis-specific guidance or suggested next step (Q-30.1). The system not only considers sepsis as a cause, for example, but also identifies a missing action (no fluids charted) and suggests next actions. Prioritising alerts according to type and severity of organ failure (Q-31.1) aids in ranking the urgency of response and suggesting remedial actions. Other clinicians considered time to act as necessary (Q-32.1), presenting both the required response and the urgency to act.\u003c/p\u003e\u003cp\u003eParticipants noted specific diagnoses with a pre-determined response within established protocols reduced cognitive burden, while general deterioration was more ambiguous as to cause and appropriate response. They also stressed the need for role-appropriate recommendations, particularly for junior clinicians (Q-33.1) but also for specific staff groups (Q-34.1) and different levels of expertise (Q-35.1).\u003c/p\u003e\u003cp\u003eAlso important was the desire for on-demand, flexible querying, describing a model that monitors and notifies using an interactive approach according to pre-specified rules (Q-36.1).\u003c/p\u003e\u003cp\u003e\u003cem\u003eSub-theme 4 \u0026ndash; More Than a Number: Supporting Interpretation of Deterioration Alerts\u003c/em\u003e\u003c/p\u003e\u003cp\u003eClinicians wanted to understand the alerts given by AI models (Q-37.1), wanting clinically meaningful AI explanations. Simply reporting a probability of deterioration was insufficient, as clinicians often trust their own judgement unless information is provided that complements or challenges their reasoning (Q-38.1).\u003c/p\u003e\u003cp\u003eWhile some clinicians valued patient-specific details (Q-39), others considered cohort-level outputs as providing more useful insights and context (Q-40). Preferences varied between those seeking simplicity (e.g., clear trends) (Q-41), and those wanting more in-depth, feature-based information (Q-42). Metrics of model confidence in its predictions were considered important (Q-43) for gauging reliability, and senior clinicians in particular desired supporting evidence about model performance from validation studies (Q-44).\u003c/p\u003e\u003cp\u003eFinally, explanations were seen as necessary for both real-time decision support and training and adoption (Q-45).\u003c/p\u003e\u003cp\u003e\u003cem\u003eSub-theme 5 \u0026ndash; Suggesting priority of actions but not mandating them\u003c/em\u003e\u003c/p\u003e\u003cp\u003eRanking patient risk can help separate high-risk from low-risk patients, thereby lessening alert fatigue and failure to recognise critical deterioration (Q-46). High-risk patients with life threatening conditions (Q-47), for which established treatments are available, receive priority (Q-48). However, the system should not mandate particular responses because different alerts may have similar levels of criticality (Q-49), and clinicians must decide which alert warrants immediate action (Q-50).\u003c/p\u003e\u003cp\u003e\u003cem\u003eSub-theme 6 - Responsive systems for real-world clinical complexity\u003c/em\u003e\u003c/p\u003e\u003cp\u003eAlert systems should facilitate model refinement and improved accuracy through continuous clinician feedback (Q-51), particularly when clinicians disagree with system outputs (Q-52 - Q-54). Any new AI-based tool should include adaptability to clinicians\u0026rsquo; intuitive, experience-based gestalt (Q-55), rather than relying solely on quantitative data from EMRs (Q-56) with their inherent technical and infrastructure limitations (Q-57).\u003c/p\u003e\u003cp\u003eClinicians expressed contrary positions regarding personal customisation of the system (Q-58, Q-59), depending on their roles, desiring critical patient deterioration data calibrated to the level of expertise, while also optimising communication among different staff (Q-60), again highlighting the importance of adaptative UIs.\u003c/p\u003e\u003cp\u003e\u003cem\u003eSub-theme 7 \u0026ndash; Single Watchtower vs Local Patrols\u003c/em\u003e\u003c/p\u003e\u003cp\u003eIn a centralised system, alerts are directed to a dedicated team responsible for monitoring and escalation. In a distributed system, alerts go directly to clinicians caring for individual patients, with escalation managed locally. Clinicians were undecided about which system to adopt. Some nurses envisioned a centralised overview of patient alerts (Q-61), while some doctors preferred a mixed approach (Q-62). Doctors valued seeing outputs specific to their patients which facilitated response coordination, streamlined workflows and time savings (Q-63). One doctor described a mixed system (Q-64) in which both nurses and doctors shared access to the same interface and could independently update alert status (e.g., from active to deactivated). A nurse (Q-65) cautioned that alerts sent only to the clinician on duty, who had to interpret prediction outputs, could create a single point of failure. In contrast, a mixed approach allows alerts to be managed collaboratively by a team with junior and more senior clinicians, an important safeguard given the \u0026ldquo;fear of speaking up\u0026rdquo; often experienced by junior clinicians. The method for delivering an alert can vary (Q-66) to a device carried by clinicians (Q-67) as one option, especially one with a dedicated UI (Q-68). However, staff will not actively interact with a device throughout all working hours, as duties vary, so the device would have to be shared or monitored by others. This requires the new system to support individual logins and logouts for each clinician, so that urgent calls can be managed more efficiently (Q-69). One nurse noted that having alerts delivered at the bedside using an audible sound made her more likely to respond (Q-70).\u003c/p\u003e\u003cp\u003e\u003cem\u003eSub-theme 8 - More than just accurate: a system that works for us!\u003c/em\u003e\u003c/p\u003e\u003cp\u003eThis sub-theme covers UI features raised by participants not covered in previous sub-themes.\u003c/p\u003e\u003cp\u003eAlerts should be visible but not distracting (Q-71), understandable (Q-72), convey information clearly (Q-73), consider the patient\u0026rsquo;s clinical status (Q-74), be pertinent to the user (Q-75), and be amenable to customisation (Q-76). Moreover, each alert should include a \u0026lsquo;status\u0026rsquo; indicator (e.g., \u0026lsquo;active\u0026rsquo;, \u0026lsquo;acknowledged\u0026rsquo;) to help clinicians track and manage them effectively (Q-77). In ward settings, multiple modalities (e.g., text, sounds, visual cues) (Q-78) could help nurses prioritise urgent cases and provide reassurance to patients, when they hear or see these alerts, that their own, less urgent needs will be addressed (Q-79).\u003c/p\u003e\u003cp\u003eParticipants noted clinical deterioration tools can serve as a safety net (Q-80), particularly for junior staff (Q-81), while maintaining clinician control to select issues of most concern (Q-82).\u003c/p\u003e\u003cp\u003eThe need for rigorous evaluation of system utility (Q-83) in building trust and supporting adoption (Q-84) was emphasised, with evidence of effectiveness preceding implementation (Q-85) in reducing resistance to adoption.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec12\" class=\"Section3\"\u003e\u003ch2\u003e2.2.4. Theme 4 - The Ghost in the Machine\u003c/h2\u003e\u003cp\u003eMO-EWS brings a fundamental tension between AI\u0026rsquo;s intended role as a clinical aid versus becoming an intrusive or unreliable agent. For some participants, MO-EWS risk fragmenting attention and creating cognitive overload (Q-86), alert fatigue (Q-87), desensitisation and eventual dismissal of potentially critical outputs (Q-88).\u003c/p\u003e\u003cp\u003eOverreliance on system outputs with little or no critical appraisal (Q-89) (i.e. automation bias), could inadvertently erode critical thinking skills, particularly among junior staff (Q-90). How clinicians with different expertise levels respond to AI recommendations raises concerns in that system performance is only as effective in guiding care as its user\u0026rsquo;s ability to consider and apply its outputs appropriately. This is more so if outdated or incomplete input data also risks generating flawed recommendations (Q-91).\u003c/p\u003e\u003cp\u003eOther concerns relate to how AI outputs might unintentionally distort priorities in care delivery, whether due to technical design limitations (Q-92), constraints in the wider healthcare system (Q-93), or the AI\u0026rsquo;s inability to dynamically adapt to evolving clinical exigencies (Q-94).\u003c/p\u003e\u003cp\u003e2.2.5. Theme 5 \u0026ndash; Ghost in the loop: The Role of User Training in Shaping the Machine\u003c/p\u003e\u003cp\u003eSuccessful adoption of AI in clinical settings depends on comprehensive, well-timed staff training (Q-95), involvement of clinical champions or \u0026lsquo;super users\u0026rsquo; in tool design and training (Q-96), attention to generational differences in AI familiarity, from paper-based tools to emerging generative AI systems (Q-97), and positioning AI as an always-available support tool for less experienced clinicians dealing with urgent situations (Q-98). Training should occur prior to AI system implementation and be reinforced through in-service education, ensuring staff understand the system\u0026rsquo;s purpose and benefits (Q-99).\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec13\" class=\"Section3\"\u003e\u003ch2\u003e2.2.6. Theme 6 - Beyond the ghost: the ecosystem matters\u003c/h2\u003e\u003cp\u003eBeyond the AI algorithm, factors such as digital infrastructure, organisational policies, communication channels, and social dynamics critically influence the success of an AI system. Changed workflows due to system implementation must be standardised and accepted by clinicians, making this a clinical governance issue (Q-100, Q-105). In addition, a new clinical deterioration tool should be integrated into the current EMR platform in avoiding time consuming dispersal of outputs across different systems (Q-101) and ensuring easy and quick clinician access to information (Q-102). The AI system should aim to ameliorate communication breakdown between different staff (Q-103), and between staff and patients and their families (Q-104), while cognisant of constraints in the wider healthcare system (Q-106).\u003c/p\u003e\u003c/div\u003e\u003c/div\u003e\n\u003ch3\u003e3. Mapping of themes to system components and deriving recommendations\u003c/h3\u003e\n\u003cp\u003eMapping these clinician-identified themes to system components of UI, AI model, and workflow [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e] allowed derivation of recommendations for designing MO-EWS, summarised in Table\u0026nbsp;1 (acknowledging most of them can be useful for single output systems). In Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e2\u003c/span\u003e, we propose a categorisation of the feasibility of these different design options for each of the three system components based on consensus following discussion among all authors.\u003c/p\u003e\u003cp\u003e\u003cem\u003eTable I Recommendations according to theme. Abbreviations: UI, User Interface, AI, Artificial Intelligence\u003c/em\u003e\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"No\" id=\"Taba\" border=\"1\"\u003e\u003ccolgroup cols=\"3\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003eUI\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eAI\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003eWorkflow\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003ctr\u003e\u003cth align=\"left\" colspan=\"3\" nameend=\"c3\" namest=\"c1\"\u003e\u003cp\u003eIt cried wolf too many times but sometimes not at all\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eReduce alert noise through intuitive visualization of trends, patient-specific thresholds, and contextual cues that support clinician judgment or multiple AI-based prediction outputs.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eUse adaptive thresholds, delta changes, precision medicine, and calibrated predictions to minimise false positives and better capture subtle deterioration.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eDeliver timely, role-appropriate alerts while preserving opportunities for hands-on scrutiny of patient data.\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colspan=\"3\" nameend=\"c3\" namest=\"c1\"\u003e\u003cp\u003e\u003cb\u003eHunting for answers in a fractured system\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eIntegrate vitals, labs, documentation, and multiple predictions into a clean, role-adapted dashboard with trends and quick search functions related to the multiple prediction outputs\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eEnable cross-referencing of predictions with clinical data, provide search and filtering functions, and align predictions with clinicians\u0026rsquo; mental models to support rapid interpretation.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eEmbed predictions seamlessly into core EMR workflows, timed and tailored to clinicians\u0026rsquo; roles and responsibilities.\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colspan=\"3\" nameend=\"c3\" namest=\"c1\"\u003e\u003cp\u003e\u003cb\u003eNavigating complexity: single vs multi-predictions\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eDesign layered, drill-down interfaces that allow clinicians to explore multiple condition-specific alerts without overwhelming the main view; support proactive exploration while minimising cognitive overload.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eProvide multiple, detailed AI-based predictions that support differential diagnosis and augment clinician reasoning without priming towards specific conclusions; allow clinicians to filter, prioritise, or dismiss alerts as needed.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eIntegrate alert presentation into workflow in ways that prevent simultaneous alert overload, ensuring clinicians can respond effectively during busy shifts and maintain situational awareness.\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colspan=\"3\" nameend=\"c3\" namest=\"c1\"\u003e\u003cp\u003e\u003cb\u003eFrom chasing the problems to catching them early\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eKeep observations and trends visible in the interface to allow early recognition of deterioration; use dashboards that highlight subtle changes before alerts trigger.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eProvide proactive, context-aware predictions to identify early signs of deterioration and enable preventive actions.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eSupport clinicians in initiating preventive care before deterioration occurs; integrate proactive alerts with centralised communication and hospital resources to improve efficiency and patient safety.\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colspan=\"3\" nameend=\"c3\" namest=\"c1\"\u003e\u003cp\u003e\u003cb\u003eIt is only useful if I can act of it\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003ePresent alerts in actionable formats with clear severity ranking, prioritization, and role-specific guidance; allow on-demand querying and interactive exploration of patient data.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eGenerate predictions with actionable recommendations, urgency indicators, and clinical guidance tailored to existing protocols and staff expertise.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eIntegrate alerts into workflow to ensure timely responses, support role-appropriate actions (especially for juniors), and highlight priorities to prevent missed or delayed interventions.\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colspan=\"3\" nameend=\"c3\" namest=\"c1\"\u003e\u003cp\u003e\u003cb\u003eMore than a number: supporting interpretation of deterioration alerts\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eDisplay AI explanations in a clear, interpretable way, offering both simple trends and deeper feature-based insights; indicate model confidence and provide supporting evidence to enhance trust.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eProvide meaningful, patient-specific and cohort-level explanations that complement clinician reasoning; include metrics of prediction reliability and links to validation studies.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eIntegrate explanations into the clinical workflow to support real-time decisions, training, and adoption without slowing urgent actions.\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colspan=\"3\" nameend=\"c3\" namest=\"c1\"\u003e\u003cp\u003e\u003cb\u003eSuggesting priority of actions but not mandating them\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eDisplay alerts with clear prioritization cues for high-risk patients without forcing actions; use visual hierarchy to highlight urgency.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003ePrioritise high-risk predictions based on severity and likelihood of life-threatening conditions while allowing clinician discretion.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eSupport clinician decision-making by filtering and ranking patients according to risk, avoiding alert fatigue, and keeping clinicians in control of prioritization and actions.\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colspan=\"3\" nameend=\"c3\" namest=\"c1\"\u003e\u003cp\u003e\u003cb\u003eResponsive systems for real clinical complexity\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eImplement adaptive, role-specific user interfaces that allow clinicians to personalise alert displays and visualizations; enable intuitive interaction to provide feedback and additional data.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eIncorporate clinician gestalt and experience-based judgment into AI predictions; allow AI models to learn from continuous clinician feedback to refine accuracy and relevance.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eMaintain clinician control over choice of response; support communication among staff and integrate feedback loops to improve system performance and coordination in patient care.\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colspan=\"3\" nameend=\"c3\" namest=\"c1\"\u003e\u003cp\u003e\u003cb\u003eSingle watchtower vs local patrol\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eDesign interfaces that support both centralised and distributed alert viewing; provide login/logout flexibility and device-agnostic delivery; use auditory or visual cues to increase responsiveness.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eEnsure AI alerts can be routed to multiple recipients, track acknowledgement of receipt, and support collaborative management; adapt alert prioritisation based on real-time context and user roles.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eEnable collaborative alert management across teams to prevent single points of failure; integrate mixed centralised/distributed approaches for efficiency, coordination, and safeguarding junior clinicians.\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colspan=\"3\" nameend=\"c3\" namest=\"c1\"\u003e\u003cp\u003e\u003cb\u003eMore than just accurate: system that works for us!\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eDesign alerts that are visible but non-distracting, clear, understandable, and use multiple modalities (text, sound, visuals); include status indicators (active, acknowledged, dismissed); allow user customisation.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003ePrioritise alerts based on patient status and context; ensure relevance to the clinician receiving it; support safety-net functionality for new or junior staff.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eIntegrate alerts into clinical workflow as supportive tools rather than mandates; provide evidence of effectiveness to build trust and facilitate adoption; allow clinicians to select which alerts to act on while maintaining oversight of critical cases.\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colspan=\"3\" nameend=\"c3\" namest=\"c1\"\u003e\u003cp\u003e\u003cb\u003eTheme 4 \u0026ndash; The ghost in the machine\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eDesign interfaces that provide alert visibility and prioritisation but avoid overwhelming the user.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eAI should provide interpretable, context-aware predictions, indicate confidence levels, and dynamically adapt to evolving patient status while mitigating alarm fatigue.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eWorkflow should preserve clinician critical thinking; system should support appropriate verification of AI predictions, prevent desensitisation, and account for varying expertise levels and real-world clinical constraints.\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colspan=\"3\" nameend=\"c3\" namest=\"c1\"\u003e\u003cp\u003e\u003cb\u003eTheme 5 - Ghost in the loop: the role of user training in shaping the machine\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eInterface should be intuitive and user-friendly to support learning; include guided tutorials, contextual help, and clear visualizations to reinforce understanding.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eAI should act as an on-demand expert, offering real-time guidance, explanations, and actionable insights to bridge experience gaps.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eWorkflow should incorporate structured training programs, engage clinical champions, and provide ongoing reinforcement; consider generational differences in AI familiarity and ensure alignment with clinical reasoning.\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colspan=\"3\" nameend=\"c3\" namest=\"c1\"\u003e\u003cp\u003e\u003cb\u003eTheme 6 \u0026ndash; Beyond the ghost: the ecosystem matters\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eThe interface should integrate information from multiple sources into a single, coherent view, reducing fragmentation and enabling quick access.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eAI can support communication by highlighting relevant alerts or updates to appropriate staff, while considering system constraints.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eStandardised workflows, policies, and governance should be established before system design; integration into existing systems is crucial to streamline care and avoid inefficiencies.\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003eFeasibility took into consideration aspects of UI, AI, and workflow that are technically moderately or highly feasible versus those constrained by hospital infrastructure and organisational factors. These recommendations serve as a checklist for co-design and prototyping sessions, ensuring tools are grounded in clinician experience while effectively leveraging AI capabilities.\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eAssessment of feasibility of different design options for system components. Abbreviation: UI, User Interface, AI, Artificial Intelligence.\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"3\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003eUI\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eAI\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003eWorkflow\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e\u003cb\u003eHigh feasibility\u003c/b\u003e: Layered dashboards, status indicators, hierarchical structures, trend visualizations, search/filter functions, multi-modal alerts (text, sound, visual). These are mostly standard UI/UX practices and achievable with modern software development.\u003c/p\u003e\u003cp\u003e\u003cb\u003eModerate feasibility\u003c/b\u003e: Fully adaptive and role-specific interfaces, drill-down views for multiple AI predictions, integrating explanations and model confidence dynamically. Requires significant development effort and close collaboration with clinicians.\u003c/p\u003e\u003cp\u003e\u003cb\u003eLow feasibility\u003c/b\u003e: Real-time integration of multiple AI prediction outputs while avoiding cognitive overload; presenting all relevant patient data without clutter for proactive monitoring. Technically complex, and financially challenging, especially for legacy/proprietary EMR systems.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e\u003cb\u003eHigh feasibility\u003c/b\u003e: Generating multiple condition-specific outputs, providing patient-specific and cohort-level insights, integrating confidence scores, supporting prioritization based on severity. Current ML/AI methods can achieve this.\u003c/p\u003e\u003cp\u003e\u003cb\u003eModerate feasibility\u003c/b\u003e: Adaptive AI that learns continuously from clinician feedback, accounts for clinician gestalt, and dynamically updates predictions in real time. Conceptually possible but requires extensive validation and regulatory approval.\u003c/p\u003e\u003cp\u003e\u003cb\u003eLow feasibility\u003c/b\u003e: AI that perfectly mimics clinician reasoning, dynamically balances multiple alerts to prevent fatigue, and customises predictions per staff experience level. This is still cutting-edge and challenging to deploy safely.\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e\u003cb\u003eHigh feasibility\u003c/b\u003e: Delivering alerts in centralised or distributed systems, supporting collaborative alert management, providing actionable recommendations and role-appropriate guidance, integrating alerts into existing workflows.\u003c/p\u003e\u003cp\u003e\u003cb\u003eModerate feasibility\u003c/b\u003e: Seamless proactive monitoring before alerts trigger, integrating multi-source data into a single coherent workflow view, supporting on-demand querying. Requires workflow redesign and clinician input.\u003c/p\u003e\u003cp\u003e\u003cb\u003eLow feasibility\u003c/b\u003e: Personalising alert prioritisation across staff while maintaining safety and efficiency, dynamically adapting workflows in real time based on evolving patient states. Complex due to human, policy, and infrastructural constraints.\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e"},{"header":"4. Discussion","content":"\u003cp\u003eThis study explored clinicians\u0026rsquo; experiences with current and envisioned clinical deterioration tools, revealing a disconnect between the intended CDSS function and its real-world implementation [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e, \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e, \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. A MO-EWS can help bridge this gap by offering context-sensitive, adaptive outputs rather than a single fixed deterioration score. By learning patient-specific baselines and trajectories, such systems can support nuanced decision-making while reducing false alarms. When paired with transparent UIs, co-designed workflows, and customisable levels of detail, they can minimise alert fatigue, avoid tool overreliance, and preserve critical thinking [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]. Continuous feedback loops [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e, \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e] further ensure predictions evolve with additional quantitative data and clinician input.\u003c/p\u003e\u003cdiv id=\"Sec16\" class=\"Section2\"\u003e\u003ch2\u003e4.1. Single vs. Multiple AI-based Prediction Outputs\u003c/h2\u003e\u003cp\u003eAI predictions with multiple versus single outputs are not inherently better. Senior clinicians (P01-D, 30y; P04-D, 37y) disagreed in their choice of either, showing experience shapes views on usefulness, risk, and cognitive load. Accordingly, MO-EWS should adopt a customisable interface supporting both single and multiple outputs as desired by clinicians. This can be achieved through progressive disclosure (providing a key prediction by default, with the option to expand into multiple outputs), adaptive presentation (tailoring detail to patient risk level or clinician role), and co-design with diverse clinical groups in ensuring alignment with varying workflows, cognitive needs, and decision-making styles.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec17\" class=\"Section2\"\u003e\u003ch2\u003e4.2. Proactive systems\u003c/h2\u003e\u003cp\u003eDoctors desire proactive systems, not just reactive alerts, that monitor trends, respond to queries, and disclose context-aware information before alerts trigger, supporting clinicians to act preventively. Displaying patient-specific trends and trajectories across multiple conditions and offering layered outputs beginning with a synthesised summary but allowing drill-down into detailed predictions when required, can all help manage informational complexity without adding cognitive burden [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. Such systems can foster human\u0026ndash;AI teaming [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e], whereby AI is most effective when it reflects how clinicians think, not just delivers facts they need [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e]. The \u0026ldquo;layered\u0026rdquo; interface metaphor aligns with iterative reasoning and differential diagnosis, especially when coupled with explanations that make clinical sense [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e].\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec18\" class=\"Section2\"\u003e\u003ch2\u003e4.3. Situational awareness or tailored user experience\u003c/h2\u003e\u003cp\u003eCombining centralised and distributed methods of alert delivery can enhance team communication and decision-making, user-centred personalisation of output presentation tailored to cognitive style, role, expertise, or setting [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e] may further support workflow [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e] but could create information silos and undermine multidisciplinary collaboration and shared situational awareness. A more standardised approach would support consistency, coordination, and safety in acute care. This tension reveals a design dilemma, reflecting differing assumptions about hierarchy, role boundaries, and knowledge parity. Effective CDSS design should try to accommodate both perspectives with role-aware and modifiable interfaces, adapted to user preferences where possible.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec19\" class=\"Section2\"\u003e\u003ch2\u003e4.4. Authority or safety net\u003c/h2\u003e\u003cp\u003eSome clinicians value AI as a safety net or second opinion, while others resist systems that dictate decisions or priorities and threaten clinician autonomy and accountability [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e]. Safety concerns around clinicians overriding alerts highlights the need to balance automation with human judgment (\u0026ldquo;human-in-the-loop AI\u0026rdquo;) and take account of medico-legal liability if patients are harmed by inaccurate or misinterpreted AI systems. MO-EWS should offer context-aware recommendations while preserving clinicians\u0026rsquo; autonomy to interpret, adapt, and act.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec20\" class=\"Section2\"\u003e\u003ch2\u003e4.5. Reliability, integration and interpretability\u003c/h2\u003e\u003cp\u003eSystem reliability and integration are critical since out-of-date data and fragmented systems can delay or distort decision-making. Cognitive ergonomics and system design aligned with how clinicians think and act under pressure are required. Clear, timely and easily interpretable predictions delivered on the appropriate platform [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e] are also essential for trust and adoption [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. Failing to prioritise cognitive usability over computational precision can lead to alert fatigue, overload, and mistrust [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e].\u003c/p\u003e\u003cp\u003eRegarding feasibility, most UI and AI features are technically achievable with existing software development and machine learning methods. Major bottlenecks lie in workflow integration within hospital settings. High-level UI/AI capabilities risk being undermined if alerts, data, and predictions cannot be seamlessly integrated into clinical workflows. Grading of feasibility depends not only on technical UI/AI advances but also on the level of hospital digital infrastructure, organisational policies, and clinician preferences. Legacy EMRs and proprietary systems, associated with poor interoperability and organisational inertia, make MO-EWS implementation particularly challenging.\u003c/p\u003e\u003c/div\u003e\n\u003ch3\u003e5. Study implications\u003c/h3\u003e\n\u003cp\u003eClinician adoption of EWS rests on meeting diverse, often subjective expectations [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e]. Even well-designed tools may fail if expectations are unrealistic and not addressed through iterative, user-centred design and evaluation. Preserving autonomy and shared mental models requires careful design trade-offs, adaptive UIs, and frameworks centred on clinical reasoning needs rather than accuracy alone. Designers may underestimate workflow disruption and alert burden; clinician\u0026ndash;system dynamics should be understood, not treated as a straightforward technical problem to fix. This study proposes an opportunity for clinicians to compare perspectives, invites developers to collaborate with medical staff and organisations, and encourages researchers to use the identified themes as an investigative checklist that ensures EWS accuracy is not prioritised at the expense of usability and workflow integration.\u003c/p\u003e\n\u003ch3\u003e6. Limitations\u003c/h3\u003e\n\u003cp\u003eThis study is limited by reliance on self-reported experiences from volunteer clinicians in specific hospitals with particular EMR systems, which brings selection bias and may not generalise to other contexts. The 22 participants constrained subgroup exploration, limiting transferability of themes, although this is not a methodological weakness in RTA. Negative experiences may be overrepresented, introducing potential bias against beneficial redesign. Finally, conducting RTA from a single positional perspective may limit interpretive diversity, although clinician authors [name deleted to maintain the integrity of the review process] with expertise in deterioration reviewed the analyses.\u003c/p\u003e"},{"header":"5. Conclusion","content":"\u003cp\u003eThis study of clinician perspectives on current EWTs and future MO-EWS highlights persistent challenges in optimising CDSS. Effective solutions require real-time monitoring, responsive queries, user-centred modelling, and calibrated predictions, without violating cognitive usability principles[\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. More AI does not guarantee better support; interfaces must fit clinicians\u0026rsquo; mental models and workflows so the system, not the user, carries the burden of sense-making. Even small design\u0026ndash;practice mismatches risk unsafe decisions, missed detections, or delayed responses.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eTun HM, Rahman HA, Naing L, Malik OA (Dec. 2024) Trust in AI-Based Clinical Decision Support Systems Among Healthcare Workers: A Systematic Review (Preprint). J Med Internet Res. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.2196/69678\u003c/span\u003e\u003cspan address=\"10.2196/69678\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eYuan S, Yang Z, Li J, Wu C, Liu S (June 2025) AI-Powered early warning systems for clinical deterioration significantly improve patient outcomes: a meta-analysis. BMC Med Inf Decis Mak 25(1):203. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1186/s12911-025-03048-x\u003c/span\u003e\u003cspan address=\"10.1186/s12911-025-03048-x\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eVan Der Vegt AH et al (Jan. 2024) Systematic review and longitudinal analysis of implementing Artificial Intelligence to predict clinical deterioration in adult hospitals: what is known and what remains uncertain. J Am Med Inf Assoc 31(2):509\u0026ndash;524. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1093/jamia/ocad220\u003c/span\u003e\u003cspan address=\"10.1093/jamia/ocad220\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eNoselli M et al (2025) Presenting Artificial Intelligence Predictions Based on Electronic Medical Records to Clinicians in Hospitals: A Systematic Review, \u003cem\u003eIEEE J. Biomed. Health Inform.\u003c/em\u003e, vol. 29, no. 10, pp. 7619\u0026ndash;7632, Oct. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1109/JBHI.2025.3572494\u003c/span\u003e\u003cspan address=\"10.1109/JBHI.2025.3572494\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eJayawardena T, Baysari M, Bamgboje-Ayodele A (2025 Apr) Interface design features of clinical decision support systems for real-time detection of deterioration: A scoping review. Int J Med Informatics 22:105946\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eVerma AA (Oct. 2024) Toward the Rigorous Evaluation of Early Warning Scores. JAMA Netw Open 7(10):e2438966. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1001/jamanetworkopen.2024.38966\u003c/span\u003e\u003cspan address=\"10.1001/jamanetworkopen.2024.38966\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eC\u0026aacute;novas-Segura B, Morales A, Juarez JM, Campos M (July 2023) Meaningful time-related aspects of alerts in Clinical Decision Support Systems. A unified framework. J Biomed Inf 143:104397. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.jbi.2023.104397\u003c/span\u003e\u003cspan address=\"10.1016/j.jbi.2023.104397\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBlythe R et al (Sept. 2024) Clinician perspectives and recommendations regarding design of clinical prediction models for deteriorating patients in acute care. BMC Med Inf Decis Mak 24(1):241. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1186/s12911-024-02647-4\u003c/span\u003e\u003cspan address=\"10.1186/s12911-024-02647-4\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eOwoyemi A, Okpara E, Salwei M, Boyd A (Oct. 2024) End user experience of a widely used artificial intelligence based sepsis system. JAMIA Open 7(4):ooae096. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1093/jamiaopen/ooae096\u003c/span\u003e\u003cspan address=\"10.1093/jamiaopen/ooae096\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eZhang S et al (2024) Rethinking Human-AI Collaboration in Complex Medical Decision Making: A Case Study in Sepsis Diagnosis, in \u003cem\u003eProceedings of the CHI Conference on Human Factors in Computing Systems\u003c/em\u003e, May pp. 1\u0026ndash;18. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1145/3613904.3642343\u003c/span\u003e\u003cspan address=\"10.1145/3613904.3642343\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eButler JM et al (2025) \u0026lsquo;Be Really Careful about That\u0026rsquo;: Clinicians\u0026rsquo; Perceptions of an Intelligence Augmentation Tool for In-Hospital Deterioration Detection, \u003cem\u003eAppl. Clin. Inform.\u003c/em\u003e, vol. 16, no. 02, pp. 377\u0026ndash;392, Mar. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1055/a-2505-7743\u003c/span\u003e\u003cspan address=\"10.1055/a-2505-7743\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMalterud K, Siersma VD, Guassora AD (2016) Sample Size in Qualitative Interview Studies: Guided by Information Power, \u003cem\u003eQual. Health Res.\u003c/em\u003e, vol. 26, no. 13, pp. 1753\u0026ndash;1760, Nov. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1177/1049732315617444\u003c/span\u003e\u003cspan address=\"10.1177/1049732315617444\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBraun V, Clarke V (2024) Supporting best practice in reflexive thematic analysis reporting in \u003cem\u003ePalliative Medicine\u003c/em\u003e : A review of published research and introduction to the \u003cem\u003eReflexive Thematic Analysis Reporting Guidelines\u003c/em\u003e (RTARG), \u003cem\u003ePalliat. Med.\u003c/em\u003e, vol. 38, no. 6, pp. 608\u0026ndash;616, June \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1177/02692163241234800\u003c/span\u003e\u003cspan address=\"10.1177/02692163241234800\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eByrne D (June 2022) A worked example of Braun and Clarke\u0026rsquo;s approach to reflexive thematic analysis. Qual Quant 56(3):1391\u0026ndash;1412. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/s11135-021-01182-y\u003c/span\u003e\u003cspan address=\"10.1007/s11135-021-01182-y\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eAzeem M, Salfi NA, USAGE OF NVIVO SOFTWARE FOR QUALITATIVE DATA ANALYSIS (2012), vol. 2, no. 1\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eAmmenwerth E (2019) Technology Acceptance Models in ealth nformatics: TAM and UTAUT. Stud Health Technol Inf 263:64\u0026ndash;71. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3233/SHTI190111\u003c/span\u003e\u003cspan address=\"10.3233/SHTI190111\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eTong A, Sainsbury P, Craig J (2007) Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups, \u003cem\u003eInt. J. Qual. Health Care\u003c/em\u003e, vol. 19, no. 6, pp. 349\u0026ndash;357, Sept. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1093/intqhc/mzm042\u003c/span\u003e\u003cspan address=\"10.1093/intqhc/mzm042\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eCampbell V et al (Aug. 2020) Predicting clinical deterioration with Q-ADDS compared to NEWS, Between the Flags, and eCART track and trigger tools. Resuscitation 153:28\u0026ndash;34. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.resuscitation.2020.05.027\u003c/span\u003e\u003cspan address=\"10.1016/j.resuscitation.2020.05.027\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eCho H et al (May 2022) Assessing the Usability of a Clinical Decision Support System: Heuristic Evaluation. JMIR Hum Factors 9:e31758. no. 210.2196/31758\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eAbdulnour R-EE, Gin B, Boscardin CK (2025) Educational Strategies for Clinical Supervision of Artificial Intelligence Use, \u003cem\u003eN. Engl. J. Med.\u003c/em\u003e, vol. 393, no. 8, pp. 786\u0026ndash;797, Aug. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1056/NEJMra2503232\u003c/span\u003e\u003cspan address=\"10.1056/NEJMra2503232\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eWeigand L, Viers T, Tipton E (2025) Integration of Rapid Response Teams and Early Warning Systems to Reduce Cardiac Arrests and Intensive Care Unit Readmissions, \u003cem\u003eCrit. Care Nurse\u003c/em\u003e, vol. 45, no. 4, pp. 49\u0026ndash;56, Aug. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.4037/ccn2025131\u003c/span\u003e\u003cspan address=\"10.4037/ccn2025131\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eGandhi TK et al (July 2023) How can artificial intelligence decrease cognitive and work burden for front line practitioners? JAMIA Open 6(3):ooad079. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1093/jamiaopen/ooad079\u003c/span\u003e\u003cspan address=\"10.1093/jamiaopen/ooad079\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eWang L et al (June 2023) Human-centered design and evaluation of AI-empowered clinical decision support systems: a systemtic review. Front Comput Sci 5:1187299. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3389/fcomp.2023.1187299\u003c/span\u003e\u003cspan address=\"10.3389/fcomp.2023.1187299\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eElgin CY, Elgin C (Dec. 2024) Ethical implications of AI-driven clinical decision support systems on healthcare resource allocation: a qualitative study of healthcare professionals\u0026rsquo; perspectives. BMC Med Ethics 25(1):148. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1186/s12910-024-01151-8\u003c/span\u003e\u003cspan address=\"10.1186/s12910-024-01151-8\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSirajuddin AM, Osheroff JA, Sittig DF, Chuo J, Velasco F, Collins DA (2012) Implementation Pearls from a New Guidebook on Improving Medication Use and Outcomes with Clinical Decision Support:\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eRaita E, Oulasvirta A (July 2011) Too good to be bad: Favorable product expectations boost subjective usability ratings. Interact Comput 23(4):363\u0026ndash;371. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.intcom.2011.04.002\u003c/span\u003e\u003cspan address=\"10.1016/j.intcom.2011.04.002\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"},{"header":"Box","content":"\u003cp\u003eBox is available in the Supplementary Files section.\u003c/p\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"The University of Queensland","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Artificial Intelligence, Clinical decision support system, Human Factors, Qualitative analysis","lastPublishedDoi":"10.21203/rs.3.rs-8071546/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8071546/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAI-enabled clinical decision support systems (CDSS) can help detect patient deterioration, but their value depends on outputs that fit workflows, support reasoning, and avoid alert fatigue.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eObjective.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eLittle is known about clinician perspectives on multi-output Early Warning Systems (MO-EWS), which provide simultaneous predictions (e.g., overall deterioration, organ- or diagnosis-specific). This study explores clinician views on future MO-EWS and offers user-centred recommendations to improve design, usability, and adoption.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eReflexive thematic analysis was used to examine clinicians’ perceptions of existing early warning systems (EWS) and envisioned AI-based tools for predicting clinical deterioration. Between January and June 2025, 22 clinicians from Australian hospitals with experience in managing clinical deterioration were interviewed using a semi-structured questionnaire. Resulting themes were mapped to system components (user interface [UI], AI model, clinical workflow) to inform practical design recommendations which were then assessed for their feasibility.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eParticipants found existing EWS distracting and sometimes unsafe. They envisioned AI-based systems that provide multiple, contextualised outputs (e.g., causes of deterioration, organ dysfunction, clinical severity) as layered, adaptable tools to support reasoning, prioritisation, and communication. While many UI improvements are readily achievable, model development is more complex, and workflow integration will depend on infrastructure, policies, and clinician preferences.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusion\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eClinicians perceive potential benefits in an AI-enabled system that provides more insightful information when predicting clinical deterioration in individual patients, but various design and implementation challenges remain.\u003c/p\u003e","manuscriptTitle":"Clinicians’ perspectives on the design of early warning systems presenting AI-based multiple outputs for clinical deterioration: A qualitative analysis","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-11-11 17:22:40","doi":"10.21203/rs.3.rs-8071546/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"52ded57c-504f-4748-b011-bb9e5a7b36f7","owner":[],"postedDate":"November 11th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":57688433,"name":"Medical Informatics"},{"id":57688434,"name":"Software Engineering"},{"id":57688435,"name":"Artificial Intelligence and Machine Learning"}],"tags":[],"updatedAt":"2025-11-11T17:22:41+00:00","versionOfRecord":[],"versionCreatedAt":"2025-11-11 17:22:40","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8071546","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8071546","identity":"rs-8071546","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}