Rule-Based Electronic Sepsis Alerts Identify High-Risk Patients Despite Poor Diagnostic Accuracy: A Real-World Evaluation and Implications for Machine Learning

Rule-Based Electronic Sepsis Alerts Identify High-Risk Patients Despite Poor Diagnostic Accuracy: A Real-World Evaluation and Implications for Machine Learning | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Rule-Based Electronic Sepsis Alerts Identify High-Risk Patients Despite Poor Diagnostic Accuracy: A Real-World Evaluation and Implications for Machine Learning Eanna L Lowney, Steven G Hirth, Laura Fanning BPharm, Graeme J Duke, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8188665/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 5 You are reading this latest preprint version Abstract Objective To evaluate the diagnostic accuracy of an electronic sepsis alert system (ESAS) in an acute care hospital within an electronic medical record (eMR) system. Design Single-centre observational study of prospectively collected data from the eMR incorporating a third-party electronic sepsis surveillance and alerting system. Clinical eMR and administrative coding data for all patient records were analysed. Performance characteristics of the ESAS were compared with the presence or absence of clinical sepsis. Setting A university-affiliated hospital in Melbourne, Australia with 25,000 multiday-stay admissions per annum. Participants All adult multiday-stay admissions between January 1 st , 2018, and December 31 st , 2019, inclusive. Main Outcome measures Sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) of the ESAS. Results 149,053 records were included in the study, of which 4,011 triggered an electronic sepsis alert. The sensitivity and PPV of the ESAS were 26.3% [95% CI, 25.1-27.6%] and 33.2% [95% CI, 31.7-34.7%] respectively, while its specificity and NPV were 98% [95% CI, 98.0-98.1%] and 97.3% [95% CI, 97.2–97.4%] respectively. Conclusion The ESAS was highly specific but lacked sensitivity for reliable clinical application. The activation of ESAS was associated with a longer length of stay, higher rates of Intensive Care Unit admission and in-hospital mortality. The ESAS ultimately identified a cohort at risk of clinical deterioration. These results highlight fundamental limitations of rule-based approaches and underscore the need for adaptive machine learning systems that can better integrate complex clinical patterns for early sepsis detection. Sepsis Infection Electronic Sepsis Alert Electronic Medical Record Clinical Decision Support Systems Machine Learning Artificial Intelligence Implementation Science. Figures Figure 1 Figure 2 Introduction Sepsis is a leading global cause of preventable morbidity and mortality. 1 – 4 Early diagnosis, administration of antimicrobials and source control are the cornerstones of management and have been shown to increase survival. 5 – 8 Sepsis is defined as life-threatening organ dysfunction caused by a dysregulated host response to acute infection 9 and is associated with a significant clinical and rising financial burden. 10 Early diagnosis of sepsis in the hospital population is hampered by the absence of a gold standard diagnostic test and the need to collate diverse clinical and laboratory features to prompt clinical suspicion and distinguish sepsis from non-infectious diseases. 11 , 12 The introduction of electronic medical record (eMR) systems has provided a unique opportunity for automated early warning alerts for the potential presence of sepsis. 13 – 21 Numerous rule-based electronic sepsis alert systems (ESAS), often utilising outdated sepsis definitions, have been trialled in emergency departments and inpatient settings. 22 While these studies have primarily focused on outcome measures such as mortality, hospital length of stay (LOS), and time to antimicrobial administration, few have assessed the diagnostic accuracy of an ESAS as a reliable sepsis surveillance tool. 14 , 15 , 18 , 22 Moreover, many of these alerts rely on criteria based on the systemic inflammatory response syndrome (SIRS) rather than the Sepsis-3 international definition of sepsis, creating a highly sensitive tool but one with a low positive predictive value (PPV). 23 This has been shown to lead to the misdiagnosis of sepsis, inappropriate administration of antimicrobials 17 and potentially contributes to alert fatigue amongst clinicians. 24 , 25 Despite efforts to optimise alert criteria, balancing sensitivity and specificity remains a challenge. 19 Recent advances in machine learning (ML) and artificial intelligence (AI) offer promising alternatives to rule-based systems. However, before ML-based approaches can be widely adopted, it is essential to rigorously evaluate the performance of existing rule-based systems in real-world clinical settings to establish baseline performance and identify specific areas for improvement. 26 We sought to evaluate the diagnostic accuracy of a rule-based ESAS that was designed in accordance with the latest international sepsis definition. Our study was conducted in a metropolitan teaching hospital in Australia equipped with a comprehensive hospital-wide eMR system. Categorical data are presented as frequency, percentage and mean with standard deviation (SD); continuous data are presented as median and interquartile range (IQR). Statistical comparison between groups was performed using t-test or Wilcoxon rank sum, respectively. All data linkage and statistical analysis were conducted in StataMP™ v17.0 (2019, College Station, TX). Methods Electronic Sepsis Alert System The Millennium™ (Cerner Corp., Kansas City, MO) eMR was introduced throughout the hospital in October 2017 and included a third-party electronic sepsis surveillance and alerting system. 27 The rule-based sepsis alert system was redesigned to align with the Sepsis-3 international definition of sepsis. 9 It continuously monitors patient vital signs and laboratory data collected in real-time by the eMR and compares these data with predetermined diagnostic criteria consistent with the presence of sepsis. Two levels of alerts - Possible Sepsis Alert (PSA) and Sepsis Alert (SA) - are triggered according to the number and severity of clinical criteria present within a specific time window. At least three minor criteria must reach threshold values to trigger a PSA; whereas an SA is triggered by the presence of any two minor criteria plus at least one major feature suggesting organ dysfunction (Table 1 ). The presence of either alert then triggers an automated message to the bedside nurse, as well as to the treating doctor in the event of a SA. This message describes the clinical criteria present with the date/time stamp. In addition, the eMR prompts the primary nurse with the option and guidance to initiate investigations and treatment for suspected sepsis while waiting for assessment by the medical team. The eMR also provides the medical team with an interactive pathway (PowerPlan) for investigation and treatment of suspected sepsis. These data were collected prospectively and stored in the eMR. Table 1 Vital Sign and Organ Dysfunction Parameters for Possible Sepsis Alert and Sepsis alert respectively. Calling Criteria (unit of measure) Less than Greater than Vital Sign Respiratory Rate (breaths/min) 11 24 Heart Rate (bpm) 51 119 Temperature (centigrade) 35.5 38.5 WCC (x10^9/L) 4 12 Bands Absolute - 59% Change in Behaviour (Yes/No) - - Organ Dysfunction Creatinine (umol/L) ∆ > 44.2 umol/L SBP (mmHg) 90 MAP (mmHg) 65 Lactate (mmol/L) 2 Bilirubin (umol/L) 1.71 34.2 Study setting, Population and Design This single-centre observational study was undertaken at a university-affiliated hospital in Australia with 25,000 multiday-stay admissions per annum, a 24-hour cardiac catheter laboratory, 10-bed Intensive Care Unit and comprehensive acute inpatient specialty services. Inclusion criteria were all adult (≥ 18 years) multiday-stay inpatient admissions between January 1st, 2018, and December 31st, 2019, inclusive. Paediatric admissions (< 18 years) and Emergency Department (ED) attendances that were not admitted to an inpatient unit were excluded from the study. Sepsis alerts triggered during ED assessment and resuscitation were excluded from our analysis, as physiological derangements from various infectious and non-infectious conditions may mimic sepsis and confound the results. All available data for PSA and SA including clinical triggers, timing, interventions, outcomes and date of hospital admission were extracted from the eMR and linked with administrative diagnosis coded data containing International Classification of Diseases and Health Related Problems 10th edition – Australian Modification (ICD-10-AM) using unique patient identifiers and admission and discharge dates. The presence of either sepsis alert (PSA or SA) at any time during the inpatient acute care phase were established. Where multiple alerts for the same patient occurred during a single hospital episode, only the first PSA and first SA were included. The presence or absence of sepsis was determined from the clinical diagnoses coded in the administrative dataset. This is based on a published (“synchronous”) methodology that uses ICD-10-AM diagnosis codes and was validated against the reference standard of clinical chart review. 28 Sepsis was deemed to be present if at least one diagnosis code for an acute infection was accompanied by a second, synchronous diagnosis consistent with acute organ dysfunction. This method has been shown to yield excellent positive and negative predictive values and outperforms previously published (“explicit” and “implicit”) methods for identifying sepsis in administrative data. Note that the explicit term “sepsis,” found in many ICD-10-AM diagnoses, is alone insufficient and poorly predictive of sepsis. 29 Clinical (acute and comorbid) diagnoses and demographic characteristics (age, sex, ethnicity, admission source) were extracted from the administrative dataset. The burden of comorbid disease was quantified using both the Charlson and Elixhauser methods 30 and the presence of clinical frailty using the Gilbert method. 31 Patient and episode identifiers were retained for data linkage of eMR and administrative datasets and then removed prior to analysis. During the study period, clinical staff were informed of all sepsis alerts (as described above) and all management decisions, including response to any sepsis alert, were determined by the treating team with usual care. This study was conducted in accordance with the National Statement on Ethical Conduct in Human Research (NHMRC, 2007, updated 2018) and the principles of the Declaration of Helsinki (2013). The Eastern Health Human Research Ethics Committee approved this investigation (LR68-2018) and the need for patient consent was waived due to the unblinded and observational methodology, analysis of de-identified data and reporting of aggregated results. Statistical Analysis Performance metrics for the ESAS were determined by comparison with the sepsis population identified from the administrative dataset as the reference standard. Primary outcomes included sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) for any sepsis alert (PSA or SA); and were calculated with the user written Stata (statistical software) command “diagt”. 32 The derived area under the receiver operating characteristic curve (AUROC) value provides a combined estimate of both sensitivity and specificity. A value above 0.7 was deemed acceptable but above 0.8 was desirable. Secondary outcomes included the prevalence of acute infection with or without sepsis, admission to the Intensive Care Unit (ICU), hospital survival and hospital length of stay (LOS). Analyses were repeated separately for either PSA or SA and for any (PSA or SA) alert and the latter are reported herein, unless otherwise stated. Sensitivity analyses were undertaken for the following subgroups: patient diagnosed with acute infection, patients aged > 65 years, and patients who were admitted following an inter-hospital transfer. Results From January 2018 to December 2019 there were 149,075 multiday-stay admissions (Fig. 1 ). Following the exclusion of paediatric records, 142,053 (95%) inpatient admissions were included in the analysis. There were 25,619 (18.0; 95% CI 17.8–18.3%) admissions associated with a diagnosis of acute infection at any time during the hospital episode and 5,058 (3.6%) patient admissions were classified as clinical sepsis, giving a prevalence rate of 3.56 (95% CI, 3.46–3.66) per 100 admissions. The median lead time to the first sepsis alert was 2 days [IQR, 1–6] (Appendix; supplementary table 1 ). A comparison of the study cohorts which did and did not trigger any sepsis alert (PSA and SA) are provided in Table 2 (and Appendix; supplementary table 1 ). Notably, those who experienced any sepsis alert were older (median, 69 [IQR, 53–79] v 64 [IQR, 45–78] years; P < 0.001), more likely to follow an emergency admission (87% vs 43%; P < 0.001) and experienced a longer LOS (mean, 7.4 [SD, 12.2] v 1.8 [SD, 4.3]; P < 0.001) compared to those who did not trigger any sepsis alert. Both Elixhauser and Charlson comorbidity scores were significantly higher in those that triggered any sepsis alert (73% v 48%; P < 0.001 and 48% v 24%; P < 0.001, respectively). Furthermore, those who triggered a sepsis alert were more frequently admitted to ICU (13% vs 2%; P < 0.001) or died in-hospital (6% vs 0.8%; P < 0.001) than those with no sepsis alert. Table 2 Demographic characteristics of Study Population. Data presented as n (%), unless otherwise indicated. Separations Any alert No alert p-value n (%) 142,053 4,011 (2.8) 138,042 (97.2) Age (years), median [IQR] 64 [46–78] 69 [53–79] 64 [45–78] < 0.001 Male 69,790 (49) 2,237 (56) 67,553 (49) < 0.001 Aged-care resident 2,941 (2) 82 (2) 2,859 (2) 0.586 Emergency admission 63,291 (46) 3,482 (87) 59,809 (43) < 0.001 Inter-hospital transfer 3,855 (3) 1,246 (31) 2,609 (2) < 0.001 Indigenous 420 (0.3) 10 (0.2) 410 (0.3) 0.64 Infectious disease 25,619 (18) 2,356 (59) 23,263 (17) < 0.001 Sepsis present 5,058 (4) 1,331 (33) 3,727 (3) 0 68,556 (48) 2,935 (73) 65,621 (48) 0 34,718 (24) 1,922 (48) 32,796 (24) < 0.001 Mean LOS, Days [SD] 1.9 [4.8] 7.4 [12.2] 1.8 [4.3] < 0.001 Lead time to first alert, day [IQR] 2 [1–6] 2 [1–5] na na Performance metrics for the sepsis alert system are depicted in Fig. 2 and summarised in Tables 3 and 4 . The sensitivity and PPV for the presence of sepsis were 26.3% [95% CI, 25.1–27.6%] and 33.2% [95% CI, 31.7–34.7%] while the specificity and NPV were 98% [95% CI, 98-98.1%] and 97.3% [95% CI, 97.2–97.4%], respectively. The AUROC of 0.62 suggests an inefficient test. Table 3 Performance metrics for the sepsis alert system (combined PSA and SA). Metric Mean 95% confidence interval Prevalence 3.6% 3.5–3.66% Sensitivity 26.3% 25.1–27.6% Specificity 98% 98.0–98.1% AUROC 62% 61.6–62.8% Positive predictive value 33.2% 31.7–34.7% Negative predictive value 97.3% 97.2–97.4% Table 4 Total numbers of true positives, true negatives, false positives and false negatives for the sepsis alert system Sepsis No Sepsis Total Any Alert 1,331 2,680 4,011 No Alert 3,727 134,315 138,042 Toal 5,058 136,995 142,053 Sensitivity analyses in subgroups with acute infection, or those aged > 65years, did not reveal any improvement in performance of sepsis alerts (Appendix; supplementary table 2). Some improvement in sensitivity 91.9% [95% CI, 89.4–94%] and PPV 44.5% [95% CI, 41.7–47.4%] was identified in the subgroup following inter-hospital transfer. The green circle denotes admissions classified as clinical sepsis, the blue circle denotes Sepsis Alerts (SA), and the red circle denotes Possible Sepsis Alerts (PSA). The grey square represents the total inpatient study population. Overlap between circles indicates true-positive alerting, while sepsis cases outside the alert circles represent missed cases (false negatives) and alert activations outside the sepsis circle represent false positives. Circle size is proportional to the number of admissions in each group. Discussion We screened nearly 150,000 consecutive hospital separations and their corresponding admission data identifying 18.0% with an acute infection and 3.6% with sepsis. Analysis of over 4,000 automated sepsis alerts generated by the hospital eMR revealed that the alert algorithm had a poor PPV (33.2% [95% CI, 31.7–34.7%]) despite a high NPV (97.3% [95% CI, 97.2–97.4%]) for the presence of sepsis. The automated sepsis alerts did not perform with sufficient reliability as a clinical prompt for sepsis. In response to the results of this study and considering the impending COVID-19 pandemic, the study hospital elected to discontinue the automated sepsis alert system. Retaining a low sensitivity alert system may have led to missed sepsis cases and a delay in key interventions. This case illustrates that technical deployment of health information technology must be accompanied by ongoing performance monitoring and a readiness to modify or withdraw systems when warranted. These observations are relevant to the future development of automated clinical alerts, including next-generation systems that may incorporate adaptive or machine-learning-based approaches. Despite its poor performance in identifying sepsis, the algorithm did appear to identify a patient cohort with a higher risk of prolonged hospital stay, transfer to ICU or in-hospital death (Table 2 ). Therefore, similar automated alert systems may help clinicians identify a cohort at higher risk of clinical deterioration and adverse events in line with Standard-8 of the Australian National Healthcare Standards. 33 An expanded version of the existing algorithm may find clinical application for this purpose and warrants further investigation. Sepsis diagnosis requires the triangulation of vital signs and laboratory parameters together integrated with clinical findings that indicate the presence of infection, while also excluding confounding factors such as non-infectious inflammatory states that can mimic severe infection. 11 The low sensitivity and PPV of this alert system are consistent with other electronic sepsis alerts developed for inpatient and critical care settings. 13 , 19 – 21 These findings reflect the non-specific nature of the diagnostic criteria for sepsis and its tendency to overlap with many non-infectious diseases that lead to significant organ dysfunction. The balance between early warning and excessive over-treatment is challenging. 17 While our study demonstrated that this ESAS had high specificity, its low sensitivity and PPV limit its clinical reliability. This reflects inherent limitations of rule-based systems that depend on fixed thresholds and predefined criteria. These systems lack contextual awareness, perform poorly in heterogeneous clinical settings, and are unable to adapt to evolving patient data or changing sepsis definitions. 13 , 34 Our findings align with a growing body of evidence demonstrating that rule-based approaches are insufficient for complex clinical predictions like sepsis, where subtle patterns and temporal relationships are critical. 13 In contrast, machine learning (ML)-based approaches have shown promise in overcoming many of these challenges. ML models can incorporate an array of continuous and categorical variables, model non-linear relationships, and detect subtle temporal patterns that may precede overt clinical deterioration. Several studies have demonstrated that ML algorithms can outperform rule-based systems, achieving higher sensitivity and PPV, with improved AUROC values when predicting sepsis. 26 , 35 When combined with timely provider review, ML-based early warning systems have shown potential to reduce mortality and hospital length of stay in sepsis patients. 34 , 36 Additionally, these systems may offer significant cost-saving potential by enabling earlier intervention and reducing the need for ICU admissions. 37 This study provides a real-world evaluation of a widely deployed sepsis alert system and demonstrates the essential role of post-implementation assessment in health informatics. Our findings serve as an important benchmark for evaluating rule-based approaches and provide a foundation for developing more sophisticated, adaptive systems. However, further research is warranted to determine whether ML-based alerts can substantially improve diagnostic performance while avoiding new risks such as alert fatigue, lack of clinical transparency, or inappropriate treatment of non-infectious conditions under the mistaken presumption of sepsis. 17 , 24 The strengths of this study include a large study population and whole-of-hospital cohort with a sufficiently high prevalence of sepsis (3.6%). The two-month eMR run-in period, prior to study commencement, permitted time for education of staff including familiarity with the sepsis alert system. Study data were collected over a two-year period capturing seasonal and case mix variation in the frequency of sepsis. Data collection and provisional analysis were completed prior to the SARS-COV19 pandemic. To date, this is the largest published study to address the clinical validity of automated sepsis alerts in Australia. There are several important limitations in our study. The findings of this single centre study may not necessarily be generalisable to other sites. Clinical diagnosis and treatment following a sepsis alert were left to the judgement of the treating team and the study hospital does not employ protocolised early-goal directed therapy for sepsis. Additionally, the presence or absence of sepsis was determined from the clinical diagnoses coded in the administrative dataset. While the use of ICD-10-AM coding data has been shown to over or under report cases of sepsis when compared to the gold standard of prospective chart review for objective markers of infection, 3,29 the method utilised in our study outperforms other methods for identifying sepsis in administrative data. 28 . Although this inaccuracy may affect the calculated performance metrics of the electronic sepsis alerts potentially leading to deviations from the true diagnostic accuracy that would be achieved through comprehensive chart review, our methodology permitted a larger study population over a longer duration with limited research resources and the (SARS-COV19) pandemic imminent. Conclusion We investigated the performance and reliability of an automated clinical alert system for early identification of in-hospital sepsis. While the algorithm appears to be highly specific, it lacked sufficient sensitivity for clinical purposes. Further research and clinical validation are required to refine these potentially attractive systems to improve patient care without increasing unnecessary workload burden or exposing patients to risks of unnecessary investigations and treatment. Abbreviations ESAS Electronic Sepsis Alert System eMR Electronic Medical Record ICU Intensive Care Unit LOS Length of Stay PPV Positive Predictive Value NPV Negative Predictive Value AUROC Area Under the Receiver Operating Characteristic Curve ICD-10-AM International Classification of Diseases, 10th Revision, Australian Modification Declarations Ethics Approval and consent to participate This study was conducted in accordance with the National Statement on Ethical Conduct in Human Research (NHMRC, 2007; updated 2018) and the principles of the Declaration of Helsinki (2013). Ethical approval was granted by the Eastern Health Human Research Ethics Committee (Reference number LR68-2018 ). The requirement for informed consent to participate was waived by the ethics committee due to the retrospective observational design, use of routinely collected clinical and administrative data, analysis of de-identified information, and reporting of aggregated results. Consent for publication Not applicable. This study did not include identifiable individual patient data. Authorship Contributions Eanna Lowney: Conceptualisation, Methodology, investigation, writing- original draft, review and editing, project administration. Laura Fanning: Conceptualisation, Methodology, investigation, formal analysis, writing- review and editing. Steve Hirth : Conceptualisation, formal analysis, investigation. Graeme Duke : Supervision, conceptualisation, methodology, investigation, writing- original draft, review and editing, project administration. Owen Roodenburg : Supervision, conceptualisation, methodology, writing- review and editing, project administration. Acknowledgements Not Applicable. Funding The authors received no financial support for the research, authorship, and/or publication of this article. Data availability statement The authors confirm that the data supporting the findings of this study are available within the article and its supplementary materials. Competing Interests The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. References Vincent J-L, Marshall JC, Ñamendys-Silva SA, et al. Assessment of the worldwide burden of critical illness: the Intensive Care Over Nations (ICON) audit. Lancet Respir Med. 2014;2:380–6. Rudd KE, Johnson SC, Agesa KM, et al. Global, regional, and national sepsis incidence and mortality, 1990–2017: analysis for the Global Burden of Disease Study. Lancet Lond Engl. 2020;395:200–11. Angus DC, Linde-Zwirble WT, Lidicker J, et al. Epidemiology of severe sepsis in the United States: Analysis of incidence, outcome, and associated costs of care. Crit Care Med. 2001;29:1303–10. Fleischmann C, Scherag A, Adhikari NKJ, et al. Assessment of Global Incidence and Mortality of Hospital-treated Sepsis. Current Estimates and Limitations. Am J Resp Crit Care. 2016;193:259–72. Seymour CW, Gesten F, Prescott HC, et al. Time to Treatment and Mortality during Mandated Emergency Care for Sepsis. New Engl J Med. 2017;376:2235–44. Ferrer R, Martin-Loeches I, Phillips G, et al. Empiric Antibiotic Treatment Reduces Mortality in Severe Sepsis and Septic Shock From the First Hour. Crit Care Med. 2014;42:1749–55. Dellinger RP, Levy MM, Rhodes A, et al. Surviving Sepsis Campaign: International Guidelines for Management of Severe Sepsis and Septic Shock, 2012. Intensive Care Med. 2013;39:165–228. Reitz KM, Kennedy J, Li SR, et al. Association Between Time to Source Control in Sepsis and 90-Day Mortality. JAMA Surg. 2022;157:817–26. Singer M, Deutschman CS, Seymour CW, et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016;315:801–10. Li L, Sunderland N, Rathnayake K, et al. Sepsis epidemiology in Australian Public Hospitals, a nationwide longitudinal study (2013–2018). Infect Dis Heal. 2021;26:S9. Vincent J-L. The Clinical Challenge of Sepsis Identification and Monitoring. Plos Med. 2016;13:e1002022. Duncan CF, Youngstein T, Kirrane MD, et al. Diagnostic Challenges in Sepsis. Curr Infect Dis Rep. 2021;23:22. Makam AN, Nguyen OK, Auerbach AD. Diagnostic accuracy and effectiveness of automated electronic sepsis alert systems: A systematic review. J Hosp Med. 2015;10:396–402. Manaktala S, Claypool SR. Evaluating the impact of a computerized surveillance algorithm and decision support system on sepsis mortality. J Am Med Inf Assn. 2017;24:88–95. Austrian JS, Jamin CT, Doty GR, et al. Impact of an emergency department electronic sepsis surveillance system on patient mortality and length of stay. J Am Med Inf Assn. 2017;25:523–9. Warttig S, Alderson P, Evans DJ et al. Automated monitoring compared to standard care for the early detection of sepsis in critically ill patients. Cochrane Db Syst Rev. Epub ahead of print 2018. 10.1002/14651858.cd012404.pub2 Seetharaman S, Wilson C, Landrum M et al. Does use of Electronic Alerts for Systemic Inflammatory Response Syndrome (SIRS) to Identify Patients with Sepsis Improve Mortality? Am J Med. Epub ahead of print 2019. 10.1016/j.amjmed.2019.01.032 Downing NL, Rolnick J, Poole SF et al. Electronic health record-based clinical decision support alert for severe sepsis: a randomised evaluation. Bmj Qual Saf 2019; bmjqs-2018-008765. Li L, Rathnayake K, Green M, et al. Improving the Performance of Clinical Decision Support for Early Detection of Sepsis: A Retrospective Observational Cohort Study. Stud Heal Technol Inf. 2019;264:679–83. Giannini HM, Ginestra JC, Chivers C, et al. A Machine Learning Algorithm to Predict Severe Sepsis and Septic Shock: Development, Implementation, and Impact on Clinical Practice*. Crit Care Med. 2019;47:1485–92. Topiwala R, Patel K, Twigg J, et al. Retrospective Observational Study of the Clinical Performance Characteristics of a Machine Learning Approach to Early Sepsis Identification. Crit Care Explor. 2019;1:e0046. Kim H-J, Ko R-E, Lim SY, et al. Sepsis Alert Systems, Mortality, and Adherence in Emergency Departments. JAMA Netw Open. 2024;7:e2422823. Hwang MI, Bond WF, Powell ES. Sepsis Alerts in Emergency Departments: A Systematic Review of Accuracy and Quality Measure Impact. West J Emerg Med. 2020;21:1201–10. Kane-Gill SL, O’Connor MF, Rothschild JM, et al. Technologic Distractions (Part 1): Summary of Approaches to Manage Alert Quantity With Intent to Reduce Alert Fatigue and Suggestions for Alert Fatigue Metrics. Crit care Med. 2017;45:1481–8. Sawyer AM, Deal EN, Labelle AJ, et al. Implementation of a real-time computerized sepsis alert in nonintensive care unit patients* Crit Care Med 2011; 39: 469–473. Shimabukuro DW, Barton CW, Feldman MD, et al. Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial. BMJ Open Respir Res. 2017;4:e000234. Amland RC, Hahn-Cover KE. Clinical Decision Support for Early Recognition of Sepsis. Am J Med Qual. 2016;31:103–10. Duke GJ, Bishara M, Hirth S et al. Performance of hospital administrative data for detection of sepsis in Australia: The sepsis coding and documentation (SECOND) study. Health Inf Manag J 2022; 183335832211077. Martin GS, Mannino DM, Eaton S, et al. The Epidemiology of Sepsis in the United States from 1979 through 2000. New Engl J Med. 2003;348:1546–54. van Walraven C, Austin PC, Jennings A, et al. A Modification of the Elixhauser Comorbidity Measures Into a Point System for Hospital Death Using Administrative Data. Med Care. 2009;47:626–33. Gilbert T, Neuburger J, Kraindler J, et al. Development and validation of a Hospital Frailty Risk Score focusing on older people in acute care settings using electronic hospital records: an observational study. Lancet Lond Engl. 2018;391:1775–82. Seed P. DIAGT: Stata module to report summary statistics for diagnostic tests compared to true disease status. Statistical Software Components S423401, Boston College Department of Economics , https://EconPapers.repec.org/RePEc:boc:bocode:s423401 . (2010). National Safety and Quality Health Service Standards. National Safety and Quality Health Service Standards. 2nd ed. Sydney: ACSQHC; 2017. Adams R, Henry KE, Sridharan A, et al. Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis. Nat Med. 2022;28:1455–60. Bedoya AD, Futoma J, Clement ME, et al. Machine learning for early detection of sepsis: an internal and temporal validation study. JAMIA Open. 2020;3:252–60. Henry KE, Adams R, Parent C, et al. Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timing. Nat Med. 2022;28:1447–54. Ericson O, Hjelmgren J, Sjövall F, et al. The Potential Cost and Cost-Effectiveness Impact of Using a Machine Learning Algorithm for Early Detection of Sepsis in Intensive Care Units in Sweden. J Heal Econ Outcomes Res. 2022;9:101–10. Additional Declarations No competing interests reported. Supplementary Files Supplement.docx Cite Share Download PDF Status: Under Review Version 1 posted Reviewers invited by journal 08 Jan, 2026 Editor assigned by journal 05 Jan, 2026 Editor invited by journal 17 Dec, 2025 Submission checks completed at journal 14 Dec, 2025 First submitted to journal 14 Dec, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8188665","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":572728871,"identity":"7fe6c130-b0ec-4b4f-92e5-d579327a89e8","order_by":0,"name":"Eanna L Lowney","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA5klEQVRIiWNgGAWjYNCCAjDJ+ABI8PARp8UATDKDKB42UrSwSYBJQor52c8ee/DDwCZxO3uPWeXXHDsZNgbmh49u4NEi2ZOXbthjkJa4s+eM2W3ZbclAh7EZG+fgc9KBHDMJHoPDuRtu5JjdltzGDNTCwyaNT4v9+Tdmkn9AWu6/MSuW3FZPWIuBRI6ZNMQWHjPGj9sOE9YiceONmbSMQVr9hjNpxdKM247zsDET8At/f46Z5JsKG2OD44c3fvy5rdqen7354WN8WpAAhwEzD4hmJk45CLA/YPxBvOpRMApGwSgYQQAA9MBEE1Ziq2MAAAAASUVORK5CYII=","orcid":"","institution":"Eastern Health","correspondingAuthor":true,"prefix":"","firstName":"Eanna","middleName":"L","lastName":"Lowney","suffix":""},{"id":572728873,"identity":"26e6c6da-2c38-4896-a913-65e342893f78","order_by":1,"name":"Steven G Hirth","email":"","orcid":"","institution":"Eastern Health","correspondingAuthor":false,"prefix":"","firstName":"Steven","middleName":"G","lastName":"Hirth","suffix":""},{"id":572728874,"identity":"56ebec03-a8a7-40eb-8631-d877d5aae296","order_by":2,"name":"Laura Fanning BPharm","email":"","orcid":"","institution":"Eastern Health","correspondingAuthor":false,"prefix":"","firstName":"Laura","middleName":"Fanning","lastName":"BPharm","suffix":""},{"id":572728875,"identity":"d31cf70c-5d59-442e-bbcf-4bb53dcf772e","order_by":3,"name":"Graeme J Duke","email":"","orcid":"","institution":"Eastern Health","correspondingAuthor":false,"prefix":"","firstName":"Graeme","middleName":"J","lastName":"Duke","suffix":""},{"id":572728876,"identity":"132a60c6-db6d-4ea5-8221-6d55af0fd3ff","order_by":4,"name":"Owen Roodenburg","email":"","orcid":"","institution":"Eastern Health","correspondingAuthor":false,"prefix":"","firstName":"Owen","middleName":"","lastName":"Roodenburg","suffix":""}],"badges":[],"createdAt":"2025-11-24 03:08:12","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8188665/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8188665/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":100366003,"identity":"8cc262b8-28a6-4ca5-9bf8-e602fb0a88f1","added_by":"auto","created_at":"2026-01-16 07:55:51","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":216932,"visible":true,"origin":"","legend":"","description":"","filename":"ESASEL2.docx","url":"https://assets-eu.researchsquare.com/files/rs-8188665/v1/e5036845e1ab41dfeb164e9b.docx"},{"id":100124590,"identity":"dbf5076a-8091-4bb6-a014-c33c8d6dadbc","added_by":"auto","created_at":"2026-01-13 09:11:48","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":6145,"visible":true,"origin":"","legend":"","description":"","filename":"08619bec61b04746b953253b3e59cf67.json","url":"https://assets-eu.researchsquare.com/files/rs-8188665/v1/13b4bab7b881f584c87ada03.json"},{"id":100365881,"identity":"5dde73c5-c589-4f78-a252-3ab358e72080","added_by":"auto","created_at":"2026-01-16 07:55:42","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":18803,"visible":true,"origin":"","legend":"","description":"","filename":"Supplement.docx","url":"https://assets-eu.researchsquare.com/files/rs-8188665/v1/7b1364e0a568b3328534133a.docx"},{"id":100124601,"identity":"3c9e207a-0d62-4854-87bc-81c333b311e2","added_by":"auto","created_at":"2026-01-13 09:11:49","extension":"xml","order_by":3,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":98231,"visible":true,"origin":"","legend":"","description":"","filename":"08619bec61b04746b953253b3e59cf671enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-8188665/v1/d17ee5f242450378b15baadc.xml"},{"id":100124598,"identity":"6732a155-4005-48ee-84e2-4b3ca46a9407","added_by":"auto","created_at":"2026-01-13 09:11:49","extension":"png","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":33541,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-8188665/v1/a9db48ef9d0a00d16b6fb9e6.png"},{"id":100365894,"identity":"a081cbeb-cea6-4f1e-9fe4-cdbfd78eb821","added_by":"auto","created_at":"2026-01-16 07:55:43","extension":"png","order_by":7,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":16395,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-8188665/v1/874c394d2c7c6d387ca114d4.png"},{"id":100124597,"identity":"3ec48c11-56ba-49d7-9ccf-b06fbac2f079","added_by":"auto","created_at":"2026-01-13 09:11:49","extension":"xml","order_by":8,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":94891,"visible":true,"origin":"","legend":"","description":"","filename":"08619bec61b04746b953253b3e59cf671structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-8188665/v1/cafd9c4be2994ef75cc1af08.xml"},{"id":100366462,"identity":"3da7b1e8-dc67-40ad-9b73-dc0bdba8edb0","added_by":"auto","created_at":"2026-01-16 07:56:19","extension":"html","order_by":9,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":106046,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-8188665/v1/d29ae49885727892b1f4d362.html"},{"id":100366715,"identity":"7b989ce7-31ab-4d7f-bda4-805c65ec7e5f","added_by":"auto","created_at":"2026-01-16 07:56:30","extension":"jpeg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":63988,"visible":true,"origin":"","legend":"\u003cp\u003eStrobe Diagram\u003c/p\u003e","description":"","filename":"floatimage1.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-8188665/v1/62d7ed862de7900848a8df6e.jpeg"},{"id":100124596,"identity":"b6683532-40fb-4c9c-a98a-99bbbf593154","added_by":"auto","created_at":"2026-01-13 09:11:49","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":48265,"visible":true,"origin":"","legend":"\u003cp\u003eVenn diagram of sepsis alerts and possible sepsis alerts in relation to cohort with identified sepsis and total study population\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-8188665/v1/b434b062599e8d3ba68be29a.png"},{"id":100857659,"identity":"f26581a9-3ce2-45d9-8457-17b0184473f3","added_by":"auto","created_at":"2026-01-22 07:17:26","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":796367,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8188665/v1/758ef59c-dbd7-4ec5-a6d4-ee11d9b7d2c5.pdf"},{"id":100124592,"identity":"24f30f8c-0c88-4fe6-b7d8-b2b3ce335a71","added_by":"auto","created_at":"2026-01-13 09:11:48","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":18803,"visible":true,"origin":"","legend":"","description":"","filename":"Supplement.docx","url":"https://assets-eu.researchsquare.com/files/rs-8188665/v1/40a406382ebf45375e1d4bd3.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Rule-Based Electronic Sepsis Alerts Identify High-Risk Patients Despite Poor Diagnostic Accuracy: A Real-World Evaluation and Implications for Machine Learning","fulltext":[{"header":"Introduction","content":"\u003cp\u003eSepsis is a leading global cause of preventable morbidity and mortality.\u003csup\u003e\u003cspan additionalcitationids=\"CR2 CR3\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e–\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u003c/sup\u003e Early diagnosis, administration of antimicrobials and source control are the cornerstones of management and have been shown to increase survival.\u003csup\u003e\u003cspan additionalcitationids=\"CR6 CR7\" citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e–\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u003c/sup\u003e Sepsis is defined as life-threatening organ dysfunction caused by a dysregulated host response to acute infection\u003csup\u003e\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e\u003c/sup\u003e and is associated with a significant clinical and rising financial burden.\u003csup\u003e\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u003c/sup\u003e Early diagnosis of sepsis in the hospital population is hampered by the absence of a gold standard diagnostic test and the need to collate diverse clinical and laboratory features to prompt clinical suspicion and distinguish sepsis from non-infectious diseases.\u003csup\u003e\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e,\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eThe introduction of electronic medical record (eMR) systems has provided a unique opportunity for automated early warning alerts for the potential presence of sepsis.\u003csup\u003e\u003cspan additionalcitationids=\"CR14 CR15 CR16 CR17 CR18 CR19 CR20\" citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e–\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e\u003c/sup\u003e Numerous rule-based electronic sepsis alert systems (ESAS), often utilising outdated sepsis definitions, have been trialled in emergency departments and inpatient settings.\u003csup\u003e\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e\u003c/sup\u003e While these studies have primarily focused on outcome measures such as mortality, hospital length of stay (LOS), and time to antimicrobial administration, few have assessed the diagnostic accuracy of an ESAS as a reliable sepsis surveillance tool.\u003csup\u003e\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e,\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e,\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e,\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eMoreover, many of these alerts rely on criteria based on the systemic inflammatory response syndrome (SIRS) rather than the Sepsis-3 international definition of sepsis, creating a highly sensitive tool but one with a low positive predictive value (PPV).\u003csup\u003e\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e This has been shown to lead to the misdiagnosis of sepsis, inappropriate administration of antimicrobials\u003csup\u003e\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e\u003c/sup\u003e and potentially contributes to alert fatigue amongst clinicians.\u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e,\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e\u003c/sup\u003e Despite efforts to optimise alert criteria, balancing sensitivity and specificity remains a challenge.\u003csup\u003e\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eRecent advances in machine learning (ML) and artificial intelligence (AI) offer promising alternatives to rule-based systems. However, before ML-based approaches can be widely adopted, it is essential to rigorously evaluate the performance of existing rule-based systems in real-world clinical settings to establish baseline performance and identify specific areas for improvement.\u003csup\u003e\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eWe sought to evaluate the diagnostic accuracy of a rule-based ESAS that was designed in accordance with the latest international sepsis definition. Our study was conducted in a metropolitan teaching hospital in Australia equipped with a comprehensive hospital-wide eMR system.\u003c/p\u003e \u003cdiv id=\"Sec2\" class=\"Section2\"\u003e \u003cp\u003eCategorical data are presented as frequency, percentage and mean with standard deviation (SD); continuous data are presented as median and interquartile range (IQR). Statistical comparison between groups was performed using t-test or Wilcoxon rank sum, respectively. All data linkage and statistical analysis were conducted in StataMP™ v17.0 (2019, College Station, TX).\u003c/p\u003e \u003c/div\u003e"},{"header":"Methods","content":"\u003cp\u003eElectronic Sepsis Alert System\u003c/p\u003e\u003cp\u003eThe Millennium™ (Cerner Corp., Kansas City, MO) eMR was introduced throughout the hospital in October 2017 and included a third-party electronic sepsis surveillance and alerting system.\u003csup\u003e\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e\u003c/sup\u003e The rule-based sepsis alert system was redesigned to align with the Sepsis-3 international definition of sepsis.\u003csup\u003e\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e\u003c/sup\u003e It continuously monitors patient vital signs and laboratory data collected in real-time by the eMR and compares these data with predetermined diagnostic criteria consistent with the presence of sepsis.\u003c/p\u003e\u003cp\u003eTwo levels of alerts - \u003cem\u003ePossible Sepsis Alert (PSA)\u003c/em\u003e and \u003cem\u003eSepsis Alert (SA)\u003c/em\u003e - are triggered according to the number and severity of clinical criteria present within a specific time window. At least three minor criteria must reach threshold values to trigger a PSA; whereas an SA is triggered by the presence of any two minor criteria plus at least one major feature suggesting organ dysfunction (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). The presence of either alert then triggers an automated message to the bedside nurse, as well as to the treating doctor in the event of a SA. This message describes the clinical criteria present with the date/time stamp. In addition, the eMR prompts the primary nurse with the option and guidance to initiate investigations and treatment for suspected sepsis while waiting for assessment by the medical team. The eMR also provides the medical team with an interactive pathway (PowerPlan) for investigation and treatment of suspected sepsis. These data were collected prospectively and stored in the eMR.\u003c/p\u003e\u003cp\u003e \u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eVital Sign and Organ Dysfunction Parameters for Possible Sepsis Alert and Sepsis alert respectively.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e\u003ccolgroup cols=\"3\"\u003e\u003c/colgroup\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCalling Criteria (unit of measure)\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eLess than\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eGreater than\u003c/p\u003e \u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eVital Sign\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRespiratory Rate (breaths/min)\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e11\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e24\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHeart Rate (bpm)\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e51\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e119\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTemperature (centigrade)\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e35.5\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e38.5\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eWCC (x10^9/L)\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e12\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBands Absolute\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e59%\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eChange in Behaviour (Yes/No)\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOrgan Dysfunction\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCreatinine (umol/L)\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e∆ \u0026gt; 44.2 umol/L\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSBP (mmHg)\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e90\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMAP (mmHg)\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLactate (mmol/L)\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBilirubin (umol/L)\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e1.71\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e34.2\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\u003c/div\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eStudy setting, Population and Design\u003c/p\u003e\u003cp\u003eThis single-centre observational study was undertaken at a university-affiliated hospital in Australia with 25,000 multiday-stay admissions per annum, a 24-hour cardiac catheter laboratory, 10-bed Intensive Care Unit and comprehensive acute inpatient specialty services. Inclusion criteria were all adult (≥ 18 years) multiday-stay inpatient admissions between January 1st, 2018, and December 31st, 2019, inclusive. Paediatric admissions (\u0026lt; 18 years) and Emergency Department (ED) attendances that were not admitted to an inpatient unit were excluded from the study. Sepsis alerts triggered during ED assessment and resuscitation were excluded from our analysis, as physiological derangements from various infectious and non-infectious conditions may mimic sepsis and confound the results.\u003c/p\u003e\u003cp\u003eAll available data for PSA and SA including clinical triggers, timing, interventions, outcomes and date of hospital admission were extracted from the eMR and linked with administrative diagnosis coded data containing International Classification of Diseases and Health Related Problems 10th edition – Australian Modification (ICD-10-AM) using unique patient identifiers and admission and discharge dates. The presence of either sepsis alert (PSA or SA) at any time during the inpatient acute care phase were established. Where multiple alerts for the same patient occurred during a single hospital episode, only the first PSA and first SA were included.\u003c/p\u003e\u003cp\u003eThe presence or absence of sepsis was determined from the clinical diagnoses coded in the administrative dataset. This is based on a published (“synchronous”) methodology that uses ICD-10-AM diagnosis codes and was validated against the reference standard of clinical chart review.\u003csup\u003e\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e\u003c/sup\u003e Sepsis was deemed to be present if at least one diagnosis code for an acute infection was accompanied by a second, synchronous diagnosis consistent with acute organ dysfunction. This method has been shown to yield excellent positive and negative predictive values and outperforms previously published (“explicit” and “implicit”) methods for identifying sepsis in administrative data. Note that the explicit term “sepsis,” found in many ICD-10-AM diagnoses, is alone insufficient and poorly predictive of sepsis.\u003csup\u003e\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e\u003cp\u003eClinical (acute and comorbid) diagnoses and demographic characteristics (age, sex, ethnicity, admission source) were extracted from the administrative dataset. The burden of comorbid disease was quantified using both the Charlson and Elixhauser methods\u003csup\u003e\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e\u003c/sup\u003e and the presence of clinical frailty using the Gilbert method.\u003csup\u003e\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e\u003cp\u003ePatient and episode identifiers were retained for data linkage of eMR and administrative datasets and then removed prior to analysis. During the study period, clinical staff were informed of all sepsis alerts (as described above) and all management decisions, including response to any sepsis alert, were determined by the treating team with usual care.\u003c/p\u003e\u003cp\u003e This study was conducted in accordance with the National Statement on Ethical Conduct in Human Research (NHMRC, 2007, updated 2018) and the principles of the Declaration of Helsinki (2013). The Eastern Health Human Research Ethics Committee approved this investigation (LR68-2018) and the need for patient consent was waived due to the unblinded and observational methodology, analysis of de-identified data and reporting of aggregated results.\u003c/p\u003e\u003ch2\u003eStatistical Analysis\u003c/h2\u003e\u003cp\u003ePerformance metrics for the ESAS were determined by comparison with the sepsis population identified from the administrative dataset as the reference standard. Primary outcomes included sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) for any sepsis alert (PSA or SA); and were calculated with the user written Stata (statistical software) command “diagt”.\u003csup\u003e32\u003c/sup\u003e The derived area under the receiver operating characteristic curve (AUROC) value provides a combined estimate of both sensitivity and specificity. A value above 0.7 was deemed acceptable but above 0.8 was desirable. Secondary outcomes included the prevalence of acute infection with or without sepsis, admission to the Intensive Care Unit (ICU), hospital survival and hospital length of stay (LOS). Analyses were repeated separately for either PSA or SA and for any (PSA or SA) alert and the latter are reported herein, unless otherwise stated. Sensitivity analyses were undertaken for the following subgroups: patient diagnosed with acute infection, patients aged \u0026gt; 65 years, and patients who were admitted following an inter-hospital transfer.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eFrom January 2018 to December 2019 there were 149,075 multiday-stay admissions (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Following the exclusion of paediatric records, 142,053 (95%) inpatient admissions were included in the analysis. There were 25,619 (18.0; 95% CI 17.8\u0026ndash;18.3%) admissions associated with a diagnosis of acute infection at any time during the hospital episode and 5,058 (3.6%) patient admissions were classified as clinical sepsis, giving a prevalence rate of 3.56 (95% CI, 3.46\u0026ndash;3.66) per 100 admissions.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThe median lead time to the first sepsis alert was 2 days [IQR, 1\u0026ndash;6] (Appendix; supplementary table \u003cspan refid=\"MOESM1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). A comparison of the study cohorts which did and did not trigger any sepsis alert (PSA and SA) are provided in Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e (and Appendix; supplementary table \u003cspan refid=\"MOESM1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Notably, those who experienced any sepsis alert were older (median, 69 [IQR, 53\u0026ndash;79] v 64 [IQR, 45\u0026ndash;78] years; \u003cem\u003eP\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;0.001), more likely to follow an emergency admission (87% vs 43%; \u003cem\u003eP\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;0.001) and experienced a longer LOS (mean, 7.4 [SD, 12.2] v 1.8 [SD, 4.3]; \u003cem\u003eP\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;0.001) compared to those who did not trigger any sepsis alert. Both Elixhauser and Charlson comorbidity scores were significantly higher in those that triggered any sepsis alert (73% v 48%; \u003cem\u003eP\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;0.001 and 48% v 24%; \u003cem\u003eP\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;0.001, respectively). Furthermore, those who triggered a sepsis alert were more frequently admitted to ICU (13% vs 2%; \u003cem\u003eP\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;0.001) or died in-hospital (6% vs 0.8%; \u003cem\u003eP\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;0.001) than those with no sepsis alert.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eDemographic characteristics of Study Population. Data presented as n (%), unless otherwise indicated.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSeparations\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAny alert\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eNo alert\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003ep-value\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003en (%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e142,053\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e4,011 (2.8)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e138,042 (97.2)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAge (years), median [IQR]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e64 [46\u0026ndash;78]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e69 [53\u0026ndash;79]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e64 [45\u0026ndash;78]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u0026lt;\u0026thinsp;0.001\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMale\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e69,790 (49)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e2,237 (56)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e67,553 (49)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u0026lt;\u0026thinsp;0.001\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAged-care resident\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2,941 (2)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e82 (2)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2,859 (2)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.586\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEmergency admission\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e63,291 (46)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e3,482 (87)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e59,809 (43)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u0026lt;\u0026thinsp;0.001\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInter-hospital transfer\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e3,855 (3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1,246 (31)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2,609 (2)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u0026lt;\u0026thinsp;0.001\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eIndigenous\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e420 (0.3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e10 (0.2)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e410 (0.3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.64\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInfectious disease\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e25,619 (18)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e2,356 (59)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e23,263 (17)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u0026lt;\u0026thinsp;0.001\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSepsis present\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e5,058 (4)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1,331 (33)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e3,727 (3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u0026lt;\u0026thinsp;0.001\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eElixhauser comorbidity score\u0026thinsp;\u0026gt;\u0026thinsp;0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e68,556 (48)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e2,935 (73)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e65,621 (48)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u0026lt;\u0026thinsp;0.001\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCharlson comorbidity score\u0026thinsp;\u0026gt;\u0026thinsp;0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e34,718 (24)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1,922 (48)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e32,796 (24)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u0026lt;\u0026thinsp;0.001\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMean LOS, Days [SD]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e1.9 [4.8]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e7.4 [12.2]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e1.8 [4.3]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u0026lt;\u0026thinsp;0.001\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLead time to first alert, day [IQR]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2 [1\u0026ndash;6]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e2 [1\u0026ndash;5]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ena\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003ena\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003ePerformance metrics for the sepsis alert system are depicted in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e and summarised in Tables\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e and \u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e. The sensitivity and PPV for the presence of sepsis were 26.3% [95% CI, 25.1\u0026ndash;27.6%] and 33.2% [95% CI, 31.7\u0026ndash;34.7%] while the specificity and NPV were 98% [95% CI, 98-98.1%] and 97.3% [95% CI, 97.2\u0026ndash;97.4%], respectively. The AUROC of 0.62 suggests an inefficient test.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003ePerformance metrics for the sepsis alert system (combined PSA and SA).\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMetric\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMean\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003e95% confidence interval\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePrevalence\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e3.6%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e3.5\u0026ndash;3.66%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSensitivity\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e26.3%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e25.1\u0026ndash;27.6%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSpecificity\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e98%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e98.0\u0026ndash;98.1%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAUROC\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e62%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e61.6\u0026ndash;62.8%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePositive predictive value\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e33.2%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e31.7\u0026ndash;34.7%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNegative predictive value\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e97.3%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e97.2\u0026ndash;97.4%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eTotal numbers of true positives, true negatives, false positives and false negatives for the sepsis alert system\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSepsis\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eNo Sepsis\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eTotal\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAny Alert\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1,331\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e2,680\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e4,011\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNo Alert\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e3,727\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e134,315\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e138,042\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eToal\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e5,058\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e136,995\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e142,053\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eSensitivity analyses in subgroups with acute infection, or those aged\u0026thinsp;\u0026gt;\u0026thinsp;65years, did not reveal any improvement in performance of sepsis alerts (Appendix; supplementary table 2). Some improvement in sensitivity 91.9% [95% CI, 89.4\u0026ndash;94%] and PPV 44.5% [95% CI, 41.7\u0026ndash;47.4%] was identified in the subgroup following inter-hospital transfer.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThe green circle denotes admissions classified as clinical sepsis, the blue circle denotes Sepsis Alerts (SA), and the red circle denotes Possible Sepsis Alerts (PSA). The grey square represents the total inpatient study population. Overlap between circles indicates true-positive alerting, while sepsis cases outside the alert circles represent missed cases (false negatives) and alert activations outside the sepsis circle represent false positives. Circle size is proportional to the number of admissions in each group.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eWe screened nearly 150,000 consecutive hospital separations and their corresponding admission data identifying 18.0% with an acute infection and 3.6% with sepsis. Analysis of over 4,000 automated sepsis alerts generated by the hospital eMR revealed that the alert algorithm had a poor PPV (33.2% [95% CI, 31.7\u0026ndash;34.7%]) despite a high NPV (97.3% [95% CI, 97.2\u0026ndash;97.4%]) for the presence of sepsis. The automated sepsis alerts did not perform with sufficient reliability as a clinical prompt for sepsis.\u003c/p\u003e \u003cp\u003eIn response to the results of this study and considering the impending COVID-19 pandemic, the study hospital elected to discontinue the automated sepsis alert system. Retaining a low sensitivity alert system may have led to missed sepsis cases and a delay in key interventions. This case illustrates that technical deployment of health information technology must be accompanied by ongoing performance monitoring and a readiness to modify or withdraw systems when warranted. These observations are relevant to the future development of automated clinical alerts, including next-generation systems that may incorporate adaptive or machine-learning-based approaches.\u003c/p\u003e \u003cp\u003eDespite its poor performance in identifying sepsis, the algorithm did appear to identify a patient cohort with a higher risk of prolonged hospital stay, transfer to ICU or in-hospital death (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). Therefore, similar automated alert systems may help clinicians identify a cohort at higher risk of clinical deterioration and adverse events in line with Standard-8 of the Australian National Healthcare Standards.\u003csup\u003e\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e\u003c/sup\u003e An expanded version of the existing algorithm may find clinical application for this purpose and warrants further investigation.\u003c/p\u003e \u003cp\u003eSepsis diagnosis requires the triangulation of vital signs and laboratory parameters together integrated with clinical findings that indicate the presence of infection, while also excluding confounding factors such as non-infectious inflammatory states that can mimic severe infection.\u003csup\u003e\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e The low sensitivity and PPV of this alert system are consistent with other electronic sepsis alerts developed for inpatient and critical care settings.\u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e,\u003cspan additionalcitationids=\"CR20\" citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e\u003c/sup\u003e These findings reflect the non-specific nature of the diagnostic criteria for sepsis and its tendency to overlap with many non-infectious diseases that lead to significant organ dysfunction. The balance between early warning and excessive over-treatment is challenging.\u003csup\u003e\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eWhile our study demonstrated that this ESAS had high specificity, its low sensitivity and PPV limit its clinical reliability. This reflects inherent limitations of rule-based systems that depend on fixed thresholds and predefined criteria. These systems lack contextual awareness, perform poorly in heterogeneous clinical settings, and are unable to adapt to evolving patient data or changing sepsis definitions.\u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e,\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e\u003c/sup\u003e Our findings align with a growing body of evidence demonstrating that rule-based approaches are insufficient for complex clinical predictions like sepsis, where subtle patterns and temporal relationships are critical.\u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eIn contrast, machine learning (ML)-based approaches have shown promise in overcoming many of these challenges. ML models can incorporate an array of continuous and categorical variables, model non-linear relationships, and detect subtle temporal patterns that may precede overt clinical deterioration. Several studies have demonstrated that ML algorithms can outperform rule-based systems, achieving higher sensitivity and PPV, with improved AUROC values when predicting sepsis.\u003csup\u003e\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e,\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e\u003c/sup\u003e When combined with timely provider review, ML-based early warning systems have shown potential to reduce mortality and hospital length of stay in sepsis patients.\u003csup\u003e\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e,\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e\u003c/sup\u003e Additionally, these systems may offer significant cost-saving potential by enabling earlier intervention and reducing the need for ICU admissions.\u003csup\u003e\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eThis study provides a real-world evaluation of a widely deployed sepsis alert system and demonstrates the essential role of post-implementation assessment in health informatics. Our findings serve as an important benchmark for evaluating rule-based approaches and provide a foundation for developing more sophisticated, adaptive systems. However, further research is warranted to determine whether ML-based alerts can substantially improve diagnostic performance while avoiding new risks such as alert fatigue, lack of clinical transparency, or inappropriate treatment of non-infectious conditions under the mistaken presumption of sepsis.\u003csup\u003e\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e,\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eThe strengths of this study include a large study population and whole-of-hospital cohort with a sufficiently high prevalence of sepsis (3.6%). The two-month eMR run-in period, prior to study commencement, permitted time for education of staff including familiarity with the sepsis alert system. Study data were collected over a two-year period capturing seasonal and case mix variation in the frequency of sepsis. Data collection and provisional analysis were completed prior to the SARS-COV19 pandemic. To date, this is the largest published study to address the clinical validity of automated sepsis alerts in Australia.\u003c/p\u003e \u003cp\u003eThere are several important limitations in our study. The findings of this single centre study may not necessarily be generalisable to other sites. Clinical diagnosis and treatment following a sepsis alert were left to the judgement of the treating team and the study hospital does not employ protocolised early-goal directed therapy for sepsis. Additionally, the presence or absence of sepsis was determined from the clinical diagnoses coded in the administrative dataset. While the use of ICD-10-AM coding data has been shown to over or under report cases of sepsis when compared to the gold standard of prospective chart review for objective markers of infection,\u003csup\u003e3,29\u003c/sup\u003e the method utilised in our study outperforms other methods for identifying sepsis in administrative data.\u003csup\u003e\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e\u003c/sup\u003e. Although this inaccuracy may affect the calculated performance metrics of the electronic sepsis alerts potentially leading to deviations from the true diagnostic accuracy that would be achieved through comprehensive chart review, our methodology permitted a larger study population over a longer duration with limited research resources and the (SARS-COV19) pandemic imminent.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eWe investigated the performance and reliability of an automated clinical alert system for early identification of in-hospital sepsis. While the algorithm appears to be highly specific, it lacked sufficient sensitivity for clinical purposes. Further research and clinical validation are required to refine these potentially attractive systems to improve patient care without increasing unnecessary workload burden or exposing patients to risks of unnecessary investigations and treatment.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cdiv class=\"DefinitionList\"\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eESAS\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eElectronic Sepsis Alert System\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eeMR\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eElectronic Medical Record\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eICU\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eIntensive Care Unit\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eLOS\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eLength of Stay\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003ePPV\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003ePositive Predictive Value\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eNPV\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eNegative Predictive Value\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eAUROC\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eArea Under the Receiver Operating Characteristic Curve\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eICD-10-AM\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eInternational Classification of Diseases, 10th Revision, Australian Modification\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003c/div\u003e"},{"header":"Declarations","content":"\u003ch2\u003eEthics Approval and consent to participate\u003c/h2\u003e\n\u003cp\u003eThis study was conducted in accordance with the National Statement on Ethical Conduct in Human Research (NHMRC, 2007; updated 2018) and the principles of the Declaration of Helsinki (2013). Ethical approval was granted by the \u003cstrong\u003eEastern Health Human Research Ethics Committee\u003c/strong\u003e (Reference number \u003cstrong\u003eLR68-2018\u003c/strong\u003e). The requirement for informed consent to participate was \u003cstrong\u003ewaived\u003c/strong\u003e by the ethics committee due to the retrospective observational design, use of routinely collected clinical and administrative data, analysis of de-identified information, and reporting of aggregated results.\u003c/p\u003e\n\u003ch2\u003eConsent for publication\u003c/h2\u003e\n\u003cp\u003eNot applicable. This study did not include identifiable individual patient data.\u003c/p\u003e\n\u003ch2\u003eAuthorship Contributions\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eEanna Lowney:\u003c/strong\u003e Conceptualisation, Methodology, investigation, writing- original draft, review and editing, project administration. \u003cstrong\u003eLaura Fanning:\u003c/strong\u003e Conceptualisation, Methodology, investigation, formal analysis, writing- review and editing. \u003cstrong\u003eSteve Hirth\u003c/strong\u003e: Conceptualisation, formal analysis, investigation. \u003cstrong\u003eGraeme Duke\u003c/strong\u003e:\u0026nbsp;Supervision, conceptualisation, methodology, investigation, writing- original draft, review and editing, project administration.\u003cstrong\u003e\u0026nbsp;Owen Roodenburg\u003c/strong\u003e: Supervision, conceptualisation, methodology, writing- review and editing, project administration.\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003eAcknowledgements\u003c/h2\u003e\n\u003cp\u003eNot Applicable.\u003c/p\u003e\n\u003ch2\u003eFunding\u003c/h2\u003e\n\u003cp\u003eThe authors received no financial support for the research, authorship, and/or publication of this article.\u003c/p\u003e\n\u003ch2\u003eData availability statement\u003c/h2\u003e\n\u003cp\u003eThe authors confirm that the data supporting the findings of this study are available within the article and its supplementary materials.\u003c/p\u003e\n\u003ch2\u003eCompeting Interests\u003c/h2\u003e\n\u003cp\u003eThe authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eVincent J-L, Marshall JC, \u0026Ntilde;amendys-Silva SA, et al. Assessment of the worldwide burden of critical illness: the Intensive Care Over Nations (ICON) audit. Lancet Respir Med. 2014;2:380\u0026ndash;6.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRudd KE, Johnson SC, Agesa KM, et al. Global, regional, and national sepsis incidence and mortality, 1990\u0026ndash;2017: analysis for the Global Burden of Disease Study. Lancet Lond Engl. 2020;395:200\u0026ndash;11.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAngus DC, Linde-Zwirble WT, Lidicker J, et al. Epidemiology of severe sepsis in the United States: Analysis of incidence, outcome, and associated costs of care. Crit Care Med. 2001;29:1303\u0026ndash;10.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFleischmann C, Scherag A, Adhikari NKJ, et al. Assessment of Global Incidence and Mortality of Hospital-treated Sepsis. Current Estimates and Limitations. Am J Resp Crit Care. 2016;193:259\u0026ndash;72.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSeymour CW, Gesten F, Prescott HC, et al. Time to Treatment and Mortality during Mandated Emergency Care for Sepsis. New Engl J Med. 2017;376:2235\u0026ndash;44.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFerrer R, Martin-Loeches I, Phillips G, et al. Empiric Antibiotic Treatment Reduces Mortality in Severe Sepsis and Septic Shock From the First Hour. Crit Care Med. 2014;42:1749\u0026ndash;55.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDellinger RP, Levy MM, Rhodes A, et al. Surviving Sepsis Campaign: International Guidelines for Management of Severe Sepsis and Septic Shock, 2012. Intensive Care Med. 2013;39:165\u0026ndash;228.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eReitz KM, Kennedy J, Li SR, et al. Association Between Time to Source Control in Sepsis and 90-Day Mortality. JAMA Surg. 2022;157:817\u0026ndash;26.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSinger M, Deutschman CS, Seymour CW, et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016;315:801\u0026ndash;10.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi L, Sunderland N, Rathnayake K, et al. Sepsis epidemiology in Australian Public Hospitals, a nationwide longitudinal study (2013\u0026ndash;2018). Infect Dis Heal. 2021;26:S9.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVincent J-L. The Clinical Challenge of Sepsis Identification and Monitoring. Plos Med. 2016;13:e1002022.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDuncan CF, Youngstein T, Kirrane MD, et al. Diagnostic Challenges in Sepsis. Curr Infect Dis Rep. 2021;23:22.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMakam AN, Nguyen OK, Auerbach AD. Diagnostic accuracy and effectiveness of automated electronic sepsis alert systems: A systematic review. J Hosp Med. 2015;10:396\u0026ndash;402.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eManaktala S, Claypool SR. Evaluating the impact of a computerized surveillance algorithm and decision support system on sepsis mortality. J Am Med Inf Assn. 2017;24:88\u0026ndash;95.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAustrian JS, Jamin CT, Doty GR, et al. Impact of an emergency department electronic sepsis surveillance system on patient mortality and length of stay. J Am Med Inf Assn. 2017;25:523\u0026ndash;9.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWarttig S, Alderson P, Evans DJ et al. Automated monitoring compared to standard care for the early detection of sepsis in critically ill patients. Cochrane Db Syst Rev. Epub ahead of print 2018. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1002/14651858.cd012404.pub2\u003c/span\u003e\u003cspan address=\"10.1002/14651858.cd012404.pub2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSeetharaman S, Wilson C, Landrum M et al. Does use of Electronic Alerts for Systemic Inflammatory Response Syndrome (SIRS) to Identify Patients with Sepsis Improve Mortality? Am J Med. Epub ahead of print 2019. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.amjmed.2019.01.032\u003c/span\u003e\u003cspan address=\"10.1016/j.amjmed.2019.01.032\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDowning NL, Rolnick J, Poole SF et al. Electronic health record-based clinical decision support alert for severe sepsis: a randomised evaluation. Bmj Qual Saf 2019; bmjqs-2018-008765.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi L, Rathnayake K, Green M, et al. Improving the Performance of Clinical Decision Support for Early Detection of Sepsis: A Retrospective Observational Cohort Study. Stud Heal Technol Inf. 2019;264:679\u0026ndash;83.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGiannini HM, Ginestra JC, Chivers C, et al. A Machine Learning Algorithm to Predict Severe Sepsis and Septic Shock: Development, Implementation, and Impact on Clinical Practice*. Crit Care Med. 2019;47:1485\u0026ndash;92.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTopiwala R, Patel K, Twigg J, et al. Retrospective Observational Study of the Clinical Performance Characteristics of a Machine Learning Approach to Early Sepsis Identification. Crit Care Explor. 2019;1:e0046.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKim H-J, Ko R-E, Lim SY, et al. Sepsis Alert Systems, Mortality, and Adherence in Emergency Departments. JAMA Netw Open. 2024;7:e2422823.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHwang MI, Bond WF, Powell ES. Sepsis Alerts in Emergency Departments: A Systematic Review of Accuracy and Quality Measure Impact. West J Emerg Med. 2020;21:1201\u0026ndash;10.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKane-Gill SL, O\u0026rsquo;Connor MF, Rothschild JM, et al. Technologic Distractions (Part 1): Summary of Approaches to Manage Alert Quantity With Intent to Reduce Alert Fatigue and Suggestions for Alert Fatigue Metrics. Crit care Med. 2017;45:1481\u0026ndash;8.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSawyer AM, Deal EN, Labelle AJ, et al. Implementation of a real-time computerized sepsis alert in nonintensive care unit patients* \u003cem\u003eCrit Care Med\u003c/em\u003e 2011; 39: 469\u0026ndash;473.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShimabukuro DW, Barton CW, Feldman MD, et al. Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial. BMJ Open Respir Res. 2017;4:e000234.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAmland RC, Hahn-Cover KE. Clinical Decision Support for Early Recognition of Sepsis. Am J Med Qual. 2016;31:103\u0026ndash;10.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDuke GJ, Bishara M, Hirth S et al. Performance of hospital administrative data for detection of sepsis in Australia: The sepsis coding and documentation (SECOND) study. Health Inf Manag J 2022; 183335832211077.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMartin GS, Mannino DM, Eaton S, et al. The Epidemiology of Sepsis in the United States from 1979 through 2000. New Engl J Med. 2003;348:1546\u0026ndash;54.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003evan Walraven C, Austin PC, Jennings A, et al. A Modification of the Elixhauser Comorbidity Measures Into a Point System for Hospital Death Using Administrative Data. Med Care. 2009;47:626\u0026ndash;33.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGilbert T, Neuburger J, Kraindler J, et al. Development and validation of a Hospital Frailty Risk Score focusing on older people in acute care settings using electronic hospital records: an observational study. Lancet Lond Engl. 2018;391:1775\u0026ndash;82.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSeed P. DIAGT: Stata module to report summary statistics for diagnostic tests compared to true disease status. \u003cem\u003eStatistical Software Components S423401, Boston College Department of Economics\u003c/em\u003e, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://EconPapers.repec.org/RePEc:boc:bocode:s423401\u003c/span\u003e\u003cspan address=\"https://EconPapers.repec.org/RePEc:boc:bocode:s423401\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. (2010).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNational Safety and Quality Health Service Standards. National Safety and Quality Health Service Standards. 2nd ed. Sydney: ACSQHC; 2017.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAdams R, Henry KE, Sridharan A, et al. Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis. Nat Med. 2022;28:1455\u0026ndash;60.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBedoya AD, Futoma J, Clement ME, et al. Machine learning for early detection of sepsis: an internal and temporal validation study. JAMIA Open. 2020;3:252\u0026ndash;60.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHenry KE, Adams R, Parent C, et al. Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timing. Nat Med. 2022;28:1447\u0026ndash;54.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eEricson O, Hjelmgren J, Sj\u0026ouml;vall F, et al. The Potential Cost and Cost-Effectiveness Impact of Using a Machine Learning Algorithm for Early Detection of Sepsis in Intensive Care Units in Sweden. J Heal Econ Outcomes Res. 2022;9:101\u0026ndash;10.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-informatics-and-decision-making","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"midm","sideBox":"Learn more about [BMC Medical Informatics and Decision Making](http://bmcmedinformdecismak.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/midm/default.aspx","title":"BMC Medical Informatics and Decision Making","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Sepsis, Infection, Electronic Sepsis Alert, Electronic Medical Record, Clinical Decision Support Systems, Machine Learning, Artificial Intelligence, Implementation Science.","lastPublishedDoi":"10.21203/rs.3.rs-8188665/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8188665/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eObjective\u003c/p\u003e\n\u003cp\u003eTo evaluate the diagnostic accuracy of an electronic sepsis alert system (ESAS) in an acute care hospital within an electronic medical record (eMR) system.\u003c/p\u003e\n\u003cp\u003eDesign\u003c/p\u003e\n\u003cp\u003eSingle-centre observational study of prospectively collected data from the eMR incorporating a third-party electronic sepsis surveillance and alerting system. Clinical eMR and administrative coding data for all patient records were analysed. Performance characteristics of the ESAS were compared with the presence or absence of clinical sepsis.\u003c/p\u003e\n\u003cp\u003eSetting\u003c/p\u003e\n\u003cp\u003eA university-affiliated hospital in Melbourne, Australia with 25,000 multiday-stay admissions per annum.\u003c/p\u003e\n\u003cp\u003eParticipants\u003c/p\u003e\n\u003cp\u003eAll adult multiday-stay admissions between January 1\u003csup\u003est\u003c/sup\u003e, 2018, and December 31\u003csup\u003est\u003c/sup\u003e, 2019, inclusive.\u003c/p\u003e\n\u003cp\u003eMain Outcome measures\u003c/p\u003e\n\u003cp\u003eSensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) of the ESAS.\u003c/p\u003e\n\u003cp\u003eResults\u003c/p\u003e\n\u003cp\u003e149,053 records were included in the study, of which 4,011 triggered an electronic sepsis alert. The sensitivity and PPV of the ESAS were 26.3% [95% CI, 25.1-27.6%] and 33.2% [95% CI, 31.7-34.7%] respectively, while its specificity and NPV were 98% [95% CI, 98.0-98.1%] and 97.3% [95% CI, 97.2–97.4%] respectively.\u003c/p\u003e\n\u003cp\u003eConclusion\u003c/p\u003e\n\u003cp\u003eThe ESAS was highly specific but lacked sensitivity for reliable clinical application. The activation of ESAS was associated with a longer length of stay, higher rates of Intensive Care Unit admission and in-hospital mortality. The ESAS ultimately identified a cohort at risk of clinical deterioration. These results highlight fundamental limitations of rule-based approaches and underscore the need for adaptive machine learning systems that can better integrate complex clinical patterns for early sepsis detection.\u003c/p\u003e","manuscriptTitle":"Rule-Based Electronic Sepsis Alerts Identify High-Risk Patients Despite Poor Diagnostic Accuracy: A Real-World Evaluation and Implications for Machine Learning","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-01-13 09:11:39","doi":"10.21203/rs.3.rs-8188665/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"reviewersInvited","content":"","date":"2026-01-08T07:26:30+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-01-05T09:51:53+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2025-12-17T13:10:46+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-12-15T01:53:31+00:00","index":"","fulltext":""},{"type":"submitted","content":"BMC Medical Informatics and Decision Making","date":"2025-12-15T01:48:38+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-informatics-and-decision-making","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"midm","sideBox":"Learn more about [BMC Medical Informatics and Decision Making](http://bmcmedinformdecismak.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/midm/default.aspx","title":"BMC Medical Informatics and Decision Making","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"5c5844df-3542-4879-bdb2-5386bf17e2ea","owner":[],"postedDate":"January 13th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-01-13T09:11:40+00:00","versionOfRecord":[],"versionCreatedAt":"2026-01-13 09:11:39","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8188665","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8188665","identity":"rs-8188665","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}