ESI Triage Level Assignment for Headache Patients: Comparative Analysis of ChatGPT and Gemini Performance for Supporting Care Provider Decisions and Self-triage

ESI Triage Level Assignment for Headache Patients: Comparative Analysis of ChatGPT and Gemini Performance for Supporting Care Provider Decisions and Self-triage | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article ESI Triage Level Assignment for Headache Patients: Comparative Analysis of ChatGPT and Gemini Performance for Supporting Care Provider Decisions and Self-triage Hamed Samadpour, Sharareh Rostam Niakan Kalhori, Masoumeh Tahmasebi, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5429142/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Objective This study evaluated the performance of two advanced large language models (LLMs), ChatGPT and Gemini, in supporting triage decisions for headache patients in emergency settings via the Emergency Severity Index (ESI) from both patient self-triage and healthcare provider perspectives. Methods Data, including 500 records of patients presenting with headache complaints, were obtained from the MIMIC-IV-ED database. Two distinct prompt types were created: one for self-triage to assist patients in assessing their care needs on the basis of symptom descriptions and another for healthcare providers to determine ESI levels. Each model's output was compared to actual ESI levels via precision, recall, and F1 scores to measure performance. Results ChatGPT achieved greater accuracy at lower acuity levels (ESIs 3 and 4), accurately identifying patients who did not require urgent care. Gemini demonstrated improved performance at higher acuity levels (ESIs 1 and 2), indicating its ability to recognize critical cases effectively. Both models showed stronger performance with healthcare provider prompts than with self-triage prompts, underscoring the importance of structured input for accurate triage assessments. This variation highlights the need to refine self-triage prompts to ensure safe and precise use. Conclusion ChatGPT and Gemini show promise as decision-support tools for ED triage, particularly for assisting healthcare providers in prioritizing cases on the basis of acuity. However, further refinement is needed to increase accuracy in self-triage scenarios. Future studies should validate these findings across a broader dataset and explore the integration of LLMs into clinical decision support systems to strengthen triage reliability and effectiveness. ChatGPT Gemini Emergency Severity Index Triage Large Language Models MIMIC-IV ED Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Introduction Accurate patient categorization upon arrival at the emergency department is crucial for effectively delivering emergency care. Overestimating patient acuity can misallocate limited resources, particularly during high patient volumes. Conversely, underestimation of patient severity can result in delayed diagnosis and treatment, potentially compromising patient outcomes [ 1 ]. Overcrowding in the emergency department negatively impacts several patient-oriented outcomes, including mortality, complication rates, walkouts, time to treatment, satisfaction, and duration of stay [ 2 ]. Additionally, overcrowding has been identified as a significant stressor for medical staff, leading to increased burnout [ 3 ] and diagnostic errors [ 4 ]. Triage is a structured process designed to expedite care delivery on the basis of the severity and extent of the health issue, with three distinct phases: prehospital triage, triage at the scene, and triage upon arrival at the emergency department [ 5 ]. Electronic emergency triage (E-triage) is an emerging approach that uses digital and AI-driven tools to evaluate and prioritize patients on the basis of their condition severity, improving both the efficiency and accuracy of healthcare delivery [ 6 ]. E-triage can be conducted through two methods: self-triage or clinician-led triage. In self-triage, patients assess their condition via digital tools or symptom checkers to decide if medical care is necessary [ 7 ]. Effective self-triage can enhance the clinician-led triage process by reducing patient queues, facilitating timely care, and optimizing healthcare resources [ 8 ]. Traditional triage systems face limitations, such as slow response times, limited accuracy, and high costs. These challenges have driven the development of intelligent triage solutions incorporating AI-based approaches to address these deficiencies [ 9 ]. Recent advancements in AI have led to the creation of large language models (LLMs), such as ChatGPT by OpenAI and Gemini by Google, which hold promise for various medical applications, including patient triage [ 10 ], [ 11 ]. LLMs serve as the foundation for intelligent conversational tools (ICTs), enabling real-time, human-like interactions in healthcare. By understanding and processing natural language, these models can provide swift, accurate assistance, which is especially crucial for conditions such as headaches [ 12 ], ranging from benign to life-threatening causes such as stroke or aneurysm [ 13 ]. Symptom assessment can be particularly challenging [ 14 ]. ChatGPTs and Gemini are adept at delivering contextually relevant information, enhancing triage and self-triage, and supporting patients and healthcare providers with precise, data-driven insights [ 15 ]. We selected ChatGPT and Gemini because of their strong ability to process clinical data, align decisions with medical protocols, and adaptability across diverse healthcare domains [ 16 ]. Their advanced AI technology is suitable, particularly for critical emergency triage situations, where timely and accurate information is essential. The Emergency Severity Index (ESI) is a widely utilized triage system in emergency departments to prioritize patient care. This system categorizes patients into five levels on the basis of their clinical conditions and resource requirements. Patients classified as ESI 1 or 2 exhibit emergency or urgent conditions that demand immediate attention, whereas patients designated as ESI 3, 4, or 5 are considered stable and are prioritized on the basis of the extent of resources necessary for their treatment [ 17 ]. Current systems, such as symptom checkers, are not explicitly optimized for headache assessment, highlighting the need for more targeted, AI-driven approaches [ 18 ]. An integrated system combining AI with existing triage methods such as the ESI could significantly improve triage accuracy for headache patients. Recent research has explored the application of ChatGPT and Gemini technology in patient triage. For example, Gürbüz Meral et al. [ 19 ] compared the accuracy of the ChatGPT and Gemini to that of emergency medicine specialists in ESI triage, and Williams et al. examined the ability of the GPT to assess clinical acuity in ED patients [ 20 ]. In the Fraser et al. study, deidentified ED patient data were used to analyze the diagnostic and triage capabilities of the ChatGPT models Ada and WebMD [ 21 ]. However, our literature review indicates that no studies have evaluated Gemini's performance on the basis of the ESI triage system dataset. Instead of focusing solely on emergency cases, we examined ChatGPT and Gemini’s performance from the perspectives of patients and healthcare providers via ESI. This study aims to compare and evaluate the performance of two large language models (LLMs), ChatGPT and Gemini, in assessing ESI triage from both the healthcare provider and patient perspectives. Method Overview This study utilized a robust methodology with five steps: 1. Selection of database, sample, and data preprocessing; 2. Prompt generation; 3. Analysis with LLM ChatBots; 4. The actual level of patients should be determined, and 5. Interpretations of the results with detailed descriptions of each stage are provided in the following sections and shown in Figure 1. Selection of the Database, Sample, and Data Preprocessing Database Selection: Retrospectively collected medical data offer the opportunity to improve patient care through knowledge discovery and algorithm development. The MIMIC-IV-ED dataset is an extensive, freely accessible, and deidentified data collection containing 425,000 emergency department (ED) admissions between 2011 and 2019. These data include vital signs, triage information, medication prescriptions, and final diagnoses. This tool is designed to support various educational initiatives and research studies. The patient population is a heterogeneous group with varying severity levels, ranging from mild conditions to life-threatening cardiac complications. The data model consists of six tables: edstays, diagnosis, medrecon, pyxis, triage, and vitalsign. The edstays table tracks patient stays in the emergency department. The diagnosis table contains coded diagnoses for patients on the basis of the International Classification of Diseases, either the Ninth or Tenth revision (ICD-9 or ICD-10). Trained coders assign these diagnoses after patient discharge and are primarily used for billing purposes. The medrecon table lists each patient's medications before their ED stay as part of medication reconciliation. The pyxis table details the medications dispensed through the BD Pyxis MedStation, an automated medication dispensing system used in the ED. The triage table records information collected during the triage process, where patients’ health status and reasons for their visit are assessed upon arrival. Finally, the vital table includes documented periodic vital signs for patients during their ED stay. The database was selected because of the availability of vital signs and other necessary measurable patient information, its outcome-oriented nature, and the recorded acuity level of each patient as assessed by care providers [22]. Sample Selection: A rigorous random selection process was employed from the MIMIC-ED v2.2 database, which includes 3,951 patient records with a primary complaint of headache and self-arrival. Fifteen percent of the records—592 samples—were random, and a simple sampling method was chosen via Excel software. This careful selection was performed to ensure the fairness and representativeness of the samples, thereby avoiding any potential bias in the data. The following information fields were extracted for the patients, as presented in Table 1. Table 1- Variables extracted from MIMIC-IV ED Variable Definition Gender The sex of the subjects Race demographic category used to describe a patient’s self-identified racial or ethnic background Arrival_transport mode of transportation that brought a patient to the emergency department (ED) Disposition the outcome or status of a patient at the time of discharge from the emergency department (ED) Temperature The body temperature of the subjects in degrees Fahrenheit Heart rate number of times the heart beats per minute (bpm) Respiratory rate number of breaths a person takes per minute Oxygen saturation percentage of hemoglobin in the blood that is saturated with oxygen Systolic blood pressure(mmHg) pressure in the arteries during heart contraction, representing the higher value in a blood pressure reading Diastolic blood pressure(mmHg) pressure in the arteries when the heart is at rest between beats, representing the lower value in a blood pressure reading Pain unpleasant sensory and emotional experience often associated with actual or potential tissue damage Acuity severity or intensity of a patient's condition or illness Age length of time that a person has lived or existed, typically measured in years Data Preprocessing: Using the MS SQL Server 2022 database, the edstays and triage tables were merged via the subject_id and stay_id fields. All patients were over 18 years old. Records with outliers or missing data were removed. Five hundred records were left for analysis. Prompt generation Record Conversion: The records selected for analysis were converted into prompts via the mail merge feature in Word. This process involves combining the data in the records with predefined templates to generate personalized messages. Types of promoters: Two types of prompts were created, each designed for a different purpose: Prompt for self-triage: This prompt allows patients to briefly describe their condition and receive appropriate care without providing additional explanations. This prompt is second-level and imperative. This second-level prompt contains specific contextual data (symptoms and vital signs) and asks the model to assess the required level of care on the basis of this information. The data provided include factors such as sex, race, and vital signs, which the model must analyze. This is an imperative prompt because the model is explicitly instructed to provide a concise and direct response with minimal explanation. Sample Prompt for Self-Triage: “ I have a headache, my gender is Female, and my race is BLACK/AFRICAN AMERICAN. Her temperature was 98 F, her heart rate was 74, her respiratory rate was 18, her o2 saturation was 100, her systolic blood pressure was 115 mmHg, her diastolic blood pressure was 69 mmHg, and her pain level was 7. Please let me know which level of care I need. I am worried that I will be kept in the hospital due to my symptoms. I need to be prepared for this probable stay. I need to be ready in advance for what will happen in the hospital. Inform me shortly which level of care would be considered for me. Answer me with the slightest explanation and with full frankness. Say it in one line . “ Healthcare provider decision support prompt: This prompt allows healthcare providers to assess a patient’s triage level via the Emergency Severity Index (ESI). This is a second-level and role-play prompt. This second-level prompt also includes specific and specialized contextual data that require detailed analysis. The patient’s symptoms and vital signs are provided, and the model is asked to assess the situation on the basis of a specific medical index (the Emergency Severity Index). This is a role-play prompt because the model is asked to assume the role of an emergency medicine specialist and make decisions on the basis of that role. Sample Healthcare provider decision support prompt: “ You are an emergency medicine specialist. Please assess my triage level on the basis of the EMERGENCY SEVERITY INDEX and my vital signs. I tell you my symptoms and vital signs, and you should summarize your assessment in one sentence. I have a headache, my sex is female, and my race is BLACK/AFRICAN AMERICAN. Her heart rate was 90, her respiratory rate was 16, her o2 saturation was 99, her systolic blood pressure was 134 mmHg, her diastolic blood pressure was 51 mmHg, and her pain level was 4. Shortly, let me know what my triage level is. ” Analysis with LLM ChatBots The prompts were subsequently sent to ChatGPT and Gemini. The output from the large language models (LLMs) in response to patient prompts was mapped to the five emergency severity index (ESI) levels according to the opinions of the emergency specialists, as shown in Table 2. In the healthcare provider decision support section, the level assigned by the chatbots was directly used as the result. Table 2 -LLM response sample to triage level assignment ChatGPT Sample Response Assigned Triage Level Based on your symptoms, you would likely require immediate emergency care and admission to the hospital for further evaluation and management of your condition. 1 Based on your symptoms and vital signs, you may require urgent evaluation in an emergency department or acute care setting due to the severity of your headache. 2 You likely need urgent care based on your symptoms and vital signs, but hospitalization is not typically necessary unless further evaluation indicates otherwise. 3 Based on your symptoms and vital signs, you would likely require outpatient care unless there are additional concerning factors. 4 Based on your symptoms and vital signs, you likely do not require hospitalization and can manage your condition at home. 5 Determining the actual triage level of patients The acuity field in the data and predefined rules based on the opinions of emergency specialists were used to determine the actual level of patients. These rules involved thoroughly assessing clinical signs and applying standard emergency indicators to determine the triage level. The rules employed in this process are shown in Figure 2. Interpretation of Results As a result of the consensus meetings of experts (3 medical informatics researchers), the outputs from these bots were interpreted and analyzed on the basis of the ESI levels. Differences and similarities between the bot-generated and actual assigned levels were examined. A confusion matrix containing all levels was created, and each level's precision, recall, and F1 score were calculated. Precision‒recall and ROC curves are plotted in Python via Pandas, Matplotlib, and the Sklearn library. Additionally, Kendall's Tau statistical test was conducted in SPSS version 27 to assess the correlation between the results. Kendall's Tau is a nonparametric test measuring the strength and direction of associations between ordinal variables. In this study, the variables include the actual triage level of the ESI, the ChatGPT, and the Gemini-predicted triage level. Results Following data cleaning, 500 records were retained for analysis. Among these, 152 records (30.4%) corresponded to male participants, whereas 348 records (69.6%) were from female participants. Table 3 provides a summary of the descriptive statistics for the studied variables. Pain levels were assessed via an 11-point numerical rating scale (NRS) ranging from 0 to 10. Note that there were no records with an actual ESI level of 5, and the results corresponding to this level were excluded from the analysis. Table 3 - Descriptive statistics of the variables Variable Range Mean Standard deviation Temperature 96-103 98 0.85 Heart rate 50-139 82 15.62 Respiratory rate 12-24 17 1.54 Oxygen saturation 92-100 99 1.36 Systolic blood pressure (mmHg) 86-202 134 19.81 Diastolic blood pressure (mmHg) 51-177 80 13.34 Pain 0-10 7 2.62 Acuity 1-4 3 0.5 Age* 18-87 41 16.19 *Patient age was calculated using the date of birth and anchor date in the database. Care provider’s perspective The comparative results generated by ChatGPT and Gemini on the basis of the care provider's perspective are presented in Table 4. The confusion matrixes are shown in Figures 3-4. Table 4 - -LLM comparative confusion matrix (Care Provider Prompt) Predicted Values Level 1 2 3 4 5 Total Actual Value GPT Gemini GPT Gemini GPT Gemini GPT Gemini GPT Gemini 1 0 0 6 2 4 2 1 8 1 0 12 2 2 0 33 18 34 24 12 41 2 0 83 3 1 1 90 20 210 97 79 273 11 0 391 4 0 0 1 0 6 0 6 14 1 0 14 Total 3 1 130 40 254 123 98 336 15 0 500 Patient perspective (self-triage) The comparative results generated by ChatGPT and Gemini on the basis of the patient's perspective (self-triage) are presented in Table 5. The confusion matrixes are shown in Figures 5--6. Table 5 — LLM comparative confusion matrix (Patient Prompt) Predicted Values Level 1 2 3 4 5 Total Actual Value GPT Gemini GPT Gemini GPT Gemini GPT Gemini GPT Gemini 1 3 2 5 3 2 7 1 0 1 0 12 2 3 10 36 15 22 47 21 4 1 7 83 3 33 25 109 23 106 269 139 25 4 49 391 4 1 1 4 1 2 7 7 3 0 2 14 Total 40 38 154 42 132 330 168 32 6 58 500 Analysis of Precision, Recall, and F1 Scores The classification reports for the two prompt types are shown in Tables 6--8. Table 6 -Gemini and chatGPT precision to determine the level of the ESI Precision Patient Care Provider GPT Gemini GPT Gemini Level 1 0.07 0.05 0.00 0.00 Level 2 0.23 0.36 0.25 0.45 Level 3 0.80 0.82 0.83 0.79 Level 4 0.04 0.09 0.06 0.04 Micro Avg 0.23 0.26 0.23 0.32 Weighted Avg 0.67 0.70 0.69 0.69 Table 7 - Gemini and chatGPT Recall to determine the level of ESI Recall Patient Care Provider GPT Gemini GPT Gemini Level 1 0.25 0.17 0.00 0.00 Level 2 0.43 0.18 0.40 0.22 Level 3 0.27 0.69 0.54 0.25 Level 4 0.50 0.21 0.43 1.00 Micro Avg 0.29 0.25 0.27 0.37 Weighted Avg 0.30 0.58 0.50 0.26 Table 8 - Gemini and chatGPT F1 scores for determining the level of ESI F1-Score Patient Care Provider GPT Gemini GPT Gemini Level 1 0.12 0.08 0.00 0.00 Level 2 0.30 0.24 0.31 0.29 Level 3 0.41 0.75 0.65 0.38 Level 4 0.08 0.13 0.11 0.08 Micro Avg 0.18 0.24 0.21 0.19 Weighted Avg 0.37 0.63 0.56 0.35 The correlation analysis revealed significant positive correlations between ESI actual value and chatGPT and between ESI actual value and Gemini Care provider prompt-type results. Kendall's tau_b coefficient, a measure of rank-order correlation, indicates a strong association between the ESI and these two variables, as shown in Table 9. The statistical significance of the correlations (p < 0.01) suggests that the relationships are not due to chance. Table 9 - Correlation Kendall's Tau_b test results for care provider prompt type Variable (Level) Care provider’s prompt Correlation Coefficient ESI (actual) ChatGPT Gemini ESI (actual) 1.00 0.168 0.198 ChatGPT 0.168 1.00 0.253 Gemini 0.198 0.253 1.00 All the variable correlations are significant at the sig=0.01 level (2-tailed). Another correlation analysis revealed significant positive correlations between actual ESI and chatGPT and between gemini self-triage prompt-type results, as shown in Table 10. Kendall's tau_b coefficient indicates a moderate to strong association between the ESI and these two variables. The statistical significance of the correlations (p < 0.05 for chatGPT and p< 0.01 for Gemini) suggests that the relationships are not due to chance. Table 10 - Correlations between Kendall's _b test results for the self-triage prompt type Variable (Level) Self-triage prompt Correlation Coefficient ESI (actual) ChatGPT Gemini ESI (actual) 1.00 0.093* 0.169** ChatGPT 0.093* 1.00 0.377** Gemini 0.169** 0.377** 1.00 *Correlation is significant at the sig=0.05 level (2-tailed). **Correlation is significant at the sig=0.01 level (2-tailed). Figure 7 presents precision‒recall curves and ROC curves for ChatGPT and Gemini in the care provider and patient roles, illustrating their performance across the four severity levels. Discussion This study compares ChatGPT and Gemini in supporting headache patient triage on the basis of the Emergency Severity Index (ESI). The results indicate that ChatGPT performs better at lower acuity levels (ESIs 3 and 4). Gemini is slightly more effective at higher acuity levels (ESIs 1 and 2), showing its potential for identifying critical cases. Both models performed better when used as decision support for healthcare providers than when used for self-triage, highlighting the importance of structured input and prompt design. Further research, particularly with real-world EHR data, is essential to assess the reliability of these tools across diverse patient cases and optimize their use for safe and accurate self-triage. Previous studies, including Fraser et al. (2023) and Williams et al. (2024), have shown that AI can effectively triage patients, mainly through general symptom checkers or diagnostic tools. Our study adds to this research by uniquely comparing ChatGPT and Gemini, two advanced models, specifically in headache triage—a symptom with significant diagnostic variability. While existing research, such as that of Meral et al. (2024), indicates the potential of AI in emergency settings, this study further explores how ChatGPT and Gemini handle the complexities of headache presentations, highlighting strengths and limitations. This study highlights the potential of large language models, such as ChatGPT and Gemini, as valuable aids in emergency department (ED) triage. By enhancing the precision of triage decisions, these tools could help healthcare providers allocate resources more effectively and allow patients to make informed self-triage decisions, potentially easing ED congestion. However, further refinement is essential to ensure patient safety and minimize the risks of inaccurate self-assessments. Limitations This study has several limitations. This study focused only on headache cases, which may reflect a partial scope of conditions encountered in an ED. The dataset was sourced from a single EHR system, which may limit the generalizability of the results across other patient demographics and healthcare settings. Furthermore, as AI models rely on structured data, their effectiveness with complex or unstructured inputs, such as free-text symptom descriptions, is still being determined and may vary across cultural and linguistic contexts. Future Work Future research should validate these findings with a more extensive and diverse dataset encompassing a broader range of conditions. Integrating LLMs such as ChatGPT and Gemini into existing clinical decision support systems could enable more comprehensive patient data analysis. Further advancements could focus on improving these models' ability to interpret unstructured data, such as free-text symptom descriptions, and exploring hybrid systems that leverage the strengths of both models for enhanced triage performance. Conclusion In conclusion, this study highlights the potential of ChatGPT and Gemini as supportive tools for clinical and self-triage. While promising, these models need further refinement to ensure reliable and accurate performance across diverse patient scenarios. Optimizing these technologies could significantly enhance patient care, reduce ED overcrowding, and empower individuals to make informed health decisions. Declarations Ethics approval and consent to participate: Not applicable Clinical trial number: not applicable Consent for publication: Not applicable Availability of data and materials: https://physionet.org/content/mimic-iv-ed/2.2/ Competing interests: The authors declare that they have no competing interests. Funding: No funding was received to assist with the preparation of this manuscript. Authors' contributions: H.S. analyzed the output results, wrote the Results and Discussion sections, and edited the final manuscript. F.L. designed and created the prompts, wrote the Introduction, and conducted the literature review. S.R.N.K. supervised the entire project and contributed to the study design and methodology. M.T. designed and oversaw the patient triage process. M.R. extracted information from ChatGPT and Gemini using the designed prompts. All authors reviewed and approved the final manuscript. Acknowledgements: Not applicable References Chmielewski N, Moretz J. ESI Triage Distribution in U.S. Emergency Departments, Adv. Emerg. Nurs. J. , vol. 44, no. 1, pp. 46–53, Mar. 2022, 10.1097/TME.0000000000000390 Morley C, Unwin M, Peterson GM, Stankovich J, Kinsman L. Emergency department crowding: a systematic review of causes, consequences and solutions. PLoS ONE. 2018;13(8):e0203316. Adriaenssens J, De Gucht V, Maes S. Determinants and prevalence of burnout in emergency nurses: a systematic review of 25 years of research. Int J Nurs Stud. 2015;52(2):649–61. Kulstad EB, Sikka R, Sweis RT, Kelley KM, Rzechula KH, editors. ED overcrowding is associated with an increased frequency of medication errors, Am. J. Emerg. Med. , vol. 28, no. 3, pp. 304–309, 2010. Yancey CC, O’Rourke MC. Emergency department triage, 2020. Salman OH, Taha Z, Alsabah MQ, Hussein YS, Mohammed AS, Aal-Nouman M. A review on utilizing machine learning technology in the fields of electronic emergency triage and patient priority systems in telemedicine: Coherent taxonomy, motivations, open research challenges and recommendations for intelligent future work. Comput Methods Programs Biomed. 2021;209:106357. Dickson SJ, Dewar C, Richardson A, Hunter A, Searle S, Hodgson LE. Agreement and validity of electronic patient self-triage (eTriage) with nurse triage in two UK emergency departments: a retrospective study. Eur J Emerg Med. 2022;29(1):49–55. Augusto Duenhas T. Accorsi Outcome After Self-Triage App Referral in Urgent Direct-to-Consumer Telemedicine Encounter. Telemed E-Health, 2024. Ilicki J. Challenges in evaluating the accuracy of AI-containing digital triage systems: A systematic review. PLoS ONE. 2022;17(12):e0279636. Waghmare C. Introduction to ChatGPT. in Unleashing The Power of ChatGPT: A Real World Business Applications. Springer; 2023. pp. 1–26. Islam R, Ahmed I. Gemini-the most powerful LLM: Myth or Truth, presented at the 2024 5th Information Communication Technologies Conference (ICTC), IEEE, 2024, pp. 303–308. Chiang C, et al. A large language model–based generative natural language processing framework fine-tuned on clinical notes accurately extracts headache frequency from electronic health records. Headache J Head Face Pain. 2024;64(4):400–9. Fisher L. Assessment of patients presenting with headache. Innovait, 2011. Lacerenza MR, Schoss F, Grazzi L. The multimodal treatment in headaches. J Headache Pain. 2015;16(Suppl 1):A47. De Angelis L, et al. ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health. Front Public Health. 2023;11:1166120. Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med. 2023;29(8):1930–40. Rashid K et al. Accuracy of emergency room triage using emergency severity index (esi): Independent predictor of under and over triage. Cureus, 13, 12, 2021. Ceney A, Tolond S, Glowinski A, Marks B, Swift S, Palser T. Accuracy of online symptom checkers and the potential impact on service utilization. PLoS ONE. 2021;16(7):e0254088. Meral G, Ateş S, Günay S, Öztürk A, Kuşdoğan M. Comparative analysis of ChatGPT, Gemini and emergency medicine specialist in ESI triage assessment, Am. J. Emerg. Med. , vol. 81, pp. 146–150, Jul. 2024, 10.1016/j.ajem.2024.05.001 Williams CYK, et al. Use of a Large Language Model to Assess Clinical Acuity of Adults in the Emergency Department. JAMA Netw Open. May 2024;7(5):e248895. 10.1001/jamanetworkopen.2024.8895 . Fraser H, Crossland D, Bacher I, Ranney M, Madsen T, Hilliard R. Comparison of Diagnostic and Triage Accuracy of Ada Health and WebMD Symptom Checkers, ChatGPT, and Physicians for Patients in an Emergency Department: Clinical Data Analysis Study, JMIR MHealth UHealth , vol. 11, p. e49995, Oct. 2023, 10.2196/49995 Johnson A, Bulgarelli L, Pollard T, Celi LA, Mark R, Horng S. ‘MIMIC-IV-ED’ (version 2.2). PhysioNet (2023). https://doi.org/10.13026/ntk-km72 . 2023. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5429142","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":379451893,"identity":"ba74e90d-9315-4fe8-bec3-a0a35a7a1787","order_by":0,"name":"Hamed Samadpour","email":"","orcid":"","institution":"Tehran University of Medical Sciences","correspondingAuthor":false,"prefix":"","firstName":"Hamed","middleName":"","lastName":"Samadpour","suffix":""},{"id":379451894,"identity":"88871d9a-bcba-42d4-a180-360183e4062e","order_by":1,"name":"Sharareh Rostam Niakan Kalhori","email":"","orcid":"","institution":"Tehran University of Medical Sciences","correspondingAuthor":false,"prefix":"","firstName":"Sharareh","middleName":"Rostam Niakan","lastName":"Kalhori","suffix":""},{"id":379451895,"identity":"83ba58ba-7cd1-421e-bc6e-0c814b3bedf3","order_by":2,"name":"Masoumeh Tahmasebi","email":"","orcid":"","institution":"Ilam University of Medical Sciences","correspondingAuthor":false,"prefix":"","firstName":"Masoumeh","middleName":"","lastName":"Tahmasebi","suffix":""},{"id":379451896,"identity":"b0bfcf8f-96ce-4ed0-bd22-e792a1e1ab09","order_by":3,"name":"Mahla Rakhshi","email":"","orcid":"","institution":"Ilam University of Medical Sciences","correspondingAuthor":false,"prefix":"","firstName":"Mahla","middleName":"","lastName":"Rakhshi","suffix":""},{"id":379451897,"identity":"82ad349f-cd7f-4ab5-9700-06ce15975300","order_by":4,"name":"Fatemeh Lotfi","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA6UlEQVRIie2RMYoCQRBF/zCgiWBakXOFXoRlQcGDmJQITtQgmGy09CI4VzAQvIJGpiUTmHgGcRCMO1oMDBzbYE3anXCDftkv6vELCggE/iE9iQ3QpdY9yO9c+RWFyEBG1H4oXF3BwLjI/s1n5ftk+SPdZLtCokveT0xcWIwPXqWHaKqESS/2zOVhuV5JrU1Qk1ctM9peSM/B8lCA93LuP9EpZUtKzcI4ZWnqP5UUJhrCKUYaf7ZM1Z7pbU5nCI9SvcobE+KXSrw9fvJXQs30ZG23o5dZtrb26leQHJ+C24tR7T+BQCAQ8HMDH25Q284xKmsAAAAASUVORK5CYII=","orcid":"","institution":"Tehran University of Medical Sciences","correspondingAuthor":true,"prefix":"","firstName":"Fatemeh","middleName":"","lastName":"Lotfi","suffix":""}],"badges":[],"createdAt":"2024-11-11 05:53:38","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5429142/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5429142/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":71730517,"identity":"51016170-766c-47d4-92e0-72e4260c165d","added_by":"auto","created_at":"2024-12-18 06:45:34","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":87582,"visible":true,"origin":"","legend":"\u003cp\u003eThe data preparation method of the study starts with tabular data preparation, sentence generation, and prompting us to obtain \u0026nbsp;the ICT outputs.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-5429142/v1/0b1a946c794ac69dedee6485.png"},{"id":71730549,"identity":"6b1c0bd5-77b2-4321-8918-9c20b94ba60f","added_by":"auto","created_at":"2024-12-18 06:45:37","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":70244,"visible":true,"origin":"","legend":"\u003cp\u003eRules used to determine the actual level of patient triage\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-5429142/v1/e7c358348ba9f06041d20d96.png"},{"id":71730542,"identity":"698a1fe9-818c-454c-8665-82774c08e68c","added_by":"auto","created_at":"2024-12-18 06:45:36","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":27262,"visible":true,"origin":"","legend":"\u003cp\u003eChatGPT confusion matrix for care provider prompt\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-5429142/v1/ecd7ccef6d0de8c3b80162a4.png"},{"id":71730518,"identity":"3c61f007-9e8d-493c-9914-5444160afc61","added_by":"auto","created_at":"2024-12-18 06:45:34","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":25512,"visible":true,"origin":"","legend":"\u003cp\u003eGemini confusion matrix for care provider prompt\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-5429142/v1/984f777189472b2960e5b016.png"},{"id":71730519,"identity":"2c7423e8-2496-4a32-bfe0-aa9c575053f6","added_by":"auto","created_at":"2024-12-18 06:45:34","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":28215,"visible":true,"origin":"","legend":"\u003cp\u003eChatGPT confusion matrix in a patient prompt\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-5429142/v1/3c502a1fd6f8b4d34d168992.png"},{"id":71730525,"identity":"0d628d13-3ef2-441a-8bc4-017e7c3900e0","added_by":"auto","created_at":"2024-12-18 06:45:35","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":25976,"visible":true,"origin":"","legend":"\u003cp\u003eGemini confusion matrix in a patient prompt\u003c/p\u003e","description":"","filename":"6.png","url":"https://assets-eu.researchsquare.com/files/rs-5429142/v1/239beb1f4d2f5d073377605c.png"},{"id":71730520,"identity":"d4fd3105-a4d0-4bf7-a8d0-4360ba650879","added_by":"auto","created_at":"2024-12-18 06:45:34","extension":"png","order_by":7,"title":"Figure 7","display":"","copyAsset":false,"role":"figure","size":315740,"visible":true,"origin":"","legend":"\u003cp\u003ePrecision‒recall and ROC curves of the care provider and patient self-triage prompt type\u003c/p\u003e","description":"","filename":"7.png","url":"https://assets-eu.researchsquare.com/files/rs-5429142/v1/1e08dfddc9ce0888ce7cf2e3.png"},{"id":106799261,"identity":"5e34713f-89de-41fa-a49a-8ef7d0d20955","added_by":"auto","created_at":"2026-04-13 14:28:54","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1876802,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5429142/v1/31ab05b9-35c5-4064-9596-164b27430872.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"ESI Triage Level Assignment for Headache Patients: Comparative Analysis of ChatGPT and Gemini Performance for Supporting Care Provider Decisions and Self-triage","fulltext":[{"header":"Introduction","content":"\u003cp\u003eAccurate patient categorization upon arrival at the emergency department is crucial for effectively delivering emergency care. Overestimating patient acuity can misallocate limited resources, particularly during high patient volumes. Conversely, underestimation of patient severity can result in delayed diagnosis and treatment, potentially compromising patient outcomes [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. Overcrowding in the emergency department negatively impacts several patient-oriented outcomes, including mortality, complication rates, walkouts, time to treatment, satisfaction, and duration of stay [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. Additionally, overcrowding has been identified as a significant stressor for medical staff, leading to increased burnout [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e] and diagnostic errors [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eTriage is a structured process designed to expedite care delivery on the basis of the severity and extent of the health issue, with three distinct phases: prehospital triage, triage at the scene, and triage upon arrival at the emergency department [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. Electronic emergency triage (E-triage) is an emerging approach that uses digital and AI-driven tools to evaluate and prioritize patients on the basis of their condition severity, improving both the efficiency and accuracy of healthcare delivery [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. E-triage can be conducted through two methods: self-triage or clinician-led triage. In self-triage, patients assess their condition via digital tools or symptom checkers to decide if medical care is necessary [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. Effective self-triage can enhance the clinician-led triage process by reducing patient queues, facilitating timely care, and optimizing healthcare resources [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eTraditional triage systems face limitations, such as slow response times, limited accuracy, and high costs. These challenges have driven the development of intelligent triage solutions incorporating AI-based approaches to address these deficiencies [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eRecent advancements in AI have led to the creation of large language models (LLMs), such as ChatGPT by OpenAI and Gemini by Google, which hold promise for various medical applications, including patient triage [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e], [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eLLMs serve as the foundation for intelligent conversational tools (ICTs), enabling real-time, human-like interactions in healthcare. By understanding and processing natural language, these models can provide swift, accurate assistance, which is especially crucial for conditions such as headaches [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e], ranging from benign to life-threatening causes such as stroke or aneurysm [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. Symptom assessment can be particularly challenging [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]. ChatGPTs and Gemini are adept at delivering contextually relevant information, enhancing triage and self-triage, and supporting patients and healthcare providers with precise, data-driven insights [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eWe selected ChatGPT and Gemini because of their strong ability to process clinical data, align decisions with medical protocols, and adaptability across diverse healthcare domains [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]. Their advanced AI technology is suitable, particularly for critical emergency triage situations, where timely and accurate information is essential.\u003c/p\u003e \u003cp\u003eThe Emergency Severity Index (ESI) is a widely utilized triage system in emergency departments to prioritize patient care. This system categorizes patients into five levels on the basis of their clinical conditions and resource requirements. Patients classified as ESI 1 or 2 exhibit emergency or urgent conditions that demand immediate attention, whereas patients designated as ESI 3, 4, or 5 are considered stable and are prioritized on the basis of the extent of resources necessary for their treatment [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eCurrent systems, such as symptom checkers, are not explicitly optimized for headache assessment, highlighting the need for more targeted, AI-driven approaches [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e]. An integrated system combining AI with existing triage methods such as the ESI could significantly improve triage accuracy for headache patients.\u003c/p\u003e \u003cp\u003eRecent research has explored the application of ChatGPT and Gemini technology in patient triage. For example, G\u0026uuml;rb\u0026uuml;z Meral et al. [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e] compared the accuracy of the ChatGPT and Gemini to that of emergency medicine specialists in ESI triage, and Williams et al. examined the ability of the GPT to assess clinical acuity in ED patients [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]. In the Fraser et al. study, deidentified ED patient data were used to analyze the diagnostic and triage capabilities of the ChatGPT models Ada and WebMD [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]. However, our literature review indicates that no studies have evaluated Gemini's performance on the basis of the ESI triage system dataset. Instead of focusing solely on emergency cases, we examined ChatGPT and Gemini\u0026rsquo;s performance from the perspectives of patients and healthcare providers via ESI.\u003c/p\u003e \u003cp\u003eThis study aims to compare and evaluate the performance of two large language models (LLMs), ChatGPT and Gemini, in assessing ESI triage from both the healthcare provider and patient perspectives.\u003c/p\u003e"},{"header":"Method","content":"\u003cp\u003eOverview\u003c/p\u003e\n\u003cp\u003eThis study utilized a robust methodology with five steps: 1. Selection of database, sample, and data preprocessing; 2. Prompt generation; 3. Analysis with LLM ChatBots; 4. The actual level of patients should be determined, and 5. Interpretations of the results with detailed descriptions of each stage are provided in the following sections and shown in Figure 1.\u003c/p\u003e\n\u003cp\u003eSelection of the Database, Sample, and Data Preprocessing\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDatabase Selection:\u003c/strong\u003e Retrospectively collected medical data offer the opportunity to improve patient care through knowledge discovery and algorithm development. The MIMIC-IV-ED dataset is an extensive, freely accessible, and deidentified data collection containing 425,000 emergency department (ED) admissions between 2011 and 2019. These data include vital signs, triage information, medication prescriptions, and final diagnoses. This tool is designed to support various educational initiatives and research studies. The patient population is a heterogeneous group with varying severity levels, ranging from mild conditions to life-threatening cardiac complications. The data model consists of six tables: edstays, diagnosis, medrecon, pyxis, triage, and vitalsign. The edstays table tracks patient stays in the emergency department. The diagnosis table contains coded diagnoses for patients on the basis of the International Classification of Diseases, either the Ninth or Tenth revision (ICD-9 or ICD-10). Trained coders assign these diagnoses after patient discharge and are primarily used for billing purposes. The medrecon table lists each patient\u0026apos;s medications before their ED stay as part of medication reconciliation. The pyxis table details the medications dispensed through the BD Pyxis MedStation, an automated medication dispensing system used in the ED. The triage table records information collected during the triage process, where patients\u0026rsquo; health status and reasons for their visit are assessed upon arrival. Finally, the vital table includes documented periodic vital signs for patients during their ED stay. The database was selected because of the availability of vital signs and other necessary measurable patient information, its outcome-oriented nature, and the recorded acuity level of each patient as assessed by care providers [22].\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSample Selection:\u003c/strong\u003e A rigorous random selection process was employed from the MIMIC-ED v2.2 database, which includes 3,951 patient records with a primary complaint of headache and self-arrival. Fifteen percent of the records\u0026mdash;592 samples\u0026mdash;were random, and a simple sampling method was chosen via Excel software. This careful selection was performed to ensure the fairness and representativeness of the samples, thereby avoiding any potential bias in the data. The following information fields were extracted for the patients, as presented in Table 1.\u003c/p\u003e\n\u003cp\u003eTable 1- Variables extracted from MIMIC-IV ED\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"624\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\u003cstrong\u003eVariable\u0026nbsp;\u003c/strong\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 468px;\"\u003e\u003cstrong\u003eDefinition\u003c/strong\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\u003cstrong\u003eGender\u003c/strong\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 468px;\"\u003eThe sex of the subjects\u0026nbsp;\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\u003cstrong\u003eRace\u003c/strong\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 468px;\"\u003edemographic category used to describe a patient\u0026rsquo;s self-identified racial or ethnic background\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\u003cstrong\u003eArrival_transport\u003c/strong\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 468px;\"\u003emode of transportation that brought a patient to the emergency department (ED)\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\u003cstrong\u003eDisposition\u003c/strong\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 468px;\"\u003ethe outcome or status of a patient at the time of discharge from the emergency department (ED)\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\u003cstrong\u003eTemperature\u003c/strong\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 468px;\"\u003eThe body temperature of the subjects in degrees Fahrenheit\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\u003cstrong\u003eHeart rate\u003c/strong\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 468px;\"\u003enumber of times the heart beats per minute (bpm)\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\u003cstrong\u003eRespiratory rate\u003c/strong\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 468px;\"\u003enumber of breaths a person takes per minute\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\u003cstrong\u003eOxygen saturation\u003c/strong\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 468px;\"\u003epercentage of hemoglobin in the blood that is saturated with oxygen\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\u003cstrong\u003eSystolic blood pressure(mmHg)\u003c/strong\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 468px;\"\u003epressure in the arteries during heart contraction, representing the higher value in a blood pressure reading\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\u003cstrong\u003eDiastolic blood pressure(mmHg)\u003c/strong\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 468px;\"\u003epressure in the arteries when the heart is at rest between beats, representing the lower value in a blood pressure reading\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\u003cstrong\u003ePain\u003c/strong\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 468px;\"\u003eunpleasant sensory and emotional experience often associated with actual or potential tissue damage\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\u003cstrong\u003eAcuity\u003c/strong\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 468px;\"\u003eseverity or intensity of a patient\u0026apos;s condition or illness\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\u003cstrong\u003eAge\u003c/strong\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 468px;\"\u003elength of time that a person has lived or existed, typically measured in years\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u003cstrong\u003eData Preprocessing:\u003c/strong\u003e Using the MS SQL Server 2022 database, the edstays and triage tables were merged via the subject_id and stay_id fields. All patients were over 18 years old. Records with outliers or missing data were removed. Five hundred records were left for analysis.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePrompt generation\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eRecord Conversion:\u003c/strong\u003e The records selected for analysis were converted into prompts via the mail merge feature in Word. This process involves combining the data in the records with predefined templates to generate personalized messages.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTypes of promoters:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTwo types of prompts were created, each designed for a different purpose:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePrompt for self-triage:\u0026nbsp;\u003c/strong\u003eThis prompt allows patients to briefly describe their condition and receive appropriate care without providing additional explanations. This prompt is second-level and imperative. This second-level prompt contains specific contextual data (symptoms and vital signs) and asks the model to assess the required level of care on the basis of this information. The data provided include factors such as sex, race, and vital signs, which the model must analyze. This is an imperative prompt because the model is explicitly instructed to provide a concise and direct response with minimal explanation.\u003c/p\u003e\n\u003cp\u003eSample Prompt for Self-Triage:\u003c/p\u003e\n\u003cp\u003e\u0026nbsp; \u0026nbsp; \u0026ldquo; I have a headache, my gender is Female, and my race is BLACK/AFRICAN AMERICAN. Her temperature was 98 F, her heart rate was 74, her respiratory rate was 18, her o2 saturation was 100, her systolic blood pressure was 115 mmHg, her diastolic blood pressure was 69 mmHg, and her pain level was 7. Please let me know which level of care I need. I am worried that I will be kept in the hospital due to my symptoms. I need to be prepared for this probable stay. I need to be ready in advance for what will happen in the hospital. Inform me shortly which level of care would be considered for me. Answer me with the slightest explanation and with full frankness. Say it in one line\u003cspan dir=\"RTL\"\u003e.\u0026nbsp;\u003c/span\u003e\u0026ldquo;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eHealthcare provider decision support prompt:\u0026nbsp;\u003c/strong\u003eThis prompt allows healthcare providers to assess a patient\u0026rsquo;s triage level via the Emergency Severity Index (ESI). This is a\u0026nbsp;second-level and role-play prompt. This second-level prompt also includes specific and specialized contextual data that require detailed analysis. The patient\u0026rsquo;s symptoms and vital signs are provided, and the model is asked to assess the situation on the basis of a specific medical index (the Emergency Severity Index).\u003cstrong\u003e\u003cspan dir=\"RTL\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/strong\u003eThis is a role-play prompt because the model is asked to assume the role of an emergency medicine specialist and make decisions on the basis of that role.\u003c/p\u003e\n\u003cp\u003eSample Healthcare provider decision support prompt:\u003c/p\u003e\n\u003cp\u003e\u003cspan dir=\"RTL\"\u003e\u0026nbsp; \u0026ldquo;\u003c/span\u003eYou are an emergency medicine specialist. Please assess my triage level on the basis of the EMERGENCY SEVERITY INDEX and my vital signs. I tell you my symptoms and vital signs, and you should summarize your assessment in one sentence. I have a headache, my sex is female, and my race is BLACK/AFRICAN AMERICAN. Her heart rate was 90, her respiratory rate was 16, her o2 saturation was 99, her systolic blood pressure was 134 mmHg, her diastolic blood pressure was 51 mmHg, and her pain level was 4. Shortly, let me know what my triage level is.\u003cspan dir=\"RTL\"\u003e\u0026rdquo;\u003c/span\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cspan dir=\"RTL\"\u003e\u0026nbsp;\u003c/span\u003eAnalysis with LLM ChatBots\u003c/p\u003e\n\u003cp\u003eThe prompts were subsequently sent to ChatGPT and Gemini. The output from the large language models (LLMs) in response to patient prompts was mapped to the five emergency severity index (ESI) levels\u0026nbsp;according to the opinions of the emergency specialists, as shown in Table 2. In the healthcare provider decision support section, the level assigned by the chatbots was directly used as the result.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003e2\u003c/strong\u003e-LLM response sample to triage level assignment\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 558px;\"\u003e\u003cstrong\u003eChatGPT Sample Response\u003c/strong\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd style=\"width: 66px;\"\u003e\u003cstrong\u003eAssigned Triage Level\u003c/strong\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 558px;\"\u003eBased on your symptoms, you would likely require immediate emergency care and admission to the hospital for further evaluation and management of your condition.\u003cbr\u003e\u003c/td\u003e\n \u003ctd style=\"width: 66px;\"\u003e1\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 558px;\"\u003eBased on your symptoms and vital signs, you may require urgent evaluation in an emergency department or acute care setting due to the severity of your headache.\u003cbr\u003e\u003c/td\u003e\n \u003ctd style=\"width: 66px;\"\u003e2\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 558px;\"\u003eYou likely need urgent care based on your symptoms and vital signs, but hospitalization is not typically necessary unless further evaluation indicates otherwise.\u003cbr\u003e\u003c/td\u003e\n \u003ctd style=\"width: 66px;\"\u003e3\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 558px;\"\u003eBased on your symptoms and vital signs, you would likely require outpatient care unless there are additional concerning factors.\u003cbr\u003e\u003c/td\u003e\n \u003ctd style=\"width: 66px;\"\u003e4\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 558px;\"\u003eBased on your symptoms and vital signs, you likely do not require hospitalization and can manage your condition at home.\u003cbr\u003e\u003c/td\u003e\n \u003ctd style=\"width: 66px;\"\u003e5\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eDetermining the actual triage level of patients\u003c/p\u003e\n\u003cp\u003eThe acuity field in the data and predefined rules based on the opinions of emergency specialists were used to determine the actual level of patients. These rules involved thoroughly assessing clinical signs and applying standard emergency indicators to determine the triage level. The rules employed in this process are shown in Figure 2.\u003c/p\u003e\n\u003ctable cellpadding=\"0\" cellspacing=\"0\" width=\"100%\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003eInterpretation of Results\u003cp\u003e\u003c/p\u003e\n\u003cp\u003eAs a result of\u0026nbsp;the\u0026nbsp;consensus meetings of experts (3 medical informatics\u003cspan dir=\"RTL\"\u003e\u0026nbsp;\u003c/span\u003eresearchers), the outputs from these bots were interpreted and analyzed on the basis of the ESI levels. Differences and similarities between the bot-generated and actual assigned levels were examined. A confusion matrix containing all levels was created, and each level\u0026apos;s precision, recall, and F1 score were calculated. Precision‒recall and ROC curves are plotted in Python via Pandas, Matplotlib, and the Sklearn library. Additionally, Kendall\u0026apos;s Tau statistical test was conducted in SPSS version 27 to assess the correlation between the results. Kendall\u0026apos;s Tau is a nonparametric test measuring the strength and direction of associations between ordinal variables. In this study, the variables include the actual triage level of the ESI, the ChatGPT, and the Gemini-predicted triage level.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eFollowing data cleaning, 500 records were retained for analysis. Among these, 152 records (30.4%) corresponded to male participants, whereas 348 records (69.6%) were from female participants. Table 3 provides a summary of the descriptive statistics for the studied variables. Pain levels were assessed via an 11-point numerical rating scale (NRS) ranging from 0 to 10. Note that there were no records with an actual ESI level of 5, and the results corresponding to this level were excluded from the analysis.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003e3\u003c/strong\u003e- Descriptive statistics of the variables\u003c/p\u003e\n\u003cdiv align=\"\"\u003e\n \u003ctable border=\"0\" cellspacing=\"0\" cellpadding=\"0\" width=\"582\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 258px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eVariable\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 96px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eRange\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 78px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eMean\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eStandard deviation\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 258px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eTemperature\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 96px;\"\u003e\n \u003cp\u003e96-103\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 78px;\"\u003e\n \u003cp\u003e98\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e0.85\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 258px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eHeart rate\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 96px;\"\u003e\n \u003cp\u003e50-139\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 78px;\"\u003e\n \u003cp\u003e82\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e15.62\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 258px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eRespiratory rate\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 96px;\"\u003e\n \u003cp\u003e12-24\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 78px;\"\u003e\n \u003cp\u003e17\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e1.54\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 258px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eOxygen saturation\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 96px;\"\u003e\n \u003cp\u003e92-100\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 78px;\"\u003e\n \u003cp\u003e99\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e1.36\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 258px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eSystolic blood pressure\u003c/strong\u003e\u003cstrong\u003e(mmHg)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 96px;\"\u003e\n \u003cp\u003e86-202\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 78px;\"\u003e\n \u003cp\u003e134\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e19.81\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 258px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eDiastolic blood pressure\u003c/strong\u003e\u003cstrong\u003e(mmHg)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 96px;\"\u003e\n \u003cp\u003e51-177\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 78px;\"\u003e\n \u003cp\u003e80\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e13.34\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 258px;\"\u003e\n \u003cp\u003e\u003cstrong\u003ePain\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 96px;\"\u003e\n \u003cp\u003e0-10\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 78px;\"\u003e\n \u003cp\u003e7\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e2.62\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 258px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eAcuity\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 96px;\"\u003e\n \u003cp\u003e1-4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 78px;\"\u003e\n \u003cp\u003e3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e0.5\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 258px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eAge*\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 96px;\"\u003e\n \u003cp\u003e18-87\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 78px;\"\u003e\n \u003cp\u003e41\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e16.19\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003e*Patient age was calculated using the date of birth and anchor date in the database.\u003c/p\u003e\n\u003ch2\u003eCare provider\u0026rsquo;s perspective\u003c/h2\u003e\n\u003cp\u003eThe comparative results generated by ChatGPT and Gemini on the basis of the care provider\u0026apos;s perspective are presented in Table 4. The confusion matrixes are shown in Figures 3-4.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003e4\u003c/strong\u003e\u003cspan dir=\"RTL\"\u003e-\u003c/span\u003e-LLM comparative confusion matrix (Care Provider Prompt)\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"683\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 53px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"10\" valign=\"top\" style=\"width: 539px;\"\u003e\n \u003cp\u003e\u003cstrong\u003ePredicted Values\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 53px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003eLevel\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"top\" style=\"width: 98px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e1\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"top\" style=\"width: 111px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e2\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"top\" style=\"width: 108px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e3\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"top\" style=\"width: 114px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e4\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"top\" style=\"width: 108px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e5\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"5\" style=\"width: 53px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eActual Value\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 39px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGPT\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGemini\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 53px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGPT\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGemini\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGPT\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGemini\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 56px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGPT\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGemini\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGPT\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGemini\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e1\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 39px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 53px;\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 56px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e8\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e12\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e2\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 39px;\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 53px;\"\u003e\n \u003cp\u003e33\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e18\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e34\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e24\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 56px;\"\u003e\n \u003cp\u003e12\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e41\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e83\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e3\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 39px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 53px;\"\u003e\n \u003cp\u003e90\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e20\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e210\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e97\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 56px;\"\u003e\n \u003cp\u003e79\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e273\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e11\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e391\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e4\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 39px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 53px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 56px;\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e14\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e14\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 53px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 39px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e3\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e1\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 53px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e130\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e40\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e254\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e123\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 56px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e98\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e336\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e15\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e500\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003ch2\u003ePatient perspective (self-triage)\u003c/h2\u003e\n\u003cp\u003eThe comparative results generated by ChatGPT and Gemini on the basis of the patient\u0026apos;s perspective (self-triage) are presented in Table 5. The confusion matrixes are shown in Figures 5--6.\u003c/p\u003e\n\u003cp\u003eTable 5\u003cspan dir=\"RTL\"\u003e\u0026mdash;\u003c/span\u003eLLM comparative confusion matrix (Patient Prompt)\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"683\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 53px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"10\" valign=\"top\" style=\"width: 539px;\"\u003e\n \u003cp\u003e\u003cstrong\u003ePredicted Values\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 53px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003eLevel\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"top\" style=\"width: 98px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e1\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"top\" style=\"width: 111px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e2\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"top\" style=\"width: 108px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e3\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"top\" style=\"width: 114px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e4\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"top\" style=\"width: 108px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e5\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"5\" style=\"width: 53px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eActual Value\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 39px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGPT\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGemini\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 53px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGPT\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGemini\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGPT\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGemini\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 56px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGPT\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGemini\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGPT\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGemini\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e1\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 39px;\"\u003e\n \u003cp\u003e3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 53px;\"\u003e\n \u003cp\u003e5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e7\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 56px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e12\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e2\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 39px;\"\u003e\n \u003cp\u003e3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e10\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 53px;\"\u003e\n \u003cp\u003e36\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e15\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e22\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e47\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 56px;\"\u003e\n \u003cp\u003e21\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e7\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e83\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e3\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 39px;\"\u003e\n \u003cp\u003e33\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e25\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 53px;\"\u003e\n \u003cp\u003e109\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e23\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e106\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e269\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 56px;\"\u003e\n \u003cp\u003e139\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e25\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e49\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e391\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e4\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 39px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 53px;\"\u003e\n \u003cp\u003e4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e7\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 56px;\"\u003e\n \u003cp\u003e7\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e14\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 53px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 39px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e40\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e38\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 53px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e154\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e42\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e132\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e330\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 56px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e168\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e32\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 50px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e6\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e58\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e500\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp dir=\"RTL\"\u003e\u003cbr\u003e\u003c/p\u003e\n\u003ch2\u003eAnalysis of Precision, Recall, and F1 Scores\u003c/h2\u003e\n\u003cp\u003eThe classification reports for the two prompt types are shown in Tables 6--8.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003e6\u003c/strong\u003e-Gemini and chatGPT \u003cstrong\u003eprecision\u003c/strong\u003e to determine the level of the ESI\u003c/p\u003e\n\u003cdiv align=\"\"\u003e\n \u003ctable border=\"0\" cellspacing=\"0\" cellpadding=\"0\" width=\"436\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"2\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003ePrecision\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"top\" style=\"width: 135px;\"\u003e\n \u003cp\u003e\u003cstrong\u003ePatient\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"top\" style=\"width: 151px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eCare Provider\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 77px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGPT\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGemini\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 93px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGPT\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGemini\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eLevel 1\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 77px;\"\u003e\n \u003cp\u003e0.07\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.05\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 93px;\"\u003e\n \u003cp\u003e0.00\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.00\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eLevel 2\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 77px;\"\u003e\n \u003cp\u003e0.23\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.36\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 93px;\"\u003e\n \u003cp\u003e0.25\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.45\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eLevel 3\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 77px;\"\u003e\n \u003cp\u003e0.80\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.82\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 93px;\"\u003e\n \u003cp\u003e0.83\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.79\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eLevel 4\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 77px;\"\u003e\n \u003cp\u003e0.04\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.09\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 93px;\"\u003e\n \u003cp\u003e0.06\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.04\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eMicro Avg\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 77px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.23\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.26\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 93px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.23\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.32\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eWeighted Avg\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 77px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.67\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.70\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 93px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.69\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.69\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003e7\u003c/strong\u003e- Gemini and chatGPT \u003cstrong\u003eRecall\u003c/strong\u003e to determine the level of ESI\u003c/p\u003e\n\u003cdiv align=\"\"\u003e\n \u003ctable border=\"0\" cellspacing=\"0\" cellpadding=\"0\" width=\"436\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"2\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eRecall\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"top\" style=\"width: 135px;\"\u003e\n \u003cp\u003e\u003cstrong\u003ePatient\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"top\" style=\"width: 151px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eCare Provider\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 77px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGPT\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGemini\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 93px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGPT\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGemini\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eLevel 1\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 77px;\"\u003e\n \u003cp\u003e0.25\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.17\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 93px;\"\u003e\n \u003cp\u003e0.00\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.00\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eLevel 2\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 77px;\"\u003e\n \u003cp\u003e0.43\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.18\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 93px;\"\u003e\n \u003cp\u003e0.40\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.22\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eLevel 3\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 77px;\"\u003e\n \u003cp\u003e0.27\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.69\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 93px;\"\u003e\n \u003cp\u003e0.54\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.25\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eLevel 4\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 77px;\"\u003e\n \u003cp\u003e0.50\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.21\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 93px;\"\u003e\n \u003cp\u003e0.43\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e1.00\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eMicro Avg\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 77px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.29\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.25\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 93px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.27\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.37\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eWeighted Avg\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 77px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.30\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.58\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 93px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.50\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.26\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp dir=\"RTL\"\u003e\u003cspan dir=\"LTR\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003e8\u003c/strong\u003e- Gemini and chatGPT \u003cstrong\u003eF1\u003c/strong\u003e scores for determining the level of ESI\u003c/p\u003e\n\u003cdiv align=\"\"\u003e\n \u003ctable border=\"0\" cellspacing=\"0\" cellpadding=\"0\" width=\"436\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"2\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eF1-Score\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"top\" style=\"width: 135px;\"\u003e\n \u003cp\u003e\u003cstrong\u003ePatient\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"top\" style=\"width: 151px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eCare Provider\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 77px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGPT\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGemini\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 93px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGPT\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGemini\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eLevel 1\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 77px;\"\u003e\n \u003cp\u003e0.12\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.08\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 93px;\"\u003e\n \u003cp\u003e0.00\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.00\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eLevel 2\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 77px;\"\u003e\n \u003cp\u003e0.30\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.24\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 93px;\"\u003e\n \u003cp\u003e0.31\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.29\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eLevel 3\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 77px;\"\u003e\n \u003cp\u003e0.41\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.75\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 93px;\"\u003e\n \u003cp\u003e0.65\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.38\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eLevel 4\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 77px;\"\u003e\n \u003cp\u003e0.08\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.13\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 93px;\"\u003e\n \u003cp\u003e0.11\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e0.08\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eMicro Avg\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 77px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.18\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.24\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 93px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.21\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.19\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 150px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eWeighted Avg\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 77px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.37\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.63\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 93px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.56\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.35\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003eThe correlation analysis revealed significant positive correlations between ESI actual value and chatGPT and between ESI actual value and Gemini Care provider prompt-type results. Kendall\u0026apos;s tau_b coefficient, a measure of rank-order correlation, indicates a strong association between the ESI and these two variables, as shown in Table 9. The statistical significance of the correlations (p \u0026lt; 0.01) suggests that the relationships are not due to chance.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003e9\u003c/strong\u003e - Correlation Kendall\u0026apos;s Tau_b test results for care provider prompt type\u003c/p\u003e\n\u003cdiv align=\"\"\u003e\n \u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"2\" valign=\"top\" style=\"width: 216px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eVariable (Level)\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003eCare provider\u0026rsquo;s prompt\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"3\" valign=\"top\" style=\"width: 312px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eCorrelation Coefficient\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 108px;\"\u003e\n \u003cp\u003eESI (actual)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 114px;\"\u003e\n \u003cp\u003eChatGPT\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 90px;\"\u003e\n \u003cp\u003eGemini\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 216px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eESI (actual)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 108px;\"\u003e\n \u003cp\u003e1.00\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 114px;\"\u003e\n \u003cp\u003e0.168\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 90px;\"\u003e\n \u003cp\u003e0.198\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 216px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eChatGPT\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 108px;\"\u003e\n \u003cp\u003e0.168\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 114px;\"\u003e\n \u003cp\u003e1.00\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 90px;\"\u003e\n \u003cp\u003e0.253\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 216px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGemini\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 108px;\"\u003e\n \u003cp\u003e0.198\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 114px;\"\u003e\n \u003cp\u003e0.253\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 90px;\"\u003e\n \u003cp\u003e1.00\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003eAll the variable correlations are significant at the sig=0.01 level (2-tailed).\u003c/p\u003e\n\u003cp\u003eAnother correlation analysis revealed significant positive correlations between actual ESI and chatGPT and between gemini self-triage prompt-type results, as shown in Table 10. Kendall\u0026apos;s tau_b coefficient indicates a moderate to strong association between the ESI and these two variables. The statistical significance of the correlations (p \u0026lt; 0.05 for chatGPT and p\u0026lt; 0.01 for Gemini) suggests that the relationships are not due to chance.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003e10\u003c/strong\u003e- Correlations between Kendall\u0026apos;s _b test results for the self-triage prompt type\u003c/p\u003e\n\u003cdiv align=\"\"\u003e\n \u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"2\" valign=\"top\" style=\"width: 216px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eVariable (Level)\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003eSelf-triage prompt\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"3\" valign=\"top\" style=\"width: 312px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eCorrelation Coefficient\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 108px;\"\u003e\n \u003cp\u003eESI (actual)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 114px;\"\u003e\n \u003cp\u003eChatGPT\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 90px;\"\u003e\n \u003cp\u003eGemini\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 216px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eESI (actual)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 108px;\"\u003e\n \u003cp\u003e1.00\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 114px;\"\u003e\n \u003cp\u003e0.093*\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 90px;\"\u003e\n \u003cp\u003e0.169**\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 216px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eChatGPT\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 108px;\"\u003e\n \u003cp\u003e0.093*\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 114px;\"\u003e\n \u003cp\u003e1.00\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 90px;\"\u003e\n \u003cp\u003e0.377**\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 216px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eGemini\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 108px;\"\u003e\n \u003cp\u003e0.169**\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 114px;\"\u003e\n \u003cp\u003e0.377**\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 90px;\"\u003e\n \u003cp\u003e1.00\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003e*Correlation is significant at the sig=0.05 level (2-tailed).\u003c/p\u003e\n\u003cp\u003e**Correlation is significant at the sig=0.01 level (2-tailed).\u003c/p\u003e\n\u003cp\u003eFigure 7 presents precision‒recall curves and ROC curves for ChatGPT and Gemini in the care provider and patient roles, illustrating their performance across the four severity levels.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eThis study compares ChatGPT and Gemini in supporting headache patient triage on the basis of the Emergency Severity Index (ESI). The results indicate that ChatGPT performs better at lower acuity levels (ESIs 3 and 4). Gemini is slightly more effective at higher acuity levels (ESIs 1 and 2), showing its potential for identifying critical cases. Both models performed better when used as decision support for healthcare providers than when used for self-triage, highlighting the importance of structured input and prompt design. Further research, particularly with real-world EHR data, is essential to assess the reliability of these tools across diverse patient cases and optimize their use for safe and accurate self-triage.\u003c/p\u003e \u003cp\u003ePrevious studies, including Fraser et al. (2023) and Williams et al. (2024), have shown that AI can effectively triage patients, mainly through general symptom checkers or diagnostic tools. Our study adds to this research by uniquely comparing ChatGPT and Gemini, two advanced models, specifically in headache triage\u0026mdash;a symptom with significant diagnostic variability. While existing research, such as that of Meral et al. (2024), indicates the potential of AI in emergency settings, this study further explores how ChatGPT and Gemini handle the complexities of headache presentations, highlighting strengths and limitations.\u003c/p\u003e \u003cp\u003eThis study highlights the potential of large language models, such as ChatGPT and Gemini, as valuable aids in emergency department (ED) triage. By enhancing the precision of triage decisions, these tools could help healthcare providers allocate resources more effectively and allow patients to make informed self-triage decisions, potentially easing ED congestion. However, further refinement is essential to ensure patient safety and minimize the risks of inaccurate self-assessments.\u003c/p\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003eLimitations\u003c/h2\u003e \u003cp\u003eThis study has several limitations. This study focused only on headache cases, which may reflect a partial scope of conditions encountered in an ED. The dataset was sourced from a single EHR system, which may limit the generalizability of the results across other patient demographics and healthcare settings. Furthermore, as AI models rely on structured data, their effectiveness with complex or unstructured inputs, such as free-text symptom descriptions, is still being determined and may vary across cultural and linguistic contexts.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003eFuture Work\u003c/h2\u003e \u003cp\u003eFuture research should validate these findings with a more extensive and diverse dataset encompassing a broader range of conditions. Integrating LLMs such as ChatGPT and Gemini into existing clinical decision support systems could enable more comprehensive patient data analysis. Further advancements could focus on improving these models' ability to interpret unstructured data, such as free-text symptom descriptions, and exploring hybrid systems that leverage the strengths of both models for enhanced triage performance.\u003c/p\u003e \u003c/div\u003e"},{"header":"Conclusion","content":"\u003cp\u003eIn conclusion, this study highlights the potential of ChatGPT and Gemini as supportive tools for clinical and self-triage. While promising, these models need further refinement to ensure reliable and accurate performance across diverse patient scenarios. Optimizing these technologies could significantly enhance patient care, reduce ED overcrowding, and empower individuals to make informed health decisions.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003eEthics approval and consent to participate:\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e\n\u003cp\u003eClinical trial number:\u0026nbsp;\u003c/p\u003e\n\u003cp\u003enot applicable\u003c/p\u003e\n\u003cp\u003eConsent for publication:\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e\n\u003cp\u003eAvailability of data and materials:\u0026nbsp;\u003c/p\u003e\n\u003cp\u003ehttps://physionet.org/content/mimic-iv-ed/2.2/\u003c/p\u003e\n\u003cp\u003eCompeting interests:\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe authors declare that they have no competing interests.\u003c/p\u003e\n\u003cp\u003eFunding:\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eNo funding was received to assist with the preparation of this manuscript.\u003c/p\u003e\n\u003cp\u003eAuthors\u0026apos; contributions:\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eH.S. analyzed the output results, wrote the Results and Discussion sections, and edited the final manuscript. F.L. designed and created the prompts, wrote the Introduction, and conducted the literature review. S.R.N.K. supervised the entire project and contributed to the study design and methodology. M.T. designed and oversaw the patient triage process. M.R. extracted information from ChatGPT and Gemini using the designed prompts. All authors reviewed and approved the final manuscript.\u003c/p\u003e\n\u003cp\u003eAcknowledgements:\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eChmielewski N, Moretz J. ESI Triage Distribution in U.S. Emergency Departments, \u003cem\u003eAdv. Emerg. Nurs. J.\u003c/em\u003e, vol. 44, no. 1, pp. 46\u0026ndash;53, Mar. 2022, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1097/TME.0000000000000390\u003c/span\u003e\u003cspan address=\"10.1097/TME.0000000000000390\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMorley C, Unwin M, Peterson GM, Stankovich J, Kinsman L. Emergency department crowding: a systematic review of causes, consequences and solutions. PLoS ONE. 2018;13(8):e0203316.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAdriaenssens J, De Gucht V, Maes S. Determinants and prevalence of burnout in emergency nurses: a systematic review of 25 years of research. Int J Nurs Stud. 2015;52(2):649\u0026ndash;61.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKulstad EB, Sikka R, Sweis RT, Kelley KM, Rzechula KH, editors. ED overcrowding is associated with an increased frequency of medication errors, \u003cem\u003eAm. J. Emerg. Med.\u003c/em\u003e, vol. 28, no. 3, pp. 304\u0026ndash;309, 2010.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYancey CC, O\u0026rsquo;Rourke MC. Emergency department triage, 2020.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSalman OH, Taha Z, Alsabah MQ, Hussein YS, Mohammed AS, Aal-Nouman M. A review on utilizing machine learning technology in the fields of electronic emergency triage and patient priority systems in telemedicine: Coherent taxonomy, motivations, open research challenges and recommendations for intelligent future work. Comput Methods Programs Biomed. 2021;209:106357.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDickson SJ, Dewar C, Richardson A, Hunter A, Searle S, Hodgson LE. Agreement and validity of electronic patient self-triage (eTriage) with nurse triage in two UK emergency departments: a retrospective study. Eur J Emerg Med. 2022;29(1):49\u0026ndash;55.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAugusto Duenhas T. Accorsi Outcome After Self-Triage App Referral in Urgent Direct-to-Consumer Telemedicine Encounter. Telemed E-Health, 2024.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eIlicki J. Challenges in evaluating the accuracy of AI-containing digital triage systems: A systematic review. PLoS ONE. 2022;17(12):e0279636.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWaghmare C. Introduction to ChatGPT. in Unleashing The Power of ChatGPT: A Real World Business Applications. Springer; 2023. pp. 1\u0026ndash;26.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eIslam R, Ahmed I. Gemini-the most powerful LLM: Myth or Truth, presented at the 2024 5th Information Communication Technologies Conference (ICTC), IEEE, 2024, pp. 303\u0026ndash;308.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChiang C, et al. A large language model\u0026ndash;based generative natural language processing framework fine-tuned on clinical notes accurately extracts headache frequency from electronic health records. Headache J Head Face Pain. 2024;64(4):400\u0026ndash;9.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFisher L. Assessment of patients presenting with headache. Innovait, 2011.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLacerenza MR, Schoss F, Grazzi L. The multimodal treatment in headaches. J Headache Pain. 2015;16(Suppl 1):A47.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDe Angelis L, et al. ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health. Front Public Health. 2023;11:1166120.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eThirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med. 2023;29(8):1930\u0026ndash;40.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRashid K et al. Accuracy of emergency room triage using emergency severity index (esi): Independent predictor of under and over triage. Cureus, 13, 12, 2021.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCeney A, Tolond S, Glowinski A, Marks B, Swift S, Palser T. Accuracy of online symptom checkers and the potential impact on service utilization. PLoS ONE. 2021;16(7):e0254088.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMeral G, Ateş S, G\u0026uuml;nay S, \u0026Ouml;zt\u0026uuml;rk A, Kuşdoğan M. Comparative analysis of ChatGPT, Gemini and emergency medicine specialist in ESI triage assessment, \u003cem\u003eAm. J. Emerg. Med.\u003c/em\u003e, vol. 81, pp. 146\u0026ndash;150, Jul. 2024, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.ajem.2024.05.001\u003c/span\u003e\u003cspan address=\"10.1016/j.ajem.2024.05.001\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWilliams CYK, et al. Use of a Large Language Model to Assess Clinical Acuity of Adults in the Emergency Department. JAMA Netw Open. May 2024;7(5):e248895. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1001/jamanetworkopen.2024.8895\u003c/span\u003e\u003cspan address=\"10.1001/jamanetworkopen.2024.8895\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFraser H, Crossland D, Bacher I, Ranney M, Madsen T, Hilliard R. Comparison of Diagnostic and Triage Accuracy of Ada Health and WebMD Symptom Checkers, ChatGPT, and Physicians for Patients in an Emergency Department: Clinical Data Analysis Study, \u003cem\u003eJMIR MHealth UHealth\u003c/em\u003e, vol. 11, p. e49995, Oct. 2023, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.2196/49995\u003c/span\u003e\u003cspan address=\"10.2196/49995\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJohnson A, Bulgarelli L, Pollard T, Celi LA, Mark R, Horng S. \u0026lsquo;MIMIC-IV-ED\u0026rsquo; (version 2.2). PhysioNet (2023). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.13026/ntk-km72\u003c/span\u003e\u003cspan address=\"10.13026/ntk-km72\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. 2023.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"ChatGPT, Gemini, Emergency Severity Index, Triage, Large Language Models, MIMIC-IV ED","lastPublishedDoi":"10.21203/rs.3.rs-5429142/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5429142/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eObjective\u003c/h2\u003e \u003cp\u003eThis study evaluated the performance of two advanced large language models (LLMs), ChatGPT and Gemini, in supporting triage decisions for headache patients in emergency settings via the Emergency Severity Index (ESI) from both patient self-triage and healthcare provider perspectives.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eData, including 500 records of patients presenting with headache complaints, were obtained from the MIMIC-IV-ED database. Two distinct prompt types were created: one for self-triage to assist patients in assessing their care needs on the basis of symptom descriptions and another for healthcare providers to determine ESI levels. Each model's output was compared to actual ESI levels via precision, recall, and F1 scores to measure performance.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eChatGPT achieved greater accuracy at lower acuity levels (ESIs 3 and 4), accurately identifying patients who did not require urgent care. Gemini demonstrated improved performance at higher acuity levels (ESIs 1 and 2), indicating its ability to recognize critical cases effectively. Both models showed stronger performance with healthcare provider prompts than with self-triage prompts, underscoring the importance of structured input for accurate triage assessments. This variation highlights the need to refine self-triage prompts to ensure safe and precise use.\u003c/p\u003e\u003ch2\u003eConclusion\u003c/h2\u003e \u003cp\u003eChatGPT and Gemini show promise as decision-support tools for ED triage, particularly for assisting healthcare providers in prioritizing cases on the basis of acuity. However, further refinement is needed to increase accuracy in self-triage scenarios. Future studies should validate these findings across a broader dataset and explore the integration of LLMs into clinical decision support systems to strengthen triage reliability and effectiveness.\u003c/p\u003e","manuscriptTitle":"ESI Triage Level Assignment for Headache Patients: Comparative Analysis of ChatGPT and Gemini Performance for Supporting Care Provider Decisions and Self-triage","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-12-18 06:45:28","doi":"10.21203/rs.3.rs-5429142/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"aae7139e-f24a-4aaf-b4d9-7c17715efe86","owner":[],"postedDate":"December 18th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-04-13T14:27:39+00:00","versionOfRecord":[],"versionCreatedAt":"2024-12-18 06:45:28","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5429142","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5429142","identity":"rs-5429142","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}