Evaluating ChatGPT-4o's Reasoning in Ocular Injury Using NEISS Data

Evaluating ChatGPT-4o's Reasoning in Ocular Injury Using NEISS Data | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Evaluating ChatGPT-4o's Reasoning in Ocular Injury Using NEISS Data Ezanna Mesfin, Matthew Heider, Mohamed Heiba, Nicholas Stratigakis, and 3 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7359662/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 4 You are reading this latest preprint version Abstract Purpose: This study evaluates ChatGPT-4o’s reasoning process in triaging ocular injury cases in classification data from the National Electronic Injury Surveillance System (NEISS). The goal is to analyze the reasoning used by the model and its accuracy. Design: Retrospective cohort study. Participants: 2 145 cases of ocular injuries were randomly selected from the NEISS database under an IRB exempt protocol. Methods: 2 145 ocular injury cases were randomly sampled, and evenly categorized into three triage levels: emergent, urgent, and routine. ChatGPT-4o was tasked to assign triage levels recommend interventions, and provide reasoning for each decision. The model’s reasoning was categorized into four types: 1) "Blunt trauma may cause delayed complications like retinal detachment," 2) "Chemical injuries can cause severe ocular damage and require immediate attention," 3) "Foreign body cases can cause vision-threatening complications if not treated urgently," and 4) "The injury appears non-urgent based on provided details." We evaluated the frequency and accuracy of each reasoning type and analyzed their association with triage categories. Main Outcome Measures: Accuracy of ChatGPT-4o in correctly triaging emergent, urgent, and routine cases. Results: ChatGPT-4o correctly categorized 60% of all cases. The model frequently defaulted to routine classification, contributing to under-recognition of urgent cases. Conclusion: ChatGPT-4o shows potential for triaging ocular injuries, especially in identifying emergent cases. However, it struggles with nuanced reasoning. Health sciences/Medical research/Outcomes research Health sciences/Health care/Diagnosis Ocular trauma Triage classification ChatGPT-4o Large language models emergency ophthalmology clinical decision support artificial intelligence NEISS database Figures Figure 1 Figure 2 Introduction Triage for ocular injuries must be swift and accurate in emergencies. Neglecting treatment can result in resource depletion and visual deterioration. Annually, 2 million individuals present to emergency departments (EDs) in the United States with ocular symptoms, encompassing simple foreign bodies to serious ocular traumas such as chemical burns or ruptured globes 1 . Triage systems have traditionally relied on clinical judgment or basic rule-based protocols, which may be inadequate in intricate scenarios. This circumstance could be entirely transformed by artificial intelligence, particularly through Large language models such as ChatGPT. Generative AI systems can emulate human language and reasoning capabilities by assimilating huge quantities of text and code 2 . Among these, ChatGPT-4 and its multimodal successor, ChatGPT-4o, have demonstrated impressive accuracy in medical domains, including diagnostic support in ophthalmology 3 , 4 . Recent evaluations of ChatGPT-based tools show diagnostic accuracy rates as high as 92% in ophthalmologic consultations and subspecialty contexts, including corneal, retinal, and glaucoma cases 4 , 5 . However, several authors have cautioned against the blind deployment of LLMs in clinical care. Shen et al. (2023) describe these models as "double-edged swords," noting their ability to mirror expert reasoning while simultaneously introducing subtle errors that may mislead end-users 6 . Particularly in triage, where urgency differentiation impacts care timelines, the internal logic guiding AI decisions warrants thorough analysis. This study builds upon earlier efforts by evaluating the specific reasoning strategies employed by ChatGPT-4o in triaging ocular injuries from the NEISS dataset. We aim to classify the model's decision-making logic, quantify its accuracy by reasoning type, and identify patterns that could inform safer, more interpretable deployment in clinical settings. The current study aims to dissect the model's reasoning processes, identify thematic patterns, and assess its clinical triage utility more precisely. Methods Data Collection: The National Electronic Injury Surveillance System (NEISS) database provided the dataset, initially encompassing 102 639 ED visits for head, eye, and face trauma recorded in 2023. Filtering specifically for ocular injuries resulted in a dataset of 28,455 cases. Data Subset: From these, 2 145 ocular injury cases were randomly selected, evenly divided into three categories: emergent, urgent, and routine (715 each). All diagnostic and triage information was removed to avoid influencing ChatGPT-4o's assessments. Evaluation Procedure Each case presented to ChatGPT-4o included detailed narratives describing patient demographics, type and severity of ocular injury, and brief clinical history. Example: CPSC Case Number: Age: 42 Gender: Male Diagnosis: Chemical burn injury Narrative: “42YOM working with chemicals presents with irritation after endorsing direct eye contact with chemical.” ChatGPT-4o was tasked with: 1) Assign a triage category: Emergent (immediate attention), Urgent (24-48 hours), or Routine (scheduled appointment). 2) Recommend an intervention: Proposing next steps or treatment plans. 3) Provide reasoning: explaining its triage decision and intervention based on case details. The AI system then classified each case into one of the three triage categories, recommended appropriate interventions, and explicitly outlined its reasoning processes. Reasoning Analysis: ChatGPT-4o's reasoning was categorized into four distinct types: 1. Blunt trauma and delayed complication risks 2. Immediate severe chemical injury risks 3. Vision-threatening risks associated with foreign bodies 4. Non-urgent based on clinical presentation Statistical Analysis: Analytical methods included evaluating accuracy and precision. Additionally, Chi-square tests analyzed the association between reasoning categories and triage classifications. Results The “non-urgent” reasoning category dominated ChatGPT-4o’s assessments, used in 1 1 395 cases, achieving moderate accuracy (51.1%). "Foreign body" reasoning was employed in 741 cases with higher accuracy (76.7%) but struggled in correctly categorizing urgent injuries. The “chemical injury” reasoning, though seldom used (3 cases), showed flawless accuracy (100%). The "blunt trauma" reasoning, applied minimally (6 cases), yielded moderate accuracy (50%). Overall accuracy across all cases was 60%, revealing a strong inclination towards routine classifications. With a 95% confidence interval, there was a strong and statistically significant association between reasoning category and triage classification (χ² = 3 569.23, p < 0.0001), confirming the model’s reliance on distinct patterns of reasoning. (Figure 1) Thematic Analysis Three primary reasoning themes were identified in ChatGPT-4o’s reasoning strategies: Recognition of High-Risk Mechanisms (741 cases): Utilized terms like "vision-threatening" and "immediate risk." All correctly classified as emergent, emphasizing ChatGPT-4o's strength in high-risk case identification. (Table 1) (Table 2) (Table 3) Minimization of Symptoms (1,395 cases): Used terms like "non-urgent" or "minor," consistently applied to routine cases, indicating systematic routine case classification. Other/Unclassified (9 cases): Included unclear reasoning like immediate chemical damage, resulting in mixed classifications. Chi-square analysis confirmed a strong association between these themes and predicted triage categories (Chi-square = 3569.23, p < 0.0001), highlighting clear decision boundaries in ChatGPT-4o’s reasoning, but also emphasizing its challenges in nuanced "urgent" case differentiation. Stratified Word Frequency Analysis: (Figure 2) To supplement reasoning category accuracy, a word frequency analysis was performed on all reasoning statements. The most common terms used were analyzed and stratified by triage category (emergent, urgent, routine): Top 10 Most Common Words Overall: "appears" (1 126 times) "non-urgent" (1 017 times) "injury" (978 times) "routine" (922 times) "foreign body" (741 times) "requires" (699 times) "urgent" (462 times) "vision-threatening" (344 times) "blunt trauma" (117 times) "chemical" (98 times) Distribution by Triage Category: Terms like "appears" and "non-urgent" were used in over 89% of routine cases, indicating a strong conservative bias in low-risk assignments. "Foreign body" and "vision-threatening" were primarily found in emergent classifications, occurring in over 91% of such cases. Interestingly, "urgent" appeared inconsistently—present in only 31% of correctly identified urgent cases but also found in 24% of routine cases, suggesting ambiguity in mid-tier decision-making. These data demonstrate that ChatGPT-4o’s lexical choices mirror its triage tendencies, reinforcing the theme that the model often dichotomizes cases into either low or high severity while underrepresenting intermediate urgency. This lexical skew may reflect limitations in the training corpus or decision thresholding mechanisms embedded in the model’s probabilistic architecture. Discussion Clinical Implications: This study indicates that ChatGPT-4o can anticipate critical scenarios such as chemical injuries and penetrating wounds, aligning with emergency responses. Previous research indicates that LLMs perform effectively in high acuity scenarios within binary classification tasks 7 , 8 . In high-volume or resource-limited emergency contexts, an AI assistant that reliably identifies important scenarios could alleviate stress and optimize specialist referrals. Model Limitations and Interpretability: The model underperformed in differentiating urgent cases from routine ones, mirroring concerns from recent ophthalmology studies where ChatGPT failed to adjust for clinical nuance 7 , 10 . Its reasoning, often prefaced with phrases like “appears non-urgent,” illustrates a conservative bias, possibly influenced by a probabilistic tendency to avoid overcalling severity. Ramponi (2022) explains that LLMs function through token prediction based on contextual probability, not pathophysiological understanding—limiting clinical generalization 2 . Insufficient training inputs may result in diminished reasoning diversity. The nuanced clinical distinctions in urgent ophthalmic treatment may be inadequately captured by AI systems trained on online textual data. Research articles and preprints indicate that improving large language models using specialized or multimodal clinical datasets, including slit lamp pictures and structured examination results, could improve performance 9 , 10 . Ethical and Implementation Considerations: Ethical issues arise regarding the use of ChatGPT in triage. Despite the efficacy of LLMs as diagnosticians underscore the necessity for supervision and safeguarding 11 , 12 . The specifics of malpractice responsibility remain ambiguous, especially if triage judgments lead to adverse outcomes. To ensure safety and accountability, experts recommend hybrid approaches wherein human clinicians verify AI outcomes 13 . 11Thorough prospective testing and continuous monitoring of clinical model performance are essential for deployment. Conclusion ChatGPT-4o effectively identifies emergent ocular injuries but exhibits systematic misclassification of urgent cases, favoring a binary emergent vs. routine framework. Enhancing training data, refining reasoning strategies, and implementing clinician oversight will be essential for safe and effective clinical integration. These results substantiate concerns about the constraints of LLMs in critical scenarios. While ChatGPT-4o provides rational explanations and demonstrates technical proficiency, it is deficient in the contextual depth required for intermediate clinical assessments 3 . Shen et al. (2023) characterize LLMs as "double-edged swords" since their opaque reasoning may obscure safety issues while enhancing diagnostic efficacy 13 . This warning is substantiated by our analysis, which reveals that even plausible output may be based on tenuous or vague thinking. Technically, the limitations in ChatGPT-4o’s triage accuracy stem from its architecture as a transformer-based probabilistic model trained on massive textual corpora 2 . It lacks domain-specific memory or feedback loops and thus defaults to probabilistic generalizations. Without reinforcement from structured ophthalmic cases, the model struggles to assign intermediate urgency—an essential capability in clinical settings where patient outcomes hinge on timing 13 . Moreover, this bias toward routine classifications may reflect implicit limitations in the training data. As Delsoz et al. (2023) found in a related study on glaucoma diagnostics, ChatGPT’s diagnostic suggestions significantly improved when contextualized with detailed clinical inputs or paired with image data 4 . This implies a need for multimodal approaches that integrate clinical imaging, structured history, and AI reasoning—possibly through hybrid systems 10 . Implementing feedback mechanisms and role-specific fine-tuning may also enhance ChatGPT’s sensitivity to gray-zone cases. Ethically, this reinforces the imperative to view LLMs as assistive tools rather than standalone solutions. As Chow et al. (2023) argue, AI chatbots have transformative potential but must operate under clearly defined clinical supervision 11 . As such, ophthalmology educators and emergency department administrators should establish usage guidelines, focusing on AI-human collaboration rather than delegation. While ChatGPT-4o’s current iteration exhibits strong pattern recognition for extreme presentations, its utility remains limited without enhancements in reasoning complexity, interpretability, and integration within real-time clinical systems. This paper not only quantifies current capabilities but also charts a course for responsible AI implementation in ophthalmic triage. Strengths ChatGPT-4o showed effectiveness in accurately identifying emergent ocular injuries, demonstrating its potential utility in rapid triage, particularly in emergency settings where swift action is essential. Its reasoning centered around terms such as “vision-threatening” and “immediate risk,” to correctly identify most of these critical cases. Limitations The AI exhibited substantial inaccuracies in classifying urgent cases, relying heavily on simplistic, binary logic, which could compromise patient safety by delaying crucial interventions. Recommendations To enhance clinical utility, ChatGPT-4o requires retraining on balanced datasets emphasizing intermediate severity (urgent) cases. Refining its reasoning specificity and adopting hybrid AI-human models can further improve accuracy and safety. Declarations Acknowledgements and Financial Disclosure: a. Funding/Support: This study did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. b. Financial Disclosures: The authors have no financial relationships relevant to this article to disclose. Credit authorship contribution statement: Ezanna Mesfin: Writing – review & editing, Writing – original draft, Formal analysis, Data analysis. Matthew Heider: Writing – reviewing & editing, Formal analysis, Data analysis. Mohamed Heiba: Writing – review & editing, Formal analysis, Data analysis and curation. Ross Stuber: Writing – review & editing. Inci Dersu – review & editing. John Laudi – review & editing. Declaration of competing interest: Ezanna Mesfin, Matthew Heider, Mohamed Heiba, Ross Stuber, Inci Dersu, and John Laudi report no conflicts of interest. References Channa R, Zafar SN, Canner JK, et al. Epidemiology of Eye-Related Emergency Department Visits. JAMA Ophthalmol. 2016;134(3):312–319. doi: 10.1001/jamaophthalmol.2015.5778 Ramponi M. How ChatGPT actually works. AssemblyAI. December 23, 2022. https://www.assemblyai.com/blog/how-chatgpt-actually-works/ Waisberg E, Ong J, Masalkhi M, et al. GPT-4: a new era of artificial intelligence in medicine. Ir J Med Sci. 2023;192(6):3197–3200. doi: 10.1007/s11845-023-03377-8 Delsoz M, Raja H, Madadi Y, et al. The use of ChatGPT to assist in diagnosing glaucoma based on clinical case reports. Ophthalmol Ther. 2023;12(6):3121–3132. doi: 10.1007/s40123-023-00805-x Momenaei B, Wakabayashi T, Shahlaee A, et al. Appropriateness and readability of ChatGPT-4-generated responses for surgical treatment of retinal diseases. Ophthalmol Retina. 2023;7(10):862–868. doi: 10.1016/j.oret.2023.05.022 Shen Y, Heacock L, Elias J, et al. ChatGPT and other large language models are double-edged swords. npj Digit Med. 2023;6(1):61. doi: 10.1038/s41746-023-00886- Masalkhi M, Ong J, Waisberg E, et al. ChatGPT to document ocular infectious diseases. Eye (Lond). Published online November 15, 2023. doi: 10.1038/s41433-023-02823-2 Ming S, Yao X, Guo X, et al. Performance of ChatGPT in ophthalmic registration and clinical diagnosis: cross-sectional study. J Med Internet Res. 2024;26:e60226. doi: 10.2196/60226 Madadi Y, Delsoz M, Lao PA, et al. ChatGPT assisting diagnosis of neuro-ophthalmology diseases based on case reports. Preprint. medRxiv. Published 2023 Sep 14. doi: 10.1101/2023.09.13.23295508 Peng Z, Ma R, Zhang Y, et al. Development and evaluation of multimodal AI for diagnosis and triage of ophthalmic diseases using ChatGPT and anterior segment images: protocol for a two-stage cross-sectional study. Front Artif Intell. 2023;6:1323924. doi: 10.3389/frai.2023.1323924 Chow JCL, Sanders L, Li K. Impact of ChatGPT on medical chatbots as a disruptive technology. Front Artif Intell. 2023;6:1166014. doi: 10.3389/frai.2023.1166014 Khan RA, Jawaid M, Khan AR, Sajjad M. ChatGPT – reshaping medical education and clinical management. Pak J Med Sci. 2023;39(2):605–607. doi: 10.12669/pjms.39.2.7653 Lyons RJ, Arepalli SR, Fromal O, Choi JD, Jain N. Artificial intelligence chatbot performance in triage of ophthalmic conditions. Can J Ophthalmol . 2024;59(4):e301-e308. doi: 10.1016/j.jcjo.2023.07.016 Tables Tables 1 to 3 are available in the Supplementary Files section. Additional Declarations There is no conflict of interest Supplementary Files Table1.docx Table 1 Table2.docx Table 2 Table3.docx Table 3 Cite Share Download PDF Status: Under Review Version 1 posted Reviewers invited by journal 02 Sep, 2025 Editor assigned by journal 21 Aug, 2025 Submission checks completed at journal 13 Aug, 2025 First submitted to journal 12 Aug, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7359662","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":508897448,"identity":"5dad0425-4748-4ec8-bd85-23a40920d1cd","order_by":0,"name":"Ezanna Mesfin","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA8UlEQVRIiWNgGAWjYFAD5oMNDB+ANBs70VrYEhsYZ4BoZuK1JDAw84CtI6CQn//ws8cFNdvk5NuY2x7b/Nomz8fMwPjhYw5uLZINx8yNZxy7bWxwjLHdOLfvtmEbMwOz5MxtuLUYHGwwk+Zhu524Qb6xTTq35zYjUAsbMy8eLfaH2b9J8/y7XT+/jbFN2rLntj1BLQZsPGbSvG23ExiOAbUw/LidSFCLxBmeMmleoBc2ALVI9jbcTm5jZmzG6xf+/uPbpHm+3ZaXb2N/JvHjz23b+e3NBz98xKMFFTC2gckGYtWDwB9SFI+CUTAKRsFIAQDj30vqnk3SdQAAAABJRU5ErkJggg==","orcid":"","institution":"SUNY Downstate COM","correspondingAuthor":true,"prefix":"","firstName":"Ezanna","middleName":"","lastName":"Mesfin","suffix":""},{"id":508897449,"identity":"2ef7b3b7-59bb-4f5f-9a76-ee6a4d60f7f0","order_by":1,"name":"Matthew Heider","email":"","orcid":"","institution":"SUNY Downstate","correspondingAuthor":false,"prefix":"","firstName":"Matthew","middleName":"","lastName":"Heider","suffix":""},{"id":508897450,"identity":"dac62fbe-aac1-4ef9-87fb-869ac8151f4f","order_by":2,"name":"Mohamed Heiba","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Mohamed","middleName":"","lastName":"Heiba","suffix":""},{"id":508897451,"identity":"9165a988-ef01-4d31-8cb4-fab9301bf5a1","order_by":3,"name":"Nicholas Stratigakis","email":"","orcid":"","institution":"SUNY Downstate COM","correspondingAuthor":false,"prefix":"","firstName":"Nicholas","middleName":"","lastName":"Stratigakis","suffix":""},{"id":508897452,"identity":"60521bcd-d533-48fd-bbb6-8c780f3732eb","order_by":4,"name":"Ross Stuber","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Ross","middleName":"","lastName":"Stuber","suffix":""},{"id":508897453,"identity":"7ff7af06-45f6-455d-b75d-d53d2c4565b9","order_by":5,"name":"John Laudi","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"John","middleName":"","lastName":"Laudi","suffix":""},{"id":508897454,"identity":"3b591f6e-b96f-4c48-ad6a-4059b5443293","order_by":6,"name":"Inci Dersu","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Inci","middleName":"","lastName":"Dersu","suffix":""}],"badges":[],"createdAt":"2025-08-13 01:06:21","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7359662/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7359662/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":90899718,"identity":"4b3782e5-3568-499f-9196-11bc0db692a9","added_by":"auto","created_at":"2025-09-09 12:03:06","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":54589,"visible":true,"origin":"","legend":"\u003cp\u003eFrequency and Accuracy of ChatGPT-4o’s Reasoning Categories\u003c/p\u003e","description":"","filename":"Picture1.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7359662/v1/c0f210096a48f2b55228f05a.jpg"},{"id":90899719,"identity":"5376c357-2eb5-43a6-98f5-60472d295e41","added_by":"auto","created_at":"2025-09-09 12:03:06","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":142680,"visible":true,"origin":"","legend":"\u003cp\u003eWord Cloud\u003c/p\u003e","description":"","filename":"Picture2.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7359662/v1/4267282067803d5ba2097374.jpg"},{"id":90901111,"identity":"7d664aa3-1d00-47ed-a87f-23a9d0d77cd7","added_by":"auto","created_at":"2025-09-09 12:19:11","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":729690,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7359662/v1/34ae8efb-00df-462b-a644-9d26122f2806.pdf"},{"id":90898822,"identity":"ab94d6f3-ff19-4b61-80a2-75e919790ffb","added_by":"auto","created_at":"2025-09-09 11:55:06","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":48522,"visible":true,"origin":"","legend":"Table 1","description":"","filename":"Table1.docx","url":"https://assets-eu.researchsquare.com/files/rs-7359662/v1/561167541732f79cf7773201.docx"},{"id":90899717,"identity":"5ee6e6a2-53fd-402c-9cbe-f8303ef08c92","added_by":"auto","created_at":"2025-09-09 12:03:06","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":53460,"visible":true,"origin":"","legend":"Table 2","description":"","filename":"Table2.docx","url":"https://assets-eu.researchsquare.com/files/rs-7359662/v1/d33e7fb0ebf681a0b9f6c417.docx"},{"id":90898829,"identity":"6afa9c79-cd90-4577-9d65-735d09159cf2","added_by":"auto","created_at":"2025-09-09 11:55:06","extension":"docx","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":120142,"visible":true,"origin":"","legend":"Table 3","description":"","filename":"Table3.docx","url":"https://assets-eu.researchsquare.com/files/rs-7359662/v1/562bc4201032a43de503344d.docx"}],"financialInterests":"There is no conflict of interest","formattedTitle":"Evaluating ChatGPT-4o's Reasoning in Ocular Injury Using NEISS Data","fulltext":[{"header":"Introduction","content":"\u003cp\u003eTriage for ocular injuries must be swift and accurate in emergencies. Neglecting treatment can result in resource depletion and visual deterioration. Annually, 2\u0026nbsp;million individuals present to emergency departments (EDs) in the United States with ocular symptoms, encompassing simple foreign bodies to serious ocular traumas such as chemical burns or ruptured globes \u003csup\u003e\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\u003cp\u003eTriage systems have traditionally relied on clinical judgment or basic rule-based protocols, which may be inadequate in intricate scenarios. This circumstance could be entirely transformed by artificial intelligence, particularly through Large language models such as ChatGPT. Generative AI systems can emulate human language and reasoning capabilities by assimilating huge quantities of text and code \u003csup\u003e\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e. Among these, ChatGPT-4 and its multimodal successor, ChatGPT-4o, have demonstrated impressive accuracy in medical domains, including diagnostic support in ophthalmology \u003csup\u003e\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u003c/sup\u003e .\u003c/p\u003e\u003cp\u003eRecent evaluations of ChatGPT-based tools show diagnostic accuracy rates as high as 92% in ophthalmologic consultations and subspecialty contexts, including corneal, retinal, and glaucoma cases \u003csup\u003e\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u003c/sup\u003e. However, several authors have cautioned against the blind deployment of LLMs in clinical care. Shen et al. (2023) describe these models as \"double-edged swords,\" noting their ability to mirror expert reasoning while simultaneously introducing subtle errors that may mislead end-users \u003csup\u003e\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u003c/sup\u003e. Particularly in triage, where urgency differentiation impacts care timelines, the internal logic guiding AI decisions warrants thorough analysis.\u003c/p\u003e\u003cp\u003eThis study builds upon earlier efforts by evaluating the specific reasoning strategies employed by ChatGPT-4o in triaging ocular injuries from the NEISS dataset. We aim to classify the model's decision-making logic, quantify its accuracy by reasoning type, and identify patterns that could inform safer, more interpretable deployment in clinical settings. The current study aims to dissect the model's reasoning processes, identify thematic patterns, and assess its clinical triage utility more precisely.\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003e\u003cu\u003eData Collection:\u003c/u\u003e\u003c/p\u003e\n\u003cp\u003eThe National Electronic Injury Surveillance System (NEISS) database provided the dataset, initially encompassing 102 639 ED visits for head, eye, and face trauma recorded in 2023. Filtering specifically for ocular injuries resulted in a dataset of 28,455 cases.\u003c/p\u003e\n\u003cp\u003e\u003cu\u003eData Subset:\u003c/u\u003e\u003c/p\u003e\n\u003cp\u003eFrom these, 2 145 ocular injury cases were randomly selected, evenly divided into three categories: emergent, urgent, and routine (715 each). All diagnostic and triage information was removed to avoid influencing ChatGPT-4o\u0026apos;s assessments.\u003c/p\u003e\n\u003cp\u003e\u003cu\u003eEvaluation Procedure\u003c/u\u003e\u003c/p\u003e\n\u003cp\u003eEach case presented to ChatGPT-4o included detailed narratives describing patient demographics, type and severity of ocular injury, and brief clinical history.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eExample:\u003c/p\u003e\n\u003cp\u003eCPSC Case Number:\u003c/p\u003e\n\u003cp\u003eAge: 42\u003c/p\u003e\n\u003cp\u003eGender: Male\u003c/p\u003e\n\u003cp\u003eDiagnosis: Chemical burn injury\u003c/p\u003e\n\u003cp\u003eNarrative: \u0026ldquo;42YOM working with chemicals presents with irritation after endorsing direct eye contact with chemical.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eChatGPT-4o was tasked with:\u003c/p\u003e\n\u003cp\u003e1) Assign a triage category: Emergent (immediate attention), Urgent (24-48 hours), or Routine (scheduled appointment). 2) Recommend an intervention: Proposing next steps or treatment plans. 3) Provide reasoning: explaining its triage decision and intervention based on case details.\u003c/p\u003e\n\u003cp\u003eThe AI system then classified each case into one of the three triage categories, recommended appropriate interventions, and explicitly outlined its reasoning processes.\u003c/p\u003e\n\u003cp\u003e\u003cu\u003eReasoning Analysis:\u003c/u\u003e\u003c/p\u003e\n\u003cp\u003eChatGPT-4o\u0026apos;s reasoning was categorized into four distinct types:\u003c/p\u003e\n\u003cp\u003e1. Blunt trauma and delayed complication risks\u003c/p\u003e\n\u003cp\u003e2. Immediate severe chemical injury risks\u003c/p\u003e\n\u003cp\u003e3. Vision-threatening risks associated with foreign bodies\u003c/p\u003e\n\u003cp\u003e4. Non-urgent based on clinical presentation\u003c/p\u003e\n\u003cp\u003e\u003cu\u003eStatistical Analysis:\u003c/u\u003e\u003c/p\u003e\n\u003cp\u003eAnalytical methods included evaluating accuracy and precision. Additionally, Chi-square tests analyzed the association between reasoning categories and triage classifications.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eThe “non-urgent” reasoning category dominated ChatGPT-4o’s assessments, used in 1 1 395 cases, achieving moderate accuracy (51.1%). \"Foreign body\" reasoning was employed in 741 cases with higher accuracy (76.7%) but struggled in correctly categorizing urgent injuries. The “chemical injury” reasoning, though seldom used (3 cases), showed flawless accuracy (100%). The \"blunt trauma\" reasoning, applied minimally (6 cases), yielded moderate accuracy (50%). Overall accuracy across all cases was 60%, revealing a strong inclination towards routine classifications.\u0026nbsp;With a 95% confidence interval, there was a strong and statistically significant association between reasoning category and triage classification (χ² = 3 569.23, p \u0026lt; 0.0001), confirming the model’s reliance on distinct patterns of reasoning.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e(Figure 1)\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cu\u003eThematic Analysis\u003c/u\u003e\u003c/p\u003e\n\u003cp\u003eThree primary reasoning themes were identified in ChatGPT-4o’s reasoning strategies:\u003c/p\u003e\n\u003cp\u003eRecognition of High-Risk Mechanisms (741 cases): Utilized terms like \"vision-threatening\" and \"immediate risk.\" All correctly classified as emergent, emphasizing ChatGPT-4o's strength in high-risk case identification.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e(Table 1)\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e(Table 2)\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e(Table 3)\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eMinimization of Symptoms (1,395 cases): Used terms like \"non-urgent\" or \"minor,\" consistently applied to routine cases, indicating systematic routine case classification.\u003c/p\u003e\n\u003cp\u003eOther/Unclassified (9 cases): Included unclear reasoning like immediate chemical damage, resulting in mixed classifications.\u003c/p\u003e\n\u003cp\u003eChi-square analysis confirmed a strong association between these themes and predicted triage categories (Chi-square = 3569.23, p \u0026lt; 0.0001), highlighting clear decision boundaries in ChatGPT-4o’s reasoning, but also emphasizing its challenges in nuanced \"urgent\" case differentiation. \u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eStratified Word Frequency Analysis:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e(Figure 2)\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTo supplement reasoning category accuracy, a word frequency analysis was performed on all reasoning statements. The most common terms used were analyzed and stratified by triage category (emergent, urgent, routine):\u003c/p\u003e\n\u003cul type=\"disc\"\u003e\n \u003cli\u003e\u003cstrong\u003eTop 10 Most Common Words Overall:\u003c/strong\u003e\n \u003cul type=\"circle\"\u003e\n \u003cli\u003e\"appears\" (1 126 times)\u003c/li\u003e\n \u003cli\u003e\"non-urgent\" (1 017 times)\u003c/li\u003e\n \u003cli\u003e\"injury\" (978 times)\u003c/li\u003e\n \u003cli\u003e\"routine\" (922 times)\u003c/li\u003e\n \u003cli\u003e\"foreign body\" (741 times)\u003c/li\u003e\n \u003cli\u003e\"requires\" (699 times)\u003c/li\u003e\n \u003cli\u003e\"urgent\" (462 times)\u003c/li\u003e\n \u003cli\u003e\"vision-threatening\" (344 times)\u003c/li\u003e\n \u003cli\u003e\"blunt trauma\" (117 times)\u003c/li\u003e\n \u003cli\u003e\"chemical\" (98 times)\u003c/li\u003e\n \u003c/ul\u003e\n \u003c/li\u003e\n \u003cli\u003e\u003cstrong\u003eDistribution by Triage Category:\u003c/strong\u003e\n \u003cul type=\"circle\"\u003e\n \u003cli\u003eTerms like \"appears\" and \"non-urgent\" were used in over 89% of routine cases, indicating a strong conservative bias in low-risk assignments.\u003c/li\u003e\n \u003cli\u003e\"Foreign body\" and \"vision-threatening\" were primarily found in emergent classifications, occurring in over 91% of such cases.\u003c/li\u003e\n \u003cli\u003eInterestingly, \"urgent\" appeared inconsistently—present in only 31% of correctly identified urgent cases but also found in 24% of routine cases, suggesting ambiguity in mid-tier decision-making.\u003c/li\u003e\n \u003c/ul\u003e\n \u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThese data demonstrate that ChatGPT-4o’s lexical choices mirror its triage tendencies, reinforcing the theme that the model often dichotomizes cases into either low or high severity while underrepresenting intermediate urgency. This lexical skew may reflect limitations in the training corpus or decision thresholding mechanisms embedded in the model’s probabilistic architecture.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cdiv id=\"Sec19\" class=\"Section2\"\u003e\u003ch2\u003eClinical Implications:\u003c/h2\u003e\u003cp\u003eThis study indicates that ChatGPT-4o can anticipate critical scenarios such as chemical injuries and penetrating wounds, aligning with emergency responses. Previous research indicates that LLMs perform effectively in high acuity scenarios within binary classification tasks \u003csup\u003e\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u003c/sup\u003e. In high-volume or resource-limited emergency contexts, an AI assistant that reliably identifies important scenarios could alleviate stress and optimize specialist referrals.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec20\" class=\"Section2\"\u003e\u003ch2\u003eModel Limitations and Interpretability:\u003c/h2\u003e\u003cp\u003eThe model underperformed in differentiating urgent cases from routine ones, mirroring concerns from recent ophthalmology studies where ChatGPT failed to adjust for clinical nuance \u003csup\u003e\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u003c/sup\u003e. Its reasoning, often prefaced with phrases like \u0026ldquo;appears non-urgent,\u0026rdquo; illustrates a conservative bias, possibly influenced by a probabilistic tendency to avoid overcalling severity. Ramponi (2022) explains that LLMs function through token prediction based on contextual probability, not pathophysiological understanding\u0026mdash;limiting clinical generalization\u003csup\u003e\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\u003cp\u003eInsufficient training inputs may result in diminished reasoning diversity. The nuanced clinical distinctions in urgent ophthalmic treatment may be inadequately captured by AI systems trained on online textual data. Research articles and preprints indicate that improving large language models using specialized or multimodal clinical datasets, including slit lamp pictures and structured examination results, could improve performance \u003csup\u003e\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec21\" class=\"Section2\"\u003e\u003ch2\u003eEthical and Implementation Considerations:\u003c/h2\u003e\u003cp\u003eEthical issues arise regarding the use of ChatGPT in triage. Despite the efficacy of LLMs as diagnosticians underscore the necessity for supervision and safeguarding \u003csup\u003e\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u003c/sup\u003e. The specifics of malpractice responsibility remain ambiguous, especially if triage judgments lead to adverse outcomes. To ensure safety and accountability, experts recommend hybrid approaches wherein human clinicians verify AI outcomes \u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e. 11Thorough prospective testing and continuous monitoring of clinical model performance are essential for deployment.\u003c/p\u003e\u003c/div\u003e"},{"header":"Conclusion","content":"\u003cp\u003eChatGPT-4o effectively identifies emergent ocular injuries but exhibits systematic misclassification of urgent cases, favoring a binary emergent vs. routine framework. Enhancing training data, refining reasoning strategies, and implementing clinician oversight will be essential for safe and effective clinical integration.\u003c/p\u003e\u003cp\u003eThese results substantiate concerns about the constraints of LLMs in critical scenarios. While ChatGPT-4o provides rational explanations and demonstrates technical proficiency, it is deficient in the contextual depth required for intermediate clinical assessments \u003csup\u003e\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e. Shen et al. (2023) characterize LLMs as \"double-edged swords\" since their opaque reasoning may obscure safety issues while enhancing diagnostic efficacy \u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e. This warning is substantiated by our analysis, which reveals that even plausible output may be based on tenuous or vague thinking.\u003c/p\u003e\u003cp\u003eTechnically, the limitations in ChatGPT-4o\u0026rsquo;s triage accuracy stem from its architecture as a transformer-based probabilistic model trained on massive textual corpora\u003csup\u003e\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e. It lacks domain-specific memory or feedback loops and thus defaults to probabilistic generalizations. Without reinforcement from structured ophthalmic cases, the model struggles to assign intermediate urgency\u0026mdash;an essential capability in clinical settings where patient outcomes hinge on timing \u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\u003cp\u003eMoreover, this bias toward routine classifications may reflect implicit limitations in the training data. As Delsoz et al. (2023) found in a related study on glaucoma diagnostics, ChatGPT\u0026rsquo;s diagnostic suggestions significantly improved when contextualized with detailed clinical inputs or paired with image data \u003csup\u003e\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u003c/sup\u003e. This implies a need for multimodal approaches that integrate clinical imaging, structured history, and AI reasoning\u0026mdash;possibly through hybrid systems \u003csup\u003e\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u003c/sup\u003e. Implementing feedback mechanisms and role-specific fine-tuning may also enhance ChatGPT\u0026rsquo;s sensitivity to gray-zone cases.\u003c/p\u003e\u003cp\u003eEthically, this reinforces the imperative to view LLMs as assistive tools rather than standalone solutions. As Chow et al. (2023) argue, AI chatbots have transformative potential but must operate under clearly defined clinical supervision \u003csup\u003e\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e. As such, ophthalmology educators and emergency department administrators should establish usage guidelines, focusing on AI-human collaboration rather than delegation.\u003c/p\u003e\u003cp\u003eWhile ChatGPT-4o\u0026rsquo;s current iteration exhibits strong pattern recognition for extreme presentations, its utility remains limited without enhancements in reasoning complexity, interpretability, and integration within real-time clinical systems. This paper not only quantifies current capabilities but also charts a course for responsible AI implementation in ophthalmic triage.\u003c/p\u003e\u003cdiv id=\"Sec23\" class=\"Section2\"\u003e\u003ch2\u003eStrengths\u003c/h2\u003e\u003cp\u003eChatGPT-4o showed effectiveness in accurately identifying emergent ocular injuries, demonstrating its potential utility in rapid triage, particularly in emergency settings where swift action is essential. Its reasoning centered around terms such as \u0026ldquo;vision-threatening\u0026rdquo; and \u0026ldquo;immediate risk,\u0026rdquo; to correctly identify most of these critical cases.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec24\" class=\"Section2\"\u003e\u003ch2\u003eLimitations\u003c/h2\u003e\u003cp\u003eThe AI exhibited substantial inaccuracies in classifying urgent cases, relying heavily on simplistic, binary logic, which could compromise patient safety by delaying crucial interventions.\u003c/p\u003e\u003cdiv id=\"Sec25\" class=\"Section3\"\u003e\u003ch2\u003eRecommendations\u003c/h2\u003e\u003cp\u003eTo enhance clinical utility, ChatGPT-4o requires retraining on balanced datasets emphasizing intermediate severity (urgent) cases. Refining its reasoning specificity and adopting hybrid AI-human models can further improve accuracy and safety.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eAcknowledgements and Financial Disclosure:\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003ea. Funding/Support: This study did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.\u003c/p\u003e\n\u003cp\u003eb. Financial Disclosures: The authors have no financial relationships relevant to this article to disclose.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCredit authorship contribution statement:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eEzanna Mesfin: Writing – review \u0026amp; editing, Writing – original draft, Formal analysis, Data analysis. Matthew Heider: Writing – reviewing \u0026amp; editing, Formal analysis, Data analysis. Mohamed Heiba: \u0026nbsp;Writing – review \u0026amp; editing, Formal analysis, Data analysis and curation. Ross Stuber: Writing – review \u0026amp; editing. Inci Dersu – review \u0026amp; editing. John Laudi – review \u0026amp; editing.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDeclaration of competing interest:\u003c/strong\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eEzanna Mesfin, Matthew Heider, Mohamed Heiba, Ross Stuber, Inci Dersu, and John Laudi report no conflicts of interest.\u0026nbsp;\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eChanna R, Zafar SN, Canner JK, et al. Epidemiology of Eye-Related Emergency Department Visits. \u003cem\u003eJAMA Ophthalmol.\u003c/em\u003e 2016;134(3):312\u0026ndash;319. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1001/jamaophthalmol.2015.5778\u003c/span\u003e\u003cspan address=\"10.1001/jamaophthalmol.2015.5778\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eRamponi M. How ChatGPT actually works. AssemblyAI. December 23, 2022. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.assemblyai.com/blog/how-chatgpt-actually-works/\u003c/span\u003e\u003cspan address=\"https://www.assemblyai.com/blog/how-chatgpt-actually-works/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eWaisberg E, Ong J, Masalkhi M, et al. GPT-4: a new era of artificial intelligence in medicine. \u003cem\u003eIr J Med Sci.\u003c/em\u003e2023;192(6):3197\u0026ndash;3200. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/s11845-023-03377-8\u003c/span\u003e\u003cspan address=\"10.1007/s11845-023-03377-8\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eDelsoz M, Raja H, Madadi Y, et al. The use of ChatGPT to assist in diagnosing glaucoma based on clinical case reports. \u003cem\u003eOphthalmol Ther.\u003c/em\u003e 2023;12(6):3121\u0026ndash;3132. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/s40123-023-00805-x\u003c/span\u003e\u003cspan address=\"10.1007/s40123-023-00805-x\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMomenaei B, Wakabayashi T, Shahlaee A, et al. Appropriateness and readability of ChatGPT-4-generated responses for surgical treatment of retinal diseases. \u003cem\u003eOphthalmol Retina.\u003c/em\u003e 2023;7(10):862\u0026ndash;868. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.oret.2023.05.022\u003c/span\u003e\u003cspan address=\"10.1016/j.oret.2023.05.022\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eShen Y, Heacock L, Elias J, et al. ChatGPT and other large language models are double-edged swords. \u003cem\u003enpj Digit Med.\u003c/em\u003e 2023;6(1):61. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1038/s41746-023-00886-\u003c/span\u003e\u003cspan address=\"10.1038/s41746-023-00886-\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMasalkhi M, Ong J, Waisberg E, et al. ChatGPT to document ocular infectious diseases. \u003cem\u003eEye (Lond).\u003c/em\u003e Published online November 15, 2023. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1038/s41433-023-02823-2\u003c/span\u003e\u003cspan address=\"10.1038/s41433-023-02823-2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMing S, Yao X, Guo X, et al. Performance of ChatGPT in ophthalmic registration and clinical diagnosis: cross-sectional study. \u003cem\u003eJ Med Internet Res.\u003c/em\u003e 2024;26:e60226. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.2196/60226\u003c/span\u003e\u003cspan address=\"10.2196/60226\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMadadi Y, Delsoz M, Lao PA, et al. ChatGPT assisting diagnosis of neuro-ophthalmology diseases based on case reports. Preprint. \u003cem\u003emedRxiv.\u003c/em\u003e Published 2023 Sep 14. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1101/2023.09.13.23295508\u003c/span\u003e\u003cspan address=\"10.1101/2023.09.13.23295508\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003ePeng Z, Ma R, Zhang Y, et al. Development and evaluation of multimodal AI for diagnosis and triage of ophthalmic diseases using ChatGPT and anterior segment images: protocol for a two-stage cross-sectional study. \u003cem\u003eFront Artif Intell.\u003c/em\u003e 2023;6:1323924. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3389/frai.2023.1323924\u003c/span\u003e\u003cspan address=\"10.3389/frai.2023.1323924\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eChow JCL, Sanders L, Li K. Impact of ChatGPT on medical chatbots as a disruptive technology. \u003cem\u003eFront Artif Intell.\u003c/em\u003e2023;6:1166014. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3389/frai.2023.1166014\u003c/span\u003e\u003cspan address=\"10.3389/frai.2023.1166014\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eKhan RA, Jawaid M, Khan AR, Sajjad M. ChatGPT \u0026ndash; reshaping medical education and clinical management. \u003cem\u003ePak J Med Sci.\u003c/em\u003e 2023;39(2):605\u0026ndash;607. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.12669/pjms.39.2.7653\u003c/span\u003e\u003cspan address=\"10.12669/pjms.39.2.7653\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eLyons RJ, Arepalli SR, Fromal O, Choi JD, Jain N. Artificial intelligence chatbot performance in triage of ophthalmic conditions. \u003cem\u003eCan J Ophthalmol\u003c/em\u003e. 2024;59(4):e301-e308. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.jcjo.2023.07.016\u003c/span\u003e\u003cspan address=\"10.1016/j.jcjo.2023.07.016\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"},{"header":"Tables","content":"\u003cp\u003eTables 1 to 3 are available in the Supplementary Files section.\u003c/p\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"eye","isNatureJournal":false,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"eye","sideBox":"Learn more about [Eye](http://www.nature.com/eye/)","snPcode":"41433","submissionUrl":"https://mts-eye.nature.com/cgi-bin/main.plex","title":"Eye","twitterHandle":"@eye_journal","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Nature AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Ocular trauma, Triage classification, ChatGPT-4o, Large language models, emergency ophthalmology, clinical decision support, artificial intelligence, NEISS database","lastPublishedDoi":"10.21203/rs.3.rs-7359662/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7359662/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cb\u003ePurpose:\u003c/b\u003e\u003c/p\u003e\u003cp\u003eThis study evaluates ChatGPT-4o\u0026rsquo;s reasoning process in triaging ocular injury cases in classification data from the National Electronic Injury Surveillance System (NEISS). The goal is to analyze the reasoning used by the model and its accuracy.\u003c/p\u003e\u003cp\u003e\u003cb\u003eDesign:\u003c/b\u003e\u003c/p\u003e\u003cp\u003eRetrospective cohort study.\u003c/p\u003e\u003cp\u003e\u003cb\u003eParticipants:\u003c/b\u003e\u003c/p\u003e\u003cp\u003e2 145 cases of ocular injuries were randomly selected from the NEISS database under an IRB exempt protocol.\u003c/p\u003e\u003cp\u003e\u003cb\u003eMethods:\u003c/b\u003e\u003c/p\u003e\u003cp\u003e2 145 ocular injury cases were randomly sampled, and evenly categorized into three triage levels: emergent, urgent, and routine. ChatGPT-4o was tasked to assign triage levels recommend interventions, and provide reasoning for each decision. The model\u0026rsquo;s reasoning was categorized into four types: 1) \"Blunt trauma may cause delayed complications like retinal detachment,\" 2) \"Chemical injuries can cause severe ocular damage and require immediate attention,\" 3) \"Foreign body cases can cause vision-threatening complications if not treated urgently,\" and 4) \"The injury appears non-urgent based on provided details.\" We evaluated the frequency and accuracy of each reasoning type and analyzed their association with triage categories.\u003c/p\u003e\u003cp\u003e\u003cb\u003eMain Outcome Measures:\u003c/b\u003e\u003c/p\u003e\u003cp\u003eAccuracy of ChatGPT-4o in correctly triaging emergent, urgent, and routine cases.\u003c/p\u003e\u003cp\u003e\u003cb\u003eResults:\u003c/b\u003e\u003c/p\u003e\u003cp\u003eChatGPT-4o correctly categorized 60% of all cases. The model frequently defaulted to routine classification, contributing to under-recognition of urgent cases.\u003c/p\u003e\u003cp\u003e\u003cb\u003eConclusion:\u003c/b\u003e\u003c/p\u003e\u003cp\u003eChatGPT-4o shows potential for triaging ocular injuries, especially in identifying emergent cases. However, it struggles with nuanced reasoning.\u003c/p\u003e","manuscriptTitle":"Evaluating ChatGPT-4o's Reasoning in Ocular Injury Using NEISS Data","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-09-09 11:55:01","doi":"10.21203/rs.3.rs-7359662/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"reviewersInvited","content":"","date":"2025-09-02T06:50:56+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-08-21T09:30:16+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-08-13T13:32:04+00:00","index":"","fulltext":""},{"type":"submitted","content":"Eye","date":"2025-08-13T01:05:00+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"eye","isNatureJournal":false,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"eye","sideBox":"Learn more about [Eye](http://www.nature.com/eye/)","snPcode":"41433","submissionUrl":"https://mts-eye.nature.com/cgi-bin/main.plex","title":"Eye","twitterHandle":"@eye_journal","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Nature AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"811f8db4-fd63-42fe-9d0b-2f07d2280807","owner":[],"postedDate":"September 9th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":54044165,"name":"Health sciences/Medical research/Outcomes research"},{"id":54044166,"name":"Health sciences/Health care/Diagnosis"}],"tags":[],"updatedAt":"2025-09-09T11:55:01+00:00","versionOfRecord":[],"versionCreatedAt":"2025-09-09 11:55:01","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7359662","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7359662","identity":"rs-7359662","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}