Assessing The Performance of Multimodal Large Language Models in Diagnosing and Staging Diabetic Retinopathy: An External Validation Study of Large Language Models

Assessing The Performance of Multimodal Large Language Models in Diagnosing and Staging Diabetic Retinopathy: An External Validation Study of Large Language Models | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Assessing The Performance of Multimodal Large Language Models in Diagnosing and Staging Diabetic Retinopathy: An External Validation Study of Large Language Models Chuin-Hen Liew, Ee Xion Tan, Nur Izzati Mohd Salim, Nur Ainaa Najwa Razali, and 6 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5909259/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Diabetic retinopathy (DR) is a leading cause of visual impairment, requiring effective and scalable screening tools for early detection. Existing methods are complex, expensive, and reliant on specialized personnel, limiting their use in primary care. This study evaluates the potential of a multimodal large language model (LLM), for detecting DR, staging DR, and identifying diabetic maculopathy. This external validation study assessed the performance of LLMs using 228 fundus images captured at Tuanku Ampuan Najihah Hospital. Models evaluated include GPT-4, Google’s Gemini 1.5, Anthropic Claude 3 Haiku, and Mistral Large. Sensitivity, specificity, and predictive value were assessed, and results were validated with human ophthalmologist evaluations. As a results, GPT-4 achieved good sensitivity for detecting DR (82%) and referable DR (80%), meeting UK NICE criteria. However, all LLMs, including GPT-4, performed poorly in staging DR and detecting diabetic maculopathy. While GPT-4 shows promise in identifying DR, its limitations in detailed DR staging and maculopathy detection highlight cautious implementation. Health sciences/Diseases/Eye diseases Health sciences/Diseases/Metabolic disorders Health sciences/Health care/Diagnosis Health sciences/Health care/Health services Health sciences/Health care/Medical imaging Figures Figure 1 Figure 2 Figure 3 Figure 4 INTRODUCTION Diabetic retinopathy (DR), a progressive retinal disorder resulting from the microvascular complications (small blood vessel disease) of diabetes mellitus (DM), is the second leading cause of visual impairment in Malaysia after cataracts [ 1 ]. Within two decades of a DM diagnosis, approximately two-thirds of patients will develop DR [ 2 ]. The stages of DR include mild non-proliferative DR (NPDR), moderate NPDR, severe NPDR, proliferative DR (PDR), and advanced diabetic eye disease (ADED), listed in order of increasing severity [ 3 ]. Patients with DR stages beyond mild NPDR are at risk of visual impairment and therefore require early referral to an ophthalmologist for detailed assessment and treatment planning to prevent blindness. Alarmingly, half of the individuals with diabetes remain unaware of their condition, and most patients are not aware of DR due to its asymptomatic nature in the early stages [ 4 ]. These challenges highlight the importance of early and effective DR screening, as early detection and treatment are crucial in preventing blindness. Various techniques are available for DR detection, including ophthalmoscopy, fundus photography, 7-field stereoscopic color retinal photography, fundus fluorescein angiography (FFA), and optical coherence tomography angiography (OCT). In summary, these methods are too complex, time-consuming, costly, and require huge investments in specialized equipment & skilled personnel, making them not feasible for large-scale DR screening. In this context, single-field fundus photography using a camera-based system emerges as a highly feasible method for DR detection in primary care settings. This approach is highly recommended by the UK National Screening Committee and the Health Technology Board for Scotland. According to American Diabetic Association guidelines, all patients diagnosed with DM should undergo DR screening at diagnosis, followed by annual evaluations for those with normal results [ 5 ]. Patients with referable DR (moderate NPDR, severe NPDR, PDR) should be referred to an ophthalmologist. To address the high prevalence of DR (39%) in Malaysia [ 6 ], health clinics have implemented a DR Screening Program utilizing fundus cameras. However, an audit conducted in Penang, Malaysia revealed that nearly half of the fundoscopy exams were inaccurately assessed, due to a shortage of skilled personnel [ 7 ]. Furthermore, the screening rate was low. According to the 2020 National Diabetic Registry, only 53% of DM patients had undergone DR screening. Artificial intelligence (AI) may offer a promising solution to streamline DR screening. The AI is extensively used in ophthalmology for conditions such as DR, age-related macular degeneration, glaucoma, and cataracts [ 8 ]. Importantly, the IDx-DR system is the first AI diagnostic tool approved by the FDA for ophthalmology, demonstrating a sensitivity of 87.2% and a specificity of 90.7% in detecting DR [ 9 ]. This convolutional neural network-based AI system is costly and challenging to implement in resource-limited settings, with each screening costing near to USD 25 [ 10 ]. The recent development of transformer-based large language models (LLMs) has advanced the application of AI. The LLM is a category of AI designed to generate human-like responses and perform various language-related tasks. At present, the state-of-the-art multimodal transformers now incorporate an encoder-decoder architecture, allowing them to interpret a wide range of unstructured data types, including written text, static medical images, dynamic videos, and audio, thereby broadening their applicability. By combining the capabilities of natural language processing (NLP) and visual image analysis, multimodal LLMs are establishing new standards in AI's ability to comprehend data and generate information. Such versatility is particularly crucial in healthcare, where medical decisions frequently depend on various information sources, including textual laboratory data, medical images, and radiological images. In our literature review, we pinpointed several gaps. Although LLMs have shown promise in accurately answering ophthalmology board examination questions [ 11 , 12 ], peer-reviewed research validating their capabilities in diagnosing and classifying DR is limited. Additionally, most studies rely on public datasets for training and validating LLMs, which raised concerns about the generalizability of findings to real-world scenarios, particularly in the Malaysian context. Thus, it is crucial to rigorously evaluate the generalizability and trustworthiness of these LLMs before integrating them into healthcare systems. Our research aims to address the above research gaps by conducting an external clinical validation of multimodal LLM with vision capabilities (image processing capabilities). These include Open AI’s GPT-4, Google’s Gemini 1.5, Claude 3 Haiku, and Mistral Large models. We examined the performance of these LLMs in detecting DR, categorizing DR stages, and identifying diabetic maculopathy. Additionally, the study also evaluates the limitations of these LLMs (failure to respond to text prompts, and providing ambiguous answers). Our focus is on open proprietary LLMs, as they are easily accessible through web platforms with a simple user interface. This accessibility is crucial for clinicians without advanced computing skills, enabling them to implement these AI tools with ease. METHODOLOGY Research Approach and Data Source This external validation study was conducted retrospectively using secondary data analyzed through a cross-sectional design. The data, including fundus images and patient demographic details (age, ethnicity, and sex), were collected from the ophthalmology clinic of Tuanku Ampuan Najihah Hospital. The data collection spanned from January 1, 2018, to December 31, 2023. During routine clinical evaluations, single-field fundus photographs with a 45-degree field of view were captured using a Zeiss Visucam 500. These images were archived using the Zeiss Forum 4.4 system and stored on the clinic’s on-premise server, maintained at their original resolution of 2100 x 2100 pixels in JPG format. Study Participants The study included patients aged 18 and over, diagnosed with type 2 DM who had undergone retinal imaging using the fundus camera. Patients who had received treatments for DR, such as intravitreal anti-vascular endothelial growth factor (VEGF) therapy, pan-retinal photocoagulation, or vitrectomy, were excluded. Since type 2 DM is uncommon in children and adolescents, our research focused only on the adult population. Retinal findings in patients treated with anti-VEGF therapy may reveal new hemorrhages or retinal detachment [ 13 ], while those treated with pan-retinal photocoagulation may display scarring in their retinal images. These factors could confound the identification of advanced diabetic eye disease, so such cases were excluded from the study. Definition of Study Variables and Outcome Definition of Study Variables and Outcome The International Clinical Diabetic Retinopathy (ICDR) and Diabetic Macular Edema Disease Severity Scales were used to define and stage DR [ 3 ], as detailed in Table 1 . Referable DR includes moderate NPDR, severe NPDR, PDR, and ADED. Diabetic maculopathy was defined by the presence of exudate in the fovea within one optic disc diameter, exudates in the macula, or microaneurysms/hemorrhages within the optic one disc diameter of the macula [ 14 ]. Table 1 The Definition of Diabetic Retinopathy Stages Stages of DR ICDR Scale Findings No retinopathy 0 No abnormalities Mild NPDR 1 Microaneurysms only Moderate NPDR 2 More than just microaneurysms but less than severe NPDR Severe NPDR 3 Any of the following: - exceed 20 intraretinal hemorrhages in each of the four quadrants; - presence of definite venous beading in two quadrants; - presence of intraretinal microvascular abnormalities in at least one quadrant with no evidence of proliferative retinopathy PDR 4 One or more of the following: - neovascularization - vitreous/preretinal hemorrhage Note: Table adopted from Wilkinson et al.(2003)[ 3 ]. DR = diabetic retinopathy, NPDR = non-proliferative diabetic retinopathy, PDR = proliferative diabetic retinopathy, ICDR = International Clinical Diabetic Retinopathy. Sampling This study aims to assess whether LLMs are sufficiently sensitive to serve as screening tools for detecting DR. The study’s null hypothesis specifies that the LLM is not sensitive to detect DR, with sensitivity set at 50%, while the alternative hypothesis expects a higher sensitivity threshold, set at a minimum of 70% [ 15 ]. According to a Power Analysis and Sample Size table developed by Bujang et al., if the prevalence of DR is estimated at 30% [ 6 ], a sample size of 163 subjects is necessary to achieve an 80% power to detect an increase in screening sensitivity from 50–70%, assuming a target significance level of 0.05 [ 15 ]. An additional 20% sample size was included to account for excluded images that may not achieve complete annotation agreement among the ophthalmologists, making the estimated minimum sample size to be 200 fundus images. Between January 1, 2018, and December 31, 2023, there were 963 patients met the study's eligibility criteria. A stratified random sampling technique selected 250 patients (exceeding the estimated sample size). This process involved assigning a unique research identifier to each patient and using a computer-generated random number sequence (Microsoft Excel 2016, Version 2205) to randomly select the participants. Random stratified sampling is important as it ensures that all study groups (DR vs. non-DR) are adequately represented, minimizing selection bias. Data Collection and Data Annotations The sampled fundus images and demographic details were downloaded onto the clinic’s computer and the structured data were extracted into a spreadsheet. The fundus images were stored in their original JPG format (no resizing or editing) for data annotation by an ophthalmologist and for testing the multimodal LLM. Two experienced ophthalmologists independently reviewed the fundus images for the presence or absence of DR, the stage of retinopathy, and the presence or absence of maculopathy, without access to each other’s assessments or the patients’ information. The assessments were guided by the ICDR recommendations. They reached an agreement on 91% (228/250) of the images, and the agreed images were then used for the external validation of the multimodal LLM. The complete agreement among the ophthalmologists on the presence of DR, stages of DR, and the presence of maculopathy was used as the benchmark (reference standard) for defining study outcomes. Testing the Multimodal LLM We tested a variety of LLMs (index tests) using the following website platforms Open AI’s GPT-4 ( https://openai.com/ ) Google’s Gemini 1.5 ( https://aistudio.google.com ) Anthropic Claude 3 Haiku ( https://poe.com/Claude-3-Haiku ) Mistral’s Large ( https://chat.mistral.ai/login%20 ) Fundus images were uploaded to the chat interface with the research standardized text prompt: “Act as an ophthalmologist. Your task is to provide a report on fundus images of patients with diabetes mellitus captured using a fundus camera during retinopathy screening. Please report with the following order. Firstly, describe the findings observed in the fundus images. Secondly, comment on the presence or absence of retinopathy. Thirdly, if retinopathy is present, provide comments on the stages of retinopathy, where options include mild non-proliferative retinopathy, moderate non-proliferative retinopathy, severe non-proliferative retinopathy, proliferative retinopathy, or advanced diabetic eye disease.” Next, researchers reviewed the LLM’s responses to identify the presence of DR, the stages of DR (rated using the ICDR scale), and any findings of diabetic maculopathy. These findings were documented in a spreadsheet and validated by an ophthalmology researcher. Figure 1 illustrates the Open AI’s ChatGPT LLM interface with the study prompt, uploaded fundus image, and the AI-generated response. Model Evaluation The LLM's performance was assessed by constructing a confusion matrix for each study outcome and comparing the LLM's classification results with the ophthalmologists' actual labels. Performance metrics such as the area under the receiver operating characteristic curve (AUROC), sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), F1 score, and accuracy were calculated. The specific formulas used to calculate these metrics are provided in Supplementary Table 1. The number of failed and ambiguous responses from GPT-4 was also recorded. A failed response was noted when the LLM refused to reply to the text prompt. Ambiguous responses were said to occur when the LLM response could not precisely determine the stage of DR, providing a range (e.g., mild to moderate DR) instead of a specific stage. Data Management and Analysis Before analysis, the structured dataset was reviewed, and no typographical errors, outliers, or missing values were found. The age variable distribution was non-normal (Kolmogorov–Smirnov test p-value < 0.05), and thus reported using median and interquartile ranges. Categorical data (ethnicity and sex) were reported as frequencies and percentages. Data analyses were performed using the Statistical Package for the Social Sciences (SPSS version 26.0). The calculation of performance metrics was performed using the Python library sklearn.metrics (sklearn version: 1.3.0). Ethical Considerations This study was conducted in accordance with the Declaration of Helsinki and received approval from the Malaysia Medical Research Ethics Committee (NMRR ID-24-00315-WXU). The need for informed consent was waived for this retrospective study. RESULTS Study Participation and Demographic Details Of the 250 fundus images sampled, 228 (91%) achieved complete agreement between the two ophthalmology researchers on the annotations for DR, DR stages, and maculopathy (Fig. 2 ). The study cohort (n = 228) had a median age of 57.0 years (IQR 40.8–65.0), with a slight female predominance at 54.4%. The ethnicity distribution of the cohort was predominantly Malay 168 (73.7%), followed by Chinese 32 (14.0%) and Indian 28 (12.3%), showing a similar proportion of main ethnicities in Malaysia (see Table 2 ). Half of the patients (n = 122, 53.5%) had DR, with the distribution of DR stages illustrated in Fig. 3 . Among those with DR, 82% had referable DR (100 out of 122 fundus images indicated severity beyond minimal NPDR) and 80 (35.1%) out of 228 images with evidence of maculopathy Table 2 Demographic Characteristics of Study Participants Demographic Details Result (n = 228) Age (years), median (IQR) 57.0 (40.8–65.0) Sex Male Female 104 (45.6%) 124 (54.4%) Ethnicity Malay Chinese Indian 168 (73.7%) 32 (14.0%) 28 (12.3%) Stages of DR No retinopathy Retinopathy - Non-referable DR - Referable DR 106 (46.5%) 122 (53.5%) 22 out of 122 (18.0%) 100 out of 122 (82.0%) Note : IQR = interquartile range, DR = diabetic retinopathy Validation of Large Language Models for Diabetic Retinopathy Detection The GPT-4 model demonstrated the highest performance in detecting DR, with an AUROC of 0.81. It achieved the highest sensitivity at 82% and a precision (PPV) of 82% in accurately classifying the presence of DR. In contrast, the performance of other models, including Gemini 1.5, Claude3 Haiku, and Mistral Large, was markedly lower, with AUROC values ranging from 0.50 to 0.54 (see Table 3 ). Table 3 Performance of Large Language Models for Detection of Diabetic Retinopathy Open AI’s GPT-4 Google Gemini 1.5 Claude 3 Haiku Mistral Large AUROC 0.806 0.539 0.540 0.500 Sensitivity 0.820 0.984 0.984 1 Specificity 0.793 0.094 0.094 0 PPV 0.820 0.556 0.556 0.535 NPV 0.792 0.833 0.833 0 Accuracy 0.807 0.570 0.570 0.535 F1 0.820 0.710 0.710 0.697 Note: AUROC = area under the receiver operating characteristic curve, PPV = positive predictive value, NPV = negative predictive value Validation of Large Language Models for Staging Diabetic Retinopathy Gemini 1.5 stood out as being best in identifying moderate NPDR, achieving an AUROC of 0.81 and a sensitivity of 83%. However, its precision was low, with a PPV of 31%. All LLMs demonstrated poor performance in detecting mild NPDR, severe NPDR, and PDR (see Supplementary Table 2–5). Identifying referable DR is crucial in primary care, as any stage beyond mild NPDR requires referral to an ophthalmologist for detailed assessment and prompt intervention to prevent blindness. The GPT-4 model performed the best in classifying referable DR, with an AUROC of 0.80 compared to other models. The GPT-4 achieved a sensitivity of 80% and a precision (PPV) of 76% in identifying referable DR (see Table 4 ). Table 4 Performance of Large Language Models for Classifying Referrable DR Open AI’s GPT-4 Google Gemini 1.5 Claude 3 Haiku Mistral’s AI AUROC 0.802 0.793 0.558 0.505 Sensitivity 0.800 0.680 0.960 0.620 Specificity 0.805 0.906 0.156 0.391 PPV 0.762 0.850 0.471 0.443 NPV 0.837 0.784 0.833 0.568 Accuracy 0.803 0.807 0.509 0.491 F1 0.781 0.756 0.632 0.517 Note: DR = diabetic retinopathy, AUROC = area under the receiver operating characteristic curve, PPV = positive predictive value, NPV = negative predictive value Validation of Large Language Models for Maculopathy Detection Overall, all LLMs showed low performance in detecting maculopathy, with AUROC values ranging from 0.49 to 0.65 (see Table 5 ). Although Gemini 1.5 demonstrated high sensitivity (83%) in identifying maculopathy, its precision was low, with a PPV of only 37%. Table 5 Performance of Large Language Models for Detection of Diabetic Maculopathy Open AI’s GPT-4 Google Gemini 1.5 Claude 3 Haiku Mistral’s AI AUROC 0.645 0.527 0.503 0.491 Sensitivity 0.425 0.825 0.100 0.400 Specificity 0.865 0.230 0.905 0.581 PPV 0.630 0.367 0.364 0.340 NPV 0.736 0.708 0.650 0.642 Accuracy 0.711 0.439 0.623 0.518 F1 0.507 0.508 0.157 0.368 Note: AUROC = area under the receiver operating characteristic curve, PPV = positive predictive value, NPV = negative predictive value Limitations of Large Language Models To recap, an LLM was considered to have failed in responding when it refused to answer the text prompt (refused to respond to a medical issue). The GPT-4 had the highest failure rate at 8%, followed by Gemini 1.5 at 3%, and Mistral Large at 1%. The Claude3 Haiku model responded coherently to all text prompts during validation. Additionally, GPT-4 had the highest rate (13%) of requiring text re-prompting to clarify the output generated, due to frequently providing ambiguous answers, followed by Claude3 Haiku at 9%. In contrast, Gemini 1.5 & Mistral AI provided clear responses 100% of the time (Table 6 ). Table 6 Limitations of Large Language Models Open AI’s GPT-4 Google Gemini 1.5 Claude 3 Haiku Mistral’s AI Failure to Response 19 (8.3%) 6 (2.6%) 0 2 (0.9%) Ambiguous Response 30 (13.1%) 0 20 (8.8%) 0 DISCUSSION To recap, our study aimed to validate the multimodal LLMs for the performance of detecting DR, categorizing DR stages, and identifying diabetic maculopathy. Our study demonstrated that the GPT-4 exhibited good performance in detecting the presence of DR with a sensitivity of 82% and referable DR with a sensitivity of 80%, meeting the UK National Institute for Clinical Excellence (NICE) criteria for DR screening tools (> 80% sensitivity). However, all LLMs' effectiveness in staging DR, as well as detecting diabetic maculopathy, was found to be inadequate. Additionally, our study showed that GPT-4 has the highest chance among all LLMs to refuse answering medical-related prompts, followed by Gemini 1.5. The LLMs may refuse to respond to medical queries due to safety concerns. These generalist models are not specifically trained for healthcare applications, as their training focuses on general NLP tasks rather than specialized medical knowledge which can lead to the generation of incorrect information known as hallucinations. To mitigate the risk of harm, developers often impose restrictions on these models to prevent them from providing medical advice or diagnoses. This is done through various techniques, including fine-tuning the model with instructions to avoid engaging in medical queries or incorporating filters that detect and block responses to certain health-related topics [ 16 ]. Besides, these LLMs may be trained to encourage users to consult qualified medical professionals rather than relying on AI responses. Our study is comparable to the research conducted by Xu et al., which also examined the use of GPT-4 with fundus images. They reported that GPT-4 achieved a correct response rate of only 13.7%, although their investigation was not exclusively focused on DR [ 17 ]. Hence, the direct comparison was challenging due to the unspecified proportion of DR in their study sample. Next, Raghu et al. assessed GPT-4's ability to predict DR using central retinal thickness measurements obtained with OCT, reporting a sensitivity of 67% and specificity of 68% [ 18 ]. Antaki et al. also evaluated the Gemini LLM using OCT images, revealing its limited ability to detect macular disease features, with an average F1 score of just 10.7% [ 19 ]. However, these results were based on OCT image validation and not specifically on fundus images from diabetic patients. Despite using resource-intensive OCT measurements, the LLMs' performance in detecting abnormal ophthalmological features was suboptimal. All studies highlight the challenge of using generalized AI models in specialized clinical contexts. Our validation study showcase several strengths. Firstly, we validated LLMs using a well-generalized multi-racial dataset, providing a comprehensive range of fundus images representative of Malaysia. The use of an external validation design enhances the applicability of results across diverse patient populations. Additionally, employing stratified random sampling minimized selection bias, ensuring the generalizability of results. The study includes multiple LLMs, such as GPT-4, Gemini 1.5, Claude 3, and Mistral AI, allowing for a comprehensive comparison of their performance in detection and staging. Besides, the utilization of detailed performance metrics (sensitivity, specificity, PPV, NPV, F1 score, and accuracy) provides a thorough evaluation of each model's efficacy. Our data annotations and AI output verifications by researchers were robust. Fundus images were labeled independently by two experienced ophthalmologists following international standards (Proposed International Clinical Diabetic Retinopathy and Diabetic Macular Edema Disease Severity Scales). The AI output responses were also verified by two researchers independently before data analysis, ensuring reliable data annotations and interpretation of AI outputs. Moreover, we shared all fundus images and the Python script to obtain the performance metrics in an open data repository and ensure reporting transparency. Additionally, our studies also focus on practical implementation aspects of the AI which aligns with responsible AI principles (fairness, transparency, explainability, data privacy), ensuring the ethical application of the findings in clinical settings. We conducted a subgroup analysis examining GPT-4's DR detection across different ethnicities. We observed a decrease in PPV within the Chinese subgroup, likely due to the imbalanced ethnic representation during the external validation (Supplementary Table 6). Nonetheless, the overall performance was comparable across all ethnicities (AUROC of DR detection: 0.74–0.82), suggesting minimal ethnic bias when using GPT-4. Concerning AI transparency, all proprietary LLM models tested in our study is lacking of clear transparency regarding data quantity and quality, hyperparameters, and the specific transformer architecture used. Additionally, AI explainability is essential in healthcare AI applications. All LLMs, with their complex transformer architecture, are inherently less interpretable than traditional machine learning algorithms. This complexity makes the reasoning behind LLM outputs difficult to understand, making them largely black-box systems. As a result, LLMs may not be the most suitable AI solution for healthcare facilities that prioritize transparency and result explainability. A discussion on responsible AI is incomplete without addressing data privacy. ChatGPT Plus (GPT-4 model) and Claude 3 both offer users the ability to opt out of data sharing for future model training. This feature is particularly important as it ensures patients’ data remains private and is not utilized for further training of the AI models. Conversely, AI Studio (Gemini 1.5 model) and Mistral AI (Mistral Large model) provide the option to disable data sharing only for subscription-based users. This limitation means that users who do not subscribe to these platforms may have their uploaded data included in future model training processes. This distinction highlights the approaches to patients’ data privacy across different AI systems, which is a crucial consideration for healthcare AI applications while maintaining strict data privacy standards. This study has limitations. Image annotations (presence of DR, stages of DR, and maculopathy) were determined by independent agreement among ophthalmologists rather than using gold-standard diagnostic tools like OCT and FFA. We do not have such data due to the limited clinical application of these advanced tools in the study site and the major ophthalmology centers in Malaysia. These methods are only viable as gold standards in prospective research. Despite these limitations, the fundus images labeled by ophthalmologists provide a broad spectrum of DR stages (Fig. 3 ) for external validation of LLMs. Besides, the small sample sizes for subgroup analyses present a challenge in validating AI performance for specific ethnicities. While over 200 fundus images were collected for validation, the representation of minor ethnic groups such as Chinese and Indian was relatively small compared to the Malay group (see Supplementary Table 6). Future studies should aim for a balanced proportional representation of all major ethnic groups using stratified sampling methods to enhance generalizability. Moving beyond clinical validation, integrating GPT-4 as an AI system for DR screening involves several steps. We propose the following plan for primary care centers looking to incorporate GPT-4 into their DR screening program, specifically to identify DR and classify referable DR. The process involves a comprehensive pipeline from image capture to result reporting. The initial phase of the screening process involves the use of the Zeiss Visucam 500 fundus camera, interfaced with the Zeiss Forum 4.4 system. Fundus images are captured in JPG format and stored on an on-premise server, ensuring immediate availability and data security. The core component of the pipeline is the integration of the GPT-4 model for the analysis of fundus images. The GPT-4 application programming interface (API) requires a secure environment which is Health Insurance Portability and Accountability Act (HIPAA) compliant for processing the medical images. The process begins with a programmed script that retrieves stored images and sends them to the GPT-4 API. The model analyzes each image, generating a detailed diagnostic report that includes the presence of DR and type of DR with recommendations for further action. The AI-generated results are then formatted into a structured report. This report includes fundus image findings, stages of DR, and clinical recommendations, ensuring that healthcare professionals can easily interpret the results. These reports are stored securely on the on-premise server and made accessible through the Zeiss Forum 4.4 system. To ensure the system's ongoing effectiveness, a feedback loop is established. The referral-receiving ophthalmologist reviews the AI-generated reports, providing feedback that is used to continuously update and retrain the model. This iterative process is essential for maintaining high diagnostic accuracy. Encryption and secured access protocols are needed to safeguard sensitive patient data and ensure compliance with regulations. Compliance with data privacy regulations, such as HIPAA, is a priority throughout the implementation. Regular audits and stringent security measures are in place to protect patient data and ensure the system's integrity. By implementing this automated workflow (see Fig. 4 ), the clinic leverages AI technology to enhance the efficiency and accuracy of DR screening. This integration not only reduces the workload on healthcare professionals but also ensures timely and accurate diagnoses, ultimately improving patient outcomes. As AI technology advances, the AI pipeline must stay adaptive to incorporate newer algorithms once they are clinically validated for better DR classification tasks. It is recommended that healthcare facilities to wait for the development of more advanced and precise AI models if their primary focus is on the detailed classification of DR stages. In conclusion, this study validates GPT-4's potential as a multimodal LLM for detecting the presence of DR and classifying referable DR. It is among the few external clinical validations assessing GPT-4's performance in DR classification using fundus images. Integrating AI into DR screening in primary care has the potential to improve screening efficiency for this high-burden non-communicable disease in Malaysia. The GPT-4's ability to identify the presence of DR and referable DR could streamline the screening process, facilitating expedited referrals to ophthalmologists for detailed assessment and intervention, crucial in preventing vision loss. Given the current limitations, the use of AI in DR screening necessitates cautious implementation, and it should complement rather than replace conventional clinical pathways and consultations. Declarations Data Availability: All fundus images used for validating the LLMs are available at the following link: doi:10.6084/m9.figshare.28057571 Code Availability: The codes used for the analysis in this study are available at https://github.com/chuinhen/ResearchProject_DR/blob/main/codes/codes.ipynb Acknowledgments: We thank the Director-General of Health Malaysia for giving permission to publish this article. Author Contributions: CHL contributed to the conception and design of the study. NIMS, SS, AHS, and SYT made substantial contributions to data collection. CHL performed data analysis. CHL, EXT, QZN, SM, and KSM contributed to data interpretation. CHL wrote the first draft of the manuscript. All authors revised the manuscript for important intellectual content, approved the final version to be published, and agreed to be accountable for all aspects of their work. Competing Interest: The authors have no competing interests to disclose. Funding: This study received no funding References Chew, F. L. M. et al. Estimates of visual impairment and its causes from the National Eye Survey in Malaysia (NESII). PLOS ONE 13, e0198799, doi: 10.1371/journal.pone.0198799 (2018). Fong, D. S. et al. Retinopathy in diabetes. Diabetes care 27, s84-s87 (2004). Wilkinson, C. P. et al. Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology 110, 1677–1682, doi: 10.1016/S0161-6420(03)00475-5 (2003). Saeedi, P. et al. Global and regional diabetes prevalence estimates for 2019 and projections for 2030 and 2045: Results from the International Diabetes Federation Diabetes Atlas. Diabetes research and clinical practice 157, 107843 (2019). Solomon, S. D. et al. Diabetic Retinopathy: A Position Statement by the American Diabetes Association. Diabetes Care 40, 412–418, doi: 10.2337/dc16-2641 (2017). Abougalambou, S. S. I. & Abougalambou, A. S. Risk factors associated with diabetic retinopathy among type 2 diabetes patients at teaching hospital in Malaysia. Diabetes & Metabolic Syndrome: Clinical Research & Reviews 9, 98–103, doi: https://doi.org/10.1016/j.dsx.2014.04.019 (2015). Lim, A. K. E. & Khaw, K. W. Audit of diabetic retinopathy referrals to Penang Hospital, a tertiary Ophthalmology centre in Malaysia. Asian J Ophthalmol 11, 17–19 (2009). Benet, D. & Pellicer-Valero, O. J. Artificial intelligence: the unstoppable revolution in ophthalmology. survey of ophthalmology 67, 252–270 (2022). Abràmoff, M. D., Lavin, P. T., Birch, M., Shah, N. & Folk, J. C. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. npj Digital Medicine 1, 39, doi: 10.1038/s41746-018-0040-6 (2018). Chen, E. M., Chen, D., Chilakamarri, P., Lopez, R. & Parikh, R. Economic challenges of artificial intelligence adoption for diabetic retinopathy. Ophthalmology 128, 475–477 (2021). Lin, J. C., Younessi, D. N., Kurapati, S. S., Tang, O. Y. & Scott, I. U. Comparison of GPT-3.5, GPT-4, and human user performance on a practice ophthalmology written examination. Eye 37, 3694–3695, doi: 10.1038/s41433-023-02564-2 (2023). Taloni, A. et al. Comparative performance of humans versus GPT-4.0 and GPT-3.5 in the self-assessment program of American Academy of Ophthalmology. Scientific Reports 13, 18562, doi: 10.1038/s41598-023-45837-2 (2023). Ghasemi Falavarjani, K. & Nguyen, Q. Adverse events and complications associated with intravitreal injection of anti-VEGF agents: a review of literature. Eye 27, 787–794 (2013). Peto, T. & Tadros, C. Screening for Diabetic Retinopathy and Diabetic Macular Edema in the United Kingdom. Current Diabetes Reports 12, 338–345, doi: 10.1007/s11892-012-0285-4 (2012). Bujang, M. A. & Adnan, T. H. Requirements for Minimum Sample Size for Sensitivity and Specificity Analysis. J Clin Diagn Res 10, Ye01-ye06, doi: 10.7860/jcdr/2016/18129.8744 (2016). Clusmann, J. et al. The future landscape of large language models in medicine. Communications medicine 3, 141 (2023). Xu, P. et al. Evaluation of a digital ophthalmologist app built by GPT4-V(ision). medRxiv , 2023.2011.2027.23299056, doi: 10.1101/2023.11.27.23299056 (2023). Raghu, K. et al. The Utility of ChatGPT in Diabetic Retinopathy Risk Assessment: A Comparative Study with Clinical Diagnosis. Clin Ophthalmol 17, 4021–4031, doi: 10.2147/opth.S435052 (2023). Antaki, F., Chopra, R. & Keane, P. A. Vision-Language Models for Feature Detection of Macular Diseases on Optical Coherence Tomography. JAMA Ophthalmology 142, 573–576, doi: 10.1001/jamaophthalmol.2024.1165 (2024). Additional Declarations No competing interests reported. Supplementary Files supplementraymaterial.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5909259","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":408286163,"identity":"d210c5ab-5943-4966-802d-36c36b325faa","order_by":0,"name":"Chuin-Hen Liew","email":"","orcid":"","institution":"Hospital Tuanku Ampuan Najihah, Ministry of Health","correspondingAuthor":false,"prefix":"","firstName":"Chuin-Hen","middleName":"","lastName":"Liew","suffix":""},{"id":408286164,"identity":"58640210-d452-43ea-996e-ce61d8bceba5","order_by":1,"name":"Ee Xion Tan","email":"","orcid":"","institution":"IMU University","correspondingAuthor":false,"prefix":"","firstName":"Ee","middleName":"Xion","lastName":"Tan","suffix":""},{"id":408286165,"identity":"298fa110-1e2e-4d91-a56f-7a85d94b0887","order_by":2,"name":"Nur Izzati Mohd Salim","email":"","orcid":"","institution":"Hospital Tuanku Ampuan Najihah, Ministry of Health","correspondingAuthor":false,"prefix":"","firstName":"Nur","middleName":"Izzati Mohd","lastName":"Salim","suffix":""},{"id":408286166,"identity":"ba3fb2c1-e16a-4dac-9d11-01b6b9c9c211","order_by":3,"name":"Nur Ainaa Najwa Razali","email":"","orcid":"","institution":"Hospital Tuanku Ampuan Najihah, Ministry of Health","correspondingAuthor":false,"prefix":"","firstName":"Nur","middleName":"Ainaa Najwa","lastName":"Razali","suffix":""},{"id":408286167,"identity":"dfd40586-115c-4e1c-8c08-7de0e8a8610d","order_by":4,"name":"Abdul Hadi Sharifudin","email":"","orcid":"","institution":"Hospital Tuanku Ampuan Najihah, Ministry of Health","correspondingAuthor":false,"prefix":"","firstName":"Abdul","middleName":"Hadi","lastName":"Sharifudin","suffix":""},{"id":408286168,"identity":"cb0b220f-7599-406b-9409-0451692880cc","order_by":5,"name":"Sin Yee Teo","email":"","orcid":"","institution":"Hospital Tuanku Ja'afar, Ministry of Health Malaysia","correspondingAuthor":false,"prefix":"","firstName":"Sin","middleName":"Yee","lastName":"Teo","suffix":""},{"id":408286169,"identity":"2a5f03db-2dcd-4f66-8168-a584ebe9be7d","order_by":6,"name":"Sangeetha Subramaniam","email":"","orcid":"","institution":"Hospital Tuanku Ampuan Najihah, Ministry of Health","correspondingAuthor":false,"prefix":"","firstName":"Sangeetha","middleName":"","lastName":"Subramaniam","suffix":""},{"id":408286170,"identity":"740c1a1c-f41e-432f-90e3-b714a52417c3","order_by":7,"name":"Qi Zhe Ngoo","email":"","orcid":"","institution":"Universiti Sains Malaysia","correspondingAuthor":false,"prefix":"","firstName":"Qi","middleName":"Zhe","lastName":"Ngoo","suffix":""},{"id":408286171,"identity":"d31a2fb1-c233-46da-8c18-fb4abf4e9653","order_by":8,"name":"Saravanan Muthaiyah","email":"","orcid":"","institution":"IMU University","correspondingAuthor":false,"prefix":"","firstName":"Saravanan","middleName":"","lastName":"Muthaiyah","suffix":""},{"id":408286172,"identity":"a0270b21-83af-45d8-b91f-8dd222495ae1","order_by":9,"name":"Kalaiarasi Sonai Muthu","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA50lEQVRIiWNgGAWjYBACCQbmBoYEEIuZ+QBE6ABBLYwwLWwJJGiBAB4D4rRIth9s/PCg5p68fDvP10032xjk+G4ksG7mwaNFmiexWSLhWLHhhsO8227ntjEYS95IYLuNT4scQ2KDRAJbAuMGZoiWxA0EtfA/bP6R8C/Bfn4zzzOQlnqCWqQlEtuAKCGx4TAPG0hLggEhLZIzHrZZJPYlJG84zGZ2O+echOHMMw/bbs7Bo0XifPLhmz++JdjO7z/87HZOmY083/HkYzfe4NGCYQQQMzYw4XMYdsD4g2Qto2AUjIJRMIwBAA2sU34WarPwAAAAAElFTkSuQmCC","orcid":"","institution":"Multimedia University","correspondingAuthor":true,"prefix":"","firstName":"Kalaiarasi","middleName":"Sonai","lastName":"Muthu","suffix":""}],"badges":[],"createdAt":"2025-01-27 04:23:16","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5909259/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5909259/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":75188065,"identity":"679f1dcd-f699-4db1-9e41-f206cbe095ac","added_by":"auto","created_at":"2025-01-31 17:59:59","extension":"jpeg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":493879,"visible":true,"origin":"","legend":"\u003cp\u003eThe ChatGPT Interface Displays the Study Prompt, Uploaded Fundus Image, and the AI-generated response. The image illustrates a fundus image of a patient with diabetes mellitus, which was uploaded to the large language model. The task assigned to it was to generate a clinical report that describes the findings in the fundus image, determines the presence of retinopathy, and classifies its stage if present.\u003c/p\u003e","description":"","filename":"floatimage1.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-5909259/v1/e17445f6b49fec55df3a9316.jpeg"},{"id":75188061,"identity":"7ca7e379-7063-4dff-b037-bdff843f04ee","added_by":"auto","created_at":"2025-01-31 17:59:58","extension":"jpeg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":47749,"visible":true,"origin":"","legend":"\u003cp\u003eStudy Participation Flow Chart. This figure outlines the patient selection steps taken to identify a cohort of fundus images for studying diabetic retinopathy.\u003c/p\u003e","description":"","filename":"floatimage2.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-5909259/v1/1626ad4a19dde4c02bfe1e72.jpeg"},{"id":75188071,"identity":"9aa75fb4-e3c6-44db-bac9-cf712bc5bca7","added_by":"auto","created_at":"2025-01-31 17:59:59","extension":"jpeg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":88027,"visible":true,"origin":"","legend":"\u003cp\u003eStages of Diabetic Retinopathy Included in the Study. This figure illustrates the proportions of patients in each stage of diabetic retinopathy.\u003c/p\u003e\n\u003cp\u003eNPDR= non-proliferative diabetic retinopathy, PDR= proliferative diabetic retinopathy, ADED= advanced diabetic eye disease\u003c/p\u003e","description":"","filename":"floatimage3.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-5909259/v1/0c897d75c306f50819009166.jpeg"},{"id":75191524,"identity":"2af9173f-1c89-4f49-bd82-01700cca0c61","added_by":"auto","created_at":"2025-01-31 18:23:59","extension":"jpeg","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":57955,"visible":true,"origin":"","legend":"\u003cp\u003eArtificial Intelligence Pipeline for Diabetic Retinopathy Screening. This flowchart illustrates the end-to-end process employed for the suggested automated analysis and reporting of diabetic retinopathy screening images, integrating fundus imaging, data storage, and AI-based analysis and reporting.\u003c/p\u003e","description":"","filename":"floatimage4.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-5909259/v1/009c40f581074ab4d464316c.jpeg"},{"id":75659967,"identity":"812c2441-8f89-4d73-8551-98f7bfd251cf","added_by":"auto","created_at":"2025-02-06 22:46:29","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1557691,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5909259/v1/dfc0d093-96e7-4c47-978e-c9bc05ea148e.pdf"},{"id":75190314,"identity":"0868a67a-8f0c-420e-aa6c-ebfebbb783b2","added_by":"auto","created_at":"2025-01-31 18:07:59","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":180611,"visible":true,"origin":"","legend":"","description":"","filename":"supplementraymaterial.docx","url":"https://assets-eu.researchsquare.com/files/rs-5909259/v1/3735851379a4f311bd7f934e.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Assessing The Performance of Multimodal Large Language Models in Diagnosing and Staging Diabetic Retinopathy: An External Validation Study of Large Language Models","fulltext":[{"header":"INTRODUCTION","content":"\u003cp\u003eDiabetic retinopathy (DR), a progressive retinal disorder resulting from the microvascular complications (small blood vessel disease) of diabetes mellitus (DM), is the second leading cause of visual impairment in Malaysia after cataracts [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. Within two decades of a DM diagnosis, approximately two-thirds of patients will develop DR [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. The stages of DR include mild non-proliferative DR (NPDR), moderate NPDR, severe NPDR, proliferative DR (PDR), and advanced diabetic eye disease (ADED), listed in order of increasing severity [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. Patients with DR stages beyond mild NPDR are at risk of visual impairment and therefore require early referral to an ophthalmologist for detailed assessment and treatment planning to prevent blindness. Alarmingly, half of the individuals with diabetes remain unaware of their condition, and most patients are not aware of DR due to its asymptomatic nature in the early stages [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. These challenges highlight the importance of early and effective DR screening, as early detection and treatment are crucial in preventing blindness.\u003c/p\u003e \u003cp\u003eVarious techniques are available for DR detection, including ophthalmoscopy, fundus photography, 7-field stereoscopic color retinal photography, fundus fluorescein angiography (FFA), and optical coherence tomography angiography (OCT). In summary, these methods are too complex, time-consuming, costly, and require huge investments in specialized equipment \u0026amp; skilled personnel, making them not feasible for large-scale DR screening. In this context, single-field fundus photography using a camera-based system emerges as a highly feasible method for DR detection in primary care settings. This approach is highly recommended by the UK National Screening Committee and the Health Technology Board for Scotland. According to American Diabetic Association guidelines, all patients diagnosed with DM should undergo DR screening at diagnosis, followed by annual evaluations for those with normal results [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. Patients with referable DR (moderate NPDR, severe NPDR, PDR) should be referred to an ophthalmologist. To address the high prevalence of DR (39%) in Malaysia [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e], health clinics have implemented a DR Screening Program utilizing fundus cameras. However, an audit conducted in Penang, Malaysia revealed that nearly half of the fundoscopy exams were inaccurately assessed, due to a shortage of skilled personnel [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. Furthermore, the screening rate was low. According to the 2020 National Diabetic Registry, only 53% of DM patients had undergone DR screening.\u003c/p\u003e \u003cp\u003eArtificial intelligence (AI) may offer a promising solution to streamline DR screening. The AI is extensively used in ophthalmology for conditions such as DR, age-related macular degeneration, glaucoma, and cataracts [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. Importantly, the IDx-DR system is the first AI diagnostic tool approved by the FDA for ophthalmology, demonstrating a sensitivity of 87.2% and a specificity of 90.7% in detecting DR [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. This convolutional neural network-based AI system is costly and challenging to implement in resource-limited settings, with each screening costing near to USD 25 [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe recent development of transformer-based large language models (LLMs) has advanced the application of AI. The LLM is a category of AI designed to generate human-like responses and perform various language-related tasks. At present, the state-of-the-art multimodal transformers now incorporate an encoder-decoder architecture, allowing them to interpret a wide range of unstructured data types, including written text, static medical images, dynamic videos, and audio, thereby broadening their applicability. By combining the capabilities of natural language processing (NLP) and visual image analysis, multimodal LLMs are establishing new standards in AI's ability to comprehend data and generate information. Such versatility is particularly crucial in healthcare, where medical decisions frequently depend on various information sources, including textual laboratory data, medical images, and radiological images. In our literature review, we pinpointed several gaps. Although LLMs have shown promise in accurately answering ophthalmology board examination questions [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e], peer-reviewed research validating their capabilities in diagnosing and classifying DR is limited. Additionally, most studies rely on public datasets for training and validating LLMs, which raised concerns about the generalizability of findings to real-world scenarios, particularly in the Malaysian context. Thus, it is crucial to rigorously evaluate the generalizability and trustworthiness of these LLMs before integrating them into healthcare systems.\u003c/p\u003e \u003cp\u003eOur research aims to address the above research gaps by conducting an external clinical validation of multimodal LLM with vision capabilities (image processing capabilities). These include Open AI\u0026rsquo;s GPT-4, Google\u0026rsquo;s Gemini 1.5, Claude 3 Haiku, and Mistral Large models. We examined the performance of these LLMs in detecting DR, categorizing DR stages, and identifying diabetic maculopathy. Additionally, the study also evaluates the limitations of these LLMs (failure to respond to text prompts, and providing ambiguous answers). Our focus is on open proprietary LLMs, as they are easily accessible through web platforms with a simple user interface. This accessibility is crucial for clinicians without advanced computing skills, enabling them to implement these AI tools with ease.\u003c/p\u003e"},{"header":"METHODOLOGY","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eResearch Approach and Data Source\u003c/h2\u003e \u003cp\u003eThis external validation study was conducted retrospectively using secondary data analyzed through a cross-sectional design. The data, including fundus images and patient demographic details (age, ethnicity, and sex), were collected from the ophthalmology clinic of Tuanku Ampuan Najihah Hospital. The data collection spanned from January 1, 2018, to December 31, 2023.\u003c/p\u003e \u003cp\u003eDuring routine clinical evaluations, single-field fundus photographs with a 45-degree field of view were captured using a Zeiss Visucam 500. These images were archived using the Zeiss Forum 4.4 system and stored on the clinic\u0026rsquo;s on-premise server, maintained at their original resolution of 2100 x 2100 pixels in JPG format.\u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003eStudy Participants\u003c/h3\u003e\n\u003cp\u003eThe study included patients aged 18 and over, diagnosed with type 2 DM who had undergone retinal imaging using the fundus camera. Patients who had received treatments for DR, such as intravitreal anti-vascular endothelial growth factor (VEGF) therapy, pan-retinal photocoagulation, or vitrectomy, were excluded. Since type 2 DM is uncommon in children and adolescents, our research focused only on the adult population. Retinal findings in patients treated with anti-VEGF therapy may reveal new hemorrhages or retinal detachment [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e], while those treated with pan-retinal photocoagulation may display scarring in their retinal images. These factors could confound the identification of advanced diabetic eye disease, so such cases were excluded from the study.\u003c/p\u003e\n\u003ch3\u003eDefinition of Study Variables and Outcome\u003c/h3\u003e\n\u003cdiv class=\"Heading\"\u003eDefinition of Study Variables and Outcome\u003c/div\u003e \u003cp\u003eThe International Clinical Diabetic Retinopathy (ICDR) and Diabetic Macular Edema Disease Severity Scales were used to define and stage DR [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e], as detailed in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. Referable DR includes moderate NPDR, severe NPDR, PDR, and ADED. Diabetic maculopathy was defined by the presence of exudate in the fovea within one optic disc diameter, exudates in the macula, or microaneurysms/hemorrhages within the optic one disc diameter of the macula [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e].\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eThe Definition of Diabetic Retinopathy Stages\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eStages of DR\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eICDR Scale\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eFindings\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNo retinopathy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eNo abnormalities\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMild NPDR\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMicroaneurysms only\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModerate NPDR\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMore than just microaneurysms but less than severe NPDR\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSevere NPDR\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAny of the following:\u003c/p\u003e \u003cp\u003e- exceed 20 intraretinal hemorrhages in each of the four quadrants;\u003c/p\u003e \u003cp\u003e- presence of definite venous beading in two quadrants;\u003c/p\u003e \u003cp\u003e- presence of intraretinal microvascular abnormalities in at least one quadrant\u003c/p\u003e \u003cp\u003ewith no evidence of proliferative retinopathy\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePDR\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eOne or more of the following:\u003c/p\u003e \u003cp\u003e- neovascularization\u003c/p\u003e \u003cp\u003e- vitreous/preretinal hemorrhage\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"3\"\u003eNote: Table adopted from Wilkinson et al.(2003)[\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. DR\u0026thinsp;=\u0026thinsp;diabetic retinopathy, NPDR\u0026thinsp;=\u0026thinsp;non-proliferative diabetic retinopathy, PDR\u0026thinsp;=\u0026thinsp;proliferative diabetic retinopathy, ICDR\u0026thinsp;=\u0026thinsp;International Clinical Diabetic Retinopathy.\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e\n\u003ch3\u003eSampling\u003c/h3\u003e\n\u003cp\u003eThis study aims to assess whether LLMs are sufficiently sensitive to serve as screening tools for detecting DR. The study\u0026rsquo;s null hypothesis specifies that the LLM is not sensitive to detect DR, with sensitivity set at 50%, while the alternative hypothesis expects a higher sensitivity threshold, set at a minimum of 70% [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. According to a Power Analysis and Sample Size table developed by Bujang et al., if the prevalence of DR is estimated at 30% [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e], a sample size of 163 subjects is necessary to achieve an 80% power to detect an increase in screening sensitivity from 50\u0026ndash;70%, assuming a target significance level of 0.05 [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. An additional 20% sample size was included to account for excluded images that may not achieve complete annotation agreement among the ophthalmologists, making the estimated minimum sample size to be 200 fundus images.\u003c/p\u003e \u003cp\u003eBetween January 1, 2018, and December 31, 2023, there were 963 patients met the study's eligibility criteria. A stratified random sampling technique selected 250 patients (exceeding the estimated sample size). This process involved assigning a unique research identifier to each patient and using a computer-generated random number sequence (Microsoft Excel 2016, Version 2205) to randomly select the participants. Random stratified sampling is important as it ensures that all study groups (DR vs. non-DR) are adequately represented, minimizing selection bias.\u003c/p\u003e\n\u003ch3\u003eData Collection and Data Annotations\u003c/h3\u003e\n\u003cp\u003eThe sampled fundus images and demographic details were downloaded onto the clinic\u0026rsquo;s computer and the structured data were extracted into a spreadsheet. The fundus images were stored in their original JPG format (no resizing or editing) for data annotation by an ophthalmologist and for testing the multimodal LLM. Two experienced ophthalmologists independently reviewed the fundus images for the presence or absence of DR, the stage of retinopathy, and the presence or absence of maculopathy, without access to each other\u0026rsquo;s assessments or the patients\u0026rsquo; information. The assessments were guided by the ICDR recommendations. They reached an agreement on 91% (228/250) of the images, and the agreed images were then used for the external validation of the multimodal LLM. The complete agreement among the ophthalmologists on the presence of DR, stages of DR, and the presence of maculopathy was used as the benchmark (reference standard) for defining study outcomes.\u003c/p\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eTesting the Multimodal LLM\u003c/h2\u003e \u003cp\u003eWe tested a variety of LLMs (index tests) using the following website platforms\u003c/p\u003e \u003cp\u003e \u003col style=\"list-style-type:lower-alpha;\"\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eOpen AI\u0026rsquo;s GPT-4 (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://openai.com/\u003c/span\u003e\u003cspan address=\"https://openai.com/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e)\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eGoogle\u0026rsquo;s Gemini 1.5 (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://aistudio.google.com\u003c/span\u003e\u003cspan address=\"https://aistudio.google.com\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e)\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eAnthropic Claude 3 Haiku (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://poe.com/Claude-3-Haiku\u003c/span\u003e\u003cspan address=\"https://poe.com/Claude-3-Haiku\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e)\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eMistral\u0026rsquo;s Large (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://chat.mistral.ai/login%20\u003c/span\u003e\u003cspan address=\"https://chat.mistral.ai/login%20\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e)\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e \u003c/p\u003e \u003cp\u003eFundus images were uploaded to the chat interface with the research standardized text prompt: \u0026ldquo;Act as an ophthalmologist. Your task is to provide a report on fundus images of patients with diabetes mellitus captured using a fundus camera during retinopathy screening. Please report with the following order. Firstly, describe the findings observed in the fundus images. Secondly, comment on the presence or absence of retinopathy. Thirdly, if retinopathy is present, provide comments on the stages of retinopathy, where options include mild non-proliferative retinopathy, moderate non-proliferative retinopathy, severe non-proliferative retinopathy, proliferative retinopathy, or advanced diabetic eye disease.\u0026rdquo; Next, researchers reviewed the LLM\u0026rsquo;s responses to identify the presence of DR, the stages of DR (rated using the ICDR scale), and any findings of diabetic maculopathy. These findings were documented in a spreadsheet and validated by an ophthalmology researcher. Figure\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e1\u003c/span\u003e illustrates the Open AI\u0026rsquo;s ChatGPT LLM interface with the study prompt, uploaded fundus image, and the AI-generated response.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003eModel Evaluation\u003c/h3\u003e\n\u003cp\u003eThe LLM's performance was assessed by constructing a confusion matrix for each study outcome and comparing the LLM's classification results with the ophthalmologists' actual labels. Performance metrics such as the area under the receiver operating characteristic curve (AUROC), sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), F1 score, and accuracy were calculated. The specific formulas used to calculate these metrics are provided in Supplementary Table\u0026nbsp;1. The number of failed and ambiguous responses from GPT-4 was also recorded. A failed response was noted when the LLM refused to reply to the text prompt. Ambiguous responses were said to occur when the LLM response could not precisely determine the stage of DR, providing a range (e.g., mild to moderate DR) instead of a specific stage.\u003c/p\u003e\n\u003ch3\u003eData Management and Analysis\u003c/h3\u003e\n\u003cp\u003eBefore analysis, the structured dataset was reviewed, and no typographical errors, outliers, or missing values were found. The age variable distribution was non-normal (Kolmogorov\u0026ndash;Smirnov test p-value\u0026thinsp;\u0026lt;\u0026thinsp;0.05), and thus reported using median and interquartile ranges. Categorical data (ethnicity and sex) were reported as frequencies and percentages. Data analyses were performed using the Statistical Package for the Social Sciences (SPSS version 26.0). The calculation of performance metrics was performed using the Python library sklearn.metrics (sklearn version: 1.3.0).\u003c/p\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003eEthical Considerations\u003c/h2\u003e \u003cp\u003e This study was conducted in accordance with the Declaration of Helsinki and received approval from the Malaysia Medical Research Ethics Committee (NMRR ID-24-00315-WXU). The need for informed consent was waived for this retrospective study.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"RESULTS","content":"\u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003eStudy Participation and Demographic Details\u003c/h2\u003e \u003cp\u003eOf the 250 fundus images sampled, 228 (91%) achieved complete agreement between the two ophthalmology researchers on the annotations for DR, DR stages, and maculopathy (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e2\u003c/span\u003e). The study cohort (n\u0026thinsp;=\u0026thinsp;228) had a median age of 57.0 years (IQR 40.8\u0026ndash;65.0), with a slight female predominance at 54.4%. The ethnicity distribution of the cohort was predominantly Malay 168 (73.7%), followed by Chinese 32 (14.0%) and Indian 28 (12.3%), showing a similar proportion of main ethnicities in Malaysia (see Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). Half of the patients (n\u0026thinsp;=\u0026thinsp;122, 53.5%) had DR, with the distribution of DR stages illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e3\u003c/span\u003e. Among those with DR, 82% had referable DR (100 out of 122 fundus images indicated severity beyond minimal NPDR) and 80 (35.1%) out of 228 images with evidence of maculopathy\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eDemographic Characteristics of Study Participants\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDemographic Details\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eResult (n\u0026thinsp;=\u0026thinsp;228)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAge (years), median (IQR)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e57.0 (40.8\u0026ndash;65.0)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003eSex\u003c/span\u003e\u003c/p\u003e \u003cp\u003eMale\u003c/p\u003e \u003cp\u003eFemale\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e104 (45.6%)\u003c/p\u003e \u003cp\u003e124 (54.4%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003eEthnicity\u003c/span\u003e\u003c/p\u003e \u003cp\u003eMalay\u003c/p\u003e \u003cp\u003eChinese\u003c/p\u003e \u003cp\u003eIndian\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e168 (73.7%)\u003c/p\u003e \u003cp\u003e32 (14.0%)\u003c/p\u003e \u003cp\u003e28 (12.3%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003eStages of DR\u003c/span\u003e\u003c/p\u003e \u003cp\u003eNo retinopathy\u003c/p\u003e \u003cp\u003eRetinopathy\u003c/p\u003e \u003cp\u003e- Non-referable DR\u003c/p\u003e \u003cp\u003e- Referable DR\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e106 (46.5%)\u003c/p\u003e \u003cp\u003e122 (53.5%)\u003c/p\u003e \u003cp\u003e22 out of 122 (18.0%)\u003c/p\u003e \u003cp\u003e100 out of 122 (82.0%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"2\"\u003e\u003cem\u003eNote\u003c/em\u003e: IQR\u0026thinsp;=\u0026thinsp;interquartile range, DR\u0026thinsp;=\u0026thinsp;diabetic retinopathy\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003eValidation of Large Language Models for Diabetic Retinopathy Detection\u003c/h2\u003e \u003cp\u003eThe GPT-4 model demonstrated the highest performance in detecting DR, with an AUROC of 0.81. It achieved the highest sensitivity at 82% and a precision (PPV) of 82% in accurately classifying the presence of DR. In contrast, the performance of other models, including Gemini 1.5, Claude3 Haiku, and Mistral Large, was markedly lower, with AUROC values ranging from 0.50 to 0.54 (see Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003ePerformance of Large Language Models for Detection of Diabetic Retinopathy\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eOpen AI\u0026rsquo;s GPT-4\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eGoogle Gemini 1.5\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eClaude 3 Haiku\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eMistral Large\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAUROC\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.806\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.539\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.540\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.500\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSensitivity\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.820\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.984\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.984\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSpecificity\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.793\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.094\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.094\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePPV\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.820\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.556\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.556\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.535\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNPV\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.792\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.833\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.833\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.807\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.570\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.570\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.535\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eF1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.820\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.710\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.710\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.697\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"5\"\u003eNote: AUROC\u0026thinsp;=\u0026thinsp;area under the receiver operating characteristic curve, PPV\u0026thinsp;=\u0026thinsp;positive predictive value, NPV\u0026thinsp;=\u0026thinsp;negative predictive value\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003eValidation of Large Language Models for Staging Diabetic Retinopathy\u003c/h2\u003e \u003cp\u003eGemini 1.5 stood out as being best in identifying moderate NPDR, achieving an AUROC of 0.81 and a sensitivity of 83%. However, its precision was low, with a PPV of 31%. All LLMs demonstrated poor performance in detecting mild NPDR, severe NPDR, and PDR (see Supplementary Table\u0026nbsp;2\u0026ndash;5).\u003c/p\u003e \u003cp\u003eIdentifying referable DR is crucial in primary care, as any stage beyond mild NPDR requires referral to an ophthalmologist for detailed assessment and prompt intervention to prevent blindness. The GPT-4 model performed the best in classifying referable DR, with an AUROC of 0.80 compared to other models. The GPT-4 achieved a sensitivity of 80% and a precision (PPV) of 76% in identifying referable DR (see Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003ePerformance of Large Language Models for Classifying Referrable DR\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eOpen AI\u0026rsquo;s GPT-4\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eGoogle Gemini 1.5\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eClaude 3 Haiku\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eMistral\u0026rsquo;s AI\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAUROC\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.802\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.793\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.558\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.505\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSensitivity\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.800\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.680\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.960\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.620\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSpecificity\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.805\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.906\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.156\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.391\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePPV\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.762\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.850\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.471\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.443\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNPV\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.837\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.784\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.833\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.568\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.803\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.807\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.509\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.491\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eF1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.781\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.756\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.632\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.517\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"5\"\u003eNote: DR\u0026thinsp;=\u0026thinsp;diabetic retinopathy, AUROC\u0026thinsp;=\u0026thinsp;area under the receiver operating characteristic curve, PPV\u0026thinsp;=\u0026thinsp;positive predictive value, NPV\u0026thinsp;=\u0026thinsp;negative predictive value\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003eValidation of Large Language Models for Maculopathy Detection\u003c/h2\u003e \u003cp\u003eOverall, all LLMs showed low performance in detecting maculopathy, with AUROC values ranging from 0.49 to 0.65 (see Table\u0026nbsp;\u003cspan refid=\"Tab5\" class=\"InternalRef\"\u003e5\u003c/span\u003e). Although Gemini 1.5 demonstrated high sensitivity (83%) in identifying maculopathy, its precision was low, with a PPV of only 37%.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab5\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 5\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003ePerformance of Large Language Models for Detection of Diabetic Maculopathy\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eOpen AI\u0026rsquo;s GPT-4\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eGoogle Gemini 1.5\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eClaude 3 Haiku\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eMistral\u0026rsquo;s AI\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAUROC\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.645\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.527\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.503\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.491\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSensitivity\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.425\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.825\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.100\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.400\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSpecificity\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.865\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.230\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.905\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.581\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePPV\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.630\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.367\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.364\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.340\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNPV\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.736\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.708\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.650\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.642\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.711\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.439\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.623\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.518\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eF1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.507\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.508\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.157\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.368\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"5\"\u003eNote: AUROC\u0026thinsp;=\u0026thinsp;area under the receiver operating characteristic curve, PPV\u0026thinsp;=\u0026thinsp;positive predictive value, NPV\u0026thinsp;=\u0026thinsp;negative predictive value\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec17\" class=\"Section2\"\u003e \u003ch2\u003eLimitations of Large Language Models\u003c/h2\u003e \u003cp\u003eTo recap, an LLM was considered to have failed in responding when it refused to answer the text prompt (refused to respond to a medical issue). The GPT-4 had the highest failure rate at 8%, followed by Gemini 1.5 at 3%, and Mistral Large at 1%. The Claude3 Haiku model responded coherently to all text prompts during validation. Additionally, GPT-4 had the highest rate (13%) of requiring text re-prompting to clarify the output generated, due to frequently providing ambiguous answers, followed by Claude3 Haiku at 9%. In contrast, Gemini 1.5 \u0026amp; Mistral AI provided clear responses 100% of the time (Table\u0026nbsp;\u003cspan refid=\"Tab6\" class=\"InternalRef\"\u003e6\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab6\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 6\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eLimitations of Large Language Models\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eOpen AI\u0026rsquo;s\u003c/p\u003e \u003cp\u003eGPT-4\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eGoogle\u003c/p\u003e \u003cp\u003eGemini 1.5\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eClaude 3\u003c/p\u003e \u003cp\u003eHaiku\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eMistral\u0026rsquo;s AI\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFailure to Response\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e19 (8.3%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e6 (2.6%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e2 (0.9%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAmbiguous Response\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e30 (13.1%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e20 (8.8%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"DISCUSSION","content":"\u003cp\u003eTo recap, our study aimed to validate the multimodal LLMs for the performance of detecting DR, categorizing DR stages, and identifying diabetic maculopathy. Our study demonstrated that the GPT-4 exhibited good performance in detecting the presence of DR with a sensitivity of 82% and referable DR with a sensitivity of 80%, meeting the UK National Institute for Clinical Excellence (NICE) criteria for DR screening tools (\u0026gt;\u0026thinsp;80% sensitivity). However, all LLMs' effectiveness in staging DR, as well as detecting diabetic maculopathy, was found to be inadequate.\u003c/p\u003e \u003cp\u003eAdditionally, our study showed that GPT-4 has the highest chance among all LLMs to refuse answering medical-related prompts, followed by Gemini 1.5. The LLMs may refuse to respond to medical queries due to safety concerns. These generalist models are not specifically trained for healthcare applications, as their training focuses on general NLP tasks rather than specialized medical knowledge which can lead to the generation of incorrect information known as hallucinations. To mitigate the risk of harm, developers often impose restrictions on these models to prevent them from providing medical advice or diagnoses. This is done through various techniques, including fine-tuning the model with instructions to avoid engaging in medical queries or incorporating filters that detect and block responses to certain health-related topics [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]. Besides, these LLMs may be trained to encourage users to consult qualified medical professionals rather than relying on AI responses.\u003c/p\u003e \u003cp\u003eOur study is comparable to the research conducted by Xu et al., which also examined the use of GPT-4 with fundus images. They reported that GPT-4 achieved a correct response rate of only 13.7%, although their investigation was not exclusively focused on DR [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. Hence, the direct comparison was challenging due to the unspecified proportion of DR in their study sample. Next, Raghu et al. assessed GPT-4's ability to predict DR using central retinal thickness measurements obtained with OCT, reporting a sensitivity of 67% and specificity of 68% [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e]. Antaki et al. also evaluated the Gemini LLM using OCT images, revealing its limited ability to detect macular disease features, with an average F1 score of just 10.7% [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. However, these results were based on OCT image validation and not specifically on fundus images from diabetic patients. Despite using resource-intensive OCT measurements, the LLMs' performance in detecting abnormal ophthalmological features was suboptimal. All studies highlight the challenge of using generalized AI models in specialized clinical contexts.\u003c/p\u003e \u003cp\u003eOur validation study showcase several strengths. Firstly, we validated LLMs using a well-generalized multi-racial dataset, providing a comprehensive range of fundus images representative of Malaysia. The use of an external validation design enhances the applicability of results across diverse patient populations. Additionally, employing stratified random sampling minimized selection bias, ensuring the generalizability of results. The study includes multiple LLMs, such as GPT-4, Gemini 1.5, Claude 3, and Mistral AI, allowing for a comprehensive comparison of their performance in detection and staging. Besides, the utilization of detailed performance metrics (sensitivity, specificity, PPV, NPV, F1 score, and accuracy) provides a thorough evaluation of each model's efficacy. Our data annotations and AI output verifications by researchers were robust. Fundus images were labeled independently by two experienced ophthalmologists following international standards (Proposed International Clinical Diabetic Retinopathy and Diabetic Macular Edema Disease Severity Scales). The AI output responses were also verified by two researchers independently before data analysis, ensuring reliable data annotations and interpretation of AI outputs. Moreover, we shared all fundus images and the Python script to obtain the performance metrics in an open data repository and ensure reporting transparency.\u003c/p\u003e \u003cp\u003e Additionally, our studies also focus on practical implementation aspects of the AI which aligns with responsible AI principles (fairness, transparency, explainability, data privacy), ensuring the ethical application of the findings in clinical settings. We conducted a subgroup analysis examining GPT-4's DR detection across different ethnicities. We observed a decrease in PPV within the Chinese subgroup, likely due to the imbalanced ethnic representation during the external validation (Supplementary Table\u0026nbsp;6). Nonetheless, the overall performance was comparable across all ethnicities (AUROC of DR detection: 0.74\u0026ndash;0.82), suggesting minimal ethnic bias when using GPT-4. Concerning AI transparency, all proprietary LLM models tested in our study is lacking of clear transparency regarding data quantity and quality, hyperparameters, and the specific transformer architecture used. Additionally, AI explainability is essential in healthcare AI applications. All LLMs, with their complex transformer architecture, are inherently less interpretable than traditional machine learning algorithms. This complexity makes the reasoning behind LLM outputs difficult to understand, making them largely black-box systems. As a result, LLMs may not be the most suitable AI solution for healthcare facilities that prioritize transparency and result explainability. A discussion on responsible AI is incomplete without addressing data privacy. ChatGPT Plus (GPT-4 model) and Claude 3 both offer users the ability to opt out of data sharing for future model training. This feature is particularly important as it ensures patients\u0026rsquo; data remains private and is not utilized for further training of the AI models. Conversely, AI Studio (Gemini 1.5 model) and Mistral AI (Mistral Large model) provide the option to disable data sharing only for subscription-based users. This limitation means that users who do not subscribe to these platforms may have their uploaded data included in future model training processes. This distinction highlights the approaches to patients\u0026rsquo; data privacy across different AI systems, which is a crucial consideration for healthcare AI applications while maintaining strict data privacy standards.\u003c/p\u003e \u003cp\u003eThis study has limitations. Image annotations (presence of DR, stages of DR, and maculopathy) were determined by independent agreement among ophthalmologists rather than using gold-standard diagnostic tools like OCT and FFA. We do not have such data due to the limited clinical application of these advanced tools in the study site and the major ophthalmology centers in Malaysia. These methods are only viable as gold standards in prospective research. Despite these limitations, the fundus images labeled by ophthalmologists provide a broad spectrum of DR stages (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e3\u003c/span\u003e) for external validation of LLMs. Besides, the small sample sizes for subgroup analyses present a challenge in validating AI performance for specific ethnicities. While over 200 fundus images were collected for validation, the representation of minor ethnic groups such as Chinese and Indian was relatively small compared to the Malay group (see Supplementary Table\u0026nbsp;6). Future studies should aim for a balanced proportional representation of all major ethnic groups using stratified sampling methods to enhance generalizability.\u003c/p\u003e \u003cp\u003eMoving beyond clinical validation, integrating GPT-4 as an AI system for DR screening involves several steps. We propose the following plan for primary care centers looking to incorporate GPT-4 into their DR screening program, specifically to identify DR and classify referable DR. The process involves a comprehensive pipeline from image capture to result reporting. The initial phase of the screening process involves the use of the Zeiss Visucam 500 fundus camera, interfaced with the Zeiss Forum 4.4 system. Fundus images are captured in JPG format and stored on an on-premise server, ensuring immediate availability and data security. The core component of the pipeline is the integration of the GPT-4 model for the analysis of fundus images. The GPT-4 application programming interface (API) requires a secure environment which is Health Insurance Portability and Accountability Act (HIPAA) compliant for processing the medical images. The process begins with a programmed script that retrieves stored images and sends them to the GPT-4 API. The model analyzes each image, generating a detailed diagnostic report that includes the presence of DR and type of DR with recommendations for further action. The AI-generated results are then formatted into a structured report. This report includes fundus image findings, stages of DR, and clinical recommendations, ensuring that healthcare professionals can easily interpret the results. These reports are stored securely on the on-premise server and made accessible through the Zeiss Forum 4.4 system. To ensure the system's ongoing effectiveness, a feedback loop is established. The referral-receiving ophthalmologist reviews the AI-generated reports, providing feedback that is used to continuously update and retrain the model. This iterative process is essential for maintaining high diagnostic accuracy.\u003c/p\u003e \u003cp\u003eEncryption and secured access protocols are needed to safeguard sensitive patient data and ensure compliance with regulations. Compliance with data privacy regulations, such as HIPAA, is a priority throughout the implementation. Regular audits and stringent security measures are in place to protect patient data and ensure the system's integrity. By implementing this automated workflow (see Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e4\u003c/span\u003e), the clinic leverages AI technology to enhance the efficiency and accuracy of DR screening. This integration not only reduces the workload on healthcare professionals but also ensures timely and accurate diagnoses, ultimately improving patient outcomes. As AI technology advances, the AI pipeline must stay adaptive to incorporate newer algorithms once they are clinically validated for better DR classification tasks. It is recommended that healthcare facilities to wait for the development of more advanced and precise AI models if their primary focus is on the detailed classification of DR stages.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eIn conclusion, this study validates GPT-4's potential as a multimodal LLM for detecting the presence of DR and classifying referable DR. It is among the few external clinical validations assessing GPT-4's performance in DR classification using fundus images. Integrating AI into DR screening in primary care has the potential to improve screening efficiency for this high-burden non-communicable disease in Malaysia. The GPT-4's ability to identify the presence of DR and referable DR could streamline the screening process, facilitating expedited referrals to ophthalmologists for detailed assessment and intervention, crucial in preventing vision loss. Given the current limitations, the use of AI in DR screening necessitates cautious implementation, and it should complement rather than replace conventional clinical pathways and consultations.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003eData Availability:\u003c/p\u003e\n\u003cp\u003eAll fundus images used for validating the LLMs are available at the following link: doi:10.6084/m9.figshare.28057571\u003c/p\u003e\n\u003cp\u003eCode Availability:\u003c/p\u003e\n\u003cp\u003eThe codes used for the analysis in this study are available at https://github.com/chuinhen/ResearchProject_DR/blob/main/codes/codes.ipynb\u003c/p\u003e\n\u003cp\u003eAcknowledgments:\u003c/p\u003e\n\u003cp\u003eWe thank the Director-General of Health Malaysia for giving permission to publish this article.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAuthor Contributions:\u003c/p\u003e\n\u003cp\u003eCHL contributed to the conception and design of the study. NIMS, SS, AHS, and SYT made substantial contributions to data collection. CHL performed data analysis. CHL, EXT, QZN, SM, and KSM contributed to data interpretation. CHL wrote the first draft of the manuscript. All authors revised the manuscript for important intellectual content, approved the final version to be published, and agreed to be accountable for all aspects of their work.\u003c/p\u003e\n\u003cp\u003eCompeting Interest:\u003c/p\u003e\n\u003cp\u003eThe authors have no competing interests to disclose.\u003c/p\u003e\n\u003cp\u003eFunding:\u003c/p\u003e\n\u003cp\u003eThis study received no funding\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eChew, F. L. M. \u003cem\u003eet al.\u003c/em\u003e Estimates of visual impairment and its causes from the National Eye Survey in Malaysia (NESII). PLOS ONE 13, e0198799, doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1371/journal.pone.0198799\u003c/span\u003e\u003cspan address=\"10.1371/journal.pone.0198799\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2018).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFong, D. S. \u003cem\u003eet al.\u003c/em\u003e Retinopathy in diabetes. Diabetes care 27, s84-s87 (2004).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWilkinson, C. P. \u003cem\u003eet al.\u003c/em\u003e Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology 110, 1677\u0026ndash;1682, doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/S0161-6420(03)00475-5\u003c/span\u003e\u003cspan address=\"10.1016/S0161-6420(03)00475-5\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2003).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSaeedi, P. \u003cem\u003eet al.\u003c/em\u003e Global and regional diabetes prevalence estimates for 2019 and projections for 2030 and 2045: Results from the International Diabetes Federation Diabetes Atlas. Diabetes research and clinical practice 157, 107843 (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSolomon, S. D. \u003cem\u003eet al.\u003c/em\u003e Diabetic Retinopathy: A Position Statement by the American Diabetes Association. Diabetes Care 40, 412\u0026ndash;418, doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.2337/dc16-2641\u003c/span\u003e\u003cspan address=\"10.2337/dc16-2641\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2017).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAbougalambou, S. S. I. \u0026amp; Abougalambou, A. S. Risk factors associated with diabetic retinopathy among type 2 diabetes patients at teaching hospital in Malaysia. Diabetes \u0026amp; Metabolic Syndrome: Clinical Research \u0026amp; Reviews 9, 98\u0026ndash;103, doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.dsx.2014.04.019\u003c/span\u003e\u003cspan address=\"10.1016/j.dsx.2014.04.019\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2015).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLim, A. K. E. \u0026amp; Khaw, K. W. Audit of diabetic retinopathy referrals to Penang Hospital, a tertiary Ophthalmology centre in Malaysia. Asian J Ophthalmol 11, 17\u0026ndash;19 (2009).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBenet, D. \u0026amp; Pellicer-Valero, O. J. Artificial intelligence: the unstoppable revolution in ophthalmology. survey of ophthalmology 67, 252\u0026ndash;270 (2022).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAbr\u0026agrave;moff, M. D., Lavin, P. T., Birch, M., Shah, N. \u0026amp; Folk, J. C. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. npj Digital Medicine 1, 39, doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1038/s41746-018-0040-6\u003c/span\u003e\u003cspan address=\"10.1038/s41746-018-0040-6\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2018).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen, E. M., Chen, D., Chilakamarri, P., Lopez, R. \u0026amp; Parikh, R. Economic challenges of artificial intelligence adoption for diabetic retinopathy. Ophthalmology 128, 475\u0026ndash;477 (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLin, J. C., Younessi, D. N., Kurapati, S. S., Tang, O. Y. \u0026amp; Scott, I. U. Comparison of GPT-3.5, GPT-4, and human user performance on a practice ophthalmology written examination. Eye 37, 3694\u0026ndash;3695, doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1038/s41433-023-02564-2\u003c/span\u003e\u003cspan address=\"10.1038/s41433-023-02564-2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTaloni, A. \u003cem\u003eet al.\u003c/em\u003e Comparative performance of humans versus GPT-4.0 and GPT-3.5 in the self-assessment program of American Academy of Ophthalmology. Scientific Reports 13, 18562, doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1038/s41598-023-45837-2\u003c/span\u003e\u003cspan address=\"10.1038/s41598-023-45837-2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGhasemi Falavarjani, K. \u0026amp; Nguyen, Q. Adverse events and complications associated with intravitreal injection of anti-VEGF agents: a review of literature. Eye 27, 787\u0026ndash;794 (2013).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePeto, T. \u0026amp; Tadros, C. Screening for Diabetic Retinopathy and Diabetic Macular Edema in the United Kingdom. Current Diabetes Reports 12, 338\u0026ndash;345, doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/s11892-012-0285-4\u003c/span\u003e\u003cspan address=\"10.1007/s11892-012-0285-4\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2012).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBujang, M. A. \u0026amp; Adnan, T. H. Requirements for Minimum Sample Size for Sensitivity and Specificity Analysis. J Clin Diagn Res 10, Ye01-ye06, doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.7860/jcdr/2016/18129.8744\u003c/span\u003e\u003cspan address=\"10.7860/jcdr/2016/18129.8744\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2016).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eClusmann, J. \u003cem\u003eet al.\u003c/em\u003e The future landscape of large language models in medicine. Communications medicine 3, 141 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eXu, P. \u003cem\u003eet al.\u003c/em\u003e Evaluation of a digital ophthalmologist app built by GPT4-V(ision). \u003cem\u003emedRxiv\u003c/em\u003e, 2023.2011.2027.23299056, doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1101/2023.11.27.23299056\u003c/span\u003e\u003cspan address=\"10.1101/2023.11.27.23299056\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRaghu, K. \u003cem\u003eet al.\u003c/em\u003e The Utility of ChatGPT in Diabetic Retinopathy Risk Assessment: A Comparative Study with Clinical Diagnosis. Clin Ophthalmol 17, 4021\u0026ndash;4031, doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.2147/opth.S435052\u003c/span\u003e\u003cspan address=\"10.2147/opth.S435052\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAntaki, F., Chopra, R. \u0026amp; Keane, P. A. Vision-Language Models for Feature Detection of Macular Diseases on Optical Coherence Tomography. JAMA Ophthalmology 142, 573\u0026ndash;576, doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1001/jamaophthalmol.2024.1165\u003c/span\u003e\u003cspan address=\"10.1001/jamaophthalmol.2024.1165\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2024).\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-5909259/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5909259/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eDiabetic retinopathy (DR) is a leading cause of visual impairment, requiring effective and scalable screening tools for early detection. Existing methods are complex, expensive, and reliant on specialized personnel, limiting their use in primary care. This study evaluates the potential of a multimodal large language model (LLM), for detecting DR, staging DR, and identifying diabetic maculopathy. This external validation study assessed the performance of LLMs using 228 fundus images captured at Tuanku Ampuan Najihah Hospital. Models evaluated include GPT-4, Google\u0026rsquo;s Gemini 1.5, Anthropic Claude 3 Haiku, and Mistral Large. Sensitivity, specificity, and predictive value were assessed, and results were validated with human ophthalmologist evaluations. As a results, GPT-4 achieved good sensitivity for detecting DR (82%) and referable DR (80%), meeting UK NICE criteria. However, all LLMs, including GPT-4, performed poorly in staging DR and detecting diabetic maculopathy. While GPT-4 shows promise in identifying DR, its limitations in detailed DR staging and maculopathy detection highlight cautious implementation.\u003c/p\u003e","manuscriptTitle":"Assessing The Performance of Multimodal Large Language Models in Diagnosing and Staging Diabetic Retinopathy: An External Validation Study of Large Language Models","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-01-31 17:59:54","doi":"10.21203/rs.3.rs-5909259/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"2153916a-2400-4213-aa35-323ebcf49708","owner":[],"postedDate":"January 31st, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":43532284,"name":"Health sciences/Diseases/Eye diseases"},{"id":43532285,"name":"Health sciences/Diseases/Metabolic disorders"},{"id":43532286,"name":"Health sciences/Health care/Diagnosis"},{"id":43532287,"name":"Health sciences/Health care/Health services"},{"id":43532288,"name":"Health sciences/Health care/Medical imaging"}],"tags":[],"updatedAt":"2025-02-06T22:38:22+00:00","versionOfRecord":[],"versionCreatedAt":"2025-01-31 17:59:54","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5909259","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5909259","identity":"rs-5909259","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}