Benchmarking Multimodal Large Language Models for Binary Classification of Pediatric Chest X-Rays: A Comparative Evaluation Using a Public Pneumonia Dataset

Benchmarking Multimodal Large Language Models for Binary Classification of Pediatric Chest X-Rays: A Comparative Evaluation Using a Public Pneumonia Dataset | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Short Report Benchmarking Multimodal Large Language Models for Binary Classification of Pediatric Chest X-Rays: A Comparative Evaluation Using a Public Pneumonia Dataset Nitin Chetla, Shivam Patel, Trisha Naidu, Rahul Kumar, Luis Rodriguez, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7180985/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background/Objective: Pneumonia remains a leading cause of global mortality, with chest X-rays serving as a primary diagnostic tool. While large language models (LLMs) show promise in medical imaging, limited evidence exists regarding their performance in pediatric pneumonia classification. This study evaluates the binary classification performance of five prominent multimodal LLMs in distinguishing bacterial pneumonia from normal findings in pediatric chest X-rays. Methods: We evaluated GPT-4o, GPT-4.1, Gemini 2.5 Pro Preview, Claude 4 Sonnet, and Grok 2 Vision using 1,000 pediatric chest X-ray images (500 normal, 500 bacterial pneumonia) from the Guangzhou Women and Children's Medical Center dataset. Each model received identical binary classification prompts via their respective APIs. Performance was assessed using accuracy, sensitivity, specificity, F1-score, and confusion matrix analysis. Results: GPT-4.1 demonstrated the most balanced performance with 84% sensitivity and 76% specificity. GPT-4o achieved high sensitivity (99%) but poor specificity (18%), while Gemini 2.5 Pro showed similar patterns (97% sensitivity, 25% specificity). Claude 4 Sonnet classified all images as pneumonia (100% sensitivity, 0% specificity). Grok 2 Vision showed moderate performance with 82% sensitivity and 56% specificity. Conclusion: Substantial performance variability exists among LLMs for pediatric pneumonia detection. GPT-4.1 provided optimal clinical utility with balanced sensitivity and specificity, while other models showed concerning tendencies toward false positives. These findings underscore the necessity for rigorous benchmarking before clinical implementation of LLMs in pediatric radiology. Large Language Models Pediatric Chest X-rays Multimodal AI models Clinical Benchmarking Figures Figure 1 Introduction Pneumonia is a leading cause of mortality worldwide [ 5 , 6 ], despite chest X-ray imaging being a readily available and cost-effective diagnostic tool [ 2 , 8 ]. Large language models (LLMs) have shown that they can efficiently support clinician’s diagnostic tasks [ 9 ]. While studies have examined LLM performance for chest X-ray interpretation, there is limited evidence regarding general-purpose LLMs (GPT-4o, Gemini 2.5 Pro, Claude 4 Sonnet, and Grok-2 Vision) in classifying pediatric pneumonia from chest X-rays. LLMs could soon be a secondary screening tool alongside radiologist opinion to rule out normal vs. abnormal findings and ease radiologic workflows [ 10 ]. This study aims to evaluate the binary classification performance of five prominent LLMs in distinguishing bacterial from normal pediatric pneumonia on standardized chest X-rays from a retrospective cohort of pediatric patients from Guangzhou Women and Children’s Medical Center [ 4 ] using uniform prompts and evaluation metrics.. All low-quality or unreadable chest radiographs were removed, and two physicians graded the CXRs, with a third expert responsible for checking the final set [ 4 ]. Methods A publicly available pediatric chest X-ray database from Guangzhou Women and Children’s Medical Center was used for evaluation [ 7 ]. The dataset consisted of 1,000 JPEG images: 500 labeled as “Normal” and 500 as “Bacterial Pneumonia.” All images were used in their native resolution and format, with no compression or preprocessing. Five representative LLMs were evaluated: GPT-4o (May 2024, Open AI), GPT-4.1 (April 2025, OpenAI), Gemini 2.5 Pro Preview (June 2025, Google), Claude 4 Sonnet (May 2025, Anthropic), and Grok 2 Vision (December 2024, xAI). Each image was submitted to all models via their respective APIs: Base64-encoded image inputs for OpenAI models, native image input via Google GenerativeAI API for Gemini 2.5 Pro, Base64 image input with prompt via Anthropic Vision API, and structure prompt and base64-encoded image via xAI API for Grok 2 Vision. The models received the same binary classification prompt: “This is a question from a Board Exam: Analyze the chest X-ray image and determine if the patient has Bacterial Pneumonia or is Normal. Your response must be binary: 0 = Normal, 1 = Bacterial Pneumonia. ONLY output 0 or 1. Example: If pneumonia is present, respond EXACTLY: ‘1’.” GPT-4o and GPT-4.1 were instructed to respond concisely and without disclaimers. Predictions from each model were compared to ground-truth labels as shown in Fig. 1 . Key metrics of accuracy, sensitivity (recall for Bacterial Pneumonia), specificity (recall for Normal), F1 score, and confusion matrices were computed using scikit-learn. Each model’s performance was evaluated on the same image set for direct comparison. To rule out transmission or preprocessing errors, base64-decoded images were visually verified. Claude 4 Sonnet was additionally tested with blank images to confirm appropriate response to unreadable input. Results Below, we present the comparative performance metrics for all five LLMs (Table 1 ). GPT-4o demonstrated high sensitivity for bacterial pneumonia (0.99) but low specificity for normal images (0.18). In contrast, GPT-4.1 achieved a more balanced performance, with a sensitivity of 0.84 and a specificity of 0.76. Gemini 2.5 Pro Preview also showed strong sensitivity for pneumonia (0.97), but its low specificity (0.25) for normal cases led to an accuracy of 0.61 and F1-score of 0.55. Claude 4 Sonnet predicted “bacterial pneumonia” for all images, yielding perfect sensitivity (1.00) but zero specificity (0.00), with both accuracy and F1-score at 0.33. Grok 2 Vision demonstrated moderate performance, with a sensitivity of 0.82, specificity of 0.56, accuracy of 0.69, and F1-score of 0.68. Table 1 Model Performance Summary with 95% Confidence Intervals Model Sensitivity Specificity Accuracy F1-score GPT-4o 0.99 [0.98, 1.00] 0.18 [0.15, 0.22] 0.58 [0.55, 0.62] 0.51 [0.47, 0.54] GPT-4.1 0.84 [0.81, 0.87] 0.76 [0.72, 0.80] 0.80 [0.77, 0.82] 0.80 [0.77, 0.82] Gemini 2.5 Pro Preview 0.97 [0.95, 0.98] 0.25 [0.21, 0.29] 0.61 [0.58, 0.64] 0.55 [0.52, 0.58] Claude Sonnet 4 1.00 [1.00, 1.00] 0.00 [0.00, 0.00] 0.50 [0.47, 0.53] 0.33 [0.30, 0.36] Grok Vision 2 0.82 [0.78, 0.85] 0.56 [0.51, 0.60] 0.69 [0.66, 0.71] 0.68 [0.65, 0.71] Notably, Claude 4 Sonnet failed to predict any cases as normal, despite passing image integrity and blank-image tests. GPT-4.1 provided the most balanced results, while GPT-4o and Gemini 2.5 Pro were heavily skewed toward positive (pneumonia) predictions. Discussion Our evaluation revealed substantial performance variability. GPT-4.1 achieved the most balanced sensitivity and specificity, thus the best clinical utility among the tested models. In contrast, GPT-4o and Gemini 2.5 Pro demonstrated high sensitivity but very poor specificity, likely leading to false positives. Claude 4 Sonnet classified all images as pneumonia despite correct handling of blank images, indicating a possible prompt or model misalignment. Grok 2 Vision performed moderately, with reasonable sensitivity and specificity but lower accuracy than GPT-4.1. These findings highlight the necessity of rigorous benchmarking of LLMs before clinical integration. The tendency of several models to overcall pathology may stem from training data imbalance, limited domain adaptation, or prompt sensitivity, underscoring the importance of careful prompt engineering and model selection. Limitations include the use of a single public dataset and binary task; future work should assess model generalizability across other conditions, modalities, and real-world clinical settings. Declarations I. Funding: Not applicable. No funding was received for this study. II. Conflicts of Interest: The authors declare no conflicts of interest or competing interests. III. Ethics Approval: This study used a publicly available, de-identified dataset that is HIPAA compliant and open source. Institutional review board (IRB) approval was not required. IV. Consent to participate: N/A V. Consent for publication: N/A VI. Availability of data and material: The dataset used in this study is publicly available at: https://www.kaggle.com/datasets/paultimothymooney/chest-xray-pneumonia VII. Code Availability: All analysis scripts and code used in this study are available upon request. VIII. Authors Contributions: Nitin Chetla conceived the project and led manuscript writing. Shivam Patel assisted with code execution and data handling. Trisha Naidu performed model testing and results compilation. Rahul Kumar and Luis Rodriguez contributed to experimental design and manuscript revision. Andrew Lee and Vinisha Bonagiri provided radiology and clinical oversight. All authors reviewed and approved the final manuscript. References Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M. R., Braren, R., Kaissis, G., & Rueckert, D. (2024). Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature Medicine, 30(9), 2613. https://doi.org/10.1038/s41591-024-03097-1 Ibrahim, D. M., Elshennawy, N. M., & Sarhan, A. (2021). Deep-chest: Multi-classification deep learning model for diagnosing COVID-19, pneumonia, and lung cancer chest diseases. Computers in Biology and Medicine, 132, 104348. https://doi.org/10.1016/j.compbiomed.2021.104348 Karabacak, M., & Margetis, K. (2023). Embracing Large Language Models for Medical Applications: Opportunities and Challenges. In Cureus. Cureus, Inc. https://doi.org/10.7759/cureus.39305 Kermany, D. (2018). Labeled Optical Coherence Tomography (OCT) and Chest X-Ray Images for Classification [Data set]. Mendeley. https://doi.org/10.17632/RSCBJBR9SJ.2 Lanks, C. W., Musani, A. I., & Hsia, D. W. (2019). Community-acquired pneumonia and hospital-acquired pneumonia. Medical Clinics of North America, 103, 487. https://doi.org/https://doi.org/10.1016/j.mcna.2018.12.008 Leung, A. K. C., Wong, A. H. C., & Hon, K. E. (2018). Community-Acquired Pneumonia in Children [Review of Community-Acquired Pneumonia in Children]. Recent Patents on Inflammation & Allergy Drug Discovery, 12(2), 136. Bentham Science Publishers. https://doi.org/10.2174/1872213x12666180621163821 Sakai, H., & Lam, S. S. (2025). Large Language Models for Healthcare Text Classification: A Systematic Review. https://doi.org/10.48550/ARXIV.2503.01159 Toro, C. A. O., García‐Pedrero, Á., Lillo‐Saavedra, M., & Gonzalo‐Martín, C. (2022). Automatic detection of pneumonia in chest X-ray images using textural features. Computers in Biology and Medicine, 145, 105466. https://doi.org/10.1016/j.compbiomed.2022.105466 Zhou, S., Xu, Z., Zhang, M., Xu, C., Guo, Y., Zhan, Z., Ding, S., Wang, J., Xu, K., Fang, Y., Xia, L., Yeung, J., Zha, D., Melton, G. B., Lin, M., & Zhang, R. (2024). Large Language Models for Disease Diagnosis: A Scoping Review [Review of Large Language Models for Disease Diagnosis: A Scoping Review]. arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.2409.00097 Zoga, A. C., & Syed, A. (2018). Artificial Intelligence in Radiology: Current Technology and Future Directions [Review of Artificial Intelligence in Radiology: Current Technology and Future Directions]. Seminars in Musculoskeletal Radiology, 22(5), 540. Thieme Medical Publishers (Germany). https://doi.org/10.1055/s-0038-1673383 Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7180985","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Short Report","associatedPublications":[],"authors":[{"id":490349345,"identity":"5598acbe-2159-44f6-8f49-400d95f8d8ea","order_by":0,"name":"Nitin Chetla","email":"","orcid":"","institution":"University of Virginia School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Nitin","middleName":"","lastName":"Chetla","suffix":""},{"id":490349346,"identity":"e188de14-3ea2-4dcd-9e6c-4928f7adb53c","order_by":1,"name":"Shivam Patel","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Shivam","middleName":"","lastName":"Patel","suffix":""},{"id":490349347,"identity":"3b2c2a54-2944-4028-b317-8242373c585d","order_by":2,"name":"Trisha Naidu","email":"","orcid":"","institution":"Virginia Tech","correspondingAuthor":false,"prefix":"","firstName":"Trisha","middleName":"","lastName":"Naidu","suffix":""},{"id":490349348,"identity":"528ee8f6-addb-4fbf-bd59-51058f5fa4b1","order_by":3,"name":"Rahul Kumar","email":"","orcid":"","institution":"UMass Amherst Medical School","correspondingAuthor":false,"prefix":"","firstName":"Rahul","middleName":"","lastName":"Kumar","suffix":""},{"id":490349349,"identity":"8959f590-e16d-4b11-86a9-71d234268978","order_by":4,"name":"Luis Rodriguez","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABMklEQVRIie2RvWrDMBRGJQTK4lqrTIa8gkzAxSRtXiXGUC8NGbp4VDCoi9uszdQX6NIlZLTxKugDeGno2sGmQ5wSaJUfiCF25kJ9Bl3xcc+9CAHQ0PAH0QHkqiCAjxmK3zNVcKtawRUKds2nbUF1yn5wOdOstnYSlhTqBl++3xvr5MVcrhf9MSERa/e/5x0dAZjltxWKIwwpPVvQzy4z5I0949GwO3pMTYEAMmbzSgVORMIwlRY11YXFPHJHYQqVgtFFpRLkB+WycMQPGySQJ3aYDs4o3NgpJLRALCLGMIIBKFKnVtGWwuDSU1vwnTGRLqNSOQ88dYVSq95CWt5Hzv0e60yT13y9uGZk+rYqik169XwfxFl+qhyhQ7D/1i1Q7E5e13zYF5V7NuebGxoaGv4VvxdaZ+70ZLTYAAAAAElFTkSuQmCC","orcid":"","institution":"Johns Hopkins University School of Medicine","correspondingAuthor":true,"prefix":"","firstName":"Luis","middleName":"","lastName":"Rodriguez","suffix":""},{"id":490349350,"identity":"6b27b393-b78d-4ec0-9aed-3eadeabf52fe","order_by":5,"name":"Andrew Lee","email":"","orcid":"","institution":"St. Luke's Hospital","correspondingAuthor":false,"prefix":"","firstName":"Andrew","middleName":"","lastName":"Lee","suffix":""},{"id":490349351,"identity":"5daafba0-1fc2-4656-a31f-869eb8db0675","order_by":6,"name":"Vinisha Bonagiri","email":"","orcid":"","institution":"NTR University of Health Sciences Dr. NTR University of Health Sciences","correspondingAuthor":false,"prefix":"","firstName":"Vinisha","middleName":"","lastName":"Bonagiri","suffix":""}],"badges":[],"createdAt":"2025-07-21 22:08:12","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7180985/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7180985/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":87708706,"identity":"f4fb6ce8-c2c2-4d2e-9fd2-e81e6cbaf64f","added_by":"auto","created_at":"2025-07-28 08:22:35","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":243674,"visible":true,"origin":"","legend":"\u003cp\u003eConfusion matrices for all evaluated multimodal large language models (LLMs) on the pediatric chest X-ray binary classification task.\u003c/p\u003e\n\u003cp\u003eEach matrix displays the number of true positive, false negative, false positive, and true negative predictions for bacterial pneumonia versus normal findings.\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7180985/v1/3971510df708c96cbeaf74be.png"},{"id":91354533,"identity":"e1e7fcd1-8f86-4b8b-9073-939b7f6ceca7","added_by":"auto","created_at":"2025-09-15 15:17:06","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":487906,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7180985/v1/ad598050-e7cd-49d4-b239-8c7b119cec65.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Benchmarking Multimodal Large Language Models for Binary Classification of Pediatric Chest X-Rays: A Comparative Evaluation Using a Public Pneumonia Dataset","fulltext":[{"header":"Introduction","content":"\u003cp\u003ePneumonia is a leading cause of mortality worldwide [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e], despite chest X-ray imaging being a readily available and cost-effective diagnostic tool [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. Large language models (LLMs) have shown that they can efficiently support clinician’s diagnostic tasks [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. While studies have examined LLM performance for chest X-ray interpretation, there is limited evidence regarding general-purpose LLMs (GPT-4o, Gemini 2.5 Pro, Claude 4 Sonnet, and Grok-2 Vision) in classifying pediatric pneumonia from chest X-rays. LLMs could soon be a secondary screening tool alongside radiologist opinion to rule out normal vs. abnormal findings and ease radiologic workflows [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. This study aims to evaluate the binary classification performance of five prominent LLMs in distinguishing bacterial from normal pediatric pneumonia on standardized chest X-rays from a retrospective cohort of pediatric patients from Guangzhou Women and Children’s Medical Center [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e] using uniform prompts and evaluation metrics.. All low-quality or unreadable chest radiographs were removed, and two physicians graded the CXRs, with a third expert responsible for checking the final set [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e].\u003c/p\u003e\u003cp\u003e\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003eA publicly available pediatric chest X-ray database from Guangzhou Women and Children’s Medical Center was used for evaluation [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. The dataset consisted of 1,000 JPEG images: 500 labeled as “Normal” and 500 as “Bacterial Pneumonia.” All images were used in their native resolution and format, with no compression or preprocessing. Five representative LLMs were evaluated: GPT-4o (May 2024, Open AI), GPT-4.1 (April 2025, OpenAI), Gemini 2.5 Pro Preview (June 2025, Google), Claude 4 Sonnet (May 2025, Anthropic), and Grok 2 Vision (December 2024, xAI). Each image was submitted to all models via their respective APIs: Base64-encoded image inputs for OpenAI models, native image input via Google GenerativeAI API for Gemini 2.5 Pro, Base64 image input with prompt via Anthropic Vision API, and structure prompt and base64-encoded image via xAI API for Grok 2 Vision.\u003c/p\u003e\u003cp\u003eThe models received the same binary classification prompt: “This is a question from a Board Exam: Analyze the chest X-ray image and determine if the patient has Bacterial Pneumonia or is Normal. Your response must be binary: 0 = Normal, 1 = Bacterial Pneumonia. ONLY output 0 or 1. Example: If pneumonia is present, respond EXACTLY: ‘1’.” GPT-4o and GPT-4.1 were instructed to respond concisely and without disclaimers.\u003c/p\u003e\u003cp\u003ePredictions from each model were compared to ground-truth labels as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. Key metrics of accuracy, sensitivity (recall for Bacterial Pneumonia), specificity (recall for Normal), F1 score, and confusion matrices were computed using scikit-learn. Each model’s performance was evaluated on the same image set for direct comparison. To rule out transmission or preprocessing errors, base64-decoded images were visually verified. Claude 4 Sonnet was additionally tested with blank images to confirm appropriate response to unreadable input.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eBelow, we present the comparative performance metrics for all five LLMs (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). GPT-4o demonstrated high sensitivity for bacterial pneumonia (0.99) but low specificity for normal images (0.18). In contrast, GPT-4.1 achieved a more balanced performance, with a sensitivity of 0.84 and a specificity of 0.76. Gemini 2.5 Pro Preview also showed strong sensitivity for pneumonia (0.97), but its low specificity (0.25) for normal cases led to an accuracy of 0.61 and F1-score of 0.55. Claude 4 Sonnet predicted \u0026ldquo;bacterial pneumonia\u0026rdquo; for all images, yielding perfect sensitivity (1.00) but zero specificity (0.00), with both accuracy and F1-score at 0.33. Grok 2 Vision demonstrated moderate performance, with a sensitivity of 0.82, specificity of 0.56, accuracy of 0.69, and F1-score of 0.68.\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eModel Performance Summary with 95% Confidence Intervals\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"5\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003eModel\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eSensitivity\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003eSpecificity\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c4\"\u003e\u003cp\u003eAccuracy\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c5\"\u003e\u003cp\u003eF1-score\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eGPT-4o\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.99 [0.98, 1.00]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.18 [0.15, 0.22]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e0.58 [0.55, 0.62]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e0.51 [0.47, 0.54]\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eGPT-4.1\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.84 [0.81, 0.87]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.76 [0.72, 0.80]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e0.80 [0.77, 0.82]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e0.80 [0.77, 0.82]\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eGemini 2.5 Pro Preview\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.97 [0.95, 0.98]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.25 [0.21, 0.29]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e0.61 [0.58, 0.64]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e0.55 [0.52, 0.58]\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eClaude Sonnet 4\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e1.00 [1.00, 1.00]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.00 [0.00, 0.00]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e0.50 [0.47, 0.53]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e0.33 [0.30, 0.36]\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eGrok Vision 2\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.82 [0.78, 0.85]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.56 [0.51, 0.60]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e0.69 [0.66, 0.71]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e0.68 [0.65, 0.71]\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003eNotably, Claude 4 Sonnet failed to predict any cases as normal, despite passing image integrity and blank-image tests. GPT-4.1 provided the most balanced results, while GPT-4o and Gemini 2.5 Pro were heavily skewed toward positive (pneumonia) predictions.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eOur evaluation revealed substantial performance variability. GPT-4.1 achieved the most balanced sensitivity and specificity, thus the best clinical utility among the tested models. In contrast, GPT-4o and Gemini 2.5 Pro demonstrated high sensitivity but very poor specificity, likely leading to false positives. Claude 4 Sonnet classified all images as pneumonia despite correct handling of blank images, indicating a possible prompt or model misalignment. Grok 2 Vision performed moderately, with reasonable sensitivity and specificity but lower accuracy than GPT-4.1.\u003c/p\u003e\u003cp\u003eThese findings highlight the necessity of rigorous benchmarking of LLMs before clinical integration. The tendency of several models to overcall pathology may stem from training data imbalance, limited domain adaptation, or prompt sensitivity, underscoring the importance of careful prompt engineering and model selection.\u003c/p\u003e\u003cp\u003eLimitations include the use of a single public dataset and binary task; future work should assess model generalizability across other conditions, modalities, and real-world clinical settings.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003eI. Funding: Not applicable. No funding was received for this study.\u003c/p\u003e\n\u003cp\u003eII. Conflicts of Interest: The authors declare no conflicts of interest or competing interests.\u003c/p\u003e\n\u003cp\u003eIII. Ethics Approval: This study used a publicly available, de-identified dataset that is HIPAA compliant and open source. Institutional review board (IRB) approval was not required.\u003c/p\u003e\n\u003cp\u003eIV. Consent to participate: N/A\u003c/p\u003e\n\u003cp\u003eV. Consent for publication: N/A\u003c/p\u003e\n\u003cp\u003eVI. Availability of data and material: The dataset used in this study is publicly available at:\u003c/p\u003e\n\u003cp\u003ehttps://www.kaggle.com/datasets/paultimothymooney/chest-xray-pneumonia\u003c/p\u003e\n\u003cp\u003eVII. Code Availability: All analysis scripts and code used in this study are available upon request.\u003c/p\u003e\n\u003cp\u003eVIII. Authors Contributions: Nitin Chetla conceived the project and led manuscript writing. Shivam Patel assisted with code execution and data handling. Trisha Naidu performed model testing and results compilation. Rahul Kumar and Luis Rodriguez contributed to experimental design and manuscript revision. Andrew Lee and Vinisha Bonagiri provided radiology and clinical oversight. All authors reviewed and approved the final manuscript.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eHager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M. R., Braren, R., Kaissis, G., \u0026amp; Rueckert, D. (2024). Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature Medicine, 30(9), 2613. https://doi.org/10.1038/s41591-024-03097-1\u003c/li\u003e\n\u003cli\u003eIbrahim, D. M., Elshennawy, N. M., \u0026amp; Sarhan, A. (2021). Deep-chest: Multi-classification deep learning model for diagnosing COVID-19, pneumonia, and lung cancer chest diseases. Computers in Biology and Medicine, 132, 104348. https://doi.org/10.1016/j.compbiomed.2021.104348\u003c/li\u003e\n\u003cli\u003eKarabacak, M., \u0026amp; Margetis, K. (2023). Embracing Large Language Models for Medical Applications: Opportunities and Challenges. In Cureus. Cureus, Inc. https://doi.org/10.7759/cureus.39305\u003c/li\u003e\n\u003cli\u003eKermany, D. (2018). Labeled Optical Coherence Tomography (OCT) and Chest X-Ray Images for Classification [Data set]. Mendeley. https://doi.org/10.17632/RSCBJBR9SJ.2\u003c/li\u003e\n\u003cli\u003eLanks, C. W., Musani, A. I., \u0026amp; Hsia, D. W. (2019). Community-acquired pneumonia and hospital-acquired pneumonia. Medical Clinics of North America, 103, 487. https://doi.org/https://doi.org/10.1016/j.mcna.2018.12.008\u003c/li\u003e\n\u003cli\u003eLeung, A. K. C., Wong, A. H. C., \u0026amp; Hon, K. E. (2018). Community-Acquired Pneumonia in Children [Review of Community-Acquired Pneumonia in Children]. Recent Patents on Inflammation \u0026amp; Allergy Drug Discovery, 12(2), 136. Bentham Science Publishers. https://doi.org/10.2174/1872213x12666180621163821\u003c/li\u003e\n\u003cli\u003eSakai, H., \u0026amp; Lam, S. S. (2025). Large Language Models for Healthcare Text Classification: A Systematic Review. https://doi.org/10.48550/ARXIV.2503.01159\u003c/li\u003e\n\u003cli\u003eToro, C. A. O., Garc\u0026iacute;a‐Pedrero, \u0026Aacute;., Lillo‐Saavedra, M., \u0026amp; Gonzalo‐Mart\u0026iacute;n, C. (2022). Automatic detection of pneumonia in chest X-ray images using textural features. Computers in Biology and Medicine, 145, 105466. https://doi.org/10.1016/j.compbiomed.2022.105466\u003c/li\u003e\n\u003cli\u003eZhou, S., Xu, Z., Zhang, M., Xu, C., Guo, Y., Zhan, Z., Ding, S., Wang, J., Xu, K., Fang, Y., Xia, L., Yeung, J., Zha, D., Melton, G. B., Lin, M., \u0026amp; Zhang, R. (2024). Large Language Models for Disease Diagnosis: A Scoping Review [Review of Large Language Models for Disease Diagnosis: A Scoping Review]. arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.2409.00097\u003c/li\u003e\n\u003cli\u003eZoga, A. C., \u0026amp; Syed, A. (2018). Artificial Intelligence in Radiology: Current Technology and Future Directions [Review of Artificial Intelligence in Radiology: Current Technology and Future Directions]. Seminars in Musculoskeletal Radiology, 22(5), 540. Thieme Medical Publishers (Germany). https://doi.org/10.1055/s-0038-1673383\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Large Language Models, Pediatric Chest X-rays, Multimodal AI models, Clinical Benchmarking","lastPublishedDoi":"10.21203/rs.3.rs-7180985/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7180985/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground/Objective:\u003c/strong\u003ePneumonia remains a leading cause of global mortality, with chest X-rays serving as a primary diagnostic tool. While large language models (LLMs) show promise in medical imaging, limited evidence exists regarding their performance in pediatric pneumonia classification. This study evaluates the binary classification performance of five prominent multimodal LLMs in distinguishing bacterial pneumonia from normal findings in pediatric chest X-rays.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods:\u003c/strong\u003e We evaluated GPT-4o, GPT-4.1, Gemini 2.5 Pro Preview, Claude 4 Sonnet, and Grok 2 Vision using 1,000 pediatric chest X-ray images (500 normal, 500 bacterial pneumonia) from the Guangzhou Women and Children's Medical Center dataset. Each model received identical binary classification prompts via their respective APIs. Performance was assessed using accuracy, sensitivity, specificity, F1-score, and confusion matrix analysis.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults:\u003c/strong\u003e GPT-4.1 demonstrated the most balanced performance with 84% sensitivity and 76% specificity. GPT-4o achieved high sensitivity (99%) but poor specificity (18%), while Gemini 2.5 Pro showed similar patterns (97% sensitivity, 25% specificity). Claude 4 Sonnet classified all images as pneumonia (100% sensitivity, 0% specificity). Grok 2 Vision showed moderate performance with 82% sensitivity and 56% specificity.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusion:\u003c/strong\u003e Substantial performance variability exists among LLMs for pediatric pneumonia detection. GPT-4.1 provided optimal clinical utility with balanced sensitivity and specificity, while other models showed concerning tendencies toward false positives. These findings underscore the necessity for rigorous benchmarking before clinical implementation of LLMs in pediatric radiology.\u003c/p\u003e","manuscriptTitle":"Benchmarking Multimodal Large Language Models for Binary Classification of Pediatric Chest X-Rays: A Comparative Evaluation Using a Public Pneumonia Dataset","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-07-28 08:22:30","doi":"10.21203/rs.3.rs-7180985/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"f10a599b-4a3f-4b43-99f5-6891804bce5e","owner":[],"postedDate":"July 28th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-09-15T15:08:53+00:00","versionOfRecord":[],"versionCreatedAt":"2025-07-28 08:22:30","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7180985","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7180985","identity":"rs-7180985","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}