Large Language Models in Radiology Exams: A Comparative Analysis of Performance in Turkish and English

Large Language Models in Radiology Exams: A Comparative Analysis of Performance in Turkish and English | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Large Language Models in Radiology Exams: A Comparative Analysis of Performance in Turkish and English Şahinde ATLANOĞLU, Mehmet Ali GEDİK This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8642109/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background : The primary objective of this study is to evaluate the success levels of Large Language Models regarding radiology-related questions and to analyze performance variations between the Turkish and English languages. Furthermore, the consistency of the models' responses to the same questions over different time periods was examined, and the obtained data were analyzed in comparison with the performance levels of radiology residents. Materials and Methods: This study evaluated the performance of ChatGPT-5, Grok-4, Claude 4.5 Sonnet, and Gemini 2.5 Pro using 100 multiple-choice radiology questions across five subspecialties. To assess linguistic impact, ChatGPT-5 and Gemini 2.5 Pro were tested in both Turkish and English. Temporal reliability was examined by re-testing ChatGPT-5, Claude 4.5 Sonnet, and Grok-4 after a one-week interval. Finally, AI outputs were benchmarked against a control group of 18 radiology residents (1–3 years of seniority). Results: Gemini 2.5 Pro achieved the highest accuracy (89%), followed by Claude 4.5 Sonnet (86%), ChatGPT-5 (85%), and Grok-4 (84%). All LLMs and 3rd-year residents (75.8%) significantly outperformed 1st-year (58.7%) and 2nd-year (66%) residents. Subspecialty analysis showed 3rd-year residents excelled in musculoskeletal radiology, while Claude 4.5 and Gemini 2.5 Pro significantly surpassed 1st-year residents in abdominal radiology. No significant performance gap was found between Turkish and English outputs for ChatGPT-5 and Gemini 2.5 Pro (p = 1.000), indicating good linguistic agreement (κ ≈ 0.73). Regarding temporal reliability, Claude 4.5 Sonnet demonstrated “very good” consistency over one week (κ = 0.872), whereas Grok-4 (κ = 0.575) and ChatGPT-5 (κ = 0.559) showed only “moderate” reliability. Conclusion: Our findings demonstrate that high-performance LLMs, such as Gemini 2.5 Pro, ChatGPT-5, and Grok-4, provide fundamental radiology knowledge with high accuracy and comparable efficiency. These models show significant potential as supportive tools for optimizing radiology medical education. However, further research incorporating image-based datasets is essential to determine their actual clinical efficacy in real-world radiological practice. Artificial Intelligence ChatGPT Google Gemini Grok Claude Sonnet Large Language Models Radiology Figures Figure 1 Figure 2 Background The incorporation of Artificial Intelligence (AI) into clinical practice has become a focal point in recent medical scholarship [ 1 ]. Within this domain, ChatGPT has emerged as a prominent Large Language Model (LLM), notable for its swift uptake and transformative influence [ 2 ]. LLMs employ natural language processing (NLP) architectures that generate contextual, dynamic responses to complex textual queries by leveraging extensive datasets, deep‑learning algorithms, and transformer frameworks. Unlike rule‑based systems, these models demonstrate sophisticated reasoning capabilities through the identification and interpretation of semantic relationships in input data [ 3 ]. Their applications span academic research, clinical decision support, and medical education [ 4 ]. Although AI in diagnostic radiology has historically centered on image analysis, recent progress with LLMs has broadened their use to text‑heavy tasks such as differential diagnosis, disease categorization, and radiological instruction [ 5 ]. Given the critical importance of precise diagnoses in radiology and the escalating demand for automation, integrating these technologies into routine workflows is essential. Scientific endorsement of such integration depends on LLMs achieving diagnostic accuracy that matches or surpasses that of radiologists and residents [ 6 ]. Despite an explosion of LLM research, few studies have assessed radiology‑specific expertise or benchmarked model performance against human proficiency. This investigation aims to deliver a thorough evaluation of the theoretical capabilities, linguistic variations, and temporal stability of contemporary LLMs in the field of radiology. Methods Question Set Creation and Content Validity A set of 100 multiple‑choice items was assembled for this investigation, sourced from a widely used core radiology textbook [ 7 ] and a standard board‑exam question bank [ 8 ]. Two senior radiologists, each with more than ten years of clinical experience, crafted the questions to provide an even distribution across five subspecialties—thorax, neuroradiology, musculoskeletal system, abdomen, and interventional radiology—with 20 items per domain. The non‑visual questions were subsequently translated into English by a certified translation agency and underwent linguistic validation to guarantee precision and consistency. Large Language Models and Application Protocol The study employed paid subscriptions of ChatGPT‑5, Grok‑4, Claude 4.5 Sonnet, and Gemini 2.5 Pro as its primary analytic instruments. Each question was submitted singly to the respective model interfaces in distinct sessions to reduce potential response bias. For an initial performance comparison, questions were presented to radiology residents in Turkish while the language models received them in English. Comparative Analysis Design The study was structured across three primary comparative layers Language-Based Performance : The questions were administered to ChatGPT and Gemini models in both Turkish and English to assess the influence of language on model success rates. Temporal Reliability : The full question set was re-submitted after a one-week interval to ChatGPT, Grok, and Claude models to assess response stability over time. Version Comparison : Questions were posed in English to successive releases of each model—ChatGPT v5.1, v5.0, v4.0; Gemini 2.5 Pro and 2.5 Flash—to identify performance shifts attributable to model updates. Statistical Analysis Data were analyzed using IBM SPSS version 23. To compare accuracy rates between two different artificial intelligence models, Yates’s Correction was applied. For comparisons across multiple AI models and residents, the Pearson Chi-Square Test and Fisher’s Exact Test with Monte Carlo correction were used; pairwise comparisons were conducted using the Z-test with Bonferroni correction. The McNemar Test assessed differences in correct response rates between two versions of the same AI model, between responses in two languages, and between repeated measurements. Cochran’s Q Test was employed to examine accuracy differences among three versions of the same model. Agreement between AI models, between responses in different languages, and between repeated responses was evaluated using the Kappa Test. Fleiss’ Kappa Test was used to assess agreement among responses from three different AI models. Categorical variables were reported as frequency (percentage). Statistical significance was set at \(\:p<0.050\) . Kappa agreement levels were interpreted according to the Landis & Koch (1977) classification. Kappa < 0 Poor agreement 0.0-0.20 Slight agreement 0.21–0.40 Fair agreement 0.41–0.60 Moderete agreement 0.61–0.80 Substantial agreement 0.81-1.00 Almost perpect agreement Results Among the 100 radiology questions analyzed, Gemini 2.5 Pro achieved the highest success rate among the LLMs, with 89 correct responses. When radiology residents were grouped by year of residency, correct response rates were 75.8% for third-year, 66% for second-year, and 58.7% for first-year residents. Subspecialty analysis revealed that third-year residents significantly outperformed first-year residents in musculoskeletal radiology. In abdominal radiology, Claude 4.5 Sonnet and Gemini 2.5 Pro demonstrated superior performance compared to first-year residents. Overall, the success rates of all LLMs and third-year residents were statistically higher than those of first- and second-year residents across the entire question set (Table 1 ). Table 1 Performance Comparison of Large Language Models and Radiology Residents: Total and Subspecialty Accuracy Rates LLM Test Statistic p Grok 4 Claude Sonnet 4.5 ChatGPT5 Gemini 2.5 Pro 3. yıl 2. yıl 1. yıl Interventional False 3 (15) 4 (20) 4 (20) 3 (15) 45 (28,1) 27 (45) 58 (41,4) 18,483 0,005 x True 17 (85) 16 (80) 16 (80) 17 (85) 115 (71,9) 33 (55) 82 (58,6) Musculoskeletal False 6 (30) 2 (10) 4 (20) 2 (10) 36 (22,5) 19 (31,7) 57 (40,7) 20,253 0,002 x True 14 (70) ab 18 (90) ab 16 (80) ab 18 (90) ab 124 (77,5) b 41 (68,3) ab 83 (59,3) a Neuro False 4 (20) 6 (30) 3 (15) 3 (15) 47 (29,4) 19 (31,7) 62 (44,3) 15,958 0,011 x True 16 (80) 14 (70) 17 (85) 17 (85) 113 (70,6) 41 (68,3) 78 (55,7) Thorax False 1 (5) 1 (5) 1 (5) 2 (10) 34 (21,3) 18 (30) 52 (37,1) 28,275 < 0,001 x True 19 (95) 19 (95) 19 (95) 18 (90) 126 (78,8) 42 (70) 88 (62,9) Abdomen False 2 (10) 1 (5) 3 (15) 1 (5) 32 (20) 19 (31,7) 60 (42,9) 36,867 < 0,001 x True 18 (90) ab 19 (95) b 17 (85) ab 19 (95) b 128 (80) b 41 (68,3) ab 80 (57,1) a Total False 16 (16) 14 (14) 15 (15) 11 (11) 194 (24,3) 102 (34) 289 (41,3) 107,795 < 0,001 y True 84 (84) a 86 (86) a 85 (85) a 89 (89) a 606 (75,8) a 198 (66) b 411 (58,7) b x Monte Carlo Düzeltmeli Fisher’s Exact Testi; y Pearson Ki-Kare Testi; Frekans (yüzde); a−b : Aynı harfe sahip gruplar arasında bir fark yoktur To assess the reliability and temporal consistency of Large Language Models, the same set of questions was administered to Grok-4, Claude 4.5 Sonnet, and ChatGPT-5 in two separate sessions, one week apart. The agreement between the models' responses in the initial and follow-up sessions was evaluated using the Cohen’s Kappa coefficient. The analysis demonstrated that Claude 4.5 Sonnet achieved a "very good" level of internal consistency (kappa = 0.872). In contrast, Grok-4 and ChatGPT-5 exhibited "moderate" agreement (kappa = 0.575 and kappa = 0.559, respectively) (Table 2 ). Table 2 Temporal Accuracy Comparison and Agreement Analysis of Large Language Models Grok 4 Claude Sonnet 4.5 ChatGPT5 1st Assessment False 16 (16) 14 (14) 15 (15) True 84 (84) 86 (86) 85 (85) 2 nd Assessment False 18 (18) 13 (13) 11 (11) True 82 (82) 87 (87) 89 (89) McNemar 0,774 1,000 0,344 Kappa/p 0,575/ <0,001 0,872/ <0,001 0,559/ <0,001 Frequency (percentage) To evaluate language-based performance variations, questions were administered separately in Turkish and English to ChatGPT-5 and Gemini 2.5 Pro (Fig. 1 , 2). The analysis revealed no statistically significant difference in success rates between the two languages for either model (McNemar p = 1.000). Assessment of agreement between responses in different languages showed that both ChatGPT-5 (kappa = 0.733) and Gemini 2.5 Pro (kappa = 0.734) demonstrated a "good" level of linguistic consistency (Table 3 ). Table 3 Evaluation of Accuracy and Agreement in Large Language Models' Responses to English and Turkish Questions LLM Test İstatistiği p x ChatGPT5 Gemini2.5 English False 16 (16) 10 (10) 1,105 0,293 True 84 (84) 90 (90) Turkish False 15 (15) 11 (11) 0,398 0,528 True 85 (85) 89 (89) McNemar 1,000 1,000 Kappa/p 0,733/ <0,001 0,734/ <0,001 x Yates's Correction; Frequency (percentage) In the comparative analysis across different versions of ChatGPT, no statistically significant difference was observed in overall accuracy rates (p = 0.282). Fleiss’ Kappa analysis, conducted to assess response consistency among versions, revealed a "moderate" level of agreement (kappa = 0.600; p < 0.001) (Table 4 ). Table 4 Comparison of Accuracy and Evaluation of Agreement for Responses to Questions Asked Across ChatGPT Versions ChatGPT Fleiss Kappa/p Cohran's Q p ChatGPT5 ChatGPT4 ChatGPT5.1 False 15 (15) 17 (17) 12 (12) 0,600/ <0,001 2,533 0,282 True 85 (85) 83 (83) 88 (88) Frequency (percentage) In the comparison between different versions of the Gemini models, the rates of correct responses were statistically similar, with no significant performance difference observed (p = 0.508). Response-based consistency analysis revealed a "moderate" level of agreement between the versions (kappa = 0.589; p < 0.001) (Table 5 ). Table 5 Comparison of Accuracy and Evaluation of Agreement for Responses to Questions Asked Across Gemini Versions Gemini Kappa/p McNemar Gemini 2.5 Pro Gemini 2.5 Flash False 11 (11) 14 (14) 0,589/ <0,001 0,508 True 89 (89) 86 (86) Frequency (percentage) 4o Discussion Large Language Models are transforming medical decision-support processes by leveraging deep learning architectures to analyze extensive natural language data [ 4 ]. The rapid evolution of artificial intelligence algorithms is steadily increasing their potential for integration into clinical applications [ 9 ]. Unlike previous studies, this research stands out by simultaneously evaluating four contemporary models and conducting a cross-linguistic analysis to compare Turkish and English performances of two leading models. The success rates observed in our study, ranging from 84% to 89%, represent a notable improvement over earlier findings. Previous studies reported lower accuracy rates: 40.8%–62.6% for ChatGPT-3.5; 65%–81% for ChatGPT-4; 64.89%–88.1% for ChatGPT-4o; 38.8% for Google Bard; 55.73% for Google Gemini; and 85.7% for Gemini Advanced [ 2 , 3 , 10 – 12 ]. The 85% success rate achieved by ChatGPT-5 in our study indicates a significant enhancement in clinical reasoning compared to earlier versions, GPT-3.5 and GPT-4 [ 13 ]. Similarly, the 89% success rate of Gemini 2.5 Pro highlights substantial technological progress, likely due to its access to up-to-date datasets [ 4 ]. Nevertheless, since radiology practice fundamentally relies on visual data analysis, the current text-based models alone are insufficient for direct clinical use. The future role of artificial intelligence in radiology will be better defined by models with image interpretation capabilities. Promising developments in medicine-specific, structured multimodal LLMs—such as Google’s Med-PaLM M and Microsoft’s LLaVA-M—show highly encouraging results for radiological diagnostics [ 14 , 15 ]. In this study, Gemini 2.5 Pro and Claude Sonnet achieved the highest success rates in abdominal and musculoskeletal radiology, while Gemini 2.5 Pro and Grok 4 performed best in interventional radiology. In neuroradiology, Gemini 2.5 Pro and ChatGPT-5 were the top performers, and Claude Sonnet, Grok 4, and ChatGPT-5 excelled in thoracic radiology. The 85% success rates for both ChatGPT-5 and Gemini 2.5 Pro in neuroradiology were notably higher than those reported by Gupta et al. [ 11 ]. Our results for musculoskeletal and abdominal radiology also significantly surpassed the findings of Huang et al. [ 10 ]. Payne et al. [ 13 ] noted that ChatGPT-4 performed between first-year and second-year residents (61.9%); in contrast, our study found that Claude Sonnet and Gemini Pro outperformed first-year residents in abdominal radiology, and overall, LLMs and third-year residents achieved higher success rates than first- and second-year residents. These outcomes differ from Sarangi et al., who reported that two senior residents (63.33% and 57.5%) outperformed several AI models (Bard, Bing, and ChatGPT-3.5) [ 12 ]. The strong performance of large language models on non-visual questions highlights their potential for integration into radiology education. However, for these models to serve as reliable assistants in image interpretation and complex diagnostic processes—the core of radiology practice—further evolution of LLM architectures is necessary. Specifically, the development of multimodal capabilities is essential for effective deployment in clinical decision-support systems [ 4 , 16 ]. Within this study, all questions were presented to ChatGPT-5 and Gemini 2.5 Pro in both Turkish and English to evaluate the impact of language on model performance. ChatGPT-5 achieved 84 correct responses for English and 85 for Turkish, while Gemini 2.5 Pro scored 90 for English and 89 for Turkish. These findings contrast with Toyama et al. [ 4 ], who attributed lower performance in Japanese to the language's structural complexity. In our study, the similar success rates in both languages—despite the models’ primary training on English datasets—reflect advancements in multilingual processing and language recognition in next-generation models. Meddeb et al. [ 17 ] found that GPT-4 excelled in translating radiology reports from English to German, Greek, Thai, and Turkish, while GPT-3.5 performed best for French. Although these models demonstrated high clarity and semantic consistency, medical terminology accuracy remained moderate, highlighting the need for further research across diverse language families and technical fields. Efe et al. [ 18 ] observed that ChatGPT-4o and Google Gemini changed responses when questions were repeated, while Brin et al. [ 9 ] noted that ChatGPT-3.5 altered answers for 82.5% of repeated questions, but ChatGPT-4 did not. In our study, Grok-4 and ChatGPT-5 showed moderate agreement between initial and follow-up responses (Kappa = 0.575 and Kappa = 0.559), whereas Claude Sonnet exhibited very good agreement (Kappa = 0.872). The literature frequently highlights the risk of erroneous or fabricated outputs, known as "hallucinations," in LLMs [ 2 , 19 ]. The moderate to very good agreement coefficients in our study suggest a decreasing tendency for hallucinations and increasing response stability in next-generation models. Integrating ChatGPT into healthcare applications entails significant risks and limitations beyond its benefits. Ensuring system reliability requires careful management of ethics-based risks, including data security, patient privacy, and algorithmic bias [ 3 ]. Limitations of our study include the exclusive use of non-visual, multiple-choice questions and a restricted question pool of 100, derived from only two textbooks. Conclusion Large language models demonstrated high-accuracy performance on fundamental radiology questions, indicating their potential as supportive tools in medical education for providing feedback and optimizing learning processes. Future studies incorporating radiological images will be essential to clarify the role of LLMs in clinical radiology. For full integration into radiology practice, the success of these models in image evaluation must match their strong performance in theoretical examinations. Abbreviations AI Artificial Intelligence LLM Large Language Model NLP Neuro Linguistic Programming Declarations Ethics Approval and Consent to Participate Human Proficiency and Ethical Statement: Eighteen radiology residents completed the identical question set to serve as a human benchmark. Because the study exclusively analyzed AI‑generated responses and did not involve direct interaction with living subjects, it was deemed exempt from institutional ethics review. Consent for Publication Not applicable. Availability of Data and Materials The datasets used and/or analyzed during the current study are not publicly available due to institutional regulations and patient privacy restrictions but are available from the corresponding author on reasonable request, subject to ethical approval and data sharing agreements. Competing Interests The authors declare that they have no competing interests. Funding Not applicable Authors’ Contributions S.A. planned the study, designed the methodology, and drafted the manuscript. MAG operated LLM exams, preprocess study data, revised the manuscript. All authors read and approved the final manuscript. Acknowledgements : Not applicable References Nicikowski J, Szczepanski M, Miedziaszczyk M, Kudlinski B. The potential of ChatGPT in medicine: an example analysis of nephrology specialty exams in Poland. Clin Kidney J. 2024;17(8):sfae193. Bhayana R, Bleakney RR, Krishna S. GPT-4 in Radiology: Improvements in Advanced Reasoning. Radiology. 2023;307(5):e230987. Iqbal U, Tanweer A, Rahmanti AR, Greenfield D, Lee LT, Li YJ. Impact of large language model (ChatGPT) in healthcare: an umbrella review and evidence synthesis. J Biomed Sci. 2025;32(1):45. Toyama Y, Harigai A, Abe M, Nagano M, Kawabata M, Seki Y, Takase K. Performance evaluation of ChatGPT, GPT-4, and Bard on the official board examination of the Japan Radiology Society. Jpn J Radiol. 2024;42(2):201–7. Bhayana R, Krishna S, Bleakney RR. Performance of ChatGPT on a Radiology Board-style Examination: Insights into Current Strengths and Limitations. Radiology. 2023;307(5):e230582. Aggarwal R, Sounderajah V, Martin G, Ting DSW, Karthikesalingam A, King D, Ashrafian H, Darzi A. Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis. NPJ Digit Med. 2021;4(1):65. Sun E, Shi J, Mandell J. Core Radiology: A Visual Approach to Diagnostic Imaging. 2 ed. Philadelphia, PA: Elsevier; 2021. Weissman A. Top Score for the Radiology Board. New York: Thieme Medical Publishers, Inc.; 2018. Brin D, Sorin V, Vaid A, Soroush A, Glicksberg BS, Charney AW, Nadkarni G, Klang E. Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments. Sci Rep. 2023;13(1):16492. Huang KA, Choudhary HK, Hardin WM, Prakash N. Comparative Analysis of ChatGPT-4o and Gemini Advanced Performance on Diagnostic Radiology In-Training Exams. Cureus. 2025;17(3):e80874. Gupta R, Hamid AM, Jhaveri M, Patel N, Suthar PP. Comparative Evaluation of AI Models Such as ChatGPT 3.5, ChatGPT 4.0, and Google Gemini in Neuroradiology Diagnostics. Cureus. 2024;16(8):e67766. Sarangi PK, Narayan RK, Mohakud S, Vats A, Sahani D, Mondal H. Assessing the Capability of ChatGPT, Google Bard, and Microsoft Bing in Solving Radiology Case Vignettes. Indian J Radiol Imaging. 2024;34(2):276–82. Payne DL, Purohit K, Borrero WM, Chung K, Hao M, Mpoy M, Jin M, Prasanna P, Hill V. Performance of GPT-4 on the American College of Radiology In-training Examination: Evaluating Accuracy, Model Drift, and Fine-tuning. Acad Radiol. 2024;31(7):3046–54. Tu T, Azizi S, Driess D, Schaekermann M, Amin M, Chang P, Carroll A, Lau C, Tanno R, Ktena I et al. Towards Generalist Biomedical AI. arXiv 2023, 2307:14334. Li C, Wong C, Zhang S, Usuyama N, Liu H, Yang J, Naumann T, Poon H, Gao J. LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day. arXiv. 2023;2306:00890. Yan Z, Zhang K, Zhou R, He L, Li X, Sun L. Multimodal ChatGPT for Medical Applications: An Experimental Study of GPT-4V. arXiv 2023, 2310:19061. Meddeb A, Luken S, Busch F, Adams L, Ugga L, Koltsakis E, Tzortzakakis A, Jelassi S, Dkhil I, Klontzas ME, et al. Large Language Model Ability to Translate CT and MRI Free-Text Radiology Reports Into Multiple Languages. Radiology. 2024;313(3):e241736. Is EE, Menekseoglu AK. Comparative performance of artificial intelligence models in rheumatology board-level questions: evaluating Google Gemini and ChatGPT-4o. Clin Rheumatol. 2024;43(11):3507–13. Wang S, Zhao Z, Ouyang X, Liu T, Wang Q, Shen D. Interactive computer-aided diagnosis on medical image using large language models. Commun Eng. 2024;3(1):133. Additional Declarations No competing interests reported. Supplementary Files 7f1933cf900e84a1b3e923e8b70508ae11.docx f217416db95f6cb77d340c2e4639d8ea11.pdf ccda5aba9c49313b526d42c70107677211.pdf Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8642109","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":577025082,"identity":"cfbd47f6-402d-4c7c-bf32-f0d3f39ac289","order_by":0,"name":"Şahinde ATLANOĞLU","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA5klEQVRIiWNgGAWjYHACxgMgko+BGURLyBChg5kBrIWNgS0BpIWHFC08BiCasBb59vMHDvzcYWPXJpHz+dWNGgseBvbDRzfg02JwJpnhYO+ZtOQ2idxt1jnHgA7jSUu7gVcLQzLDAd62w8lsQC3GOWxALRI8Zni1yPc/Zjj4F6wl55lxzj8itDDcSGY4DLTFDqiF+XFuGxFaDG48Njgs25aWwMbzzIw5t0+Ch42QX+T7Ex8+fNtmY8/Pnvz4c863Ojl+9sPH8DsMChIbgFEjAWKxEaMcBOyBmPkDsapHwSgYBaNgZAEAaPtGRWSl928AAAAASUVORK5CYII=","orcid":"","institution":"Kütahya Sağlık Bilimleri Üniversitesi","correspondingAuthor":true,"prefix":"","firstName":"Şahinde","middleName":"","lastName":"ATLANOĞLU","suffix":""},{"id":577025083,"identity":"1d251c55-d444-4fa8-b5a4-a55e68241c15","order_by":1,"name":"Mehmet Ali GEDİK","email":"","orcid":"","institution":"Kütahya Sağlık Bilimleri Üniversitesi","correspondingAuthor":false,"prefix":"","firstName":"Mehmet","middleName":"Ali","lastName":"GEDİK","suffix":""}],"badges":[],"createdAt":"2026-01-19 18:04:32","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8642109/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8642109/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":100811657,"identity":"f51e7c36-b4f9-40de-b21f-8435f550f28b","added_by":"auto","created_at":"2026-01-21 15:46:59","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1692413,"visible":true,"origin":"","legend":"","description":"","filename":"LLMINGILIZCEV411.01revised1.docx","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/e1ca1b9b8681b98f50c04359.docx"},{"id":100811640,"identity":"23032a7d-9aef-40b9-b55a-baffeccba08c","added_by":"auto","created_at":"2026-01-21 15:46:58","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":4360,"visible":true,"origin":"","legend":"","description":"","filename":"a942832b6696465985f2579439571dfe.json","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/153ac2200f540c39125bfc34.json"},{"id":100811664,"identity":"08896a04-11d0-4dab-bae2-e057f3bb43a7","added_by":"auto","created_at":"2026-01-21 15:47:00","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":68948,"visible":true,"origin":"","legend":"","description":"","filename":"7f1933cf900e84a1b3e923e8b70508ae11.docx","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/dd7c6a112974325277770618.docx"},{"id":100811662,"identity":"13be7cf0-0bcc-42b2-b9d9-3ca478afd1d3","added_by":"auto","created_at":"2026-01-21 15:46:59","extension":"pdf","order_by":3,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":3168207,"visible":true,"origin":"","legend":"","description":"","filename":"ccda5aba9c49313b526d42c70107677211.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/b40e983366309a212c413126.pdf"},{"id":100811648,"identity":"9899f7f9-766c-4f5b-a1bc-92647c810e6c","added_by":"auto","created_at":"2026-01-21 15:46:59","extension":"pdf","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":303323,"visible":true,"origin":"","legend":"","description":"","filename":"f217416db95f6cb77d340c2e4639d8ea11.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/a468283e494268abe37824da.pdf"},{"id":100857869,"identity":"1a48a5e9-34bf-417c-ab2b-32a023d9e64c","added_by":"auto","created_at":"2026-01-22 07:23:18","extension":"xml","order_by":5,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":84537,"visible":true,"origin":"","legend":"","description":"","filename":"a942832b6696465985f2579439571dfe1enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/3ed59458eaf3eac2e7705574.xml"},{"id":100858117,"identity":"f573323d-5e2c-4871-bd7f-d88467ca9a07","added_by":"auto","created_at":"2026-01-22 07:23:53","extension":"png","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":185458,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/46219e7ed1adf2021ad57bf3.png"},{"id":100811646,"identity":"cb1d80b8-acc6-463a-9999-4c56b78a48ce","added_by":"auto","created_at":"2026-01-21 15:46:58","extension":"png","order_by":7,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":248979,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/8636c6a3a813fd3e5a0f0122.png"},{"id":100858003,"identity":"46d3189a-7530-4a7f-bed7-09c85de65ac1","added_by":"auto","created_at":"2026-01-22 07:23:39","extension":"png","order_by":8,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":186210,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/7a9f95665a9cb5718d49809d.png"},{"id":100811661,"identity":"c7709553-03c8-484a-8f4e-7fb079f62705","added_by":"auto","created_at":"2026-01-21 15:46:59","extension":"png","order_by":9,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":375372,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/2dd84ea10307e67a4150726c.png"},{"id":100811659,"identity":"68b125d0-acc6-4c27-9aa6-1ecbbdaa3fcd","added_by":"auto","created_at":"2026-01-21 15:46:59","extension":"png","order_by":10,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":416146,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/75504e88ea73d70fdf7ecc1f.png"},{"id":100811663,"identity":"1b203e10-93eb-443e-94bb-8fadd20731d0","added_by":"auto","created_at":"2026-01-21 15:46:59","extension":"png","order_by":11,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":196510,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/f3f85df5722b4160c350b4b7.png"},{"id":100811649,"identity":"52e57a8e-2958-49bb-923f-2bd141a2c824","added_by":"auto","created_at":"2026-01-21 15:46:59","extension":"png","order_by":12,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":262296,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage7.png","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/e2d1ad880f6484c1f5a7bc6f.png"},{"id":100857863,"identity":"d48a7db6-8bde-4f25-9e62-eb3d988aff9d","added_by":"auto","created_at":"2026-01-22 07:23:17","extension":"png","order_by":13,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":54362,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/94f6f91778355a5ab3e62fc7.png"},{"id":100811652,"identity":"6caaa205-98ce-438d-b7e6-dc057c7fa731","added_by":"auto","created_at":"2026-01-21 15:46:59","extension":"png","order_by":14,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":64597,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/0e4fe626a0f49390685eb456.png"},{"id":100858068,"identity":"46d6de3c-b253-49e2-9897-5211d1cdd624","added_by":"auto","created_at":"2026-01-22 07:23:48","extension":"png","order_by":15,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":51065,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/a4ed1cfcc6bbcef21b640ecb.png"},{"id":100811660,"identity":"ae8b0b22-cffb-4c9d-b330-469ddc1480ca","added_by":"auto","created_at":"2026-01-21 15:46:59","extension":"png","order_by":16,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":175530,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/407ef4e5e22122afd910b7d3.png"},{"id":100811656,"identity":"f230314a-90fd-44b3-9805-51f31e68ffb8","added_by":"auto","created_at":"2026-01-21 15:46:59","extension":"png","order_by":17,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":104019,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/332d1cdaa1d940da406dab85.png"},{"id":100811655,"identity":"89fbf437-b45f-44cf-a188-d95d54339c06","added_by":"auto","created_at":"2026-01-21 15:46:59","extension":"png","order_by":18,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":93550,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/006f52281896a265184dcb83.png"},{"id":100858010,"identity":"710d698c-3f2e-420a-afe4-2d96ff349f30","added_by":"auto","created_at":"2026-01-22 07:23:40","extension":"png","order_by":19,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":69785,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage7.png","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/dbefb5c55db3e899ebb8cbef.png"},{"id":100858019,"identity":"45817b3b-8a06-4a71-8d32-c28787618e49","added_by":"auto","created_at":"2026-01-22 07:23:42","extension":"xml","order_by":20,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":83046,"visible":true,"origin":"","legend":"","description":"","filename":"a942832b6696465985f2579439571dfe1structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/998f6540fd7914476d8b4491.xml"},{"id":100858108,"identity":"3746bffa-8399-472d-bd0b-5c9b982a0481","added_by":"auto","created_at":"2026-01-22 07:23:52","extension":"html","order_by":21,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":92330,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/72333c236102b2e542b73147.html"},{"id":100811639,"identity":"9c4d7c19-1a8d-45d1-9020-0faaa6689e39","added_by":"auto","created_at":"2026-01-21 15:46:58","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":385020,"visible":true,"origin":"","legend":"\u003cp\u003eAnswers to the same question from Chat GBT 5 (Turkish) and Chat GBT 4-Gemini-Grok LLMs (English). All answered correctly.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/3900d9a6358657cf1cb544a0.png"},{"id":100811641,"identity":"f798769a-eca1-4828-92b0-07e869cb8b10","added_by":"auto","created_at":"2026-01-21 15:46:58","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":280860,"visible":true,"origin":"","legend":"\u003cp\u003eAnswers to the same question from Gemini and Claude Sonnet, Grok 4: Grok 4's answer was incorrect.\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/3bd25b97a853d4f487cad731.png"},{"id":101231329,"identity":"696cd369-aaeb-428f-a238-3a163d97b0b8","added_by":"auto","created_at":"2026-01-27 13:42:42","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1360734,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/9c2f3672-b02c-4ee8-afaf-051bd86e8e8c.pdf"},{"id":100811645,"identity":"b5a0f465-278b-48f8-8b57-73a77c39632f","added_by":"auto","created_at":"2026-01-21 15:46:58","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":68948,"visible":true,"origin":"","legend":"","description":"","filename":"7f1933cf900e84a1b3e923e8b70508ae11.docx","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/d3141b32373b6ce51cead551.docx"},{"id":100811643,"identity":"0405393a-a47f-4710-bacd-e97a51015e71","added_by":"auto","created_at":"2026-01-21 15:46:58","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":303323,"visible":true,"origin":"","legend":"","description":"","filename":"f217416db95f6cb77d340c2e4639d8ea11.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/2e8261498b28850d5de6c9f7.pdf"},{"id":100858168,"identity":"0492524b-2a6d-4bb9-8352-ba42ac175107","added_by":"auto","created_at":"2026-01-22 07:23:59","extension":"pdf","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":3168207,"visible":true,"origin":"","legend":"","description":"","filename":"ccda5aba9c49313b526d42c70107677211.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8642109/v1/4a0dadb8fc9822a02d64cd6f.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"\u003cp\u003eLarge Language Models in Radiology Exams: A Comparative Analysis of Performance in Turkish and English\u003c/p\u003e","fulltext":[{"header":"Background","content":"\u003cp\u003eThe incorporation of Artificial Intelligence (AI) into clinical practice has become a focal point in recent medical scholarship [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. Within this domain, ChatGPT has emerged as a prominent Large Language Model (LLM), notable for its swift uptake and transformative influence [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. LLMs employ natural language processing (NLP) architectures that generate contextual, dynamic responses to complex textual queries by leveraging extensive datasets, deep‑learning algorithms, and transformer frameworks. Unlike rule‑based systems, these models demonstrate sophisticated reasoning capabilities through the identification and interpretation of semantic relationships in input data [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. Their applications span academic research, clinical decision support, and medical education [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eAlthough AI in diagnostic radiology has historically centered on image analysis, recent progress with LLMs has broadened their use to text‑heavy tasks such as differential diagnosis, disease categorization, and radiological instruction [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. Given the critical importance of precise diagnoses in radiology and the escalating demand for automation, integrating these technologies into routine workflows is essential. Scientific endorsement of such integration depends on LLMs achieving diagnostic accuracy that matches or surpasses that of radiologists and residents [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. Despite an explosion of LLM research, few studies have assessed radiology‑specific expertise or benchmarked model performance against human proficiency. This investigation aims to deliver a thorough evaluation of the theoretical capabilities, linguistic variations, and temporal stability of contemporary LLMs in the field of radiology.\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003e \u003cstrong\u003eQuestion Set Creation and Content Validity\u003c/strong\u003e \u003cp\u003eA set of 100 multiple‑choice items was assembled for this investigation, sourced from a widely used core radiology textbook [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e] and a standard board‑exam question bank [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. Two senior radiologists, each with more than ten years of clinical experience, crafted the questions to provide an even distribution across five subspecialties\u0026mdash;thorax, neuroradiology, musculoskeletal system, abdomen, and interventional radiology\u0026mdash;with 20 items per domain. The non‑visual questions were subsequently translated into English by a certified translation agency and underwent linguistic validation to guarantee precision and consistency.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eLarge Language Models and Application Protocol\u003c/strong\u003e \u003cp\u003eThe study employed paid subscriptions of ChatGPT‑5, Grok‑4, Claude 4.5 Sonnet, and Gemini 2.5 Pro as its primary analytic instruments. Each question was submitted singly to the respective model interfaces in distinct sessions to reduce potential response bias. For an initial performance comparison, questions were presented to radiology residents in Turkish while the language models received them in English.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eComparative Analysis Design\u003c/strong\u003e \u003cp\u003eThe study was structured across three primary comparative layers\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eLanguage-Based Performance\u003c/b\u003e: The questions were administered to ChatGPT and Gemini models in both Turkish and English to assess the influence of language on model success rates.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eTemporal Reliability\u003c/b\u003e: The full question set was re-submitted after a one-week interval to ChatGPT, Grok, and Claude models to assess response stability over time.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eVersion Comparison\u003c/b\u003e: Questions were posed in English to successive releases of each model\u0026mdash;ChatGPT v5.1, v5.0, v4.0; Gemini 2.5 Pro and 2.5 Flash\u0026mdash;to identify performance shifts attributable to model updates.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e \u003c/p\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eStatistical Analysis\u003c/h2\u003e \u003cp\u003eData were analyzed using IBM SPSS version 23. To compare accuracy rates between two different artificial intelligence models, Yates\u0026rsquo;s Correction was applied. For comparisons across multiple AI models and residents, the Pearson Chi-Square Test and Fisher\u0026rsquo;s Exact Test with Monte Carlo correction were used; pairwise comparisons were conducted using the Z-test with Bonferroni correction. The McNemar Test assessed differences in correct response rates between two versions of the same AI model, between responses in two languages, and between repeated measurements. Cochran\u0026rsquo;s Q Test was employed to examine accuracy differences among three versions of the same model.\u003c/p\u003e \u003cp\u003eAgreement between AI models, between responses in different languages, and between repeated responses was evaluated using the Kappa Test. Fleiss\u0026rsquo; Kappa Test was used to assess agreement among responses from three different AI models. Categorical variables were reported as frequency (percentage). Statistical significance was set at \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:p\u0026lt;0.050\\)\u003c/span\u003e\u003c/span\u003e. Kappa agreement levels were interpreted according to the Landis \u0026amp; Koch (1977) classification.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"No\" id=\"Taba\" border=\"1\"\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eKappa\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u0026lt;\u0026thinsp;0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePoor agreement\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e0.0-0.20\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSlight agreement\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e0.21\u0026ndash;0.40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFair agreement\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e0.41\u0026ndash;0.60\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eModerete agreement\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e0.61\u0026ndash;0.80\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSubstantial agreement\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e0.81-1.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAlmost perpect agreement\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"Results","content":"\u003cp\u003eAmong the 100 radiology questions analyzed, Gemini 2.5 Pro achieved the highest success rate among the LLMs, with 89 correct responses. When radiology residents were grouped by year of residency, correct response rates were 75.8% for third-year, 66% for second-year, and 58.7% for first-year residents. Subspecialty analysis revealed that third-year residents significantly outperformed first-year residents in musculoskeletal radiology. In abdominal radiology, Claude 4.5 Sonnet and Gemini 2.5 Pro demonstrated superior performance compared to first-year residents. Overall, the success rates of all LLMs and third-year residents were statistically higher than those of first- and second-year residents across the entire question set (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003ePerformance Comparison of Large Language Models and Radiology Residents: Total and Subspecialty Accuracy Rates\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"11\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c10\" colnum=\"10\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c11\" colnum=\"11\"\u003e\u003c/div\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colspan=\"7\" nameend=\"c9\" namest=\"c3\"\u003e \u003cp\u003eLLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eTest Statistic\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003ep\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eGrok 4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eClaude Sonnet 4.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eChatGPT5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eGemini 2.5 Pro\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e3. yıl\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e2. yıl\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e1. yıl\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eInterventional\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFalse\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e3 (15)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e4 (20)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e4 (20)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e3 (15)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e45 (28,1)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e27 (45)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e58 (41,4)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e18,483\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e\u003cb\u003e0,005\u003c/b\u003e\u003csup\u003e\u003cb\u003ex\u003c/b\u003e\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTrue\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e17 (85)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e16 (80)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e16 (80)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e17 (85)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e115 (71,9)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e33 (55)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e82 (58,6)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eMusculoskeletal\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFalse\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e6 (30)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2 (10)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e4 (20)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e2 (10)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e36 (22,5)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e19 (31,7)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e\u003cb\u003e57 (40,7)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e20,253\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e\u003cb\u003e0,002\u003c/b\u003e\u003csup\u003e\u003cb\u003ex\u003c/b\u003e\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTrue\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e14 (70)\u003csup\u003eab\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e18 (90)\u003csup\u003eab\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e16 (80)\u003csup\u003eab\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e18 (90)\u003csup\u003eab\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e124 (77,5)\u003c/b\u003e\u003csup\u003e\u003cb\u003eb\u003c/b\u003e\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e41 (68,3)\u003csup\u003eab\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e\u003cb\u003e83 (59,3)\u003c/b\u003e\u003csup\u003e\u003cb\u003ea\u003c/b\u003e\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eNeuro\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFalse\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e4 (20)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e6 (30)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e3 (15)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e3 (15)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e47 (29,4)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e19 (31,7)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e62 (44,3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e15,958\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e\u003cb\u003e0,011\u003c/b\u003e\u003csup\u003e\u003cb\u003ex\u003c/b\u003e\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTrue\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e16 (80)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e14 (70)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e17 (85)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e17 (85)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e113 (70,6)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e41 (68,3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e78 (55,7)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eThorax\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFalse\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1 (5)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e1 (5)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1 (5)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e2 (10)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e34 (21,3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e18 (30)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e52 (37,1)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e28,275\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e\u003cb\u003e\u0026lt;\u0026thinsp;0,001\u003c/b\u003e\u003csup\u003e\u003cb\u003ex\u003c/b\u003e\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTrue\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e19 (95)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e19 (95)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e19 (95)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e18 (90)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e126 (78,8)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e42 (70)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e88 (62,9)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eAbdomen\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFalse\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e2 (10)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e1 (5)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e3 (15)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e1 (5)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e32 (20)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e19 (31,7)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e\u003cb\u003e60 (42,9)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e36,867\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e\u003cb\u003e\u0026lt;\u0026thinsp;0,001\u003c/b\u003e\u003csup\u003e\u003cb\u003ex\u003c/b\u003e\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTrue\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e18 (90)\u003csup\u003eab\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e19 (95)\u003c/b\u003e\u003csup\u003e\u003cb\u003eb\u003c/b\u003e\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e17 (85)\u003csup\u003eab\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e19 (95)\u003c/b\u003e\u003csup\u003e\u003cb\u003eb\u003c/b\u003e\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e128 (80)\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e41 (68,3)\u003csup\u003eab\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e\u003cb\u003e80 (57,1)\u003c/b\u003e\u003csup\u003e\u003cb\u003ea\u003c/b\u003e\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eTotal\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFalse\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e16 (16)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e14 (14)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e15 (15)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e11 (11)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e194 (24,3)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003e102 (34)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e\u003cb\u003e289 (41,3)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e107,795\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e\u003cb\u003e\u0026lt;\u0026thinsp;0,001\u003c/b\u003e\u003csup\u003e\u003cb\u003ey\u003c/b\u003e\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTrue\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e84 (84)\u003c/b\u003e\u003csup\u003e\u003cb\u003ea\u003c/b\u003e\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e86 (86)\u003c/b\u003e\u003csup\u003e\u003cb\u003ea\u003c/b\u003e\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e85 (85)\u003c/b\u003e\u003csup\u003e\u003cb\u003ea\u003c/b\u003e\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e89 (89)\u003c/b\u003e\u003csup\u003e\u003cb\u003ea\u003c/b\u003e\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e606 (75,8)\u003c/b\u003e\u003csup\u003e\u003cb\u003ea\u003c/b\u003e\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cb\u003e198 (66)\u003c/b\u003e\u003csup\u003e\u003cb\u003eb\u003c/b\u003e\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e\u003cb\u003e411 (58,7)\u003c/b\u003e\u003csup\u003e\u003cb\u003eb\u003c/b\u003e\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"11\"\u003e\u003csup\u003ex\u003c/sup\u003eMonte Carlo D\u0026uuml;zeltmeli Fisher\u0026rsquo;s Exact Testi; \u003csup\u003ey\u003c/sup\u003ePearson Ki-Kare Testi; Frekans (y\u0026uuml;zde); \u003csup\u003ea\u0026minus;b\u003c/sup\u003e: Aynı harfe sahip gruplar arasında bir fark yoktur\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eTo assess the reliability and temporal consistency of Large Language Models, the same set of questions was administered to Grok-4, Claude 4.5 Sonnet, and ChatGPT-5 in two separate sessions, one week apart. The agreement between the models' responses in the initial and follow-up sessions was evaluated using the Cohen\u0026rsquo;s Kappa coefficient. The analysis demonstrated that Claude 4.5 Sonnet achieved a \"very good\" level of internal consistency (kappa\u0026thinsp;=\u0026thinsp;0.872). In contrast, Grok-4 and ChatGPT-5 exhibited \"moderate\" agreement (kappa\u0026thinsp;=\u0026thinsp;0.575 and kappa\u0026thinsp;=\u0026thinsp;0.559, respectively) (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eTemporal Accuracy Comparison and Agreement Analysis of Large Language Models\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eGrok 4\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eClaude Sonnet 4.5\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eChatGPT5\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e1st Assessment\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFalse\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e16 (16)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e14 (14)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e15 (15)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTrue\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e84 (84)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e86 (86)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e85 (85)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2 nd Assessment\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFalse\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e18 (18)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e13 (13)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e11 (11)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTrue\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e82 (82)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e87 (87)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e89 (89)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMcNemar\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0,774\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1,000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0,344\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eKappa/p\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0,575/\u003cb\u003e\u0026lt;0,001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0,872/\u003cb\u003e\u0026lt;0,001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0,559/\u003cb\u003e\u0026lt;0,001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"4\"\u003eFrequency (percentage)\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eTo evaluate language-based performance variations, questions were administered separately in Turkish and English to ChatGPT-5 and Gemini 2.5 Pro (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e, 2). The analysis revealed no statistically significant difference in success rates between the two languages for either model (McNemar p\u0026thinsp;=\u0026thinsp;1.000). Assessment of agreement between responses in different languages showed that both ChatGPT-5 (kappa\u0026thinsp;=\u0026thinsp;0.733) and Gemini 2.5 Pro (kappa\u0026thinsp;=\u0026thinsp;0.734) demonstrated a \"good\" level of linguistic consistency (Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eEvaluation of Accuracy and Agreement in Large Language Models' Responses to English and Turkish Questions\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eLLM\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eTest İstatistiği\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003ep\u003csup\u003ex\u003c/sup\u003e\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eChatGPT5\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eGemini2.5\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEnglish\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFalse\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e16 (16)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e10 (10)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e1,105\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e0,293\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTrue\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e84 (84)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e90 (90)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTurkish\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFalse\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e15 (15)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e11 (11)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e0,398\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e0,528\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTrue\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e85 (85)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e89 (89)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMcNemar\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e1,000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1,000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eKappa/p\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0,733/\u003cb\u003e\u0026lt;0,001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0,734/\u003cb\u003e\u0026lt;0,001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"5\"\u003e\u003csup\u003ex\u003c/sup\u003eYates's Correction; Frequency (percentage)\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e\u003cp\u003eIn the comparative analysis across different versions of ChatGPT, no statistically significant difference was observed in overall accuracy rates (p\u0026thinsp;=\u0026thinsp;0.282). Fleiss\u0026rsquo; Kappa analysis, conducted to assess response consistency among versions, revealed a \"moderate\" level of agreement (kappa\u0026thinsp;=\u0026thinsp;0.600; p\u0026thinsp;\u0026lt;\u0026thinsp;0.001) (Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eComparison of Accuracy and Evaluation of Agreement for Responses to Questions Asked Across ChatGPT Versions\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"7\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colspan=\"3\" nameend=\"c4\" namest=\"c2\"\u003e \u003cp\u003eChatGPT\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eFleiss Kappa/p\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eCohran's Q\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003ep\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eChatGPT5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eChatGPT4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eChatGPT5.1\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFalse\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e15 (15)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e17 (17)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e12 (12)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e0,600/\u003cb\u003e\u0026lt;0,001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e2,533\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e0,282\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTrue\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e85 (85)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e83 (83)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e88 (88)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"7\"\u003eFrequency (percentage)\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eIn the comparison between different versions of the Gemini models, the rates of correct responses were statistically similar, with no significant performance difference observed (p\u0026thinsp;=\u0026thinsp;0.508). Response-based consistency analysis revealed a \"moderate\" level of agreement between the versions (kappa\u0026thinsp;=\u0026thinsp;0.589; p\u0026thinsp;\u0026lt;\u0026thinsp;0.001) (Table\u0026nbsp;\u003cspan refid=\"Tab5\" class=\"InternalRef\"\u003e5\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab5\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 5\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eComparison of Accuracy and Evaluation of Agreement for Responses to Questions Asked Across Gemini Versions\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eGemini\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eKappa/p\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eMcNemar\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eGemini 2.5 Pro\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eGemini 2.5 Flash\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFalse\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e11 (11)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e14 (14)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e0,589/\u003cb\u003e\u0026lt;0,001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e0,508\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTrue\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e89 (89)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e86 (86)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"5\"\u003eFrequency (percentage)\u003c/td\u003e\u003c/tr\u003e \u003ctr\u003e\u003ctd colspan=\"5\"\u003e4o\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eLarge Language Models are transforming medical decision-support processes by leveraging deep learning architectures to analyze extensive natural language data [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. The rapid evolution of artificial intelligence algorithms is steadily increasing their potential for integration into clinical applications [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. Unlike previous studies, this research stands out by simultaneously evaluating four contemporary models and conducting a cross-linguistic analysis to compare Turkish and English performances of two leading models.\u003c/p\u003e \u003cp\u003eThe success rates observed in our study, ranging from 84% to 89%, represent a notable improvement over earlier findings. Previous studies reported lower accuracy rates: 40.8%\u0026ndash;62.6% for ChatGPT-3.5; 65%\u0026ndash;81% for ChatGPT-4; 64.89%\u0026ndash;88.1% for ChatGPT-4o; 38.8% for Google Bard; 55.73% for Google Gemini; and 85.7% for Gemini Advanced [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan additionalcitationids=\"CR11\" citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. The 85% success rate achieved by ChatGPT-5 in our study indicates a significant enhancement in clinical reasoning compared to earlier versions, GPT-3.5 and GPT-4 [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. Similarly, the 89% success rate of Gemini 2.5 Pro highlights substantial technological progress, likely due to its access to up-to-date datasets [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. Nevertheless, since radiology practice fundamentally relies on visual data analysis, the current text-based models alone are insufficient for direct clinical use. The future role of artificial intelligence in radiology will be better defined by models with image interpretation capabilities. Promising developments in medicine-specific, structured multimodal LLMs\u0026mdash;such as Google\u0026rsquo;s Med-PaLM M and Microsoft\u0026rsquo;s LLaVA-M\u0026mdash;show highly encouraging results for radiological diagnostics [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eIn this study, Gemini 2.5 Pro and Claude Sonnet achieved the highest success rates in abdominal and musculoskeletal radiology, while Gemini 2.5 Pro and Grok 4 performed best in interventional radiology. In neuroradiology, Gemini 2.5 Pro and ChatGPT-5 were the top performers, and Claude Sonnet, Grok 4, and ChatGPT-5 excelled in thoracic radiology. The 85% success rates for both ChatGPT-5 and Gemini 2.5 Pro in neuroradiology were notably higher than those reported by Gupta et al. [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. Our results for musculoskeletal and abdominal radiology also significantly surpassed the findings of Huang et al. [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. Payne et al. [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e] noted that ChatGPT-4 performed between first-year and second-year residents (61.9%); in contrast, our study found that Claude Sonnet and Gemini Pro outperformed first-year residents in abdominal radiology, and overall, LLMs and third-year residents achieved higher success rates than first- and second-year residents. These outcomes differ from Sarangi et al., who reported that two senior residents (63.33% and 57.5%) outperformed several AI models (Bard, Bing, and ChatGPT-3.5) [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. The strong performance of large language models on non-visual questions highlights their potential for integration into radiology education. However, for these models to serve as reliable assistants in image interpretation and complex diagnostic processes\u0026mdash;the core of radiology practice\u0026mdash;further evolution of LLM architectures is necessary. Specifically, the development of multimodal capabilities is essential for effective deployment in clinical decision-support systems [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e, \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eWithin this study, all questions were presented to ChatGPT-5 and Gemini 2.5 Pro in both Turkish and English to evaluate the impact of language on model performance. ChatGPT-5 achieved 84 correct responses for English and 85 for Turkish, while Gemini 2.5 Pro scored 90 for English and 89 for Turkish. These findings contrast with Toyama et al. [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e], who attributed lower performance in Japanese to the language's structural complexity. In our study, the similar success rates in both languages\u0026mdash;despite the models\u0026rsquo; primary training on English datasets\u0026mdash;reflect advancements in multilingual processing and language recognition in next-generation models. Meddeb et al. [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e] found that GPT-4 excelled in translating radiology reports from English to German, Greek, Thai, and Turkish, while GPT-3.5 performed best for French. Although these models demonstrated high clarity and semantic consistency, medical terminology accuracy remained moderate, highlighting the need for further research across diverse language families and technical fields.\u003c/p\u003e \u003cp\u003eEfe et al. [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e] observed that ChatGPT-4o and Google Gemini changed responses when questions were repeated, while Brin et al. [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e] noted that ChatGPT-3.5 altered answers for 82.5% of repeated questions, but ChatGPT-4 did not. In our study, Grok-4 and ChatGPT-5 showed moderate agreement between initial and follow-up responses (Kappa\u0026thinsp;=\u0026thinsp;0.575 and Kappa\u0026thinsp;=\u0026thinsp;0.559), whereas Claude Sonnet exhibited very good agreement (Kappa\u0026thinsp;=\u0026thinsp;0.872). The literature frequently highlights the risk of erroneous or fabricated outputs, known as \"hallucinations,\" in LLMs [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. The moderate to very good agreement coefficients in our study suggest a decreasing tendency for hallucinations and increasing response stability in next-generation models.\u003c/p\u003e \u003cp\u003eIntegrating ChatGPT into healthcare applications entails significant risks and limitations beyond its benefits. Ensuring system reliability requires careful management of ethics-based risks, including data security, patient privacy, and algorithmic bias [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. Limitations of our study include the exclusive use of non-visual, multiple-choice questions and a restricted question pool of 100, derived from only two textbooks.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eLarge language models demonstrated high-accuracy performance on fundamental radiology questions, indicating their potential as supportive tools in medical education for providing feedback and optimizing learning processes. Future studies incorporating radiological images will be essential to clarify the role of LLMs in clinical radiology. For full integration into radiology practice, the success of these models in image evaluation must match their strong performance in theoretical examinations.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cdiv class=\"DefinitionList\"\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eAI\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eArtificial Intelligence\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eLLM\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eLarge Language Model\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eNLP\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eNeuro Linguistic Programming\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003c/div\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eEthics Approval and Consent to Participate\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eHuman Proficiency and Ethical Statement:\u003c/strong\u003e Eighteen radiology residents completed the identical question set to serve as a human benchmark. Because the study exclusively analyzed AI‑generated responses and did not involve direct interaction with living subjects, it was deemed exempt from institutional ethics review.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent for Publication Not applicable.\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAvailability of Data and Materials The datasets used and/or analyzed during the current study are not publicly available due to institutional regulations and patient privacy restrictions but are available from the corresponding author on reasonable request, subject to ethical approval and data sharing agreements.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting Interests\u003c/strong\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe authors declare that they have no competing interests.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding\u003c/strong\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eNot applicable\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthors\u0026rsquo; Contributions\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eS.A. planned the study, designed the methodology, and drafted the manuscript. MAG operated LLM exams, preprocess study data, revised the manuscript. All authors read and approved the final manuscript.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e: Not applicable\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eNicikowski J, Szczepanski M, Miedziaszczyk M, Kudlinski B. The potential of ChatGPT in medicine: an example analysis of nephrology specialty exams in Poland. Clin Kidney J. 2024;17(8):sfae193.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBhayana R, Bleakney RR, Krishna S. GPT-4 in Radiology: Improvements in Advanced Reasoning. Radiology. 2023;307(5):e230987.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eIqbal U, Tanweer A, Rahmanti AR, Greenfield D, Lee LT, Li YJ. Impact of large language model (ChatGPT) in healthcare: an umbrella review and evidence synthesis. J Biomed Sci. 2025;32(1):45.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eToyama Y, Harigai A, Abe M, Nagano M, Kawabata M, Seki Y, Takase K. Performance evaluation of ChatGPT, GPT-4, and Bard on the official board examination of the Japan Radiology Society. Jpn J Radiol. 2024;42(2):201\u0026ndash;7.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBhayana R, Krishna S, Bleakney RR. Performance of ChatGPT on a Radiology Board-style Examination: Insights into Current Strengths and Limitations. Radiology. 2023;307(5):e230582.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAggarwal R, Sounderajah V, Martin G, Ting DSW, Karthikesalingam A, King D, Ashrafian H, Darzi A. Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis. NPJ Digit Med. 2021;4(1):65.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSun E, Shi J, Mandell J. Core Radiology: A Visual Approach to Diagnostic Imaging. 2 ed. Philadelphia, PA: Elsevier; 2021.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWeissman A. Top Score for the Radiology Board. New York: Thieme Medical Publishers, Inc.; 2018.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBrin D, Sorin V, Vaid A, Soroush A, Glicksberg BS, Charney AW, Nadkarni G, Klang E. Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments. Sci Rep. 2023;13(1):16492.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHuang KA, Choudhary HK, Hardin WM, Prakash N. Comparative Analysis of ChatGPT-4o and Gemini Advanced Performance on Diagnostic Radiology In-Training Exams. Cureus. 2025;17(3):e80874.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGupta R, Hamid AM, Jhaveri M, Patel N, Suthar PP. Comparative Evaluation of AI Models Such as ChatGPT 3.5, ChatGPT 4.0, and Google Gemini in Neuroradiology Diagnostics. Cureus. 2024;16(8):e67766.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSarangi PK, Narayan RK, Mohakud S, Vats A, Sahani D, Mondal H. Assessing the Capability of ChatGPT, Google Bard, and Microsoft Bing in Solving Radiology Case Vignettes. Indian J Radiol Imaging. 2024;34(2):276\u0026ndash;82.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePayne DL, Purohit K, Borrero WM, Chung K, Hao M, Mpoy M, Jin M, Prasanna P, Hill V. Performance of GPT-4 on the American College of Radiology In-training Examination: Evaluating Accuracy, Model Drift, and Fine-tuning. Acad Radiol. 2024;31(7):3046\u0026ndash;54.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTu T, Azizi S, Driess D, Schaekermann M, Amin M, Chang P, Carroll A, Lau C, Tanno R, Ktena I et al. Towards Generalist Biomedical AI. \u003cem\u003earXiv\u003c/em\u003e 2023, 2307:14334.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi C, Wong C, Zhang S, Usuyama N, Liu H, Yang J, Naumann T, Poon H, Gao J. LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day. arXiv. 2023;2306:00890.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYan Z, Zhang K, Zhou R, He L, Li X, Sun L. Multimodal ChatGPT for Medical Applications: An Experimental Study of GPT-4V. \u003cem\u003earXiv\u003c/em\u003e 2023, 2310:19061.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMeddeb A, Luken S, Busch F, Adams L, Ugga L, Koltsakis E, Tzortzakakis A, Jelassi S, Dkhil I, Klontzas ME, et al. Large Language Model Ability to Translate CT and MRI Free-Text Radiology Reports Into Multiple Languages. Radiology. 2024;313(3):e241736.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eIs EE, Menekseoglu AK. Comparative performance of artificial intelligence models in rheumatology board-level questions: evaluating Google Gemini and ChatGPT-4o. Clin Rheumatol. 2024;43(11):3507\u0026ndash;13.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang S, Zhao Z, Ouyang X, Liu T, Wang Q, Shen D. Interactive computer-aided diagnosis on medical image using large language models. Commun Eng. 2024;3(1):133.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Artificial Intelligence, ChatGPT, Google Gemini, Grok, Claude Sonnet, Large Language Models, Radiology","lastPublishedDoi":"10.21203/rs.3.rs-8642109/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8642109/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground\u003c/strong\u003e: The primary objective of this study is to evaluate the success levels of Large Language Models regarding radiology-related questions and to analyze performance variations between the Turkish and English languages. Furthermore, the consistency of the models' responses to the same questions over different time periods was examined, and the obtained data were analyzed in comparison with the performance levels of radiology residents.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMaterials and Methods: \u003c/strong\u003eThis study evaluated the performance of ChatGPT-5, Grok-4, Claude 4.5 Sonnet, and Gemini 2.5 Pro using 100 multiple-choice radiology questions across five subspecialties. To assess linguistic impact, ChatGPT-5 and Gemini 2.5 Pro were tested in both Turkish and English. Temporal reliability was examined by re-testing ChatGPT-5, Claude 4.5 Sonnet, and Grok-4 after a one-week interval. Finally, AI outputs were benchmarked against a control group of 18 radiology residents (1–3 years of seniority).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults: \u003c/strong\u003eGemini 2.5 Pro achieved the highest accuracy (89%), followed by Claude 4.5 Sonnet (86%), ChatGPT-5 (85%), and Grok-4 (84%). All LLMs and 3rd-year residents (75.8%) significantly outperformed 1st-year (58.7%) and 2nd-year (66%) residents. Subspecialty analysis showed 3rd-year residents excelled in musculoskeletal radiology, while Claude 4.5 and Gemini 2.5 Pro significantly surpassed 1st-year residents in abdominal radiology.\u003c/p\u003e\n\u003cp\u003eNo significant performance gap was found between Turkish and English outputs for ChatGPT-5 and Gemini 2.5 Pro (p = 1.000), indicating good linguistic agreement (κ ≈ 0.73). Regarding temporal reliability, Claude 4.5 Sonnet demonstrated “very good” consistency over one week (κ = 0.872), whereas Grok-4 (κ = 0.575) and ChatGPT-5 (κ = 0.559) showed only “moderate” reliability.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusion:\u003c/strong\u003e Our findings demonstrate that high-performance LLMs, such as Gemini 2.5 Pro, ChatGPT-5, and Grok-4, provide fundamental radiology knowledge with high accuracy and comparable efficiency. These models show significant potential as supportive tools for optimizing radiology medical education. However, further research incorporating image-based datasets is essential to determine their actual clinical efficacy in real-world radiological practice.\u003c/p\u003e","manuscriptTitle":"Large Language Models in Radiology Exams: A Comparative Analysis of Performance in Turkish and English","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-01-21 15:46:53","doi":"10.21203/rs.3.rs-8642109/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"50898d14-2f99-41b5-a508-7e510478bc3a","owner":[],"postedDate":"January 21st, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-01-27T13:41:52+00:00","versionOfRecord":[],"versionCreatedAt":"2026-01-21 15:46:53","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8642109","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8642109","identity":"rs-8642109","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}