Comparative Evaluation the Knowledge of Large Language Models about Response Evaluation Criteria in Solid Tumors?

Comparative Evaluation the Knowledge of Large Language Models about Response Evaluation Criteria in Solid Tumors? | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Comparative Evaluation the Knowledge of Large Language Models about Response Evaluation Criteria in Solid Tumors? Eren Çamur, Turay Cesur, Yasin Celal Güneş This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6122883/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract PURPOSE: To evaluate the diagnostic prowess of eight cutting‐edge large language models (LLMs) in applying the RECIST 1.1 guidelines for oncologic imaging and to compare their performance with that of board‐certified radiologists. This study explores the potential of LLMs as transformative adjuncts in cancer follow‐up imaging. MATERIAL AND METHOD: In this experimental cross‐sectional study, 50 text‐based and 30 case‐based multiple‐choice questions (MCQs) derived from RECIST 1.1 were administered to eight LLMs—including ChatGPT variants, Claude (3 Opus and 3.5 Sonnet), Google Gemini 1.5 Pro, Meta Llama 3.1 405B, Mistral Large 2, and Perplexity Pro—and two junior radiologists with seven years of experience. Responses were independently scored as correct or incorrect, and non‐parametric statistical analyses were performed to compare performance across groups. RESULTS: Strikingly, all LLMs demonstrated competence comparable to that of the radiologists, with only minor performance variations. Claude 3.5 Sonnet led the pack, achieving 83.3% accuracy on case‐based and 90% on text‐based questions. Other models exhibited robust performance, with no significant differences in case‐based assessments between LLMs and radiologists. CONCLUSION: Our findings may pioneer a great change in the reporting of follow-up imaging of cancer patients, which has an important place in clinical practice. The exceptional performance of LLMs,-particularly Claude 3.5 Sonnet- and their peers underscores the promise of LLMs as revolutionary tools in oncologic imaging. These models not only support radiologist but may soon redefine clinical workflows, setting a new benchmark for diagnostic excellence in radiology. ChatGPT cancer large language models radiology RECIST Figures Figure 1 Figure 2 Figure 3 1. INTRODUCTION Large language models (LLMs) represent a remarkable breakthrough in natural language processing, capable of performing specific tasks in radiology without additional training[ 1 – 4 ]. This positions LLMs as transformative forces poised to significantly reshape radiology practice. They have the potential to usher in a new era of efficiency and excellence, both as supportive diagnostic tools and in facilitating the reporting process. Consequently, there has been a rapid increase in studies investigating the radiological knowledge of LLMs and their potential applications and contributions to radiology[ 3 – 7 ]. Although there are many studies evaluating the radiological knowledge of LLMs at different fields, the lack of studies evaluating their knowledge in oncology radiology is an important gap in this regard. The radiology report is vital in guiding patient management in oncology, requiring meticulous comparison with prior studies and assessment. Response Evaluation Criteria in Solid Tumors (RECIST) guideline, revised in 2009 to RECIST 1.1, was developed to address this need. It provides a standardized approach to reporting solid tumor measurements and defines objective criteria for assessing changes in tumor size, ensuring a consistent and reliable approach to reporting[ 8 ]. Previous studies evaluates the proficiency and knowledge of various LLMs in different spesific types of cancer[ 2 , 9 – 12 ]. Güneş et al. tested the performance of current LLMs, in particular Claude 3.5 Sonnet, in interpreting BI-RADS categories via text-based questions and found that these models achieved remarkable accuracy (up to 90%), approaching the level of expertise of breast radiologists[ 13 ]. In another study, Kaba et al. demonstrated that advanced LLMs, especially ChatGPT-4, showed high accuracy (93%) in text-based questions in interpreting thyroid imaging guidelines based on the K-TIRADS classification system and emphasized the competence of LLMs in this field[ 14 ]. To our best knowledge, there is no study has compared the performance of LLMs about RECIST 1.1, a critical guideline in the radiological reporting of follow up imaging of cancer patients. We aimed to fill this gap by evaluating the performance of different LLMs in RECIST 1.1 guideline and compare them with radiologists to reveal their possible contribution in follow-up imaging of cancer patients. 2. MATERIAL and METHOD 2.1 Study Design This experimental study utilized a cross-sectional design to assess the accuracy of different LLMs compared with radiologists in answering text-based and case-based MCQs pertaining to RECIST 1.1. The evaluated LLMs included Perplexity Pro, ChatGPT-4o, ChatGPT-o1, Claude 3 Opus, Claude 3.5 Sonnet, Google Gemini 1.5 Pro, Meta Llama 3.1 405B, and Mistral Large 2. Their answers were benchmarked against responses from two board-certified radiologists (EDiR): Radiologist 1 (Y.C.G.) (R1) and Radiologist 2 (T.C.) (R2), both with seven years of experience in general radiology. The text-based and case-based MCQs were designed based on RECIST 1.1 guideline by board-certified radiologist (EDiR), Radiologist 3 (E.Ç.) (R3), also with seven year experience in radiology. The MCQs did not include any authentic patient data or images; therefore, ethical committee approval was not required or applicable for this study. Methodological transparency and reproducibility were ensured by adhering to the Standards for Reporting Diagnostic Accuracy Studies (STARD) guideline[ 15 ]. An overview of the flowchart is presented in Fig. 1 . 2.2 Data Collection for Text-based and Case-based Multiple Choice Questions A total of 50 text-based MCQs and 30 case-based MCQs were utilized in the study. These questions comprehensively covered the all sections of RECIST 1.1 and tested the application of the information therein. Each question was carefully constructed to focus on a single, specific, and critical concept relevant to radiological practice under this guideline. Each MCQ had 5 choices and only one choice was correct. A complete list of text-based and case-based MCQs and dataset of the study are available in Supplementary Materials. 2.3 Design of Input-Output Procedures for LLMs The input prompt provided to the LLMs was: "Act as a radiology professor with 30 years of experience in radiology, particularly specialized in RECIST 1.1. Provide only one choice that correctly answers each multiple-choice question. Some questions will be case-based; carefully analyze the provided information before answering. Each question has exactly one correct response." This prompt was consistently employed across eight distinct platforms with default hyperparameters by R3 in February 2025: Anthropic’s Claude 3 Opus and 3.5 Sonnet ( https://claude.ai.com ), OpenAI’s ChatGPT-o1, ChatGPT-4o ( https://chat.openai.com ), Google Gemini 1.5 Pro ( https://gemini.google.com ), Mistral Large 2 ( https://mistral.ai ), Meta Llama 3.1 405B ( https://metaai.com ), and Perplexity Pro ( https://perplexity.ai ). The MCQs were administered sequentially within a single conversation session per LLM to maintain uniformity. None of the LLMs underwent additional pre-training or fine-tuning by the study authors, and no supplementary details that could potentially affect the study results were provided (Fig. 2 ). R3 reviewed the LLM responses and categorized them as correct (1) or incorrect (0). 2.4 Radiologist Performance Evaluation R1 and R2, independently answered the MCQs in a blinded manner in January 2025 using their personal computers. They completed text-based MCQs first, immediately followed by case-based MCQs without any interval. R3 separately evaluated their answers and categorized them as correct (1) or incorrect (0). 2.5 Statistical Analysis The Kolmogorov-Smirnov test was conducted to determine data distribution characteristics. Descriptive statistical measures such as minimum, maximum, median, interquartile range, and percentages were computed to summarize the dataset thoroughly. Given the non-normal data distribution, non-parametric tests were utilized for comparative analyses. The McNemar test, appropriate for paired nominal data, compared the correct response rates between different prompts or LLMs. Additionally, the chi-square test was applied to identify performance differences based on question types. 3. RESULTS 3.1 Case-Based MCQs Claude 3.5 Sonnet demonstrated the highest accuracy at 83.3%, followed by R2 and Gemini 1.5 Pro, both achieved 80.0%. R1 closely followed with 76.7%, while ChatGPT-4o, Llama 3.1 405B, and Mistral Large 2 each recorded 73.3%. Claude 3 Opus and ChatGPT-o1 shared an accuracy of 66.7%. Perplexity Pro exhibited the lowest accuracy among LLMs and radiologists, with 60.0% (Fig. 3 ). There was no significant difference in accuracy on case-based MCQs among LLMs and between LLMs and radiologists (Table 1 ). Table 1 Comparison of the Performance of LLMs and Radiologist on Case-based Multiple Choice Questions (p-values are obtained from McNemar test) Claude 3 Opus Claude 3.5 Sonnet Chat GPT-4o Chat GPT-o1 Mistral Large 2 Gemini 1.5 Pro Llama 3.1 405B Perplexity Pro Radiologist 1 Radiologist 2 Claude 3 Opus - 0.063 0.687 1 0.774 0.219 0.774 0.754 0.581 0.344 Claude 3.5 Sonnet 0.063 - 0.453 0.227 0.508 1 0.549 0.092 0.727 1 Chat GPT-4o 0.687 0.453 - 0.687 1 0.625 1 0.289 1 0.774 Chat GPT-o1 1 0.227 0.687 - 0.687 0.289 0.625 0.687 0.549 0.454 Mistral Large 2 0.774 0.508 1 0.687 - 0.754 1 0.344 1 0.791 Gemini 1.5 Pro 0.219 1 0.625 0.289 0.754 - 0.727 0.109 1 1 Llama 3.1 405B 0.774 0.549 1 0.625 1 0.727 - 0.344 1 0.791 Perplexity Pro 0.754 0.092 0.289 0.687 0.344 0.109 0.344 - 0.267 0.210 Radiologist 1 0.581 0.727 1 0.549 1 1 1 0.267 - 1 Radiologist 2 0.344 1 0.774 0.454 0.791 1 0.791 0.210 1 - 3.2 Text-Based MCQs Claude 3.5 Sonnet achieved the highest accuracy at 90.0%, followed by Claude 3 Opus and ChatGPT-o1, both scored 84.0%. ChatGPT-4o recorded an accuracy of 82.0%. Gemini 1.5 Pro attained 74.0%, while R2 (T.C.) followed closely with 72.0%, equaling the performance of Mistral Large 2 and Llama 3.1 405B. R1 had a slightly lower accuracy of 70.0%. Perplexity Pro demonstrated the lowest performance among all models, with an accuracy of 68.0% (Fig. 3 ). Claude 3.5 Sonnet outperformed Mistral Large 2, Llama 3.1 405B, Perplexity Pro achieving the highest scores on text-based questions (p = 0.012, p = 0.022, p = 0.007). It also demonstrated superior performance according to R1 and R2 (p = 0.021, p = 0.049). When other LLMs were compared among themselves and with radiologists, there was no significant difference in performance between them (Table 2 ). Table 2 Comparison of the Performance of LLMs and Radiologist on Text-based Multiple Choice Questions (p-values are obtained from McNemar test) Claude 3 Opus Claude 3.5 Sonnet Chat GPT-4o Chat GPT-o1 Mistral Large 2 Gemini 1.5 Pro Llama 3.1 405B Perplexity Pro Radiologist 1 Radiologist 2 Claude 3 Opus - 0.375 1 1 0.180 0.332 0.146 0.057 0.143 0.210 Claude 3.5 Sonnet 0.375 - 0.219 0.453 0.012 0.077 0.022 0.007 0.021 0.049 Chat GPT-4o 1 0.219 - 1 0.125 0.388 0.267 0.118 0.238 0.359 Chat GPT-o1 1 0.453 1 - 0.109 0.332 0.210 0.115 0.167 0.238 Mistral Large 2 0.180 0.012 0.125 0.109 - 1 1 0.791 1 1 Gemini 1.5 Pro 0.332 0.077 0.388 0.332 1 - 1 0.607 0.804 1 Llama 3.1 405B 0.146 0.022 0.267 0.210 1 1 - 0.804 1 1 Perplexity Pro 0.057 0.007 0.118 0.115 0.791 0.607 0.804 - 1 0.824 Radiologist 1 0.143 0.021 0.238 0.167 1 0.804 1 1 - 1 Radiologist 2 0.210 0.049 0.359 0.238 1 1 1 0.824 1 - 4. DISCUSSION The most striking result of our study is that LLMs included in the study have knowledge and proficiency about RECIST 1.1 comparable to radiologists. Our study uniquely examines the performance of several LLMs about RECIST guideline, comparing their performance with that of radiologists. This approach not only identifies which LLM demonstrates a more comprehensive grasp of RECIST 1.1 but also offers insights into how radiologists' performance stacks up against that of LLMs. Similar to our results, Coşkun et al. extracted 59 questions from the patient information material on prostate cancer available on European Urology Patient Information Society website and posed these questions to ChatGPT[ 16 ]. The researchers assessed the responses using a 5-point Likert scale, yielding a mean score of 3.62 ± 0.49, and concluded that both the accuracy and content quality of ChatGPT's responses were suboptimal and required improvement. Similarly, Lombardo et al. evaluated the August 2023 version of ChatGPT using 195 questions created from the EAU 2023 prostate cancer guidelines. Two expert radiologists reviewed the answers and found that 26% (50/195) were completely correct, 26% (51/195) were correct but inadequate, 24% (47/195) contained a mix of correct and incorrect information, and 24% (47/195) were incorrect. They concluded that ChatGPT demonstrates a low accuracy rate in relation to the EAU 2023 prostate cancer guidelines and suggested that further training is needed. Moreover, their findings indicated that ChatGPT’s performance varied by section, performing better on questions related to "follow-up" and "quality of life," while it fared worst in the "diagnosis" and "treatment" sections[ 17 ]. In a recent study assessing LLMs in breast cancer care, three models—GPT-3.5, GPT-4, and Google Gemini (formerly Bard)—were evaluated using 60 MCQs covering treatment, diagnostic techniques, imaging interpretation, and pathology in breast cancer. Notably, GPT-4 achieved a 95% accuracy rate, outperforming GPT-3.5 (90%) and Google Gemini (80%), with statistically significant differences observed among the models (p = 0.010). Furthermore, the models performed consistently across questions sourced from public databases and those formulated by radiologists[ 18 ]. Also, Cao at al. evaluated LLMs in hepatocellular carcinoma diagnosis and management questions and found that ChatGPT-3.5, Gemini, and Bing answered only 45%, 60%, and 30% of basic clinical questions accurately, respectively, with even fewer responses deemed both accurate and reliable[ 19 ]. Another important result of our study is that LLMs responded as well as radiologists both on text-based and case-based MCQs that require analysis of the findings and datas obtained. This result suggests that LLMs can make correct evaluation by analyzing the findings and reports of cancer patients in clinical practice. To our best knowledge, there is no studies evaluating the performance of LLMs on case-based questions about cancer. Previous studies have evaluated LLMs' knowledge of cancer and cancer-related guidelines only text-based. Çıtır reported that ChatGPT-3.5 gave largely correct answers to questions about oral cancer, 51.25% gave “very good” and 46.25% gave “good” answers, and the overall reliability was 97.5%[ 20 ]. Similarly, Yurtçu et al. ChatGPT demonstrated strong accuracy in answering frequently asked questions about cervical cancer[ 21 ]. Beyond these studies, our study uniquely demonstrates that LLMs perform quite adequately on case-based MCQs in line with the correct analysis. With this finding, we believe that our study may be a leading point for further multicenter studies that evaluates the performance of LLMs based real-patient scenarios. The impressive performance of Claude 3.5 Sonnet—achieving 83.3% accuracy on case-based MCQs and 90% on text-based MCQs—indicates that this model could be pioneer model at this field. The observed minor variations in accuracy among LLMs can be largely attributed to differences in their underlying architectures. Internet-connected models, such as Gemini 1.5 Pro, and Perplexity Pro, frequently derive their responses from non-scientific sources, which may account for the comparatively lower performance of web-enabled LLMs relative to those without internet access. In contrast, all ChatGPT and Claude models are trained on closed datasets, potentially contributing to their enhanced reliability.. 4.1 Limitations Our study has few limitations. First, the number of questions was limited, and the assessment relied solely on MCQs. LLMs may demonstrate enhanced performance in open-ended questions, where they can provide more comprehensive responses. Another critical factor influencing LLM responses is the prompt design. In our study, a single standardized prompt was used for all MCQs. Optimizing prompts for specific questions may enhance LLM performance. Second, we compared the accuracy of LLMs against two general radiologists with seven years of experience. It is likely that more experienced senior radiologists, particularly more specialized about oncological imaging, would achieve higher accuracy. Future studies should incorporate both senior radiologists and a combination of open-ended and multiple-choice questions to provide a more comprehensive evaluation. Third, MCQs used in this study were based on fictional cases. In clinical practice, real-world cases often present greater complexity, requiring more complex analysis and clinical knowledge. Lastly, this study assessed textually performance of LLMs about RECIST 1.1, while visual evaluation remains an integral component of radiological assessment. As such, the results of our study may not fully reflect the real-world applicability of LLMs in this field. To address this, future studies should focus on image-based assessment using multimodal LLMs capable of integrating visual assessment. 5. CONCLUSION Our findings may pioneer a great change in the reporting of follow-up imaging of cancer patients, which has an important place in clinical practice. In particular, the outstanding performance of Claude 3.5 Sonnet suggests that it may be the leading model in this field. Furthermore, our study shows that LLMs have promising potential as supportive tools for reporting of follow-up imaging of cancer patients, which has important place in the management of these patients. Declarations Funding: No funding was received for this study. Human Ethics and Consent to Participate : Human Ethics and Consent to Participate declarations are not applicable for this study. Consent to Participate : Consent to Participate is not applicable for this study. Consent to Publish : Consent to Publish declaration is not applicable for this study. Author Contribution Eren Çamur: Conceptualization, Data curation, Formal, analysis, Methodology, Writing- original draft, Writ-ing- review&editingTuray Cesur: Methodology, Writing- review&editingYasin Celal Güneş: Writing- review&editing Data Availability All data supporting the findings of this study are available within the paper and its Supplementary Materials. References Nakaura, T., Ito, R., Ueda, D., et al. (2024). The impact of large language models on radiology: a guide for radiologists on the latest innovations in AI. Jpn J Radiol Published online March , 29 , 1–12. 10.1007/S11604-024-01552-0/FIGURES/4 Chung, E. M., Zhang, S. C., Nguyen, A. T., Atkins, K. M., Sandler, H. M., & Kamrava, M. (2023). Feasibility and acceptability of ChatGPT generated radiology report summaries for cancer patients. Digit Health , 9 . 10.1177/20552076231221620 Keshavarz, P., Bagherieh, S., Nabipoorashrafi, S. A., et al. (2024). ChatGPT in radiology: A systematic review of performance, pitfalls, and future perspectives. Diagnostic And Interventional Imaging , 105 (7–8), 251–265. 10.1016/J.DIII.2024.04.003 Bhayana, R. (2024). Chatbots and Large Language Models in Radiology: A Practical Primer for Clinical and Research Applications. Radiology , 310 (1). 10.1148/RADIOL.232756 Bera, K., O’Connor, G., Jiang, S., Tirumani, S. H., & Ramaiya, N. (2024). Analysis of ChatGPT publications in radiology: Literature so far. Current Problems In Diagnostic Radiology , 53 (2), 215–225. 10.1067/J.CPRADIOL.2023.10.013 Akinci D’Antonoli, T., Stanzione, A., Bluethgen, C., et al. (2024). Large language models in radiology: fundamentals, applications, ethical considerations, risks, and future directions. Diagnostic and Interventional Radiology , 30 (2), 80–90. 10.4274/dir.2023.232417 Srivastav, S., Chandrakar, R., Gupta, S., et al. (2023). ChatGPT in Radiology: The Advantages and Limitations of Artificial Intelligence for Medical Imaging Diagnosis. Cureus , 15 (7). 10.7759/CUREUS.41435 Eisenhauer, E. A., Therasse, P., Bogaerts, J. New response evaluation criteria in solid tumours: Revised RECIST guideline (version 1.1). European Journal Of Cancer . 45:228–247. 10.1016/j.ejca.2008.10.026 Liu, X., Duan, C., Kim, M. K., et al. (2024). Claude 3 Opus and ChatGPT With GPT-4 in Dermoscopic Image Analysis for Melanoma Diagnosis: Comparative Performance Analysis. JMIR Med Inform , 12 . 10.2196/59273 Chiarelli, G., Stephens, A., Finati, M., et al. (2024). Adequacy of prostate cancer prevention and screening recommendations provided by an artificial intelligence-powered large language model. Int Urol Nephrol Published online April , 2 , 1–7. 10.1007/S11255-024-04009-5/METRICS Aghamaliyev, U., Karimbayli, J., Giessen-Jung, C., et al. (2024). ChatGPT’s Gastrointestinal Tumor Board Tango: A limping dance partner? European Journal Of Cancer , 205 , 114100. 10.1016/j.ejca.2024.114100 Sorin, V., Barash, Y., Konen, E., & Klang, E. (2023). Large language models for oncological applications. Journal Of Cancer Research And Clinical Oncology , 149 (11), 9505–9508. 10.1007/S00432-023-04824-W/METRICS Güneş, Y. C., Cesur, T., Çamur, E., & Günbey Karabekmez, L. (2025). Evaluating text and visual diagnostic capabilities of large language models on questions related to the Breast Imaging Reporting and Data System Atlas 5th edition. Diagn Interv Radiol . ;31(2). 10.4274/DIR.2024.242876 Kaba, E., Hürsoy, N., Solak, M., & Çeliker, F. B. (2024). Accuracy of Large Language Models in Thyroid Nodule-Related Questions Based on the Korean Thyroid Imaging Reporting and Data System (K-TIRADS). Korean Journal Of Radiology , 25 (5), 499–500. 10.3348/KJR.2024.0229 Bossuyt, P. M., Reitsma, J. B., Bruns, D. E. (2015). STARD. : An updated list of essential items for reporting diagnostic accuracy studies1. Radiology . 2015;277(3):826–832. 10.1148/RADIOL.2015151516/ASSET /IMAGES/LARGE/RADIOL.2015151516.FIG1.JPEG Coskun, B., Ocakoglu, G., Yetemen, M., & Kaygisiz, O. (2023). Can ChatGPT, an Artificial Intelligence Language Model, Provide Accurate and High-quality Patient Information on Prostate Cancer? Urology , 180 , 35–58. 10.1016/j.urology.2023.05.040 Lombardo, R., Gallo, G., Stira, J. Quality of information and appropriateness of Open AI outputs for prostate cancer. Prostate Cancer And Prostatic Diseases . Published online 2024. 10.1038/S41391-024-00789-0 Irmici, G., Cozzi, A., Della Pepa, G., et al. (2024). How do large language models answer breast cancer quiz questions? A comparative study of GPT-3.5, GPT-4 and Google Gemini. La Radiologia Medica , 129 (10). 10.1007/S11547-024-01872-1 Cao, J. J., Kwon, D. H., Ghaziani, T. T., et al. (2024). Large language models’ responses to liver cancer surveillance, diagnosis, and management questions: accuracy, reliability, readability. Abdominal Radiology , 49 (12), 4286–4294. 10.1007/S00261-024-04501-7/METRICS Çi Ti, R. M. (2025). ChatGPT and oral cancer: a study on informational reliability. Bmc Oral Health , 25 (1), 86. 10.1186/S12903-025-05479-4/TABLES/2 Yurtcu, E., Ozvural, S., & Keyif, B. (2025). Analyzing the performance of ChatGPT in answering inquiries about cervical cancer. International Journal of Gynecology & Obstetrics , 168 (2), 502–507. 10.1002/IJGO.15861 Additional Declarations No competing interests reported. Supplementary Files SupplementaryMaterial1MCQsandanswers.docx SupplementaryMaterial2Casebasedquestionsandanswers.docx SupplementaryMaterial3Thedatasetofstudy.xlsx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6122883","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":448646871,"identity":"9f67f89d-e2a2-4ba0-aa31-0fa208cafd61","order_by":0,"name":"Eren Çamur","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA3UlEQVRIiWNgGAWjYHAD5gNAwoKBgYd4LWwJDAwJEiRp4TEgTgu/9OFjDxhq7KINbp/5+Jj3h4QcP88Bxg8fc3BrkexLSzdgOJacu+Fc7mZjngQJY8neBmbJmdtwazE4w2MmwdjAnLvhDO82aaCWxA3nGdiYefFosT/D/w2opR6ohecZcVoMeHjYgFoOg7SwQbScbcCvReIMm5lEwrHjuTPPsBkbzkkD+qXnYDNev/D3MD+T+FBTndt3hvnhgzc2NsAQSz744SMeLWCQgMplbCCgfhSMglEwCkYBIQAAdgFHj/IDVkgAAAAASUVORK5CYII=","orcid":"","institution":"Ministry of Health Ankara 29 Mayis State Hospital","correspondingAuthor":true,"prefix":"","firstName":"Eren","middleName":"","lastName":"Çamur","suffix":""},{"id":448646872,"identity":"18c3d00d-d2be-4ecb-9033-85f1cb2b6c53","order_by":1,"name":"Turay Cesur","email":"","orcid":"","institution":"Ankara Mamak State Hospital","correspondingAuthor":false,"prefix":"","firstName":"Turay","middleName":"","lastName":"Cesur","suffix":""},{"id":448646873,"identity":"b0cdc9fa-6728-495d-a889-358902d1c0b4","order_by":2,"name":"Yasin Celal Güneş","email":"","orcid":"","institution":"TC Saglik Bakanligi Kirikkale Yuksek Ihtisas Hastanesi","correspondingAuthor":false,"prefix":"","firstName":"Yasin","middleName":"Celal","lastName":"Güneş","suffix":""}],"badges":[],"createdAt":"2025-02-27 16:53:13","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6122883/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6122883/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":82124903,"identity":"38037cd3-bc16-4888-bb9c-94774cb7c7e0","added_by":"auto","created_at":"2025-05-07 03:41:54","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":72234,"visible":true,"origin":"","legend":"\u003cp\u003eThe workflow of the study\u003c/p\u003e","description":"","filename":"1.jpg","url":"https://assets-eu.researchsquare.com/files/rs-6122883/v1/b021f7c0847d63291e65966d.jpg"},{"id":82123506,"identity":"ac65de68-4bff-4212-a039-4ce534afb750","added_by":"auto","created_at":"2025-05-07 03:33:54","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":63126,"visible":true,"origin":"","legend":"\u003cp\u003eThe Example of Chatsession with ChatGPT-o1\u003c/p\u003e","description":"","filename":"2.jpg","url":"https://assets-eu.researchsquare.com/files/rs-6122883/v1/8a8fd53e2a7d225e8e71d87f.jpg"},{"id":82123512,"identity":"2b63982e-b2fb-40c6-b5f3-a5a7edb420ff","added_by":"auto","created_at":"2025-05-07 03:33:54","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":75304,"visible":true,"origin":"","legend":"\u003cp\u003eThe accuracy of LLMs and Radiologists on Multiple Choice Questions\u003c/p\u003e","description":"","filename":"3.jpg","url":"https://assets-eu.researchsquare.com/files/rs-6122883/v1/a8ea1cb99c74aedcd9110c4e.jpg"},{"id":92265119,"identity":"b29e5560-9296-49de-a2d9-ba2f710b7f9c","added_by":"auto","created_at":"2025-09-26 13:23:24","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1084720,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6122883/v1/645e13b4-3577-4582-a495-feb6f6e7b8be.pdf"},{"id":82124902,"identity":"01c49e30-5887-40b0-912b-f3d79f9259e7","added_by":"auto","created_at":"2025-05-07 03:41:54","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":25142,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryMaterial1MCQsandanswers.docx","url":"https://assets-eu.researchsquare.com/files/rs-6122883/v1/934f00a062da01110bcbc028.docx"},{"id":82123509,"identity":"42617c80-1ace-49d9-a606-63a7cdd88c51","added_by":"auto","created_at":"2025-05-07 03:33:54","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":25940,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryMaterial2Casebasedquestionsandanswers.docx","url":"https://assets-eu.researchsquare.com/files/rs-6122883/v1/f583f44d8b76c9c60b4eeb9c.docx"},{"id":82124906,"identity":"950f397b-8e5f-4393-a25b-0a95540206f6","added_by":"auto","created_at":"2025-05-07 03:41:54","extension":"xlsx","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":14538,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryMaterial3Thedatasetofstudy.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-6122883/v1/7445a31f96fcc64c36245b83.xlsx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Comparative Evaluation the Knowledge of Large Language Models about Response Evaluation Criteria in Solid Tumors?","fulltext":[{"header":"1. INTRODUCTION","content":"\u003cp\u003eLarge language models (LLMs) represent a remarkable breakthrough in natural language processing, capable of performing specific tasks in radiology without additional training[\u003cspan additionalcitationids=\"CR2 CR3\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. This positions LLMs as transformative forces poised to significantly reshape radiology practice. They have the potential to usher in a new era of efficiency and excellence, both as supportive diagnostic tools and in facilitating the reporting process. Consequently, there has been a rapid increase in studies investigating the radiological knowledge of LLMs and their potential applications and contributions to radiology[\u003cspan additionalcitationids=\"CR4 CR5 CR6\" citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. Although there are many studies evaluating the radiological knowledge of LLMs at different fields, the lack of studies evaluating their knowledge in oncology radiology is an important gap in this regard.\u003c/p\u003e \u003cp\u003eThe radiology report is vital in guiding patient management in oncology, requiring meticulous comparison with prior studies and assessment. Response Evaluation Criteria in Solid Tumors (RECIST) guideline, revised in 2009 to RECIST 1.1, was developed to address this need. It provides a standardized approach to reporting solid tumor measurements and defines objective criteria for assessing changes in tumor size, ensuring a consistent and reliable approach to reporting[\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e].\u003c/p\u003e \u003cp\u003ePrevious studies evaluates the proficiency and knowledge of various LLMs in different spesific types of cancer[\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan additionalcitationids=\"CR10 CR11\" citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. G\u0026uuml;neş et al. tested the performance of current LLMs, in particular Claude 3.5 Sonnet, in interpreting BI-RADS categories via text-based questions and found that these models achieved remarkable accuracy (up to 90%), approaching the level of expertise of breast radiologists[\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. In another study, Kaba et al. demonstrated that advanced LLMs, especially ChatGPT-4, showed high accuracy (93%) in text-based questions in interpreting thyroid imaging guidelines based on the K-TIRADS classification system and emphasized the competence of LLMs in this field[\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e].\u003c/p\u003e \u003cp\u003e To our best knowledge, there is no study has compared the performance of LLMs about RECIST 1.1, a critical guideline in the radiological reporting of follow up imaging of cancer patients.\u003c/p\u003e \u003cp\u003e We aimed to fill this gap by evaluating the performance of different LLMs in RECIST 1.1 guideline and compare them with radiologists to reveal their possible contribution in follow-up imaging of cancer patients.\u003c/p\u003e"},{"header":"2. MATERIAL and METHOD","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Study Design\u003c/h2\u003e \u003cp\u003eThis experimental study utilized a cross-sectional design to assess the accuracy of different LLMs compared with radiologists in answering text-based and case-based MCQs pertaining to RECIST 1.1. The evaluated LLMs included Perplexity Pro, ChatGPT-4o, ChatGPT-o1, Claude 3 Opus, Claude 3.5 Sonnet, Google Gemini 1.5 Pro, Meta Llama 3.1 405B, and Mistral Large 2. Their answers were benchmarked against responses from two board-certified radiologists (EDiR): Radiologist 1 (Y.C.G.) (R1) and Radiologist 2 (T.C.) (R2), both with seven years of experience in general radiology. The text-based and case-based MCQs were designed based on RECIST 1.1 guideline by board-certified radiologist (EDiR), Radiologist 3 (E.\u0026Ccedil;.) (R3), also with seven year experience in radiology.\u003c/p\u003e \u003cp\u003eThe MCQs did not include any authentic patient data or images; therefore, ethical committee approval was not required or applicable for this study. Methodological transparency and reproducibility were ensured by adhering to the Standards for Reporting Diagnostic Accuracy Studies (STARD) guideline[\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eAn overview of the flowchart is presented in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Data Collection for Text-based and Case-based Multiple Choice Questions\u003c/h2\u003e \u003cp\u003eA total of 50 text-based MCQs and 30 case-based MCQs were utilized in the study. These questions comprehensively covered the all sections of RECIST 1.1 and tested the application of the information therein. Each question was carefully constructed to focus on a single, specific, and critical concept relevant to radiological practice under this guideline. Each MCQ had 5 choices and only one choice was correct. A complete list of text-based and case-based MCQs and dataset of the study are available in Supplementary Materials.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.3 Design of Input-Output Procedures for LLMs\u003c/h2\u003e \u003cp\u003eThe input prompt provided to the LLMs was: \"Act as a radiology professor with 30 years of experience in radiology, particularly specialized in RECIST 1.1. Provide only one choice that correctly answers each multiple-choice question. Some questions will be case-based; carefully analyze the provided information before answering. Each question has exactly one correct response.\" This prompt was consistently employed across eight distinct platforms with default hyperparameters by R3 in February 2025: Anthropic\u0026rsquo;s Claude 3 Opus and 3.5 Sonnet (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://claude.ai.com\u003c/span\u003e\u003cspan address=\"https://claude.ai.com\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e), OpenAI\u0026rsquo;s ChatGPT-o1, ChatGPT-4o (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://chat.openai.com\u003c/span\u003e\u003cspan address=\"https://chat.openai.com\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e), Google Gemini 1.5 Pro (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://gemini.google.com\u003c/span\u003e\u003cspan address=\"https://gemini.google.com\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e), Mistral Large 2 (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://mistral.ai\u003c/span\u003e\u003cspan address=\"https://mistral.ai\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e), Meta Llama 3.1 405B (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://metaai.com\u003c/span\u003e\u003cspan address=\"https://metaai.com\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e), and Perplexity Pro (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://perplexity.ai\u003c/span\u003e\u003cspan address=\"https://perplexity.ai\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThe MCQs were administered sequentially within a single conversation session per LLM to maintain uniformity. None of the LLMs underwent additional pre-training or fine-tuning by the study authors, and no supplementary details that could potentially affect the study results were provided (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). R3 reviewed the LLM responses and categorized them as correct (1) or incorrect (0).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e2.4 Radiologist Performance Evaluation\u003c/h2\u003e \u003cp\u003eR1 and R2, independently answered the MCQs in a blinded manner in January 2025 using their personal computers. They completed text-based MCQs first, immediately followed by case-based MCQs without any interval. R3 separately evaluated their answers and categorized them as correct (1) or incorrect (0).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e2.5 Statistical Analysis\u003c/h2\u003e \u003cp\u003eThe Kolmogorov-Smirnov test was conducted to determine data distribution characteristics. Descriptive statistical measures such as minimum, maximum, median, interquartile range, and percentages were computed to summarize the dataset thoroughly. Given the non-normal data distribution, non-parametric tests were utilized for comparative analyses. The McNemar test, appropriate for paired nominal data, compared the correct response rates between different prompts or LLMs. Additionally, the chi-square test was applied to identify performance differences based on question types.\u003c/p\u003e \u003c/div\u003e"},{"header":"3. RESULTS","content":"\u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Case-Based MCQs\u003c/h2\u003e \u003cp\u003eClaude 3.5 Sonnet demonstrated the highest accuracy at 83.3%, followed by R2 and Gemini 1.5 Pro, both achieved 80.0%. R1 closely followed with 76.7%, while ChatGPT-4o, Llama 3.1 405B, and Mistral Large 2 each recorded 73.3%. Claude 3 Opus and ChatGPT-o1 shared an accuracy of 66.7%. Perplexity Pro exhibited the lowest accuracy among LLMs and radiologists, with 60.0% (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThere was no significant difference in accuracy on case-based MCQs among LLMs and between LLMs and radiologists (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eComparison of the Performance of LLMs and Radiologist on Case-based Multiple Choice Questions (p-values are obtained from McNemar test)\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"13\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c10\" colnum=\"10\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c11\" colnum=\"11\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c12\" colnum=\"12\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c13\" colnum=\"13\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eClaude 3 Opus\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003eClaude 3.5 Sonnet\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eChat\u003c/p\u003e \u003cp\u003eGPT-4o\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eChat\u003c/p\u003e \u003cp\u003eGPT-o1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e \u003cp\u003eMistral Large 2\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c9\"\u003e \u003cp\u003eGemini 1.5 Pro\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c10\"\u003e \u003cp\u003eLlama 3.1 405B\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c11\"\u003e \u003cp\u003ePerplexity Pro\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c12\"\u003e \u003cp\u003eRadiologist 1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c13\"\u003e \u003cp\u003eRadiologist 2\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eClaude 3 Opus\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003e0.063\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.687\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.774\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.219\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e0.774\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e0.754\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c12\"\u003e \u003cp\u003e0.581\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c13\"\u003e \u003cp\u003e0.344\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eClaude 3.5 Sonnet\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003e0.063\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.453\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.227\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.508\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e0.549\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e0.092\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c12\"\u003e \u003cp\u003e0.727\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c13\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eChat\u003c/b\u003e\u003c/p\u003e \u003cp\u003e\u003cb\u003eGPT-4o\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003e0.687\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003e0.453\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.687\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.625\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e0.289\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c12\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c13\"\u003e \u003cp\u003e0.774\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eChat\u003c/b\u003e\u003c/p\u003e \u003cp\u003e\u003cb\u003eGPT-o1\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003e0.227\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.687\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.687\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.289\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e0.625\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e0.687\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c12\"\u003e \u003cp\u003e0.549\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c13\"\u003e \u003cp\u003e0.454\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eMistral Large 2\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003e0.774\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003e0.508\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.687\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.754\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e0.344\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c12\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c13\"\u003e \u003cp\u003e0.791\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eGemini 1.5 Pro\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003e0.219\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.625\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.289\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.754\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e0.727\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e0.109\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c12\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c13\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eLlama 3.1 405B\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003e0.774\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003e0.549\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.625\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.727\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e0.344\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c12\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c13\"\u003e \u003cp\u003e0.791\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003ePerplexity Pro\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003e0.754\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003e0.092\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.289\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.687\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.344\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.109\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e0.344\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c12\"\u003e \u003cp\u003e0.267\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c13\"\u003e \u003cp\u003e0.210\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eRadiologist 1\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003e0.581\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003e0.727\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.549\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e0.267\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c12\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c13\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eRadiologist 2\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003e0.344\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.774\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.454\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.791\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e0.791\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e0.210\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c12\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c13\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003e3.2 Text-Based MCQs\u003c/h2\u003e \u003cp\u003eClaude 3.5 Sonnet achieved the highest accuracy at 90.0%, followed by Claude 3 Opus and ChatGPT-o1, both scored 84.0%. ChatGPT-4o recorded an accuracy of 82.0%. Gemini 1.5 Pro attained 74.0%, while R2 (T.C.) followed closely with 72.0%, equaling the performance of Mistral Large 2 and Llama 3.1 405B. R1 had a slightly lower accuracy of 70.0%. Perplexity Pro demonstrated the lowest performance among all models, with an accuracy of 68.0% (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eClaude 3.5 Sonnet outperformed Mistral Large 2, Llama 3.1 405B, Perplexity Pro achieving the highest scores on text-based questions (p\u0026thinsp;=\u0026thinsp;0.012, p\u0026thinsp;=\u0026thinsp;0.022, p\u0026thinsp;=\u0026thinsp;0.007). It also demonstrated superior performance according to R1 and R2 (p\u0026thinsp;=\u0026thinsp;0.021, p\u0026thinsp;=\u0026thinsp;0.049).\u003c/p\u003e \u003cp\u003eWhen other LLMs were compared among themselves and with radiologists, there was no significant difference in performance between them (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eComparison of the Performance of LLMs and Radiologist on Text-based Multiple Choice Questions (p-values are obtained from McNemar test)\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"14\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c10\" colnum=\"10\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c11\" colnum=\"11\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c12\" colnum=\"12\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c13\" colnum=\"13\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c14\" colnum=\"14\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eClaude 3 Opus\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003eClaude 3.5 Sonnet\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eChat\u003c/p\u003e \u003cp\u003eGPT-4o\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eChat\u003c/p\u003e \u003cp\u003eGPT-o1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e \u003cp\u003eMistral Large 2\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"2\" nameend=\"c10\" namest=\"c9\"\u003e \u003cp\u003eGemini 1.5 Pro\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c11\"\u003e \u003cp\u003eLlama 3.1 405B\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c12\"\u003e \u003cp\u003ePerplexity Pro\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c13\"\u003e \u003cp\u003eRadiologist 1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c14\"\u003e \u003cp\u003eRadiologist 2\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eClaude 3 Opus\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003e0.375\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c9\" namest=\"c8\"\u003e \u003cp\u003e0.180\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e0.332\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e0.146\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c12\"\u003e \u003cp\u003e0.057\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c13\"\u003e \u003cp\u003e0.143\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c14\"\u003e \u003cp\u003e0.210\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eClaude 3.5 Sonnet\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003e0.375\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.219\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.453\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c9\" namest=\"c8\"\u003e \u003cp\u003e0.012\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e0.077\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e0.022\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c12\"\u003e \u003cp\u003e0.007\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c13\"\u003e \u003cp\u003e0.021\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c14\"\u003e \u003cp\u003e0.049\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eChat\u003c/b\u003e\u003c/p\u003e \u003cp\u003e\u003cb\u003eGPT-4o\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003e0.219\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c9\" namest=\"c8\"\u003e \u003cp\u003e0.125\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e0.388\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e0.267\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c12\"\u003e \u003cp\u003e0.118\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c13\"\u003e \u003cp\u003e0.238\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c14\"\u003e \u003cp\u003e0.359\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eChat\u003c/b\u003e\u003c/p\u003e \u003cp\u003e\u003cb\u003eGPT-o1\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003e0.453\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c9\" namest=\"c8\"\u003e \u003cp\u003e0.109\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e0.332\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e0.210\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c12\"\u003e \u003cp\u003e0.115\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c13\"\u003e \u003cp\u003e0.167\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c14\"\u003e \u003cp\u003e0.238\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eMistral Large 2\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003e0.180\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003e0.012\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.125\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.109\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c9\" namest=\"c8\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c12\"\u003e \u003cp\u003e0.791\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c13\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c14\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eGemini 1.5 Pro\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003e0.332\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003e0.077\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.388\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.332\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c9\" namest=\"c8\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c12\"\u003e \u003cp\u003e0.607\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c13\"\u003e \u003cp\u003e0.804\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c14\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eLlama 3.1 405B\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003e0.146\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003e0.022\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.267\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.210\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c9\" namest=\"c8\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c12\"\u003e \u003cp\u003e0.804\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c13\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c14\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003ePerplexity Pro\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003e0.057\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003e0.007\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.118\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.115\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c9\" namest=\"c8\"\u003e \u003cp\u003e0.791\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e0.607\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e0.804\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c12\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c13\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c14\"\u003e \u003cp\u003e0.824\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eRadiologist 1\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003e0.143\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003e0.021\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.238\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.167\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c9\" namest=\"c8\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e0.804\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c12\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c13\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c14\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eRadiologist 2\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003e0.210\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003e0.049\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.359\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.238\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c9\" namest=\"c8\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c11\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c12\"\u003e \u003cp\u003e0.824\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c13\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c14\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"4. DISCUSSION","content":"\u003cp\u003eThe most striking result of our study is that LLMs included in the study have knowledge and proficiency about RECIST 1.1 comparable to radiologists. Our study uniquely examines the performance of several LLMs about RECIST guideline, comparing their performance with that of radiologists. This approach not only identifies which LLM demonstrates a more comprehensive grasp of RECIST 1.1 but also offers insights into how radiologists' performance stacks up against that of LLMs.\u003c/p\u003e \u003cp\u003eSimilar to our results, Coşkun et al. extracted 59 questions from the patient information material on prostate cancer available on European Urology Patient Information Society website and posed these questions to ChatGPT[\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]. The researchers assessed the responses using a 5-point Likert scale, yielding a mean score of 3.62\u0026thinsp;\u0026plusmn;\u0026thinsp;0.49, and concluded that both the accuracy and content quality of ChatGPT's responses were suboptimal and required improvement. Similarly, Lombardo et al. evaluated the August 2023 version of ChatGPT using 195 questions created from the EAU 2023 prostate cancer guidelines. Two expert radiologists reviewed the answers and found that 26% (50/195) were completely correct, 26% (51/195) were correct but inadequate, 24% (47/195) contained a mix of correct and incorrect information, and 24% (47/195) were incorrect. They concluded that ChatGPT demonstrates a low accuracy rate in relation to the EAU 2023 prostate cancer guidelines and suggested that further training is needed. Moreover, their findings indicated that ChatGPT\u0026rsquo;s performance varied by section, performing better on questions related to \"follow-up\" and \"quality of life,\" while it fared worst in the \"diagnosis\" and \"treatment\" sections[\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. In a recent study assessing LLMs in breast cancer care, three models\u0026mdash;GPT-3.5, GPT-4, and Google Gemini (formerly Bard)\u0026mdash;were evaluated using 60 MCQs covering treatment, diagnostic techniques, imaging interpretation, and pathology in breast cancer. Notably, GPT-4 achieved a 95% accuracy rate, outperforming GPT-3.5 (90%) and Google Gemini (80%), with statistically significant differences observed among the models (p\u0026thinsp;=\u0026thinsp;0.010). Furthermore, the models performed consistently across questions sourced from public databases and those formulated by radiologists[\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e]. Also, Cao at al. evaluated LLMs in hepatocellular carcinoma diagnosis and management questions and found that ChatGPT-3.5, Gemini, and Bing answered only 45%, 60%, and 30% of basic clinical questions accurately, respectively, with even fewer responses deemed both accurate and reliable[\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eAnother important result of our study is that LLMs responded as well as radiologists both on text-based and case-based MCQs that require analysis of the findings and datas obtained. This result suggests that LLMs can make correct evaluation by analyzing the findings and reports of cancer patients in clinical practice. To our best knowledge, there is no studies evaluating the performance of LLMs on case-based questions about cancer. Previous studies have evaluated LLMs' knowledge of cancer and cancer-related guidelines only text-based. \u0026Ccedil;ıtır reported that ChatGPT-3.5 gave largely correct answers to questions about oral cancer, 51.25% gave \u0026ldquo;very good\u0026rdquo; and 46.25% gave \u0026ldquo;good\u0026rdquo; answers, and the overall reliability was 97.5%[\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]. Similarly, Yurt\u0026ccedil;u et al. ChatGPT demonstrated strong accuracy in answering frequently asked questions about cervical cancer[\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]. Beyond these studies, our study uniquely demonstrates that LLMs perform quite adequately on case-based MCQs in line with the correct analysis. With this finding, we believe that our study may be a leading point for further multicenter studies that evaluates the performance of LLMs based real-patient scenarios.\u003c/p\u003e \u003cp\u003eThe impressive performance of Claude 3.5 Sonnet\u0026mdash;achieving 83.3% accuracy on case-based MCQs and 90% on text-based MCQs\u0026mdash;indicates that this model could be pioneer model at this field. The observed minor variations in accuracy among LLMs can be largely attributed to differences in their underlying architectures. Internet-connected models, such as Gemini 1.5 Pro, and Perplexity Pro, frequently derive their responses from non-scientific sources, which may account for the comparatively lower performance of web-enabled LLMs relative to those without internet access. In contrast, all ChatGPT and Claude models are trained on closed datasets, potentially contributing to their enhanced reliability..\u003c/p\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003e4.1 Limitations\u003c/h2\u003e \u003cp\u003eOur study has few limitations. First, the number of questions was limited, and the assessment relied solely on MCQs. LLMs may demonstrate enhanced performance in open-ended questions, where they can provide more comprehensive responses. Another critical factor influencing LLM responses is the prompt design. In our study, a single standardized prompt was used for all MCQs. Optimizing prompts for specific questions may enhance LLM performance.\u003c/p\u003e \u003cp\u003eSecond, we compared the accuracy of LLMs against two general radiologists with seven years of experience. It is likely that more experienced senior radiologists, particularly more specialized about oncological imaging, would achieve higher accuracy. Future studies should incorporate both senior radiologists and a combination of open-ended and multiple-choice questions to provide a more comprehensive evaluation.\u003c/p\u003e \u003cp\u003eThird, MCQs used in this study were based on fictional cases. In clinical practice, real-world cases often present greater complexity, requiring more complex analysis and clinical knowledge.\u003c/p\u003e \u003cp\u003eLastly, this study assessed textually performance of LLMs about RECIST 1.1, while visual evaluation remains an integral component of radiological assessment. As such, the results of our study may not fully reflect the real-world applicability of LLMs in this field. To address this, future studies should focus on image-based assessment using multimodal LLMs capable of integrating visual assessment.\u003c/p\u003e \u003c/div\u003e"},{"header":"5. CONCLUSION","content":"\u003cp\u003eOur findings may pioneer a great change in the reporting of follow-up imaging of cancer patients, which has an important place in clinical practice. In particular, the outstanding performance of Claude 3.5 Sonnet suggests that it may be the leading model in this field. Furthermore, our study shows that LLMs have promising potential as supportive tools for reporting of follow-up imaging of cancer patients, which has important place in the management of these patients.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eFunding:\u003c/h2\u003e \u003cp\u003eNo funding was received for this study.\u003c/p\u003e \u003cp\u003e \u003cb\u003eHuman Ethics and Consent to Participate\u003c/b\u003e: Human Ethics and Consent to Participate declarations are not applicable for this study.\u003c/p\u003e \u003cp\u003e \u003cb\u003eConsent to Participate\u003c/b\u003e: Consent to Participate is not applicable for this study.\u003c/p\u003e \u003cp\u003e \u003cb\u003eConsent to Publish\u003c/b\u003e: Consent to Publish declaration is not applicable for this study.\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eEren \u0026Ccedil;amur: Conceptualization, Data curation, Formal, analysis, Methodology, Writing- original draft, Writ-ing- review\u0026amp;editingTuray Cesur: Methodology, Writing- review\u0026amp;editingYasin Celal G\u0026uuml;neş: Writing- review\u0026amp;editing\u003c/p\u003e\u003ch2\u003eData Availability\u003c/h2\u003e\u003cp\u003eAll data supporting the findings of this study are available within the paper and its Supplementary Materials.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eNakaura, T., Ito, R., Ueda, D., et al. (2024). The impact of large language models on radiology: a guide for radiologists on the latest innovations in AI. \u003cem\u003eJpn J Radiol Published online March\u003c/em\u003e, \u003cem\u003e29\u003c/em\u003e, 1\u0026ndash;12. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/S11604-024-01552-0/FIGURES/4\u003c/span\u003e\u003cspan address=\"10.1007/S11604-024-01552-0/FIGURES/4\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChung, E. M., Zhang, S. C., Nguyen, A. T., Atkins, K. M., Sandler, H. M., \u0026amp; Kamrava, M. (2023). Feasibility and acceptability of ChatGPT generated radiology report summaries for cancer patients. \u003cem\u003eDigit Health\u003c/em\u003e, \u003cem\u003e9\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1177/20552076231221620\u003c/span\u003e\u003cspan address=\"10.1177/20552076231221620\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKeshavarz, P., Bagherieh, S., Nabipoorashrafi, S. A., et al. (2024). ChatGPT in radiology: A systematic review of performance, pitfalls, and future perspectives. \u003cem\u003eDiagnostic And Interventional Imaging\u003c/em\u003e, \u003cem\u003e105\u003c/em\u003e(7\u0026ndash;8), 251\u0026ndash;265. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/J.DIII.2024.04.003\u003c/span\u003e\u003cspan address=\"10.1016/J.DIII.2024.04.003\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBhayana, R. (2024). Chatbots and Large Language Models in Radiology: A Practical Primer for Clinical and Research Applications. \u003cem\u003eRadiology\u003c/em\u003e, \u003cem\u003e310\u003c/em\u003e(1). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1148/RADIOL.232756\u003c/span\u003e\u003cspan address=\"10.1148/RADIOL.232756\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBera, K., O\u0026rsquo;Connor, G., Jiang, S., Tirumani, S. H., \u0026amp; Ramaiya, N. (2024). Analysis of ChatGPT publications in radiology: Literature so far. \u003cem\u003eCurrent Problems In Diagnostic Radiology\u003c/em\u003e, \u003cem\u003e53\u003c/em\u003e(2), 215\u0026ndash;225. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1067/J.CPRADIOL.2023.10.013\u003c/span\u003e\u003cspan address=\"10.1067/J.CPRADIOL.2023.10.013\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAkinci D\u0026rsquo;Antonoli, T., Stanzione, A., Bluethgen, C., et al. (2024). Large language models in radiology: fundamentals, applications, ethical considerations, risks, and future directions. \u003cem\u003eDiagnostic and Interventional Radiology\u003c/em\u003e, \u003cem\u003e30\u003c/em\u003e(2), 80\u0026ndash;90. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.4274/dir.2023.232417\u003c/span\u003e\u003cspan address=\"10.4274/dir.2023.232417\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSrivastav, S., Chandrakar, R., Gupta, S., et al. (2023). ChatGPT in Radiology: The Advantages and Limitations of Artificial Intelligence for Medical Imaging Diagnosis. \u003cem\u003eCureus\u003c/em\u003e, \u003cem\u003e15\u003c/em\u003e(7). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.7759/CUREUS.41435\u003c/span\u003e\u003cspan address=\"10.7759/CUREUS.41435\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eEisenhauer, E. A., Therasse, P., Bogaerts, J. New response evaluation criteria in solid tumours: Revised RECIST guideline (version 1.1). \u003cem\u003eEuropean Journal Of Cancer\u003c/em\u003e. 45:228\u0026ndash;247. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.ejca.2008.10.026\u003c/span\u003e\u003cspan address=\"10.1016/j.ejca.2008.10.026\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu, X., Duan, C., Kim, M. K., et al. (2024). Claude 3 Opus and ChatGPT With GPT-4 in Dermoscopic Image Analysis for Melanoma Diagnosis: Comparative Performance Analysis. \u003cem\u003eJMIR Med Inform\u003c/em\u003e, \u003cem\u003e12\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.2196/59273\u003c/span\u003e\u003cspan address=\"10.2196/59273\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChiarelli, G., Stephens, A., Finati, M., et al. (2024). Adequacy of prostate cancer prevention and screening recommendations provided by an artificial intelligence-powered large language model. \u003cem\u003eInt Urol Nephrol Published online April\u003c/em\u003e, \u003cem\u003e2\u003c/em\u003e, 1\u0026ndash;7. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/S11255-024-04009-5/METRICS\u003c/span\u003e\u003cspan address=\"10.1007/S11255-024-04009-5/METRICS\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAghamaliyev, U., Karimbayli, J., Giessen-Jung, C., et al. (2024). ChatGPT\u0026rsquo;s Gastrointestinal Tumor Board Tango: A limping dance partner? \u003cem\u003eEuropean Journal Of Cancer\u003c/em\u003e, \u003cem\u003e205\u003c/em\u003e, 114100. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.ejca.2024.114100\u003c/span\u003e\u003cspan address=\"10.1016/j.ejca.2024.114100\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSorin, V., Barash, Y., Konen, E., \u0026amp; Klang, E. (2023). Large language models for oncological applications. \u003cem\u003eJournal Of Cancer Research And Clinical Oncology\u003c/em\u003e, \u003cem\u003e149\u003c/em\u003e(11), 9505\u0026ndash;9508. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/S00432-023-04824-W/METRICS\u003c/span\u003e\u003cspan address=\"10.1007/S00432-023-04824-W/METRICS\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eG\u0026uuml;neş, Y. C., Cesur, T., \u0026Ccedil;amur, E., \u0026amp; G\u0026uuml;nbey Karabekmez, L. (2025). Evaluating text and visual diagnostic capabilities of large language models on questions related to the Breast Imaging Reporting and Data System Atlas 5th edition. \u003cem\u003eDiagn Interv Radiol\u003c/em\u003e. ;31(2). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.4274/DIR.2024.242876\u003c/span\u003e\u003cspan address=\"10.4274/DIR.2024.242876\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKaba, E., H\u0026uuml;rsoy, N., Solak, M., \u0026amp; \u0026Ccedil;eliker, F. B. (2024). Accuracy of Large Language Models in Thyroid Nodule-Related Questions Based on the Korean Thyroid Imaging Reporting and Data System (K-TIRADS). \u003cem\u003eKorean Journal Of Radiology\u003c/em\u003e, \u003cem\u003e25\u003c/em\u003e(5), 499\u0026ndash;500. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3348/KJR.2024.0229\u003c/span\u003e\u003cspan address=\"10.3348/KJR.2024.0229\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBossuyt, P. M., Reitsma, J. B., Bruns, D. E. (2015). STARD. : An updated list of essential items for reporting diagnostic accuracy studies1. \u003cem\u003eRadiology\u003c/em\u003e. 2015;277(3):826\u0026ndash;832. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1148/RADIOL.2015151516/ASSET\u003c/span\u003e\u003cspan address=\"10.1148/RADIOL.2015151516/ASSET\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e/IMAGES/LARGE/RADIOL.2015151516.FIG1.JPEG\u003c/span\u003e\u003cspan address=\"http:///IMAGES/LARGE/RADIOL.2015151516.FIG1.JPEG\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCoskun, B., Ocakoglu, G., Yetemen, M., \u0026amp; Kaygisiz, O. (2023). Can ChatGPT, an Artificial Intelligence Language Model, Provide Accurate and High-quality Patient Information on Prostate Cancer? \u003cem\u003eUrology\u003c/em\u003e, \u003cem\u003e180\u003c/em\u003e, 35\u0026ndash;58. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.urology.2023.05.040\u003c/span\u003e\u003cspan address=\"10.1016/j.urology.2023.05.040\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLombardo, R., Gallo, G., Stira, J. Quality of information and appropriateness of Open AI outputs for prostate cancer. \u003cem\u003eProstate Cancer And Prostatic Diseases\u003c/em\u003e. Published online 2024. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1038/S41391-024-00789-0\u003c/span\u003e\u003cspan address=\"10.1038/S41391-024-00789-0\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eIrmici, G., Cozzi, A., Della Pepa, G., et al. (2024). How do large language models answer breast cancer quiz questions? A comparative study of GPT-3.5, GPT-4 and Google Gemini. \u003cem\u003eLa Radiologia Medica\u003c/em\u003e, \u003cem\u003e129\u003c/em\u003e(10). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/S11547-024-01872-1\u003c/span\u003e\u003cspan address=\"10.1007/S11547-024-01872-1\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCao, J. J., Kwon, D. H., Ghaziani, T. T., et al. (2024). Large language models\u0026rsquo; responses to liver cancer surveillance, diagnosis, and management questions: accuracy, reliability, readability. \u003cem\u003eAbdominal Radiology\u003c/em\u003e, \u003cem\u003e49\u003c/em\u003e(12), 4286\u0026ndash;4294. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/S00261-024-04501-7/METRICS\u003c/span\u003e\u003cspan address=\"10.1007/S00261-024-04501-7/METRICS\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003e\u0026Ccedil;i Ti, R. M. (2025). ChatGPT and oral cancer: a study on informational reliability. \u003cem\u003eBmc Oral Health\u003c/em\u003e, \u003cem\u003e25\u003c/em\u003e(1), 86. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1186/S12903-025-05479-4/TABLES/2\u003c/span\u003e\u003cspan address=\"10.1186/S12903-025-05479-4/TABLES/2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYurtcu, E., Ozvural, S., \u0026amp; Keyif, B. (2025). Analyzing the performance of ChatGPT in answering inquiries about cervical cancer. \u003cem\u003eInternational Journal of Gynecology \u0026amp; Obstetrics\u003c/em\u003e, \u003cem\u003e168\u003c/em\u003e(2), 502\u0026ndash;507. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1002/IJGO.15861\u003c/span\u003e\u003cspan address=\"10.1002/IJGO.15861\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"ChatGPT, cancer, large language models, radiology, RECIST","lastPublishedDoi":"10.21203/rs.3.rs-6122883/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6122883/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003ePURPOSE:\u003c/strong\u003e To evaluate the diagnostic prowess of eight cutting‐edge large language models (LLMs) in applying the RECIST 1.1 guidelines for oncologic imaging and to compare their performance with that of board‐certified radiologists. This study explores the potential of LLMs as transformative adjuncts in cancer follow‐up imaging.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMATERIAL AND METHOD:\u003c/strong\u003e In this experimental cross‐sectional study, 50 text‐based and 30 case‐based multiple‐choice questions (MCQs) derived from RECIST 1.1 were administered to eight LLMs—including ChatGPT variants, Claude (3 Opus and 3.5 Sonnet), Google Gemini 1.5 Pro, Meta Llama 3.1 405B, Mistral Large 2, and Perplexity Pro—and two junior radiologists with seven years of experience. Responses were independently scored as correct or incorrect, and non‐parametric statistical analyses were performed to compare performance across groups.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eRESULTS:\u003c/strong\u003e Strikingly, all LLMs demonstrated competence comparable to that of the radiologists, with only minor performance variations. Claude 3.5 Sonnet led the pack, achieving 83.3% accuracy on case‐based and 90% on text‐based questions. Other models exhibited robust performance, with no significant differences in case‐based assessments between LLMs and radiologists.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCONCLUSION:\u003c/strong\u003e Our findings may pioneer a great change in the reporting of follow-up imaging of cancer patients, which has an important place in clinical practice. The exceptional performance of LLMs,-particularly Claude 3.5 Sonnet- and their peers underscores the promise of LLMs as revolutionary tools in oncologic imaging. These models not only support radiologist but may soon redefine clinical workflows, setting a new benchmark for diagnostic excellence in radiology.\u003c/p\u003e","manuscriptTitle":"Comparative Evaluation the Knowledge of Large Language Models about Response Evaluation Criteria in Solid Tumors?","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-05-07 03:33:50","doi":"10.21203/rs.3.rs-6122883/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"fb7988b6-d779-4e36-a46f-7e858bf31d9f","owner":[],"postedDate":"May 7th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-09-26T13:23:15+00:00","versionOfRecord":[],"versionCreatedAt":"2025-05-07 03:33:50","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6122883","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6122883","identity":"rs-6122883","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}