Evaluation of ChatGPT-4o’s and DeepSeek R1’s responses to urological problems: A comparative study

Evaluation of ChatGPT-4o’s and DeepSeek R1’s responses to urological problems: A comparative study | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Evaluation of ChatGPT-4o’s and DeepSeek R1’s responses to urological problems: A comparative study Hanbo Lu, Yusa Zhang, Zhan Wang, Yang Zhao, Jiang Liu, Dongxu Qiu, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9012062/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 11 You are reading this latest preprint version Abstract Background Urology presents unique challenges for AI systems, requiring both extensive medical knowledge and advanced reasoning. While large language models (LLMs) like GPT-4 have shown promise in medical education and decision support, their performance in urology remains underexplored. Objective To compare the performance of two advanced large language models (LLMs), ChatGPT-4o and DeepSeek R1, in answering urology-related single-choice questions, and to evaluate their accuracy, stability, and reasoning capability across different response configurations. Methods A total of 809 single-choice questions from the Chinese National Qualification Examination for Attending Physicians in Urology were administered to ChatGPT-4o and DeepSeek R1. Each model was tested under three configurations: standard mode, advanced reasoning mode, and retrieval-augmented generation (RAG). Accuracy was calculated for each configuration, and statistical comparisons were performed using McNemar’s test with effect sizes expressed as Cohen’s h. Stability across reasoning modes was assessed by comparing performance variability. Additional analyses examined performance differences between short-answer and case-based clinical questions. Results ChatGPT-4o achieved accuracy rates of 78.12%, 73.79%, and 78.99% in standard, advanced reasoning, and RAG modes, respectively. DeepSeek R1 outperformed ChatGPT-4o across all configurations, with accuracy rates of 83.19%, 81.46%, and 84.55%, respectively. All between-model differences were statistically significant (p < 0.001), with small-to-medium effect sizes (Cohen’s h = 0.129, 0.185, and 0.144). DeepSeek R1 demonstrated substantially greater internal stability across reasoning modes, whereas ChatGPT-4o showed notable variability. In subgroup analyses, DeepSeek R1 exhibited a more pronounced advantage in complex, case-based clinical questions. Both models performed consistently across urological disease categories, and findings were limited to the Chinese-language context in which the evaluation was conducted. Conclusion DeepSeek R1 showed superior performance compared with ChatGPT-4o in both accuracy and stability when answering urology-related examination questions, particularly in complex case-based scenarios. These results suggest that optimized LLMs may serve as valuable tools in medical education and clinical decision support, especially within Chinese-language environments. Further research is needed to assess their generalizability across languages, clinical settings, and more diverse task formats. Figures Figure 1 Figure 2 Figure 3 Figure 4 Background Urology, with its complex and diverse range of conditions, from urinary tract infections to urological cancers, presents a unique challenge for artificial intelligence (AI) systems( 1 , 2 ). Accurate and reliable assistance in this field requires not only vast medical knowledge but also advanced reasoning abilities to interpret nuanced clinical scenarios. Previous studies have demonstrated the potential of large language models (LLMs) in answering medical questions( 3 ). For example, research in radiology showed that GPT-4 achieved over 80% accuracy in board-exam style questions( 4 ), and in dentistry, LLMs were able to correctly answer more than two-thirds of national specialty examination questions( 5 ). However, systematic and cross-model comparisons across specialized medical disciplines remain scarce. Such comparisons are critical for medical professionals who rely on these tools for learning and decision support. With the rapid development of AI, LLMs such as OpenAI’s ChatGPT have shown promise in medical education( 6 ), diagnostic reasoning, and decision support( 7 ). Extant research has examined the performance of LLMs in standardized medical licensing examinations, including the United States Medical Licensing Examination (USMLE) ( 8 ), the American Board of Anesthesiology (ABA) examinations( 9 ), the Japanese Medical Licensing Examination (JMLE)( 10 ), and the Chinese National Medical Licensing Examination (CNMLE)( 11 ). These models can process complex clinical information and generate human-like responses, enabling novel applications in both learning and practice. In parallel, emerging models like DeepSeek R1 have been developed and fine-tuned with domain-specific data, offering potentially superior performance in non-English or specialty-specific contexts( 12 – 14 ). Despite these advances, the comparative effectiveness of LLM in specialized fields like urology remains underexplored. To address this gap, the present study conducts a head-to-head evaluation of ChatGPT-4o and DeepSeek R1 across a comprehensive set of 809 urology-related single-choice questions from the Chinese National Qualification Examination for Attending Physicians in Urology. By comparing their accuracy, reasoning stability, repeat stability and performance across different urological domains and question types, we aim to assess their potential as supportive tools in medical education and clinical decision-making within urology. Methods Study Design : This was a cross-sectional, in silico evaluation designed to compare the performance of two large language models (LLMs), ChatGPT-4o and DeepSeek R1, in solving urology-related single-choice questions. The evaluation workflow is illustrated in Fig. 1 . A total of 809 standardized single-choice questions in urology were submitted to both models in three configurations: standard mode, advanced reasoning mode (deep-thinking), and retrieval-augmented generation (RAG). All six model configurations were evaluated independently under controlled and isolated testing conditions. Potential leakage between the test set and model training data is strictly prohibited, because the questions are not accessible in the open Internet. Each model received the same set of questions, in the same format, without prompts, feedback, or iterative interactions to simulate real-world usage scenarios. Each model’s accuracy was determined by the average performance from three independent evaluations, ensuring consistency across trials. The trials were repeated three times, simulating real-world re-prompting. Data Source : All 809 questions were derived from the National Qualification Examination for Attending Physicians in Urology, ensuring that they reflect standard clinical knowledge and practice guidelines in China. Each item was accompanied by an officially released standard answer, guaranteeing the accuracy and reliability of the reference solutions. The dataset covered a wide spectrum of urological topics, including: Genitourinary tuberculosis (91 questions); Urolithiasis (69 questions); Urological trauma (90 questions); Urinary tract infections (116 questions); Urinary dysfunction (43 questions); Adrenal gland disorders (92 questions); Obstructive uropathy (68 questions); Congenital anomalies (73 questions) and Urological cancers (167 questions). Additionally, the questions were stratified into short-answer format (n = 523) and case-based clinical scenarios (n = 286). A case-based question presents a clinical scenario that requires diagnostic reasoning and management decision-making. For instance: A 35-year-old man presented with a painless right testicular mass for 2 months. Physical examination revealed an enlarged right testis with a palpable hard mass and a feeling of heaviness; the transillumination test was negative. Scrotal ultrasonography suggested a right testicular tumor. Chest X-ray showed no space-occupying lesions, while abdominopelvic CT indicated retroperitoneal lymph node metastasis. After a right radical orchiectomy, the pathological diagnosis was seminoma. What should be the next step in management? A. Radiotherapy B. Close follow-up observation C. Combination chemotherapy D. Radiotherapy followed by chemotherapy E. Retroperitoneal lymph node dissection (Correct answer: A. Radiotherapy.) In contrast, a short-answer question focuses on factual recall or key conceptual understanding, such as: Testicular tumors most commonly occur in young and middle-aged men. The diagnosis is based on the following key points, except : A. Unilateral testicular enlargement or palpable mass, smooth surface, firm consistency, nonelastic, and feeling of heaviness B. Elevated β-HCG, AFP, and lactate dehydrogenase (LDH) C. Ultrasonography and CT examination D. MRI examination E. Intermittent hematospermia (Correct answer: E. Intermittent hematospermia.) AI Models : Six large language model (LLM) configurations were evaluated in this study, encompassing two base models, ChatGPT-4o ( https://platform.openai.com/docs/models/chatgpt-4o-latest ) and DeepSeek R1 ( https://github.com/deepseek-ai/DeepSeek-R1 ), with three different response modes for each. The tested configurations were: ChatGPT-4o (Standard Mode) ChatGPT-4o with Advanced Reasoning Mode (ChatGPT-4o deep-thinking) ChatGPT-4o with Retrieval-Augmented Generation (RAG) DeepSeek R1 (Standard Mode) DeepSeek R1 with Advanced Reasoning Mode (DeepSeek R1 deep-thinking) DeepSeek R1 with Retrieval-Augmented Generation (RAG) The standard and advanced reasoning modes operated in closed environments with no internet access and relied solely on the models’ internal knowledge bases. In contrast, the RAG configurations were permitted to access external online resources in real time, simulating an open-book setting to retrieve potentially relevant information before generating answers( 15 ). Different modes are reproducible by choosing different options in the dialogue page. For each configuration, all 809 urological single-choice questions were submitted individually via direct copy-paste of the question stem and options, without additional prompts, context, or formatting changes. Each model was given the same input text in an isolated session to prevent memory leakage or carryover effects. The outputs were recorded verbatim and later scored based on correctness( 16 ). Statistical Analysis : Statistical analyses were conducted to compare the accuracy of the six AI model configurations across 809 urological single-choice questions. Accuracy was defined as the proportion of correct responses. When comparing different models evaluated on the same set of questions, McNemar’s test was used to account for paired binary outcomes. When comparing the performance of the same model across different question categories (e.g., short-answer vs. case-based) or disease subgroups, where observations were independent, Pearson’s chi-square (χ²) test was applied. Effect sizes and 95% Clopper-Pearson confidence intervals (CIs) were calculated for accuracy estimates for each model and question type. Paired accuracy differences between models were also computed, with 95% confidence intervals estimated to assess the magnitude and precision of performance gaps. A two-sided p-value < 0.05 was considered statistically significant. All analyses were performed using Python (version 3.12.7) with the pandas, numpy, and scipy.stats libraries, and visualizations were generated using matplotlib and seaborn. Results Comparative Accuracy and Robustness Across Disease Categories As shown in Fig. 2 -A, DeepSeek R1 demonstrated consistently higher accuracy across all three operational modes compared with ChatGPT-4o. In the basic mode, DeepSeek R1 achieved an accuracy of 83.19%, significantly outperforming ChatGPT-4o (78.12%, p < 0.001, Cohen’s h = 0.129). In the deep-thinking mode, DeepSeek R1 reached 81.46%, exceeding ChatGPT-4o’s 73.79% (p < 0.001, h = 0.185). Similarly, in the RAG mode, DeepSeek R1 obtained the highest overall accuracy of 84.55%, compared with 78.99% for ChatGPT-4o (p < 0.001, h = 0.144). The paired accuracy differences with 95% confidence intervals are shown in Fig. 2 -B, where all intervals exclude zero (Basic: − 6.76% to − 3.37%; Deep-thinking: − 9.71% to − 5.62%; RAG: − 7.45% to − 3.68%), indicating statistically robust and practically meaningful improvements. Overall, these results confirm a consistent performance advantage of DeepSeek R1 across all configurations, with small-to-medium effect sizes by Cohen’s h criteria. Across the nine urological disease categories, each model demonstrated relatively consistent accuracy (Fig. 2 -C). No significant inter-disease variation was observed for any single model (p > 0.05), indicating stable generalization across diverse diagnostic topics. While minor fluctuations existed, DeepSeek R1-RAG maintained both high and stable accuracy across diseases, whereas the basic versions of both models exhibited wider variability.The heatmap visualization in Fig. 2 -D further depicts detailed performance patterns across model configurations and disease types. DeepSeek R1 Exhibits Superior Intra-Model Stability and Consistency Across Multiple Reasoning Modes and Repeated Tests When evaluating performance consistency across different reasoning modes, DeepSeek R1 demonstrated notably greater stability in accuracy compared with ChatGPT-4o. The 95% confidence intervals for DeepSeek R1 across its three modes (Basic, Deep-thinking, and RAG) were narrower and showed substantial overlap, suggesting that its performance remained steady regardless of reasoning strategy. In contrast, ChatGPT-4o exhibited wider confidence intervals and greater dispersion, indicating higher variability in accuracy between modes (Fig. 3 -A). Notably, this stability was consistent even in the repeated testing of the same 809 urology-related questions. To assess test–retest stability, each model independently answered the entire question set three times under identical conditions. A question was considered unstable if any of the three responses differed. The proportion of such unstable questions was then calculated and compared between models across reasoning modes. Specifically, when evaluating the model’s accuracy across three separate trials, DeepSeek R1 demonstrated significantly fewer unstable questions. For example, in the basic mode, DeepSeek R1 exhibited only 3.8% unstable answers compared to 6.1% for ChatGPT-4o (p = 0.0021, Cohen’s h = 0.16). This pattern held across all reasoning modes—Deep-thinking (3.7% vs. 6.3%, p = 0.0005, h = 0.18) and RAG (3.2% vs. 5.8%, p = 0.0003, h = 0.17)—further reinforcing the superior stability of DeepSeek R1. As shown in the paired comparison, the 95% confidence intervals of accuracy differences between ChatGPT-4o and DeepSeek R1 remained consistently below zero across all reasoning modes, with mean paired differences ranging from − 2.9% to − 3.7% (Fig. 3 -B). DeepSeek R1 Outperforms ChatGPT-4o More Significantly in Case-Based Questions, Highlighting Superior Reasoning Capabilities To evaluate model performance across different question types, short-answer and case-based items were analyzed separately. For short-answer questions, DeepSeek R1 achieved accuracies of 82.4%, 80.7%, and 84.5% under the basic, deep-thinking, and RAG modes, whereas ChatGPT-4o scored 78.2%, 74.6%, and 80.5%. For case-based questions, DeepSeek R1 achieved 84.6%, 82.9%, and 84.6%, compared with ChatGPT-4o’s 78.0%, 72.4%, and 76.2%. When viewed through the 95% confidence interval plots, the differences are visually evident: for short-answer questions, the intervals are close, indicating a small difference between models, whereas for case-based questions, the intervals are clearly separated, highlighting a substantial performance gap in favor of DeepSeek R1 (Fig. 4 -C). Within-model comparisons showed no statistically significant differences between performance on short-answer and case-based questions for either model (Fig. 4 -A,B). However, DeepSeek R1 consistently outperforms ChatGPT-4o across question types, and its advantage is more pronounced in case-based items. This suggests a potential strength in handling context-rich, clinically oriented reasoning tasks. This outcome may result from the differences in training data sources and model tuning. Discussion Principal Findings In recent years, large language models (LLMs) have shown great promise in supporting clinical decision-making, medical education, and diagnostic reasoning( 6 ). Several studies have evaluated the reasoning capabilities of models like GPT-4 in various medical domains( 6 , 17 ), including general internal medicine( 3 ), oncology( 18 ), and radiology( 19 ). However, there is a lack of large-scale, domain-specific benchmarking focused on urology, especially one that compares different operational modes (basic, deep-thinking, and RAG) across multiple LLM platforms. In this study, we systematically evaluated six model configurations, ChatGPT-4o and DeepSeek R1, each in basic, deep-thinking, and RAG modes, across 809 urology-related questions spanning ten disease categories. Our findings demonstrate that DeepSeek R1 consistently outperforms ChatGPT-4o in overall accuracy, with DeepSeek R1-RAG achieving the highest score (84.55%). DeepSeek R1 also exhibited superior internal stability across different reasoning modes, in contrast to ChatGPT-4o which showed significant variability. Besides, during repeated tests, DeepSeek R1 demonstrated greater stability than ChatGPT across all modes. Importantly, while both models performed stably across different urological disease types, DeepSeek R1 showed more pronounced advantages in complex, context-rich case-based questions. Interpretation and Strengths Specifically, we show that while ChatGPT-4o performs well overall, DeepSeek R1—especially with RAG enabled—achieves higher and more stable accuracy in urology, a field that demands nuanced reasoning and contextual understanding. Our subgroup analysis revealed that DeepSeek R1’s advantage is particularly evident in case-based clinical questions, indicating stronger capabilities in multi-step reasoning and decision-making. This may stem from differences in training data coverage, retrieval strategies, and reasoning architecture between the two models. Compared with prior benchmarking studies on English-language medical examinations( 6 , 20 ), where Med-PaLM 2 achieved near-physician-level performance with up to 86.5% accuracy on medical exam datasets and GPT-4 outperformed GPT-3.5 on the Polish Medical Final Examination with an average accuracy of 79.7%, the performance observed in our study (78–85%) appears lower. However, these differences likely reflect the distinct dataset characteristics and linguistic complexity rather than model limitations. Importantly, our dataset—derived from the Chinese National Qualification Examination for Attending Physicians in Urology—represents a large-scale, high-quality corpus aligned with real clinical education and practice in China, which has never been used before. As such, our findings offer valuable insight into how general-purpose LLMs perform under authentic Chinese-language conditions, providing a more practically relevant benchmark for large-scale application and localized optimization in Chinese medical contexts. Our study also presents several advantages over previous work. First, we used the largest urology-specific question set to date, covering a wide range of clinically relevant subdomains. Second, we employed multiple operational modes for each model, allowing for a fine-grained comparison of reasoning strategies. Third, we incorporated statistical comparisons not only across models but also across disease types and question formats, offering a more comprehensive and multidimensional evaluation framework. Clinical and Technical Implications Differences between ChatGPT-4o and DeepSeek R1 have important implications for clinical practice( 21 ) and education in urology( 22 ). For instance, in residency training or standardized examination preparation, a more stable and accurate model like DeepSeek R1 could provide reliable automated question banks or real-time tutoring support. In clinical decision-support contexts, improved consistency reduces the risk of contradictory outputs when models are queried multiple times about diagnostic or management strategies, which is essential for maintaining clinician trust. Regarding model implementation, each model was evaluated under three built-in operational configurations—basic, deep-thinking, and retrieval-augmented generation (RAG)—as provided by their respective platforms. These modes were selected within the model interface to represent varying levels of reasoning depth and knowledge retrieval integration. The RAG configuration, in particular, enables the model to reference external or pre-indexed knowledge sources during reasoning, which may explain its superior performance in complex, case-based questions. Limitations However, several limitations should be acknowledged. First, although the question set was large and diverse, it originated from a single national platform, which may introduce regional or linguistic bias and limit generalizability beyond China( 23 , 24 ). Nevertheless, as fundamental principles of medical reasoning are largely universal, this influence on overall validity is likely minimal( 25 ). Second, the evaluation involved only single-choice questions and structured scenarios, which cannot fully represent real-world diagnostic reasoning( 26 ). This design may underestimate models’ reasoning depth but also reduces the risk of open-ended hallucinations. Future studies incorporating open-ended prompts or multi-answer questions would provide a more comprehensive assessment( 27 ). Third, although DeepSeek R1 (RAG mode) achieved an accuracy of about 85%—above both the national passing threshold (60%) and average physician scores (68.7%)—direct comparison with human experts using external resources (“Human RAG”) remains lacking due to the absence of relevant data from proctored examinations. Fourth, while the single-choice format inherently minimizes the risk of hallucination by constraining possible outputs, it does not eliminate the issue entirely. Standardized frameworks for quantitative hallucination evaluation are still urgently needed to ensure the safe and trustworthy clinical application of medical LLMs, particularly when models are used in open-ended or decision-support contexts( 28 ). Ethical Considerations The integration of large language models (LLMs) into clinical and educational settings raises important ethical considerations( 29 ). Although both ChatGPT-4o and DeepSeek R1 demonstrated high accuracy, the observed performance gaps highlight the need for cautious deployment, particularly in scenarios involving clinical decision-making. Issues such as potential bias, overreliance on AI-generated outputs, and the absence of clear accountability frameworks must be addressed before widespread implementation. Ensuring transparency in model design, maintaining clinician oversight, and establishing rigorous validation protocols will be essential to safeguard patient safety and professional integrity( 30 ). Future Directions Future research should expand the evaluation to other languages and true clinical data, integrate open-ended and multi-turn interactions, and explore alignment strategies to further improve clinical safety. Additionally, investigating user-centered metrics such as trustworthiness, interpretability, and integration into real clinical workflows would be valuable( 31 , 32 ). As LLMs continue to evolve, rigorous domain-specific benchmarking—such as the one we present here—will be critical for informing safe and effective AI deployment in medical practice. Conclusion Both ChatGPT and DeepSeek R1 maintained consistent performance across diverse urological disease categories without significant disease-specific fluctuations. However, compared with ChatGPT, DeepSeek R1-RAG achieved the higher overall accuracy and greater stability, especially in handling complex, context-rich medical reasoning tasks. It should be noted that our evaluation was limited to Chinese-language data, and the generalizability of these findings to other countries remains to be validated. These findings suggest that DeepSeek R1, especially in RAG mode, may offer more reliable and accurate assistance in urology-related AI applications. Declarations Ethics approval and consent to participate: Not applicable. Competing interests: The authors declare that they have no conflict of interest. Funding: Postdoctoral Fellowship Program of CPSF (Grant Number: GZC20230301);Beijing Natural Science Foundation (L258065); CAMS Innovation Fund for Medical Sciences(CIFMS)(2024-I2M-C&T-B-023) Availability of data and materials: The datasets used and/or analyzed during the current study are available from the corresponding author on request. Author contributions: Conceptualization, supervision and funding acquisition: YSZ; Writing original draft and data curation: HBL; Review and editing: DXQ; Visualization: YSZ; Resources: ZW, YZ, JL. Acknowledgements: None. Consent for publication: Not applicable. References Chen X, Wang L, You M, Liu W, Fu Y, Xu J, et al. Evaluating and Enhancing Large Language Models’ Performance in Domain-Specific Medicine: Development and Usability Study With DocOA. J Med Internet Res. 2024;26:e58158. Davis R, Eppler M, Ayo-Ajibola O, Loh-Doyle JC, Nabhani J, Samplaski M, et al. Evaluating the Effectiveness of Artificial Intelligence-powered Large Language Models Application in Disseminating Appropriate and Readable Health Information in Urology. J Urol. 2023;210(4):688–94. Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepano C, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2(2):e0000198. Wei B. Performance Evaluation and Implications of Large Language Models in Radiology Board Exams: Prospective Comparative Analysis. JMIR Med Educ. 2025;11:e64284. Ceki̇c EC, Tavsan O. Evaluating large language models using national endodontic specialty examination questions: are they ready for real-world dentistry? BMC Med Educ. 2025;25(1):1308. Singhal K, Tu T, Gottweis J, Sayres R, Wulczyn E, Amin M, et al. Toward expert-level medical question answering with large language models. Nat Med. 2025;31(3):943–50. Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med. 2023;29(8):1930–40. Bicknell BT, Butler D, Whalen S, Ricks J, Dixon CJ, Clark AB, et al. ChatGPT-4 Omni Performance in USMLE Disciplines and Clinical Skills: Comparative Analysis. JMIR Med Educ. 2024;10:e63430. Angel MC, Rinehart JB, Cannesson MP, Baldi P. Clinical Knowledge and Reasoning Abilities of AI Large Language Models in Anesthesiology: A Comparative Study on the American Board of Anesthesiology Examination. Anesth Analg. 2024;139(2):349–56. Takagi S, Watari T, Erabi A, Sakaguchi K. Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study. JMIR Med Educ. 2023;9:e48002. Ming S, Guo Q, Cheng W, Lei B. Influence of Model Evolution and System Roles on ChatGPT's Performance in Chinese Medical Licensing Exams: Comparative Study. JMIR Med Educ. 2024;10:e52784. Luo PW, Liu JW, Xie X, Jiang JW, Huo XY, Chen ZL, et al. DeepSeek vs ChatGPT: a comparison study of their performance in answering prostate cancer radiotherapy questions in multiple languages. Am J Clin Exp Urol. 2025;13(2):176–85. Qiu P, Wu C, Zhang X, Lin W, Wang H, Zhang Y, et al. Towards building multilingual language model for medicine. Nat Commun. 2024;15(1):8384. Peng Y, Malin BA, Rousseau JF, Wang Y, Xu Z, Xu X, et al. From GPT to DeepSeek: Significant gaps remain in realizing AI in healthcare. J Biomed Inf. 2025;163:104791. Zakka C, Shad R, Chaurasia A, Dalal AR, Kim JL, Moor M et al. Almanac - Retrieval-Augmented Language Models for Clinical Medicine. NEJM AI. 2024;1(2). Bedi S, Liu Y, Orr-Ewing L, Dash D, Koyejo S, Callahan A, et al. Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review. JAMA. 2025;333(4):319–28. Lievin V, Hother CE, Motzfeldt AG, Winther O. Can large language models reason about medical questions? Patterns (N Y). 2024;5(3):100943. Zhu M, Lin H, Jiang J, Jinia AJ, Jee J, Pichotta K, et al. Large language model trained on clinical oncology data predicts cancer progression. NPJ Digit Med. 2025;8(1):397. Rao A, Kim J, Kamineni M, Pang M, Lie W, Dreyer KJ, et al. Evaluating GPT as an Adjunct for Radiologic Decision Making: GPT-4 Versus GPT-3.5 in a Breast Imaging Pilot. J Am Coll Radiol. 2023;20(10):990–7. Rosol M, Gasior JS, Laba J, Korzeniewski K, Mlynczak M. Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final Examination. Sci Rep. 2023;13(1):20512. Collin H, Roberts MJ, Keogh K, Siriwardana A, Basto M. Improving clinical efficiency using retrieval-augmented generation in urologic oncology: A guideline-enhanced artificial intelligence approach. BJUI Compass. 2025;6(1):e427. Shah MS, Buck M. Man plus machine: large language models can augment medical student education. Nat Rev Urol. 2025. Wu J, Wang Z, Qin Y. Performance of DeepSeek-R1 and ChatGPT-4o on the Chinese National Medical Licensing Examination: A Comparative Study. J Med Syst. 2025;49(1):74. Tanaka Y, Nakata T, Aiga K, Etani T, Muramatsu R, Katagiri S, et al. Performance of Generative Pretrained Transformer on the National Medical Licensing Examination in Japan. PLOS Digit Health. 2024;3(1):e0000433. Sozen Yanik I, Sahin Hazir D, Bilgin Avsar D. Cross-lingual performance of large language models in maxillofacial prosthodontics: a comparative evaluation. BMC Oral Health. 2025;25(1):1630. Gurbuz S, Bahar H, Yavuz U, Keskin A, Karslioglu B, Solak Y. Comparative Efficacy of ChatGPT and DeepSeek in Addressing Patient Queries on Gonarthrosis and Total Knee Arthroplasty. Arthroplast Today. 2025;33:101730. Mehandru N, Miao BY, Almaraz ER, Sushil M, Butte AJ, Alaa A. Evaluating large language models as agents in the clinic. NPJ Digit Med. 2024;7(1):84. Asgari E, Montana-Brown N, Dubois M, Khalil S, Balloch J, Yeung JA, et al. A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation. NPJ Digit Med. 2025;8(1):274. Li X, Yan X, Lai H. The ethical challenges in the integration of artificial intelligence and large language models in medical education: A scoping review. PLoS ONE. 2025;20(10):e0333411. Nouis SC, Uren V, Jariwala S. Evaluating accountability, transparency, and bias in AI-assisted healthcare decision- making: a qualitative study of healthcare professionals' perspectives in the UK. BMC Med Ethics. 2025;26(1):89. Xu R, Hong Y, Zhang F, Xu H. Evaluation of the integration of retrieval-augmented generation in large language model for breast cancer nursing care responses. Sci Rep. 2024;14(1):30794. Prasad S, Langlie J, Pasick L, Chen R, Franzmann E. Evaluating advanced AI reasoning models: ChatGPT-4.0 and DeepSeek-R1 diagnostic performance in otolaryngology: a comparative analysis. Am J Otolaryngol. 2025;46(4):104667. Additional Declarations No competing interests reported. Supplementary Files Resultdataframe.xls Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 15 May, 2026 Reviews received at journal 09 Apr, 2026 Reviewers agreed at journal 09 Apr, 2026 Reviewers agreed at journal 08 Apr, 2026 Reviews received at journal 03 Apr, 2026 Reviewers agreed at journal 02 Apr, 2026 Reviewers invited by journal 02 Apr, 2026 Editor invited by journal 05 Mar, 2026 Editor assigned by journal 03 Mar, 2026 Submission checks completed at journal 03 Mar, 2026 First submitted to journal 02 Mar, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9012062","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":618446186,"identity":"f01bd0a0-575f-4cb2-957c-8cb136d900ec","order_by":0,"name":"Hanbo Lu","email":"","orcid":"","institution":"Department of Urology, Peking Union Medical College Hospital, Peking Union Medical College, Chinese Academy of Medical Sciences","correspondingAuthor":false,"prefix":"","firstName":"Hanbo","middleName":"","lastName":"Lu","suffix":""},{"id":618446187,"identity":"ef491776-a3a8-4de5-ae6a-c3d2119190bc","order_by":1,"name":"Yusa Zhang","email":"","orcid":"","institution":"Eight-year Program of Clinical Medicine，Peking Union Medical College \u0026 Chinese Academy of Sciences","correspondingAuthor":false,"prefix":"","firstName":"Yusa","middleName":"","lastName":"Zhang","suffix":""},{"id":618446191,"identity":"524f5bd2-3d43-402a-9de5-1043e56dac96","order_by":2,"name":"Zhan Wang","email":"","orcid":"","institution":"Department of Urology, Peking Union Medical College Hospital, Peking Union Medical College, Chinese Academy of Medical Sciences","correspondingAuthor":false,"prefix":"","firstName":"Zhan","middleName":"","lastName":"Wang","suffix":""},{"id":618446192,"identity":"d2673a3c-4efe-4605-983f-4ef45dc5b4ee","order_by":3,"name":"Yang Zhao","email":"","orcid":"","institution":"Department of Urology, Peking Union Medical College Hospital, Peking Union Medical College, Chinese Academy of Medical Sciences","correspondingAuthor":false,"prefix":"","firstName":"Yang","middleName":"","lastName":"Zhao","suffix":""},{"id":618446193,"identity":"99f29086-9a7c-426c-8d1b-80371d8119fe","order_by":4,"name":"Jiang Liu","email":"","orcid":"","institution":"Department of Urology, Peking Union Medical College Hospital, Peking Union Medical College, Chinese Academy of Medical Sciences","correspondingAuthor":false,"prefix":"","firstName":"Jiang","middleName":"","lastName":"Liu","suffix":""},{"id":618446194,"identity":"a4b0f36d-f009-478e-ae3d-bea3943c4174","order_by":5,"name":"Dongxu Qiu","email":"","orcid":"","institution":"Department of Urology, Peking Union Medical College Hospital, Peking Union Medical College, Chinese Academy of Medical Sciences","correspondingAuthor":false,"prefix":"","firstName":"Dongxu","middleName":"","lastName":"Qiu","suffix":""},{"id":618446195,"identity":"73b25a1b-ca07-429e-a9f0-5f3f87e4cda6","order_by":6,"name":"Yushi Zhang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABGUlEQVRIiWNgGAWjYBACAwbGBoMEMAMCEtjYGxsffCBNC8/hZsMZeLWgMRIYJNLbpDnwaDFnb24oeLij1t6c/ezh17xtdXl8kg8bpBkY7OR0G7Brsew52GCQeOY4s2VPXpo1b9vhYjbpxAbjAoZkY7MDOBx2IxGope0Ym8GBHDNj3rYDiW1ALckzGA4kbsOl5f5DsBYeg/NvQFrqEtskDzYc5sGn5QYjSEuNhMGNHOPHvG3MiW0SjI3N+LRY9oAddsDA4MYbM8Y55w4ntvEkNjPOMMDtF3P2488Mf7bV2RuczzH+8KasLnF++/HnPz5U2Mnh0gIEbMAoOQxmSPEgHIxTOQgwP2BgqAMzPv7Aq3AUjIJRMApGKgAAWLtlqdHKFoEAAAAASUVORK5CYII=","orcid":"","institution":"Department of Urology, Peking Union Medical College Hospital, Peking Union Medical College, Chinese Academy of Medical Sciences","correspondingAuthor":true,"prefix":"","firstName":"Yushi","middleName":"","lastName":"Zhang","suffix":""}],"badges":[],"createdAt":"2026-03-02 15:54:03","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9012062/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9012062/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":106404474,"identity":"023919b5-19d2-4a3e-9a04-41f193f74878","added_by":"auto","created_at":"2026-04-08 09:16:05","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":75186,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eWorkflow of Model Evaluation in Urological Question Answering. A total of 809 urology-related questions were collected from China’s largest online medical learning platform. Six model configurations, including three versions of ChatGPT-4o and three of DeepSeek R1 (basic, deep-thinking, and retrieval-augmented generation [RAG]), were tested. Model performance was evaluated based on overall accuracy, response stability and accuracy between different question types.\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"Fig1.jpg","url":"https://assets-eu.researchsquare.com/files/rs-9012062/v1/ca14544dafc07a01715dba69.jpg"},{"id":106403990,"identity":"496ae665-3043-4a69-8ca9-3127de775dd5","added_by":"auto","created_at":"2026-04-08 09:15:20","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":126893,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eComparative Performance of ChatGPT-4o and DeepSeek R1 Models on Urological Question Sets (A) Model Accuracy Comparison. Bar plots showing mean accuracy (%, y-axis) of ChatGPT-4o and DeepSeek R1 under three configurations (Basic, Deep-thinking, and RAG). Error bars indicate Clopper-Pearson 95% confidence intervals. All pairwise differences between models were statistically significant (p \u0026lt; 0.001, McNemar’s test). (B) Paired Accuracy Difference with 95% Confidence Intervals. Forest-style plot illustrating the paired differences (ChatGPT − DeepSeek R1) with corresponding 95% confidence intervals. Negative values indicate higher accuracy for DeepSeek R1 across all configurations. (C) Accuracy of Each Model on Different Urological Disease Categories. Line plots depicting the variation of accuracy across nine urological domains (x-axis). Each color represents one model configuration, and the p-values refer to within-model differences across disease categories (Chi-square test). (D) Accuracy Heatmap Across Diseases and Model Configurations. Heatmap summarizing model performance (accuracy %) across disease categories (x-axis) and model types (y-axis). Darker red shades indicate higher accuracy.\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"Fig2.jpg","url":"https://assets-eu.researchsquare.com/files/rs-9012062/v1/2d35e82dbfe2705b87b9fad1.jpg"},{"id":106404570,"identity":"c4d85d92-49b6-4741-9bbc-c0f8ba2c39b7","added_by":"auto","created_at":"2026-04-08 09:16:16","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":65648,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eStability of Accuracy Across Reasoning Modes and Repeated Testing Within Each Model. (A) DeepSeek R1 demonstrated markedly higher intra-model stability than ChatGPT-4o, as reflected by narrower and overlapping 95% confidence intervals across reasoning modes (basic, deep-thinking, and RAG). (B) In the repeated testing of the same 809 urology-related questions, DeepSeek R1 produced significantly fewer unstable responses across all modes, confirming superior test–retest reliability(McNemar’s test). The 95% confidence intervals of paired accuracy differences between ChatGPT-4o and DeepSeek R1 remained consistently below zero across all reasoning modes, further supporting the robustness of DeepSeek R1’s performance.\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"Fig3.jpg","url":"https://assets-eu.researchsquare.com/files/rs-9012062/v1/1fc5c75030b22952e88e4cb0.jpg"},{"id":106382519,"identity":"6cc6b272-8d33-4632-83e7-697d78572055","added_by":"auto","created_at":"2026-04-08 05:28:26","extension":"jpg","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":78999,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eComparison of Model Accuracy Across Question Types and AI Architectures (A) Accuracy of six models on short-answer and case-based medical questions. Bars represent mean accuracy ± standard error. No statistically significant differences were observed between short-answer and case-based formats within the same model (all p \u0026gt; 0.05, Chi-square test). (B) Heatmap showing detailed accuracy values for each model and question format. (C) Side-by-side comparison of ChatGPT-4o and DeepSeek R1 performance across three reasoning modes (Basic, Deep-thinking, and RAG) for short-answer and case-based questions. While DeepSeek R1 consistently outperformed ChatGPT-4o across all modes, the performance gap was more pronounced for case-based questions\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"Fig4.jpg","url":"https://assets-eu.researchsquare.com/files/rs-9012062/v1/10533e9f244c7ece665988bd.jpg"},{"id":106405970,"identity":"d6dc6df7-6377-4edc-b982-07c4e319ceaf","added_by":"auto","created_at":"2026-04-08 09:29:10","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2360476,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9012062/v1/6c56b989-9ebf-402d-8c8f-07009101d659.pdf"},{"id":106382517,"identity":"9f371cf8-980f-45ba-9d87-ce8ecdc92433","added_by":"auto","created_at":"2026-04-08 05:28:26","extension":"xls","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":26624,"visible":true,"origin":"","legend":"","description":"","filename":"Resultdataframe.xls","url":"https://assets-eu.researchsquare.com/files/rs-9012062/v1/97588d291a6e97a9e72e4ebd.xls"}],"financialInterests":"No competing interests reported.","formattedTitle":"Evaluation of ChatGPT-4o’s and DeepSeek R1’s responses to urological problems: A comparative study","fulltext":[{"header":"Background","content":"\u003cp\u003eUrology, with its complex and diverse range of conditions, from urinary tract infections to urological cancers, presents a unique challenge for artificial intelligence (AI) systems(\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e). Accurate and reliable assistance in this field requires not only vast medical knowledge but also advanced reasoning abilities to interpret nuanced clinical scenarios. Previous studies have demonstrated the potential of large language models (LLMs) in answering medical questions(\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e). For example, research in radiology showed that GPT-4 achieved over 80% accuracy in board-exam style questions(\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e), and in dentistry, LLMs were able to correctly answer more than two-thirds of national specialty examination questions(\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e). However, systematic and cross-model comparisons across specialized medical disciplines remain scarce. Such comparisons are critical for medical professionals who rely on these tools for learning and decision support.\u003c/p\u003e \u003cp\u003eWith the rapid development of AI, LLMs such as OpenAI\u0026rsquo;s ChatGPT have shown promise in medical education(\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e), diagnostic reasoning, and decision support(\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e). Extant research has examined the performance of LLMs in standardized medical licensing examinations, including the United States Medical Licensing Examination (USMLE) (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e), the American Board of Anesthesiology (ABA) examinations(\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e), the Japanese Medical Licensing Examination (JMLE)(\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e), and the Chinese National Medical Licensing Examination (CNMLE)(\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e). These models can process complex clinical information and generate human-like responses, enabling novel applications in both learning and practice. In parallel, emerging models like DeepSeek R1 have been developed and fine-tuned with domain-specific data, offering potentially superior performance in non-English or specialty-specific contexts(\u003cspan additionalcitationids=\"CR13\" citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e). Despite these advances, the comparative effectiveness of LLM in specialized fields like urology remains underexplored.\u003c/p\u003e \u003cp\u003eTo address this gap, the present study conducts a head-to-head evaluation of ChatGPT-4o and DeepSeek R1 across a comprehensive set of 809 urology-related single-choice questions from the Chinese National Qualification Examination for Attending Physicians in Urology. By comparing their accuracy, reasoning stability, repeat stability and performance across different urological domains and question types, we aim to assess their potential as supportive tools in medical education and clinical decision-making within urology.\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003e \u003cb\u003eStudy Design\u003c/b\u003e: This was a cross-sectional, in silico evaluation designed to compare the performance of two large language models (LLMs), ChatGPT-4o and DeepSeek R1, in solving urology-related single-choice questions. The evaluation workflow is illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. A total of 809 standardized single-choice questions in urology were submitted to both models in three configurations: standard mode, advanced reasoning mode (deep-thinking), and retrieval-augmented generation (RAG). All six model configurations were evaluated independently under controlled and isolated testing conditions. Potential leakage between the test set and model training data is strictly prohibited, because the questions are not accessible in the open Internet. Each model received the same set of questions, in the same format, without prompts, feedback, or iterative interactions to simulate real-world usage scenarios. Each model\u0026rsquo;s accuracy was determined by the average performance from three independent evaluations, ensuring consistency across trials. The trials were repeated three times, simulating real-world re-prompting.\u003c/p\u003e \u003cp\u003e\u003cb\u003eData Source\u003c/b\u003e: All 809 questions were derived from the National Qualification Examination for Attending Physicians in Urology, ensuring that they reflect standard clinical knowledge and practice guidelines in China. Each item was accompanied by an officially released standard answer, guaranteeing the accuracy and reliability of the reference solutions. The dataset covered a wide spectrum of urological topics, including: Genitourinary tuberculosis (91 questions); Urolithiasis (69 questions); Urological trauma (90 questions); Urinary tract infections (116 questions); Urinary dysfunction (43 questions); Adrenal gland disorders (92 questions); Obstructive uropathy (68 questions); Congenital anomalies (73 questions) and Urological cancers (167 questions). Additionally, the questions were stratified into short-answer format (n\u0026thinsp;=\u0026thinsp;523) and case-based clinical scenarios (n\u0026thinsp;=\u0026thinsp;286).\u003c/p\u003e \u003cp\u003eA case-based question presents a clinical scenario that requires diagnostic reasoning and management decision-making. For instance: \u003cem\u003eA 35-year-old man presented with a painless right testicular mass for 2 months. Physical examination revealed an enlarged right testis with a palpable hard mass and a feeling of heaviness; the transillumination test was negative. Scrotal ultrasonography suggested a right testicular tumor. Chest X-ray showed no space-occupying lesions, while abdominopelvic CT indicated retroperitoneal lymph node metastasis. After a right radical orchiectomy, the pathological diagnosis was seminoma. What should be the next step in management?\u003c/em\u003e\u003c/p\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eA. Radiotherapy\u003c/h2\u003e \u003cdiv id=\"Sec4\" class=\"Section3\"\u003e \u003ch2\u003eB. Close follow-up observation\u003c/h2\u003e \u003cp\u003e \u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cem\u003eC. Combination chemotherapy\u003c/em\u003e \u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cem\u003eD. Radiotherapy followed by chemotherapy\u003c/em\u003e \u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cem\u003eE. Retroperitoneal lymph node dissection\u003c/em\u003e \u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e \u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e(Correct answer: A. Radiotherapy.)\u003c/h2\u003e \u003cp\u003eIn contrast, a short-answer question focuses on factual recall or key conceptual understanding, such as: \u003cem\u003eTesticular tumors most commonly occur in young and middle-aged men. The diagnosis is based on the following key points, except\u003c/em\u003e:\u003c/p\u003e \u003cp\u003e \u003cem\u003eA. Unilateral testicular enlargement or palpable mass, smooth surface, firm consistency, nonelastic, and feeling of heaviness\u003c/em\u003e \u003c/p\u003e \u003cp\u003e \u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cem\u003eB. Elevated β-HCG, AFP, and lactate dehydrogenase (LDH)\u003c/em\u003e \u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cem\u003eC. Ultrasonography and CT examination\u003c/em\u003e \u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cem\u003eD. MRI examination\u003c/em\u003e \u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cem\u003eE. Intermittent hematospermia\u003c/em\u003e \u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003e(Correct answer: E. Intermittent hematospermia.)\u003c/em\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003eAI Models\u003c/b\u003e: Six large language model (LLM) configurations were evaluated in this study, encompassing two base models, ChatGPT-4o (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://platform.openai.com/docs/models/chatgpt-4o-latest\u003c/span\u003e\u003cspan address=\"https://platform.openai.com/docs/models/chatgpt-4o-latest\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e) and DeepSeek R1 (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://github.com/deepseek-ai/DeepSeek-R1\u003c/span\u003e\u003cspan address=\"https://github.com/deepseek-ai/DeepSeek-R1\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e), with three different response modes for each. The tested configurations were:\u003c/p\u003e \u003cp\u003eChatGPT-4o (Standard Mode)\u003c/p\u003e \u003cp\u003eChatGPT-4o with Advanced Reasoning Mode (ChatGPT-4o deep-thinking)\u003c/p\u003e \u003cp\u003eChatGPT-4o with Retrieval-Augmented Generation (RAG)\u003c/p\u003e \u003cp\u003eDeepSeek R1 (Standard Mode)\u003c/p\u003e \u003cp\u003eDeepSeek R1 with Advanced Reasoning Mode (DeepSeek R1 deep-thinking)\u003c/p\u003e \u003cp\u003eDeepSeek R1 with Retrieval-Augmented Generation (RAG)\u003c/p\u003e \u003cp\u003eThe standard and advanced reasoning modes operated in closed environments with no internet access and relied solely on the models\u0026rsquo; internal knowledge bases. In contrast, the RAG configurations were permitted to access external online resources in real time, simulating an open-book setting to retrieve potentially relevant information before generating answers(\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e). Different modes are reproducible by choosing different options in the dialogue page.\u003c/p\u003e \u003cp\u003eFor each configuration, all 809 urological single-choice questions were submitted individually via direct copy-paste of the question stem and options, without additional prompts, context, or formatting changes. Each model was given the same input text in an isolated session to prevent memory leakage or carryover effects. The outputs were recorded verbatim and later scored based on correctness(\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cb\u003eStatistical Analysis\u003c/b\u003e: Statistical analyses were conducted to compare the accuracy of the six AI model configurations across 809 urological single-choice questions. Accuracy was defined as the proportion of correct responses. When comparing different models evaluated on the same set of questions, McNemar\u0026rsquo;s test was used to account for paired binary outcomes. When comparing the performance of the same model across different question categories (e.g., short-answer vs. case-based) or disease subgroups, where observations were independent, Pearson\u0026rsquo;s chi-square (χ\u0026sup2;) test was applied.\u003c/p\u003e \u003cp\u003eEffect sizes and 95% Clopper-Pearson confidence intervals (CIs) were calculated for accuracy estimates for each model and question type. Paired accuracy differences between models were also computed, with 95% confidence intervals estimated to assess the magnitude and precision of performance gaps. A two-sided p-value\u0026thinsp;\u0026lt;\u0026thinsp;0.05 was considered statistically significant. All analyses were performed using Python (version 3.12.7) with the pandas, numpy, and scipy.stats libraries, and visualizations were generated using matplotlib and seaborn.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"Results","content":"\u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003eComparative Accuracy and Robustness Across Disease Categories\u003c/h2\u003e \u003cp\u003eAs shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e-A, DeepSeek R1 demonstrated consistently higher accuracy across all three operational modes compared with ChatGPT-4o. In the basic mode, DeepSeek R1 achieved an accuracy of 83.19%, significantly outperforming ChatGPT-4o (78.12%, p\u0026thinsp;\u0026lt;\u0026thinsp;0.001, Cohen\u0026rsquo;s h\u0026thinsp;=\u0026thinsp;0.129). In the deep-thinking mode, DeepSeek R1 reached 81.46%, exceeding ChatGPT-4o\u0026rsquo;s 73.79% (p\u0026thinsp;\u0026lt;\u0026thinsp;0.001, h\u0026thinsp;=\u0026thinsp;0.185). Similarly, in the RAG mode, DeepSeek R1 obtained the highest overall accuracy of 84.55%, compared with 78.99% for ChatGPT-4o (p\u0026thinsp;\u0026lt;\u0026thinsp;0.001, h\u0026thinsp;=\u0026thinsp;0.144).\u003c/p\u003e \u003cp\u003eThe paired accuracy differences with 95% confidence intervals are shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e-B, where all intervals exclude zero (Basic: \u0026minus;\u0026thinsp;6.76% to \u0026minus;\u0026thinsp;3.37%; Deep-thinking: \u0026minus;\u0026thinsp;9.71% to \u0026minus;\u0026thinsp;5.62%; RAG: \u0026minus;\u0026thinsp;7.45% to \u0026minus;\u0026thinsp;3.68%), indicating statistically robust and practically meaningful improvements. Overall, these results confirm a consistent performance advantage of DeepSeek R1 across all configurations, with small-to-medium effect sizes by Cohen\u0026rsquo;s h criteria.\u003c/p\u003e \u003cp\u003eAcross the nine urological disease categories, each model demonstrated relatively consistent accuracy (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e-C). No significant inter-disease variation was observed for any single model (p\u0026thinsp;\u0026gt;\u0026thinsp;0.05), indicating stable generalization across diverse diagnostic topics. While minor fluctuations existed, DeepSeek R1-RAG maintained both high and stable accuracy across diseases, whereas the basic versions of both models exhibited wider variability.The heatmap visualization in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e-D further depicts detailed performance patterns across model configurations and disease types.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eDeepSeek R1 Exhibits Superior Intra-Model Stability and Consistency Across Multiple Reasoning Modes and Repeated Tests\u003c/h2\u003e \u003cp\u003eWhen evaluating performance consistency across different reasoning modes, DeepSeek R1 demonstrated notably greater stability in accuracy compared with ChatGPT-4o. The 95% confidence intervals for DeepSeek R1 across its three modes (Basic, Deep-thinking, and RAG) were narrower and showed substantial overlap, suggesting that its performance remained steady regardless of reasoning strategy. In contrast, ChatGPT-4o exhibited wider confidence intervals and greater dispersion, indicating higher variability in accuracy between modes (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e-A).\u003c/p\u003e \u003cp\u003eNotably, this stability was consistent even in the repeated testing of the same 809 urology-related questions. To assess test\u0026ndash;retest stability, each model independently answered the entire question set three times under identical conditions. A question was considered unstable if any of the three responses differed. The proportion of such unstable questions was then calculated and compared between models across reasoning modes. Specifically, when evaluating the model\u0026rsquo;s accuracy across three separate trials, DeepSeek R1 demonstrated significantly fewer unstable questions. For example, in the basic mode, DeepSeek R1 exhibited only 3.8% unstable answers compared to 6.1% for ChatGPT-4o (p\u0026thinsp;=\u0026thinsp;0.0021, Cohen\u0026rsquo;s h\u0026thinsp;=\u0026thinsp;0.16). This pattern held across all reasoning modes\u0026mdash;Deep-thinking (3.7% vs. 6.3%, p\u0026thinsp;=\u0026thinsp;0.0005, h\u0026thinsp;=\u0026thinsp;0.18) and RAG (3.2% vs. 5.8%, p\u0026thinsp;=\u0026thinsp;0.0003, h\u0026thinsp;=\u0026thinsp;0.17)\u0026mdash;further reinforcing the superior stability of DeepSeek R1. As shown in the paired comparison, the 95% confidence intervals of accuracy differences between ChatGPT-4o and DeepSeek R1 remained consistently below zero across all reasoning modes, with mean paired differences ranging from \u0026minus;\u0026thinsp;2.9% to \u0026minus;\u0026thinsp;3.7% (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e-B).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003eDeepSeek R1 Outperforms ChatGPT-4o More Significantly in Case-Based Questions, Highlighting Superior Reasoning Capabilities\u003c/h2\u003e \u003cp\u003eTo evaluate model performance across different question types, short-answer and case-based items were analyzed separately. For short-answer questions, DeepSeek R1 achieved accuracies of 82.4%, 80.7%, and 84.5% under the basic, deep-thinking, and RAG modes, whereas ChatGPT-4o scored 78.2%, 74.6%, and 80.5%. For case-based questions, DeepSeek R1 achieved 84.6%, 82.9%, and 84.6%, compared with ChatGPT-4o\u0026rsquo;s 78.0%, 72.4%, and 76.2%. When viewed through the 95% confidence interval plots, the differences are visually evident: for short-answer questions, the intervals are close, indicating a small difference between models, whereas for case-based questions, the intervals are clearly separated, highlighting a substantial performance gap in favor of DeepSeek R1 (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e-C).\u003c/p\u003e \u003cp\u003eWithin-model comparisons showed no statistically significant differences between performance on short-answer and case-based questions for either model (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e-A,B). However, DeepSeek R1 consistently outperforms ChatGPT-4o across question types, and its advantage is more pronounced in case-based items. This suggests a potential strength in handling context-rich, clinically oriented reasoning tasks. This outcome may result from the differences in training data sources and model tuning.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"Discussion","content":"\u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003ePrincipal Findings\u003c/h2\u003e \u003cp\u003eIn recent years, large language models (LLMs) have shown great promise in supporting clinical decision-making, medical education, and diagnostic reasoning(\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e). Several studies have evaluated the reasoning capabilities of models like GPT-4 in various medical domains(\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e), including general internal medicine(\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e), oncology(\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e), and radiology(\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e). However, there is a lack of large-scale, domain-specific benchmarking focused on urology, especially one that compares different operational modes (basic, deep-thinking, and RAG) across multiple LLM platforms.\u003c/p\u003e \u003cp\u003eIn this study, we systematically evaluated six model configurations, ChatGPT-4o and DeepSeek R1, each in basic, deep-thinking, and RAG modes, across 809 urology-related questions spanning ten disease categories. Our findings demonstrate that DeepSeek R1 consistently outperforms ChatGPT-4o in overall accuracy, with DeepSeek R1-RAG achieving the highest score (84.55%). DeepSeek R1 also exhibited superior internal stability across different reasoning modes, in contrast to ChatGPT-4o which showed significant variability. Besides, during repeated tests, DeepSeek R1 demonstrated greater stability than ChatGPT across all modes. Importantly, while both models performed stably across different urological disease types, DeepSeek R1 showed more pronounced advantages in complex, context-rich case-based questions.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003eInterpretation and Strengths\u003c/h2\u003e \u003cp\u003eSpecifically, we show that while ChatGPT-4o performs well overall, DeepSeek R1\u0026mdash;especially with RAG enabled\u0026mdash;achieves higher and more stable accuracy in urology, a field that demands nuanced reasoning and contextual understanding. Our subgroup analysis revealed that DeepSeek R1\u0026rsquo;s advantage is particularly evident in case-based clinical questions, indicating stronger capabilities in multi-step reasoning and decision-making. This may stem from differences in training data coverage, retrieval strategies, and reasoning architecture between the two models.\u003c/p\u003e \u003cp\u003eCompared with prior benchmarking studies on English-language medical examinations(\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e, \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e), where Med-PaLM 2 achieved near-physician-level performance with up to 86.5% accuracy on medical exam datasets and GPT-4 outperformed GPT-3.5 on the Polish Medical Final Examination with an average accuracy of 79.7%, the performance observed in our study (78\u0026ndash;85%) appears lower. However, these differences likely reflect the distinct dataset characteristics and linguistic complexity rather than model limitations. Importantly, our dataset\u0026mdash;derived from the Chinese National Qualification Examination for Attending Physicians in Urology\u0026mdash;represents a large-scale, high-quality corpus aligned with real clinical education and practice in China, which has never been used before. As such, our findings offer valuable insight into how general-purpose LLMs perform under authentic Chinese-language conditions, providing a more practically relevant benchmark for large-scale application and localized optimization in Chinese medical contexts.\u003c/p\u003e \u003cp\u003eOur study also presents several advantages over previous work. First, we used the largest urology-specific question set to date, covering a wide range of clinically relevant subdomains. Second, we employed multiple operational modes for each model, allowing for a fine-grained comparison of reasoning strategies. Third, we incorporated statistical comparisons not only across models but also across disease types and question formats, offering a more comprehensive and multidimensional evaluation framework.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003eClinical and Technical Implications\u003c/h2\u003e \u003cp\u003eDifferences between ChatGPT-4o and DeepSeek R1 have important implications for clinical practice(\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e) and education in urology(\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e). For instance, in residency training or standardized examination preparation, a more stable and accurate model like DeepSeek R1 could provide reliable automated question banks or real-time tutoring support. In clinical decision-support contexts, improved consistency reduces the risk of contradictory outputs when models are queried multiple times about diagnostic or management strategies, which is essential for maintaining clinician trust.\u003c/p\u003e \u003cp\u003eRegarding model implementation, each model was evaluated under three built-in operational configurations\u0026mdash;basic, deep-thinking, and retrieval-augmented generation (RAG)\u0026mdash;as provided by their respective platforms. These modes were selected within the model interface to represent varying levels of reasoning depth and knowledge retrieval integration. The RAG configuration, in particular, enables the model to reference external or pre-indexed knowledge sources during reasoning, which may explain its superior performance in complex, case-based questions.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003eLimitations\u003c/h2\u003e \u003cp\u003eHowever, several limitations should be acknowledged. First, although the question set was large and diverse, it originated from a single national platform, which may introduce regional or linguistic bias and limit generalizability beyond China(\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e, \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e). Nevertheless, as fundamental principles of medical reasoning are largely universal, this influence on overall validity is likely minimal(\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e). Second, the evaluation involved only single-choice questions and structured scenarios, which cannot fully represent real-world diagnostic reasoning(\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e). This design may underestimate models\u0026rsquo; reasoning depth but also reduces the risk of open-ended hallucinations. Future studies incorporating open-ended prompts or multi-answer questions would provide a more comprehensive assessment(\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e). Third, although DeepSeek R1 (RAG mode) achieved an accuracy of about 85%\u0026mdash;above both the national passing threshold (60%) and average physician scores (68.7%)\u0026mdash;direct comparison with human experts using external resources (\u0026ldquo;Human RAG\u0026rdquo;) remains lacking due to the absence of relevant data from proctored examinations. Fourth, while the single-choice format inherently minimizes the risk of hallucination by constraining possible outputs, it does not eliminate the issue entirely. Standardized frameworks for quantitative hallucination evaluation are still urgently needed to ensure the safe and trustworthy clinical application of medical LLMs, particularly when models are used in open-ended or decision-support contexts(\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e).\u003c/p\u003e \u003cdiv id=\"Sec15\" class=\"Section3\"\u003e \u003ch2\u003eEthical Considerations\u003c/h2\u003e \u003cp\u003eThe integration of large language models (LLMs) into clinical and educational settings raises important ethical considerations(\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e). Although both ChatGPT-4o and DeepSeek R1 demonstrated high accuracy, the observed performance gaps highlight the need for cautious deployment, particularly in scenarios involving clinical decision-making. Issues such as potential bias, overreliance on AI-generated outputs, and the absence of clear accountability frameworks must be addressed before widespread implementation. Ensuring transparency in model design, maintaining clinician oversight, and establishing rigorous validation protocols will be essential to safeguard patient safety and professional integrity(\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e).\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003eFuture Directions\u003c/h2\u003e \u003cp\u003eFuture research should expand the evaluation to other languages and true clinical data, integrate open-ended and multi-turn interactions, and explore alignment strategies to further improve clinical safety. Additionally, investigating user-centered metrics such as trustworthiness, interpretability, and integration into real clinical workflows would be valuable(\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e, \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e). As LLMs continue to evolve, rigorous domain-specific benchmarking\u0026mdash;such as the one we present here\u0026mdash;will be critical for informing safe and effective AI deployment in medical practice.\u003c/p\u003e \u003c/div\u003e"},{"header":"Conclusion","content":"\u003cp\u003eBoth ChatGPT and DeepSeek R1 maintained consistent performance across diverse urological disease categories without significant disease-specific fluctuations. However, compared with ChatGPT, DeepSeek R1-RAG achieved the higher overall accuracy and greater stability, especially in handling complex, context-rich medical reasoning tasks. It should be noted that our evaluation was limited to Chinese-language data, and the generalizability of these findings to other countries remains to be validated. These findings suggest that DeepSeek R1, especially in RAG mode, may offer more reliable and accurate assistance in urology-related AI applications.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eEthics approval and consent to participate:\u0026nbsp;\u003c/strong\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests:\u0026nbsp;\u003c/strong\u003eThe authors declare that they have no conflict of interest.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding:\u0026nbsp;\u003c/strong\u003ePostdoctoral Fellowship Program of CPSF\u003c/p\u003e\n\u003cp\u003e(Grant Number: GZC20230301);Beijing Natural Science Foundation (L258065); CAMS Innovation Fund for Medical Sciences(CIFMS)(2024-I2M-C\u0026amp;T-B-023)\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAvailability of data and materials:\u0026nbsp;\u003c/strong\u003eThe datasets used and/or analyzed during the current study are available from the corresponding author on request.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor contributions:\u003c/strong\u003e Conceptualization, supervision and funding acquisition: YSZ; Writing original draft and data curation: HBL; Review and editing: DXQ; Visualization: YSZ; Resources: ZW, YZ, JL.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgements:\u003c/strong\u003e None.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent for publication:\u0026nbsp;\u003c/strong\u003eNot applicable.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eChen X, Wang L, You M, Liu W, Fu Y, Xu J, et al. Evaluating and Enhancing Large Language Models\u0026rsquo; Performance in Domain-Specific Medicine: Development and Usability Study With DocOA. J Med Internet Res. 2024;26:e58158.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDavis R, Eppler M, Ayo-Ajibola O, Loh-Doyle JC, Nabhani J, Samplaski M, et al. Evaluating the Effectiveness of Artificial Intelligence-powered Large Language Models Application in Disseminating Appropriate and Readable Health Information in Urology. J Urol. 2023;210(4):688\u0026ndash;94.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepano C, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2(2):e0000198.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWei B. Performance Evaluation and Implications of Large Language Models in Radiology Board Exams: Prospective Comparative Analysis. JMIR Med Educ. 2025;11:e64284.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCeki̇c EC, Tavsan O. Evaluating large language models using national endodontic specialty examination questions: are they ready for real-world dentistry? BMC Med Educ. 2025;25(1):1308.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSinghal K, Tu T, Gottweis J, Sayres R, Wulczyn E, Amin M, et al. Toward expert-level medical question answering with large language models. Nat Med. 2025;31(3):943\u0026ndash;50.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eThirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med. 2023;29(8):1930\u0026ndash;40.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBicknell BT, Butler D, Whalen S, Ricks J, Dixon CJ, Clark AB, et al. ChatGPT-4 Omni Performance in USMLE Disciplines and Clinical Skills: Comparative Analysis. JMIR Med Educ. 2024;10:e63430.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAngel MC, Rinehart JB, Cannesson MP, Baldi P. Clinical Knowledge and Reasoning Abilities of AI Large Language Models in Anesthesiology: A Comparative Study on the American Board of Anesthesiology Examination. Anesth Analg. 2024;139(2):349\u0026ndash;56.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTakagi S, Watari T, Erabi A, Sakaguchi K. Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study. JMIR Med Educ. 2023;9:e48002.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMing S, Guo Q, Cheng W, Lei B. Influence of Model Evolution and System Roles on ChatGPT's Performance in Chinese Medical Licensing Exams: Comparative Study. JMIR Med Educ. 2024;10:e52784.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLuo PW, Liu JW, Xie X, Jiang JW, Huo XY, Chen ZL, et al. DeepSeek vs ChatGPT: a comparison study of their performance in answering prostate cancer radiotherapy questions in multiple languages. Am J Clin Exp Urol. 2025;13(2):176\u0026ndash;85.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eQiu P, Wu C, Zhang X, Lin W, Wang H, Zhang Y, et al. Towards building multilingual language model for medicine. Nat Commun. 2024;15(1):8384.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePeng Y, Malin BA, Rousseau JF, Wang Y, Xu Z, Xu X, et al. From GPT to DeepSeek: Significant gaps remain in realizing AI in healthcare. J Biomed Inf. 2025;163:104791.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZakka C, Shad R, Chaurasia A, Dalal AR, Kim JL, Moor M et al. Almanac - Retrieval-Augmented Language Models for Clinical Medicine. NEJM AI. 2024;1(2).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBedi S, Liu Y, Orr-Ewing L, Dash D, Koyejo S, Callahan A, et al. Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review. JAMA. 2025;333(4):319\u0026ndash;28.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLievin V, Hother CE, Motzfeldt AG, Winther O. Can large language models reason about medical questions? Patterns (N Y). 2024;5(3):100943.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhu M, Lin H, Jiang J, Jinia AJ, Jee J, Pichotta K, et al. Large language model trained on clinical oncology data predicts cancer progression. NPJ Digit Med. 2025;8(1):397.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRao A, Kim J, Kamineni M, Pang M, Lie W, Dreyer KJ, et al. Evaluating GPT as an Adjunct for Radiologic Decision Making: GPT-4 Versus GPT-3.5 in a Breast Imaging Pilot. J Am Coll Radiol. 2023;20(10):990\u0026ndash;7.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRosol M, Gasior JS, Laba J, Korzeniewski K, Mlynczak M. Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final Examination. Sci Rep. 2023;13(1):20512.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCollin H, Roberts MJ, Keogh K, Siriwardana A, Basto M. Improving clinical efficiency using retrieval-augmented generation in urologic oncology: A guideline-enhanced artificial intelligence approach. BJUI Compass. 2025;6(1):e427.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShah MS, Buck M. Man plus machine: large language models can augment medical student education. Nat Rev Urol. 2025.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWu J, Wang Z, Qin Y. Performance of DeepSeek-R1 and ChatGPT-4o on the Chinese National Medical Licensing Examination: A Comparative Study. J Med Syst. 2025;49(1):74.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTanaka Y, Nakata T, Aiga K, Etani T, Muramatsu R, Katagiri S, et al. Performance of Generative Pretrained Transformer on the National Medical Licensing Examination in Japan. PLOS Digit Health. 2024;3(1):e0000433.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSozen Yanik I, Sahin Hazir D, Bilgin Avsar D. Cross-lingual performance of large language models in maxillofacial prosthodontics: a comparative evaluation. BMC Oral Health. 2025;25(1):1630.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGurbuz S, Bahar H, Yavuz U, Keskin A, Karslioglu B, Solak Y. Comparative Efficacy of ChatGPT and DeepSeek in Addressing Patient Queries on Gonarthrosis and Total Knee Arthroplasty. Arthroplast Today. 2025;33:101730.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMehandru N, Miao BY, Almaraz ER, Sushil M, Butte AJ, Alaa A. Evaluating large language models as agents in the clinic. NPJ Digit Med. 2024;7(1):84.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAsgari E, Montana-Brown N, Dubois M, Khalil S, Balloch J, Yeung JA, et al. A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation. NPJ Digit Med. 2025;8(1):274.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi X, Yan X, Lai H. The ethical challenges in the integration of artificial intelligence and large language models in medical education: A scoping review. PLoS ONE. 2025;20(10):e0333411.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNouis SC, Uren V, Jariwala S. Evaluating accountability, transparency, and bias in AI-assisted healthcare decision- making: a qualitative study of healthcare professionals' perspectives in the UK. BMC Med Ethics. 2025;26(1):89.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eXu R, Hong Y, Zhang F, Xu H. Evaluation of the integration of retrieval-augmented generation in large language model for breast cancer nursing care responses. Sci Rep. 2024;14(1):30794.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePrasad S, Langlie J, Pasick L, Chen R, Franzmann E. Evaluating advanced AI reasoning models: ChatGPT-4.0 and DeepSeek-R1 diagnostic performance in otolaryngology: a comparative analysis. Am J Otolaryngol. 2025;46(4):104667.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-urology","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"buro","sideBox":"Learn more about [BMC Urology](http://bmcurol.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/buro/default.aspx","title":"BMC Urology","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-9012062/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9012062/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e \u003cp\u003eUrology presents unique challenges for AI systems, requiring both extensive medical knowledge and advanced reasoning. While large language models (LLMs) like GPT-4 have shown promise in medical education and decision support, their performance in urology remains underexplored.\u003c/p\u003e\u003ch2\u003eObjective\u003c/h2\u003e \u003cp\u003eTo compare the performance of two advanced large language models (LLMs), ChatGPT-4o and DeepSeek R1, in answering urology-related single-choice questions, and to evaluate their accuracy, stability, and reasoning capability across different response configurations.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eA total of 809 single-choice questions from the Chinese National Qualification Examination for Attending Physicians in Urology were administered to ChatGPT-4o and DeepSeek R1. Each model was tested under three configurations: standard mode, advanced reasoning mode, and retrieval-augmented generation (RAG). Accuracy was calculated for each configuration, and statistical comparisons were performed using McNemar\u0026rsquo;s test with effect sizes expressed as Cohen\u0026rsquo;s h. Stability across reasoning modes was assessed by comparing performance variability. Additional analyses examined performance differences between short-answer and case-based clinical questions.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eChatGPT-4o achieved accuracy rates of 78.12%, 73.79%, and 78.99% in standard, advanced reasoning, and RAG modes, respectively. DeepSeek R1 outperformed ChatGPT-4o across all configurations, with accuracy rates of 83.19%, 81.46%, and 84.55%, respectively. All between-model differences were statistically significant (p\u0026thinsp;\u0026lt;\u0026thinsp;0.001), with small-to-medium effect sizes (Cohen\u0026rsquo;s h\u0026thinsp;=\u0026thinsp;0.129, 0.185, and 0.144). DeepSeek R1 demonstrated substantially greater internal stability across reasoning modes, whereas ChatGPT-4o showed notable variability. In subgroup analyses, DeepSeek R1 exhibited a more pronounced advantage in complex, case-based clinical questions. Both models performed consistently across urological disease categories, and findings were limited to the Chinese-language context in which the evaluation was conducted.\u003c/p\u003e\u003ch2\u003eConclusion\u003c/h2\u003e \u003cp\u003eDeepSeek R1 showed superior performance compared with ChatGPT-4o in both accuracy and stability when answering urology-related examination questions, particularly in complex case-based scenarios. These results suggest that optimized LLMs may serve as valuable tools in medical education and clinical decision support, especially within Chinese-language environments. Further research is needed to assess their generalizability across languages, clinical settings, and more diverse task formats.\u003c/p\u003e","manuscriptTitle":"Evaluation of ChatGPT-4o’s and DeepSeek R1’s responses to urological problems: A comparative study","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-08 05:28:22","doi":"10.21203/rs.3.rs-9012062/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-05-15T08:19:54+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-09T15:09:48+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"284955190742533157688102136093566694062","date":"2026-04-09T13:28:16+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"266741841866790368692436906966548983446","date":"2026-04-08T17:15:18+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-03T17:10:42+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"188291821693009812209784764787960207865","date":"2026-04-02T06:16:45+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-04-02T04:03:50+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2026-03-05T20:11:42+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-03-03T14:45:19+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-03-03T14:39:52+00:00","index":"","fulltext":""},{"type":"submitted","content":"BMC Urology","date":"2026-03-02T15:44:15+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-urology","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"buro","sideBox":"Learn more about [BMC Urology](http://bmcurol.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/buro/default.aspx","title":"BMC Urology","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"be8beb59-c724-45c5-bdf8-2671acecbfe1","owner":[],"postedDate":"April 8th, 2026","published":true,"recentEditorialEvents":[{"type":"decision","content":"Revision requested","date":"2026-05-15T08:19:54+00:00","index":"","fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-05-17T03:38:33+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-08 05:28:22","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9012062","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9012062","identity":"rs-9012062","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}