Evaluation of ChatGPT's Performance in Residency Training Progress Exams and Competency Exams in Orthopedics and Traumatology

Evaluation of ChatGPT's Performance in Residency Training Progress Exams and Competency Exams in Orthopedics and Traumatology | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Evaluation of ChatGPT's Performance in Residency Training Progress Exams and Competency Exams in Orthopedics and Traumatology Yaşar Mahsut DİNÇEL, Gündüz Ercan KUTLUAY, Hadi SASANİ, Mehmet Ali ŞİMŞEK, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8464449/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background Artificial intelligence (AI) technologies have rapidly expanded into the field of medical education, offering innovative tools for training and assessment.This study aimed to evaluate the performance of the ChatGPT-3.5 language model in the “Residency Training Progress Examination” (UEGS) and the “Competency Examination” administered by the Turkish Society of Orthopedics and Traumatology (TOTBID). The objective was to determine whether ChatGPT performs comparably to orthopedic residents and whether it can achieve a passing score in the Competency Exam. Methods A total of 2,000 UEGS and 1,000 Competency Exam questions (2012–2023, excluding 2020) were presented to ChatGPT-3.5 using standardized prompts designed within the Role–Goals–Context (RGC) framework. The model’s responses were statistically compared with those of orthopedic residents and specialists using the Mann–Whitney U and Kruskal–Wallis tests (p < 0.05). Results ChatGPT achieved the highest accuracy in the General Orthopedics category (62%) and the lowest in Adult Reconstructive Surgery (40%). It outperformed residents only in the Spine Surgery category (p < 0.05). In the Competency Exams, ChatGPT passed four of ten exams. Conclusion ChatGPT-3.5 demonstrated limited reliability and accuracy in orthopedic examinations and should be used cautiously as an educational support tool. Future studies involving newer multimodal versions of large language models may clarify their potential role in medical education and assessment. Health sciences/Health care Health sciences/Medical research ChatGPT Board Examination Orthopedics Traumatology Artificial Intelligence Figures Figure 1 INTRODUCTION The rapid growth in computing power and the vast amount of data have led to significant advancements in artificial intelligence (AI) in recent years, enabling its integration into various aspects of daily life. The rapid growth in computing power and data availability has accelerated the development of artificial intelligence (AI), enabling its integration into diverse aspects of life ( 1 ). In 1943, McCulloch and Pitts introduced the first artificial neuron model, regarded as the foundation of AI. Later, Alan Turing proposed the Turing Test to determine whether machines could exhibit human-like reasoning. In 1956, John McCarthy coined the term artificial intelligence and developed LISP, the first AI programming language. After a period of limited progress, advances in computing power and algorithmic design reignited interest in artificial intelligence, marking the beginning of the deep learning era that enabled the development of large language models ( 2 ). The Generative Pre-trained Transformer (GPT) model, introduced by OpenAI in 2018, is a deep learning–based system trained on massive text datasets to generate human-like responses. The ChatGPT interface, launched in 2022, made this technology accessible to the general public. The latest versions, such as GPT-4 and GPT-5, demonstrate enhanced accuracy and reasoning ability ( 3 ). In healthcare, AI applications now contribute to diagnosis, image interpretation, and patient management with increased accuracy and reduced workload for clinicians. In education, AI-based learning systems are being tested for their potential to improve comprehension and self-directed learning ( 4 – 6 ). In this study, questions from the Residency Training and Progress Examination (UEGS) and the Competency Examination , organized by the Turkish Society of Orthopedics and Traumatology (TOTBID) across various years, were presented to the ChatGPT-3.5 model. The results were compared with participant outcomes to evaluate whether ChatGPT performs better than resident physicians in the UEGS and whether it can achieve a passing score in the Competency Examination. The findings aim to clarify whether ChatGPT can serve as a supplementary educational tool in residency training. MATERIALS AND METHOD Study design and data sources This was a retrospective, descriptive, and comparative study that evaluated ChatGPT-3.5’s performance in the Residency Training Progress Examination (UEGS) and the Competency Examination administered by the Turkish Society of Orthopedics and Traumatology (TOTBID). Both exams are organized annually and publicly accessible on the official TOTBID website ( 7 , 8 ). The UEGS is a national examination designed to assess the progress of orthopedic and traumatology residents in Türkiye. It has been held annually since 2009 and includes theoretical questions from all subfields of orthopedics. The exam initially contained 100 questions but expanded to 200 questions in 2014. The present study used UEGS exams from 2012 to 2023, excluding 2020, as those questions were unavailable online. The Competency Examination, prepared by the Turkish Orthopedics and Traumatology Education Council (TOTEK), has been conducted since 2003 and consists of two stages. Only the first stage, comprising 100 multiple-choice questions, was analyzed in this study ( 9 , 10 ). Participants who answer at least 60 questions correctly are considered successful. Competency Exam questions from 2014 to 2023 were included. ChatGPT interaction and data collection procedure All questions from the UEGS and Competency Exams were presented to the ChatGPT-3.5 model (OpenAI, web interface; tested in March 2023). The free version of ChatGPT-3.5 web interface, which represents the most widely used configuration during the study period, was employed. The model was instructed to answer in Turkish to ensure linguistic compatibility with the original exam format. Example of standardized prompt used for Competency Exam: “You are an orthopedic resident preparing for the national examination. Read the following question carefully and select the most appropriate answer among the options. Then, provide a brief (one-sentence) explanation for your choice. Answer in Turkish.” The same standardized prompt structure and wording were used consistently for all questions to ensure reproducibility. No feedback or corrections were provided to the model during data entry. For the UEGS, ChatGPT was prompted to respond to each item as either correct or incorrect with a brief explanation. Each interaction was designed following the Role–Goals–Context (RGC) Framework, which defines the AI’s role (exam participant), objective (select the most accurate answer), and contextual boundaries (question content and available options) [9]. Subcategorization of questions To allow domain-specific performance analysis, all questions were classified into subcategories representing the major divisions of orthopedics and traumatology. For the UEGS, nine subcategories were defined: general orthopedics, trauma, pediatric orthopedics, spinal surgery, hand and upper extremity surgery, foot and ankle surgery, sports trauma and knee arthroscopy, orthopedic oncology, and adult reconstructive surgery For the Competency Exam, ten subcategories were used: basic sciences, pediatric orthopedics, pediatric trauma, adult trauma, upper extremity and hand surgery, lower extremity and foot surgery, arthroscopic and sports surgery, adult reconstructive surgery and arthroplasty, spinal surgery, and infections and tumors (Table 1 ). Table 1 Distribution of questions by subcategories in Competency Exam and UEGS. Competency Exam Subcategories Number of Questions (n) UEGS Number of Subcategories Questions (n) Basic Sciences 127 General Orthopedics 285 Pediatric Orthopedics 151 Trauma 257 Pediatric Trauma 110 Pediatric Orthopedics 218 Adult Trauma 130 Spinal Surgery 212 Upper Extremity and Hand Surgery 88 Hand, Wrist, and Upper Extremity Surgery 202 Lower Extremity and Foot Surgery 52 Foot and Ankle Surgery 180 Arthroscopic Surgery and Sports Traumatology 102 Sports Trauma, Arthroscopy, and Knee Surgery 247 Adult Reconstructive Surgery and Arthroplasty 96 Orthopedic Oncology 186 Spinal Surgery 80 Adult Reconstructive Surgery 213 Infections and Tumors 64 Total 1000 Total 2000 Limitations of AI interaction Because ChatGPT-3.5 does not support image interpretation, the model was unable to answer questions containing radiographs, clinical photographs, or other visual material. In total, 56 such questions were skipped: 7 (2023), 8 (2022), 17 (2021), 5 (2020), 5 (2019), 4 (2018), 1 (2017), 5 (2016), 3 (2015), and 1 (2014). These were excluded from accuracy calculations. All remaining UEGS questions (n = 2000) and non-visual Competency Exam questions (n = 944) received valid text-based responses. Statistical analysis All statistical analyses were performed using PASW Statistics for Windows , version 18.0 (SPSS Inc., Chicago, IL, USA). Data normality was assessed with the Shapiro–Wilk and Kolmogorov–Smirnov tests. Descriptive statistics were calculated as means ± standard deviation (SD) and frequency distributions. Since the data were not normally distributed, the Mann–Whitney U test was used to compare ChatGPT and resident groups in two-group analyses, while the Kruskal–Wallis test was employed for comparisons across multiple categories. A p-value < 0.05 was considered statistically significant. Ethical considerations No human participants or patient data were involved in this study. All exam data were publicly available on the official TOTBID website and analyzed in aggregate form. The artificial intelligence algorithm we used in our study is an “open access” artificial intelligence platform. Therefore, formal ethics committee approval was not required. RESULTS Performance of ChatGPT in the UEGS In the UEGS examinations conducted between 2012 and 2023 (excluding 2020), the total number of analyzed questions was 2,000, distributed across nine orthopedic subcategories as shown in Table 1 . ChatGPT achieved an overall accuracy rate of 52%, providing 1,043 correct and 957 incorrect responses. The highest accuracy was recorded in the General Orthopedics category (62%), followed by Trauma (57%) and Orthopedic Oncology (57%). The lowest accuracy was found in Adult Reconstructive Surgery (40%) and Foot and Ankle Surgery (45%). Detailed accuracy rates by subcategory are presented in Table 2 . Table 2 Accuracy of ChatGPT-3.5 by subcategories in the UEGS Subcategory Number of Questions (n) Correct Answers (n) Accuracy (%) General Orthopedics 285 176 62 Trauma 257 146 57 Pediatric Orthopedics 218 112 51 Spinal Surgery 212 117 55 Hand, Wrist, and Upper Extremity Surgery 202 93 46 Foot and Ankle Surgery 180 81 45 Sports Trauma, Arthroscopy, and Knee Surgery 247 127 51 Orthopedic Oncology 186 106 57 Adult Reconstructive Surgery 213 85 40 Total 2,000 1,043 52 When compared statistically with resident physicians’ scores obtained from the TOTBID database, ChatGPT’s total accuracy did not differ significantly (p = 0.895). However, ChatGPT outperformed residents in the Spinal Surgery category (mean = 10.64 ± 2.54 vs. 7.54 ± 2.72, p = 0.034), while residents performed significantly better in Adult Reconstructive Surgery (mean = 7.27 ± 3.20 vs. 8.71 ± 1.43, p = 0.015). Although overall accuracy levels were similar, ChatGPT produced more incorrect responses in nearly all categories. A detailed comparison of correct and incorrect responses between ChatGPT and residents is presented in Table 3. Subcategory ChatGPT Mean ± SD (Correct) Residents Mean ± SD (Correct) p-value ChatGPT Mean ± SD (Incorrect) Residents Mean ± SD (Incorrect) p-value General Orthopedics 16.00 ± 6.48 13.17 ± 6.41 0.426 9.91 ± 6.04 5.92 ± 2.82 0.070 Trauma 13.27 ± 4.34 13.29 ± 1.69 0.929 10.10 ± 4.48 5.77 ± 0.94 0.038 Pediatric Orthopedics 10.18 ± 3.63 10.29 ± 1.22 0.965 9.64 ± 3.93 5.14 ± 1.47 0.007 Spinal Surgery 10.64 ± 2.54 7.54 ± 2.72 0.034 8.64 ± 4.13 4.23 ± 1.02 0.070 Hand, Wrist, and Upper Extremity Surgery 8.45 ± 3.53 10.93 ± 3.47 0.070 9.91 ± 4.18 5.16 ± 1.42 0.003 Foot and Ankle Surgery 7.36 ± 4.92 10.75 ± 4.95 0.085 9.00 ± 5.72 5.72 ± 2.30 0.085 Sports Trauma, Arthroscopy, and Knee Surgery 11.45 ± 4.39 11.49 ± 2.47 0.757 10.91 ± 3.70 6.51 ± 1.35 0.001 Orthopedic Oncology 9.37 ± 4.86 7.90 ± 3.51 0.566 7.27 ± 3.90 3.96 ± 1.85 0.057 Adult Reconstructive Surgery 7.27 ± 3.20 8.71 ± 1.43 0.015 11.64 ± 3.44 4.31 ± 0.75 < 0.001 Total 94.10 ± 24.84 94.58 ± 8.33 0.895 87.73 ± 28.46 46.44 ± 5.07 0.010 *Statistical comparisons were made using the Mann–Whitney U test; bold p-values indicate statistical significance (p < 0.05). Table 3. Comparison of ChatGPT-3.5 and resident participants in the UEGS Performance of ChatGPT in the Competency Exams A total of 1,000 Competency Exam questions from 2014–2023 were analyzed. After excluding 56 image-based questions that the model could not interpret, 944 questions were evaluated. ChatGPT achieved its highest accuracy in Pediatric Orthopedics (68.2%), followed by Lower Extremity and Foot Surgery (65.4%), and its lowest accuracy in Infections and Tumors (38%). Figure 1 illustrates the distribution of correct response percentages across subcategories. In the ten annual Competency Exams analyzed, ChatGPT passed four exams (years 2016, 2017, 2019, and 2022) by achieving ≥ 60% correct answers (Table 4 ). Its best performance occurred in 2019 (72% accuracy), while the lowest score was in 2023 (44%). Table 4 ChatGPT-3.5 performance in the Competency Exams (2014–2023) Year Unanswered Questions (n) Correct Answers (n) Incorrect Answers (n) Result 2023 7 44 49 Failed 2022 8 61 31 Passed 2021 17 45 38 Failed 2020 5 47 48 Failed 2019 5 72 23 Passed 2018 4 59 37 Failed 2017 1 70 29 Passed 2016 5 60 35 Passed 2015 3 57 40 Failed Total 56 515 330 4 / 10 Passed DISCUSSION This study examined the performance of the ChatGPT-3.5 artificial intelligence model in Türkiye’s Residency Training and Progress Examination (UEGS) and Competency Examination , both organized by the Turkish Society of Orthopedics and Traumatology (TOTBID). ChatGPT achieved its highest accuracy in the General Orthopedics category (61%) and the lowest in the Adult Reconstructive Surgery category (39%). The model significantly outperformed participants only in the Spine Surgery category. This finding may be explained by the fact that Spine Surgery was also the category in which resident physicians had the lowest accuracy (7.98%). These results may provide insights for those involved in planning and improving orthopedic and traumatology residency training programs, as there is already a core curriculum committee established for Spine Surgery ( 11 ). When comparing the results of ChatGPT and residents in the UEGS, no significant difference was found in the total number of correct answers. However, ChatGPT gave more incorrect responses in every category compared with the participants’ average. Based on these data, ChatGPT-3.5 performed worse than orthopedic and traumatology residents in the UEGS. In the Competency Examination, ChatGPT passed four out of ten exams by correctly answering at least 60 questions out of 100. In the remaining six exams, the model failed to reach the required passing score. Among the 1,000 questions analyzed, ChatGPT-3.5 was unable to answer 56 image-based questions due to its lack of visual interpretation capability. With newer versions such as ChatGPT-4, which can interpret images, more questions could be answered, potentially resulting in higher success rates. In a comparative study using UEGS questions, ChatGPT-4 demonstrated significantly higher accuracy than ChatGPT-3.5 ( 12 ). Likewise, a recent study using questions from the Turkish Competency Exam reported that ChatGPT-4o achieved higher accuracy than human participants across all orthopedic subdomains ( 13 ). This indicates that our findings align with the overall trend of improving model performance in newer AI systems rather than contradicting it. Our findings show that ChatGPT-3.5 performs worse than resident physicians in the UEGS and fails to pass most of the Competency Exams in orthopedics and traumatology. Similar studies have demonstrated that ChatGPT performs well in some exams but poorly in others ( 14 – 19 ). In certain cases, ChatGPT outperformed participants, whereas in others, as in our study, participants achieved better results. ChatGPT became one of the fastest-growing computer programs in history, reaching 100 million active users within two months of its public release ( 20 ). Like other AI models, it draws from a wide range of data sources, including peer-reviewed journal articles, textbooks, and online content ( 21 ). As new versions are released, both the technical capabilities of the model and the size of its knowledge base expand. Thus, it is expected that future AI models will continue to improve their ability to evaluate and answer questions. This study has certain limitations. The results of residents and specialists were evaluated using publicly available data from the TOTBID website, where not all exam years had standardized or complete analyses. Some exams were not conducted during the COVID-19 pandemic, and those years were therefore excluded. These limitations may have affected the comparison process. Finally, the use of chatbots in medical education is an emerging trend supported by many educators and medical professionals. OpenAI’s ChatGPT offers several potential advantages for both students and teachers ( 22 , 23 ). Recent reviews have highlighted that generative AI tools hold promise for orthopedic education and training but also pose challenges related to reliability, ethical use, and integration into curricula. It is important to remember that these systems are still evolving and have not yet reached perfection. There remain significant gaps in both theoretical and practical orthopedic education that AI tools cannot yet fill ( 24 ). Conclusion ChatGPT-3.5 showed variable accuracy in orthopedic examinations, performing comparably to residents only in the Spine Surgery category while underperforming in most others. Although ChatGPT offers potential educational benefits, it is not yet a reliable or valid resource for independent use in orthopedics and traumatology education. As artificial intelligence systems continue to evolve, future versions with multimodal capabilities may become more effective tools in medical learning and assessment. Declarations Ethics Approval and Consent to Participate Not Applicable Consent for Publication Not Applicable Competing Interests The authors declare that they have no competing interests Clinical Trial Number Not applicable Funding Not applicable Author Contribution YMD and GEK wrote the main manuscript.HS and ME asked questions to ChatGPT.MAŞ performed the statistical work. Acknowledgements The authors would like to express their sincere gratitude to all doctors who helped impove the orthopedic examination for a better education. Data Availability The data that support the findings of this study are available from the corresponding author upon reasonable request. References Minh, D., Wang, H. X., Li, Y. F. & Nguyen, T. N. Explainable artificial intelligence: a comprehensive review. Artif. Intell. Rev. 55(5) (1), 3503–3568 (2022 June). Haenlein, M. & Kaplan, A. A Brief History of Artificial Intelligence: On the Past, Present, and Future of Artificial Intelligence. Calif. Manage. Rev. 61 (4), 5–14 (2019). Ollivier, M. et al. A deeper dive into ChatGPT: history, use and future perspectives for orthopaedic research. Knee Surg. Sports Traumatol. Arthrosc. 31 (4), 1190–1192 (2023). Liu, P. R. et al. Application of Artificial Intelligence in Medicine: An Overview. Curr. Med. Sci. 41 (6), 1105–1115 (2021). Wu, D. et al. Artificial intelligence-tutoring problem-based learning in ophthalmology clerkship. Ann. Transl Med. 8 (11), 700–700 (2020 June). Yang, Y. Y. & Shulruf, B. Expert-led and artificial intelligence (AI) system-assisted tutoring course increase confidence of Chinese medical interns on suturing and ligature skills: prospective pilot study. J. Educ. Eval Health Prof. 16 , 7 (2019). Gönen, D. E. 2012–2013 TOTBİD-TOTEK UZMANLIK EĞİTİMİ GELİŞİM SINAVI RAPORU (UEGS). Türk Ortopedi ve Travmatoloji Birliği Derneği [Internet]. [cited 2024 Sept 1]. Available from: https://totbid.org.tr/tr/ Tabatabaian, M. Prompt Engineering Using ChatGPT: Crafting Effective Interactions and Building GPT Apps 142 (Walter de Gruyter GmbH & Co KG, 2024). Benli, İ. & Acaroğlu, E. Türk Ortopedi ve Travmatoloji Birliği Derneği (TOTBİD) Türk Ortopedi ve Travmatoloji Eğitim Konseyi Yeterlik Sınavları. Acta Orthop Traumatol Turc [Internet]. [cited 2024 Sept 1];45(2). Available from: https://dergipark.org.tr/en/download/article-file/169969 Acaroğlu, E. et al. Core curriculum (CC) of spinal surgery: a step forward in defining our profession. Acta Orthop. Traumatol. Turc. 48 (5), 475–478 (2014). Ayik, G. et al. Exploring the role of artificial intelligence in Turkish orthopedic progression exams. Acta Orthop. Traumatol. Turc. 59 (1), 18–26 (2025). Yağar, H., Gümüşoğlu, E. & Mert Asfuroğlu, Z. Assessing the performance of ChatGPT-4o on the Turkish Orthopedics and Traumatology Board Examination. Jt. Dis. Relat. Surg. 36 (2), 304–310 (2025). Ruksakulpiwat, S., Kumar, A. & Ajibade, A. Using ChatGPT in Medical Research: Current Status and Future Directions. J. Multidiscip Healthc. 16 , 1513–1520 (2023). Khan, R. A., Jawaid, M., Khan, A. R. & Sajjad, M. ChatGPT - Reshaping medical education and clinical management. Pak J Med Sci [Internet]. Feb 16 [cited 2024 Sept 28];39(2). (2023). Available from: https://pjms.org.pk/index.php/pjms/article/view/7653 Oztermeli, A. D. & Oztermeli, A. ChatGPT performance in the medical specialty exam: An observational study. Med. (Baltim). 102 (32), e34673 (2023). Sumbal, A., Sumbal, R. & Amir, A. Can ChatGPT-3.5 Pass a Medical Exam? A Systematic Review of ChatGPT’s Performance in Academic Testing. J. Med. Educ. Curric. Dev. 11 , 23821205241238641 (2024). Wang, X. et al. ChatGPT Performs on the Chinese National Medical Licensing Examination. J. Med. Syst. 47 (1), 86 (2023). Aljindan, F. K. et al. ChatGPT Conquers the Saudi Medical Licensing Exam: Exploring the Accuracy of Artificial Intelligence in Medical Knowledge Assessment and Implications for Modern Medical Education. Cureus 15 (9), e45043 (2023 Sept). Alessandri Bonetti, M., Giorgino, R., Gallo Afflitto, G., De Lorenzi, F. & Egro, F. M. How Does ChatGPT Perform on the Italian Residency Admission National Exam Compared to 15,869 Medical Graduates? Ann. Biomed. Eng. 52 (4), 745–749 (2024). Massey, P. A., Montgomery, C. & Zhang, A. S. Comparison of ChatGPT–3.5, ChatGPT-4, and Orthopaedic Resident Performance on Orthopaedic Assessment Examinations. JAAOS - J. Am. Acad. Orthop. Surg. 31 (23), 1173 (2023). Huang, Y. et al. Benchmarking ChatGPT-4 on a radiation oncology in-training exam and Red Journal Gray Zone cases: potentials and challenges for ai-assisted medical education and decision making in radiation oncology. Front. Oncol. 13 , 1265024 (2023). Moritz, S., Romeike, B., Stosch, C. & Tolks, D. Generative AI (gAI) in medical education: Chat-GPT and co. GMS J. Med. Educ. 40 (4), Doc54 (2023). Atik, O. Ş. Artificial intelligence: Who must have autonomy the machine or the human? Jt. Dis. Relat. Surg. 35 (1), 1–2 (2024). Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8464449","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":571038128,"identity":"df9e68f3-4bde-4c4d-8f80-69ffb6352b25","order_by":0,"name":"Yaşar Mahsut DİNÇEL","email":"","orcid":"","institution":"Namık Kemal University","correspondingAuthor":false,"prefix":"","firstName":"Yaşar","middleName":"Mahsut","lastName":"DİNÇEL","suffix":""},{"id":571038129,"identity":"6952143f-d0d0-43ed-a346-05643369b7e9","order_by":1,"name":"Gündüz Ercan KUTLUAY","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABHElEQVRIiWNgGAWjYJACCSjNDMQWIPoASFCGSC0JIDZbAkiQhxQtPAYgHk4t8hG5B2/z/LHJ528/e9iY94dEYv+MnM+vbtRY8DCwHz66AYsWwxt5ydY8PGmWM87kJSfzJEgkzriRu8065xjQYTxpaTewaZmRYybNI3HYwIAhx/gwSEsDUItxDhtQiwSPGW4tBv8NDPjfQLTMv5HzzDjnH24t8hIgLQkHDAwkcozBDttwI4f5cW4bbi0GPG+MLeccSDaQuPHG2HBOmoTxxjPPzJhz+yR42HD4Rb49x/DGmz92Bvz9OcYSb2xsZOcdT378OedbnRw/++FjWG05gCEkkMAGjik2LMrBtjRgCPEfYP6AQ/UoGAWjYBSMTAAAhfJdUFsxS1YAAAAASUVORK5CYII=","orcid":"","institution":"ÇOSB Kapaklı Devlet Hastahanesi","correspondingAuthor":true,"prefix":"","firstName":"Gündüz","middleName":"Ercan","lastName":"KUTLUAY","suffix":""},{"id":571038132,"identity":"283a14aa-4cb1-441f-9527-f9402859f298","order_by":2,"name":"Hadi SASANİ","email":"","orcid":"","institution":"Namık Kemal University","correspondingAuthor":false,"prefix":"","firstName":"Hadi","middleName":"","lastName":"SASANİ","suffix":""},{"id":571038134,"identity":"71c64579-3d04-4f7b-9342-9f618a2754e7","order_by":3,"name":"Mehmet Ali ŞİMŞEK","email":"","orcid":"","institution":"Namık Kemal University","correspondingAuthor":false,"prefix":"","firstName":"Mehmet","middleName":"Ali","lastName":"ŞİMŞEK","suffix":""},{"id":571038135,"identity":"2805890b-2d3e-4c44-b533-64bfbf14b15c","order_by":4,"name":"Murat EREM","email":"","orcid":"","institution":"Trakya University","correspondingAuthor":false,"prefix":"","firstName":"Murat","middleName":"","lastName":"EREM","suffix":""}],"badges":[],"createdAt":"2025-12-28 07:38:09","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8464449/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8464449/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":100008824,"identity":"2b39c792-fbf6-4fe9-b96f-7fc1f9b35922","added_by":"auto","created_at":"2026-01-12 05:56:54","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":91748,"visible":true,"origin":"","legend":"","description":"","filename":"ManuscriptRevised.docx","url":"https://assets-eu.researchsquare.com/files/rs-8464449/v1/869607f2d626b5df8594ba57.docx"},{"id":100008826,"identity":"e8d80efb-08b4-4957-ae76-bf1641d4059f","added_by":"auto","created_at":"2026-01-12 05:56:55","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":6487,"visible":true,"origin":"","legend":"","description":"","filename":"259fd23b216949d18467adc32f1129df.json","url":"https://assets-eu.researchsquare.com/files/rs-8464449/v1/3550f57f199ed22aba781ead.json"},{"id":100361828,"identity":"bf2c9271-23a9-479a-8bba-e81ccd94e787","added_by":"auto","created_at":"2026-01-16 07:45:48","extension":"xml","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":81184,"visible":true,"origin":"","legend":"","description":"","filename":"259fd23b216949d18467adc32f1129df1enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-8464449/v1/67a4c266654a5b93d961c032.xml"},{"id":100360972,"identity":"85318fca-0d03-4004-ae54-0bcb73f51535","added_by":"auto","created_at":"2026-01-16 07:44:16","extension":"png","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":14601,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-8464449/v1/f8f69cc47e935f376eeadfba.png"},{"id":100008827,"identity":"0ed54986-8cde-4861-bed4-a730b48bf3a9","added_by":"auto","created_at":"2026-01-12 05:56:55","extension":"xml","order_by":5,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":79053,"visible":true,"origin":"","legend":"","description":"","filename":"259fd23b216949d18467adc32f1129df1structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-8464449/v1/73000c238ee53ba46ad2f9c0.xml"},{"id":100362076,"identity":"cb7825ef-f735-44ec-b1a4-854de036fb71","added_by":"auto","created_at":"2026-01-16 07:46:10","extension":"html","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":90190,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-8464449/v1/deb1afcc39b7698711d6c6f7.html"},{"id":100361862,"identity":"bb562a38-2198-455e-bc22-0fa693873fe8","added_by":"auto","created_at":"2026-01-16 07:45:51","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":32594,"visible":true,"origin":"","legend":"\u003cp\u003ePercentages of correct answers by ChatGPT-3.5 in the Competency Exam subcategories.\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-8464449/v1/ff2b028b2110ff7d50243ef0.png"},{"id":101381111,"identity":"4ae044ae-55b9-42a8-a303-c8efede9252e","added_by":"auto","created_at":"2026-01-29 06:11:20","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":962150,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8464449/v1/e114f3e1-d138-4cb4-85b0-fe79dd66f5f9.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Evaluation of ChatGPT's Performance in Residency Training Progress Exams and Competency Exams in Orthopedics and Traumatology","fulltext":[{"header":"INTRODUCTION","content":"\u003cp\u003eThe rapid growth in computing power and the vast amount of data have led to significant advancements in artificial intelligence (AI) in recent years, enabling its integration into various aspects of daily life.\u003c/p\u003e \u003cp\u003eThe rapid growth in computing power and data availability has accelerated the development of artificial intelligence (AI), enabling its integration into diverse aspects of life (\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e). In 1943, McCulloch and Pitts introduced the first artificial neuron model, regarded as the foundation of AI. Later, Alan Turing proposed the Turing Test to determine whether machines could exhibit human-like reasoning. In 1956, John McCarthy coined the term \u003cem\u003eartificial intelligence\u003c/em\u003e and developed LISP, the first AI programming language. After a period of limited progress, advances in computing power and algorithmic design reignited interest in artificial intelligence, marking the beginning of the deep learning era that enabled the development of large language models (\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThe Generative Pre-trained Transformer (GPT) model, introduced by OpenAI in 2018, is a deep learning\u0026ndash;based system trained on massive text datasets to generate human-like responses. The ChatGPT interface, launched in 2022, made this technology accessible to the general public. The latest versions, such as GPT-4 and GPT-5, demonstrate enhanced accuracy and reasoning ability (\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eIn healthcare, AI applications now contribute to diagnosis, image interpretation, and patient management with increased accuracy and reduced workload for clinicians. In education, AI-based learning systems are being tested for their potential to improve comprehension and self-directed learning (\u003cspan additionalcitationids=\"CR5\" citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eIn this study, questions from the \u003cem\u003eResidency Training and Progress Examination\u003c/em\u003e (UEGS) and the \u003cem\u003eCompetency Examination\u003c/em\u003e, organized by the Turkish Society of Orthopedics and Traumatology (TOTBID) across various years, were presented to the ChatGPT-3.5 model. The results were compared with participant outcomes to evaluate whether ChatGPT performs better than resident physicians in the UEGS and whether it can achieve a passing score in the Competency Examination.\u003c/p\u003e \u003cp\u003eThe findings aim to clarify whether ChatGPT can serve as a supplementary educational tool in residency training.\u003c/p\u003e"},{"header":"MATERIALS AND METHOD","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eStudy design and data sources\u003c/h2\u003e \u003cp\u003eThis was a retrospective, descriptive, and comparative study that evaluated ChatGPT-3.5\u0026rsquo;s performance in the \u003cem\u003eResidency Training Progress Examination\u003c/em\u003e (UEGS) and the \u003cem\u003eCompetency Examination\u003c/em\u003e administered by the Turkish Society of Orthopedics and Traumatology (TOTBID). Both exams are organized annually and publicly accessible on the official TOTBID website (\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThe UEGS is a national examination designed to assess the progress of orthopedic and traumatology residents in T\u0026uuml;rkiye. It has been held annually since 2009 and includes theoretical questions from all subfields of orthopedics. The exam initially contained 100 questions but expanded to 200 questions in 2014. The present study used UEGS exams from 2012 to 2023, excluding 2020, as those questions were unavailable online.\u003c/p\u003e \u003cp\u003eThe Competency Examination, prepared by the Turkish Orthopedics and Traumatology Education Council (TOTEK), has been conducted since 2003 and consists of two stages. Only the first stage, comprising 100 multiple-choice questions, was analyzed in this study (\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e). Participants who answer at least 60 questions correctly are considered successful. Competency Exam questions from 2014 to 2023 were included.\u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003eChatGPT interaction and data collection procedure\u003c/h3\u003e\n\u003cp\u003eAll questions from the UEGS and Competency Exams were presented to the ChatGPT-3.5 model (OpenAI, web interface; tested in March 2023). The free version of ChatGPT-3.5 web interface, which represents the most widely used configuration during the study period, was employed. The model was instructed to answer in Turkish to ensure linguistic compatibility with the original exam format.\u003c/p\u003e \u003cp\u003eExample of standardized prompt used for Competency Exam: \u0026ldquo;You are an orthopedic resident preparing for the national examination. Read the following question carefully and select the most appropriate answer among the options. Then, provide a brief (one-sentence) explanation for your choice. Answer in Turkish.\u0026rdquo;\u003c/p\u003e \u003cp\u003eThe same standardized prompt structure and wording were used consistently for all questions to ensure reproducibility. No feedback or corrections were provided to the model during data entry.\u003c/p\u003e \u003cp\u003eFor the UEGS, ChatGPT was prompted to respond to each item as either \u003cem\u003ecorrect\u003c/em\u003e or \u003cem\u003eincorrect\u003c/em\u003e with a brief explanation. Each interaction was designed following the Role\u0026ndash;Goals\u0026ndash;Context (RGC) Framework, which defines the AI\u0026rsquo;s role (exam participant), objective (select the most accurate answer), and contextual boundaries (question content and available options) [9].\u003c/p\u003e\n\u003ch3\u003eSubcategorization of questions\u003c/h3\u003e\n\u003cp\u003eTo allow domain-specific performance analysis, all questions were classified into subcategories representing the major divisions of orthopedics and traumatology.\u003c/p\u003e \u003cp\u003eFor the UEGS, nine subcategories were defined: general orthopedics, trauma, pediatric orthopedics, spinal surgery, hand and upper extremity surgery, foot and ankle surgery, sports trauma and knee arthroscopy, orthopedic oncology, and adult reconstructive surgery\u003c/p\u003e \u003cp\u003eFor the Competency Exam, ten subcategories were used: basic sciences, pediatric orthopedics, pediatric trauma, adult trauma, upper extremity and hand surgery, lower extremity and foot surgery, arthroscopic and sports surgery, adult reconstructive surgery and arthroplasty, spinal surgery, and infections and tumors (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eDistribution of questions by subcategories in Competency Exam and UEGS.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eCompetency Exam Subcategories\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eNumber of Questions (n)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eUEGS\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eNumber of\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSubcategories\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eQuestions (n)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBasic Sciences\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e127\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eGeneral Orthopedics\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e285\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePediatric Orthopedics\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e151\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eTrauma\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e257\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePediatric Trauma\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e110\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003ePediatric Orthopedics\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e218\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAdult Trauma\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e130\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSpinal Surgery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e212\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eUpper Extremity and Hand Surgery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e88\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHand, Wrist, and Upper Extremity Surgery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e202\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLower Extremity and Foot Surgery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e52\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eFoot and Ankle Surgery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e180\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eArthroscopic Surgery and Sports Traumatology\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e102\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSports Trauma, Arthroscopy, and Knee Surgery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e247\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAdult Reconstructive Surgery and Arthroplasty\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e96\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eOrthopedic Oncology\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e186\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSpinal Surgery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e80\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAdult Reconstructive Surgery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e213\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInfections and Tumors\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e64\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eTotal\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e1000\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003eTotal\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e2000\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e\n\u003ch3\u003eLimitations of AI interaction\u003c/h3\u003e\n\u003cp\u003eBecause ChatGPT-3.5 does not support image interpretation, the model was unable to answer questions containing radiographs, clinical photographs, or other visual material. In total, 56 such questions were skipped: 7 (2023), 8 (2022), 17 (2021), 5 (2020), 5 (2019), 4 (2018), 1 (2017), 5 (2016), 3 (2015), and 1 (2014). These were excluded from accuracy calculations.\u003c/p\u003e \u003cp\u003eAll remaining UEGS questions (n\u0026thinsp;=\u0026thinsp;2000) and non-visual Competency Exam questions (n\u0026thinsp;=\u0026thinsp;944) received valid text-based responses.\u003c/p\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003eStatistical analysis\u003c/h2\u003e \u003cp\u003eAll statistical analyses were performed using \u003cem\u003ePASW Statistics for Windows\u003c/em\u003e, version 18.0 (SPSS Inc., Chicago, IL, USA). Data normality was assessed with the Shapiro\u0026ndash;Wilk and Kolmogorov\u0026ndash;Smirnov tests. Descriptive statistics were calculated as means\u0026thinsp;\u0026plusmn;\u0026thinsp;standard deviation (SD) and frequency distributions.\u003c/p\u003e \u003cp\u003eSince the data were not normally distributed, the Mann\u0026ndash;Whitney U test was used to compare ChatGPT and resident groups in two-group analyses, while the Kruskal\u0026ndash;Wallis test was employed for comparisons across multiple categories. A p-value\u0026thinsp;\u0026lt;\u0026thinsp;0.05 was considered statistically significant.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eEthical considerations\u003c/h2\u003e \u003cp\u003eNo human participants or patient data were involved in this study. All exam data were publicly available on the official TOTBID website and analyzed in aggregate form. The artificial intelligence algorithm we used in our study is an \u0026ldquo;open access\u0026rdquo; artificial intelligence platform. Therefore, formal ethics committee approval was not required.\u003c/p\u003e \u003c/div\u003e"},{"header":"RESULTS","content":"\u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003ePerformance of ChatGPT in the UEGS\u003c/h2\u003e \u003cp\u003eIn the UEGS examinations conducted between 2012 and 2023 (excluding 2020), the total number of analyzed questions was 2,000, distributed across nine orthopedic subcategories as shown in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. ChatGPT achieved an overall accuracy rate of 52%, providing 1,043 correct and 957 incorrect responses.\u003c/p\u003e \u003cp\u003eThe highest accuracy was recorded in the General Orthopedics category (62%), followed by Trauma (57%) and Orthopedic Oncology (57%). The lowest accuracy was found in Adult Reconstructive Surgery (40%) and Foot and Ankle Surgery (45%).\u003c/p\u003e \u003cp\u003eDetailed accuracy rates by subcategory are presented in Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eAccuracy of ChatGPT-3.5 by subcategories in the UEGS\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSubcategory\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNumber of Questions (n)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCorrect Answers (n)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAccuracy (%)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGeneral Orthopedics\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e285\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e176\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e62\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTrauma\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e257\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e146\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e57\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePediatric Orthopedics\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e218\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e112\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e51\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSpinal Surgery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e212\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e117\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e55\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHand, Wrist, and Upper Extremity Surgery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e202\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e93\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e46\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFoot and Ankle Surgery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e180\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e81\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e45\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSports Trauma, Arthroscopy, and Knee Surgery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e247\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e127\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e51\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOrthopedic Oncology\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e186\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e106\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e57\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAdult Reconstructive Surgery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e213\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e85\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eTotal\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e2,000\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e1,043\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e52\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eWhen compared statistically with resident physicians\u0026rsquo; scores obtained from the TOTBID database, ChatGPT\u0026rsquo;s total accuracy did not differ significantly (p\u0026thinsp;=\u0026thinsp;0.895). However, ChatGPT outperformed residents in the Spinal Surgery category (mean\u0026thinsp;=\u0026thinsp;10.64\u0026thinsp;\u0026plusmn;\u0026thinsp;2.54 vs. 7.54\u0026thinsp;\u0026plusmn;\u0026thinsp;2.72, p\u0026thinsp;=\u0026thinsp;0.034), while residents performed significantly better in Adult Reconstructive Surgery (mean\u0026thinsp;=\u0026thinsp;7.27\u0026thinsp;\u0026plusmn;\u0026thinsp;3.20 vs. 8.71\u0026thinsp;\u0026plusmn;\u0026thinsp;1.43, p\u0026thinsp;=\u0026thinsp;0.015).\u003c/p\u003e \u003cp\u003eAlthough overall accuracy levels were similar, ChatGPT produced more incorrect responses in nearly all categories.\u003c/p\u003e \u003cp\u003eA detailed comparison of correct and incorrect responses between ChatGPT and residents is presented in Table\u0026nbsp;3.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"No\" id=\"Taba\" border=\"1\"\u003e \u003ccolgroup cols=\"7\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\"\u0026plusmn;\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\"\u0026plusmn;\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\"\u0026plusmn;\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\"\u0026plusmn;\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSubcategory\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eChatGPT Mean\u0026thinsp;\u0026plusmn;\u0026thinsp;SD (Correct)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eResidents Mean\u0026thinsp;\u0026plusmn;\u0026thinsp;SD (Correct)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003ep-value\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eChatGPT Mean\u0026thinsp;\u0026plusmn;\u0026thinsp;SD (Incorrect)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eResidents Mean\u0026thinsp;\u0026plusmn;\u0026thinsp;SD (Incorrect)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003ep-value\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGeneral Orthopedics\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e16.00\u0026thinsp;\u0026plusmn;\u0026thinsp;6.48\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e13.17\u0026thinsp;\u0026plusmn;\u0026thinsp;6.41\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.426\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e9.91\u0026thinsp;\u0026plusmn;\u0026thinsp;6.04\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e5.92\u0026thinsp;\u0026plusmn;\u0026thinsp;2.82\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.070\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTrauma\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e13.27\u0026thinsp;\u0026plusmn;\u0026thinsp;4.34\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e13.29\u0026thinsp;\u0026plusmn;\u0026thinsp;1.69\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.929\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e10.10\u0026thinsp;\u0026plusmn;\u0026thinsp;4.48\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e5.77\u0026thinsp;\u0026plusmn;\u0026thinsp;0.94\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e0.038\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePediatric Orthopedics\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e10.18\u0026thinsp;\u0026plusmn;\u0026thinsp;3.63\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e10.29\u0026thinsp;\u0026plusmn;\u0026thinsp;1.22\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.965\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e9.64\u0026thinsp;\u0026plusmn;\u0026thinsp;3.93\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e5.14\u0026thinsp;\u0026plusmn;\u0026thinsp;1.47\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e0.007\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSpinal Surgery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e10.64\u0026thinsp;\u0026plusmn;\u0026thinsp;2.54\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e7.54\u0026thinsp;\u0026plusmn;\u0026thinsp;2.72\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e0.034\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e8.64\u0026thinsp;\u0026plusmn;\u0026thinsp;4.13\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e4.23\u0026thinsp;\u0026plusmn;\u0026thinsp;1.02\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.070\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHand, Wrist, and Upper Extremity Surgery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e8.45\u0026thinsp;\u0026plusmn;\u0026thinsp;3.53\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e10.93\u0026thinsp;\u0026plusmn;\u0026thinsp;3.47\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.070\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e9.91\u0026thinsp;\u0026plusmn;\u0026thinsp;4.18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e5.16\u0026thinsp;\u0026plusmn;\u0026thinsp;1.42\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e0.003\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFoot and Ankle Surgery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e7.36\u0026thinsp;\u0026plusmn;\u0026thinsp;4.92\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e10.75\u0026thinsp;\u0026plusmn;\u0026thinsp;4.95\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.085\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e9.00\u0026thinsp;\u0026plusmn;\u0026thinsp;5.72\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e5.72\u0026thinsp;\u0026plusmn;\u0026thinsp;2.30\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.085\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSports Trauma, Arthroscopy, and Knee Surgery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e11.45\u0026thinsp;\u0026plusmn;\u0026thinsp;4.39\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e11.49\u0026thinsp;\u0026plusmn;\u0026thinsp;2.47\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.757\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e10.91\u0026thinsp;\u0026plusmn;\u0026thinsp;3.70\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e6.51\u0026thinsp;\u0026plusmn;\u0026thinsp;1.35\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e0.001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOrthopedic Oncology\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e9.37\u0026thinsp;\u0026plusmn;\u0026thinsp;4.86\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e7.90\u0026thinsp;\u0026plusmn;\u0026thinsp;3.51\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.566\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e7.27\u0026thinsp;\u0026plusmn;\u0026thinsp;3.90\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e3.96\u0026thinsp;\u0026plusmn;\u0026thinsp;1.85\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.057\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAdult Reconstructive Surgery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e7.27\u0026thinsp;\u0026plusmn;\u0026thinsp;3.20\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e8.71\u0026thinsp;\u0026plusmn;\u0026thinsp;1.43\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e0.015\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e11.64\u0026thinsp;\u0026plusmn;\u0026thinsp;3.44\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e4.31\u0026thinsp;\u0026plusmn;\u0026thinsp;0.75\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e\u0026lt;\u0026thinsp;0.001\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eTotal\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e94.10\u0026thinsp;\u0026plusmn;\u0026thinsp;24.84\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e94.58\u0026thinsp;\u0026plusmn;\u0026thinsp;8.33\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.895\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e87.73\u0026thinsp;\u0026plusmn;\u0026thinsp;28.46\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e46.44\u0026thinsp;\u0026plusmn;\u0026thinsp;5.07\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e0.010\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e*Statistical comparisons were made using the Mann\u0026ndash;Whitney U test; bold p-values indicate statistical significance (p\u0026thinsp;\u0026lt;\u0026thinsp;0.05).\u003c/p\u003e \u003cp\u003e \u003cb\u003eTable\u0026nbsp;3.\u003c/b\u003e Comparison of ChatGPT-3.5 and resident participants in the UEGS\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003ePerformance of ChatGPT in the Competency Exams\u003c/h2\u003e \u003cp\u003eA total of 1,000 Competency Exam questions from 2014\u0026ndash;2023 were analyzed.\u003c/p\u003e \u003cp\u003eAfter excluding 56 image-based questions that the model could not interpret, 944 questions were evaluated.\u003c/p\u003e \u003cp\u003e ChatGPT achieved its highest accuracy in Pediatric Orthopedics (68.2%), followed by Lower Extremity and Foot Surgery (65.4%), and its lowest accuracy in Infections and Tumors (38%). Figure\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e illustrates the distribution of correct response percentages across subcategories.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eIn the ten annual Competency Exams analyzed, ChatGPT passed four exams (years 2016, 2017, 2019, and 2022) by achieving\u0026thinsp;\u0026ge;\u0026thinsp;60% correct answers (Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e4\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eIts best performance occurred in 2019 (72% accuracy), while the lowest score was in 2023 (44%).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eChatGPT-3.5 performance in the Competency Exams (2014\u0026ndash;2023)\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eYear\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eUnanswered Questions (n)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCorrect Answers (n)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eIncorrect Answers (n)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eResult\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2023\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e44\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e49\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eFailed\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2022\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e61\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e31\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003ePassed\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2021\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e45\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e38\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eFailed\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2020\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e47\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e48\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eFailed\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2019\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e72\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e23\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003ePassed\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2018\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e59\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e37\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eFailed\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2017\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e70\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e29\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003ePassed\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2016\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e60\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e35\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003ePassed\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2015\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e57\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eFailed\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eTotal\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e56\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e515\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e330\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e4 / 10 Passed\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"DISCUSSION","content":"\u003cp\u003eThis study examined the performance of the ChatGPT-3.5 artificial intelligence model in T\u0026uuml;rkiye\u0026rsquo;s \u003cem\u003eResidency Training and Progress Examination\u003c/em\u003e (UEGS) and \u003cem\u003eCompetency Examination\u003c/em\u003e, both organized by the Turkish Society of Orthopedics and Traumatology (TOTBID).\u003c/p\u003e \u003cp\u003eChatGPT achieved its highest accuracy in the General Orthopedics category (61%) and the lowest in the Adult Reconstructive Surgery category (39%). The model significantly outperformed participants only in the Spine Surgery category. This finding may be explained by the fact that Spine Surgery was also the category in which resident physicians had the lowest accuracy (7.98%). These results may provide insights for those involved in planning and improving orthopedic and traumatology residency training programs, as there is already a core curriculum committee established for Spine Surgery (\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eWhen comparing the results of ChatGPT and residents in the UEGS, no significant difference was found in the total number of correct answers. However, ChatGPT gave more incorrect responses in every category compared with the participants\u0026rsquo; average. Based on these data, ChatGPT-3.5 performed worse than orthopedic and traumatology residents in the UEGS.\u003c/p\u003e \u003cp\u003eIn the Competency Examination, ChatGPT passed four out of ten exams by correctly answering at least 60 questions out of 100. In the remaining six exams, the model failed to reach the required passing score. Among the 1,000 questions analyzed, ChatGPT-3.5 was unable to answer 56 image-based questions due to its lack of visual interpretation capability. With newer versions such as ChatGPT-4, which can interpret images, more questions could be answered, potentially resulting in higher success rates. In a comparative study using UEGS questions, ChatGPT-4 demonstrated significantly higher accuracy than ChatGPT-3.5 (\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e). Likewise, a recent study using questions from the Turkish Competency Exam reported that ChatGPT-4o achieved higher accuracy than human participants across all orthopedic subdomains (\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e). This indicates that our findings align with the overall trend of improving model performance in newer AI systems rather than contradicting it.\u003c/p\u003e \u003cp\u003eOur findings show that ChatGPT-3.5 performs worse than resident physicians in the UEGS and fails to pass most of the Competency Exams in orthopedics and traumatology. Similar studies have demonstrated that ChatGPT performs well in some exams but poorly in others (\u003cspan additionalcitationids=\"CR15 CR16 CR17 CR18\" citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e). In certain cases, ChatGPT outperformed participants, whereas in others, as in our study, participants achieved better results.\u003c/p\u003e \u003cp\u003eChatGPT became one of the fastest-growing computer programs in history, reaching 100\u0026nbsp;million active users within two months of its public release (\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e). Like other AI models, it draws from a wide range of data sources, including peer-reviewed journal articles, textbooks, and online content (\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e). As new versions are released, both the technical capabilities of the model and the size of its knowledge base expand. Thus, it is expected that future AI models will continue to improve their ability to evaluate and answer questions.\u003c/p\u003e \u003cp\u003eThis study has certain limitations. The results of residents and specialists were evaluated using publicly available data from the TOTBID website, where not all exam years had standardized or complete analyses. Some exams were not conducted during the COVID-19 pandemic, and those years were therefore excluded. These limitations may have affected the comparison process.\u003c/p\u003e \u003cp\u003eFinally, the use of chatbots in medical education is an emerging trend supported by many educators and medical professionals. OpenAI\u0026rsquo;s ChatGPT offers several potential advantages for both students and teachers (\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e, \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e). Recent reviews have highlighted that generative AI tools hold promise for orthopedic education and training but also pose challenges related to reliability, ethical use, and integration into curricula. It is important to remember that these systems are still evolving and have not yet reached perfection. There remain significant gaps in both theoretical and practical orthopedic education that AI tools cannot yet fill (\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e).\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eChatGPT-3.5 showed variable accuracy in orthopedic examinations, performing comparably to residents only in the Spine Surgery category while underperforming in most others. Although ChatGPT offers potential educational benefits, it is not yet a reliable or valid resource for independent use in orthopedics and traumatology education.\u003c/p\u003e \u003cp\u003eAs artificial intelligence systems continue to evolve, future versions with multimodal capabilities may become more effective tools in medical learning and assessment.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e \u003cstrong\u003eEthics Approval and Consent to Participate\u003c/strong\u003e \u003cp\u003eNot Applicable\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eConsent for Publication\u003c/strong\u003e \u003cp\u003eNot Applicable\u003c/p\u003e \u003c/p\u003e\u003cp\u003e \u003ch2\u003eCompeting Interests\u003c/h2\u003e \u003cp\u003eThe authors declare that they have no competing interests\u003c/p\u003e \u003c/p\u003e\u003cp\u003e \u003ch2\u003eClinical Trial Number\u003c/h2\u003e \u003cp\u003eNot applicable\u003c/p\u003e \u003c/p\u003e\u003ch2\u003eFunding\u003c/h2\u003e \u003cp\u003eNot applicable\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eYMD and GEK wrote the main manuscript.HS and ME asked questions to ChatGPT.MAŞ performed the statistical work.\u003c/p\u003e\u003ch2\u003eAcknowledgements\u003c/h2\u003e \u003cp\u003eThe authors would like to express their sincere gratitude to all doctors who helped impove the orthopedic examination for a better education.\u003c/p\u003e\u003ch2\u003eData Availability\u003c/h2\u003e\u003cp\u003eThe data that support the findings of this study are available from the corresponding author upon reasonable request.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eMinh, D., Wang, H. X., Li, Y. F. \u0026amp; Nguyen, T. N. Explainable artificial intelligence: a comprehensive review. \u003cem\u003eArtif. Intell. Rev.\u003c/em\u003e \u003cb\u003e55(5)\u003c/b\u003e (1), 3503\u0026ndash;3568 (2022 June).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHaenlein, M. \u0026amp; Kaplan, A. A Brief History of Artificial Intelligence: On the Past, Present, and Future of Artificial Intelligence. \u003cem\u003eCalif. Manage. Rev.\u003c/em\u003e \u003cb\u003e61\u003c/b\u003e (4), 5\u0026ndash;14 (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOllivier, M. et al. A deeper dive into ChatGPT: history, use and future perspectives for orthopaedic research. \u003cem\u003eKnee Surg. Sports Traumatol. Arthrosc.\u003c/em\u003e \u003cb\u003e31\u003c/b\u003e (4), 1190\u0026ndash;1192 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu, P. R. et al. Application of Artificial Intelligence in Medicine: An Overview. \u003cem\u003eCurr. Med. Sci.\u003c/em\u003e \u003cb\u003e41\u003c/b\u003e (6), 1105\u0026ndash;1115 (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWu, D. et al. Artificial intelligence-tutoring problem-based learning in ophthalmology clerkship. \u003cem\u003eAnn. Transl Med.\u003c/em\u003e \u003cb\u003e8\u003c/b\u003e (11), 700\u0026ndash;700 (2020 June).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYang, Y. Y. \u0026amp; Shulruf, B. Expert-led and artificial intelligence (AI) system-assisted tutoring course increase confidence of Chinese medical interns on suturing and ligature skills: prospective pilot study. \u003cem\u003eJ. Educ. Eval Health Prof.\u003c/em\u003e \u003cb\u003e16\u003c/b\u003e, 7 (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eG\u0026ouml;nen, D. E. 2012\u0026ndash;2013 TOTBİD-TOTEK UZMANLIK EĞİTİMİ GELİŞİM SINAVI RAPORU (UEGS).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eT\u0026uuml;rk Ortopedi ve Travmatoloji Birliği Derneği [Internet]. [cited 2024 Sept 1]. Available from: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://totbid.org.tr/tr/\u003c/span\u003e\u003cspan address=\"https://totbid.org.tr/tr/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTabatabaian, M. \u003cem\u003ePrompt Engineering Using ChatGPT: Crafting Effective Interactions and Building GPT Apps\u003c/em\u003e 142 (Walter de Gruyter GmbH \u0026amp; Co KG, 2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBenli, İ. \u0026amp; Acaroğlu, E. T\u0026uuml;rk Ortopedi ve Travmatoloji Birliği Derneği (TOTBİD) T\u0026uuml;rk Ortopedi ve Travmatoloji Eğitim Konseyi Yeterlik Sınavları. Acta Orthop Traumatol Turc [Internet]. [cited 2024 Sept 1];45(2). Available from: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://dergipark.org.tr/en/download/article-file/169969\u003c/span\u003e\u003cspan address=\"https://dergipark.org.tr/en/download/article-file/169969\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAcaroğlu, E. et al. Core curriculum (CC) of spinal surgery: a step forward in defining our profession. \u003cem\u003eActa Orthop. Traumatol. Turc.\u003c/em\u003e \u003cb\u003e48\u003c/b\u003e (5), 475\u0026ndash;478 (2014).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAyik, G. et al. Exploring the role of artificial intelligence in Turkish orthopedic progression exams. \u003cem\u003eActa Orthop. Traumatol. Turc.\u003c/em\u003e \u003cb\u003e59\u003c/b\u003e (1), 18\u0026ndash;26 (2025).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYağar, H., G\u0026uuml;m\u0026uuml;şoğlu, E. \u0026amp; Mert Asfuroğlu, Z. Assessing the performance of ChatGPT-4o on the Turkish Orthopedics and Traumatology Board Examination. \u003cem\u003eJt. Dis. Relat. Surg.\u003c/em\u003e \u003cb\u003e36\u003c/b\u003e (2), 304\u0026ndash;310 (2025).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRuksakulpiwat, S., Kumar, A. \u0026amp; Ajibade, A. Using ChatGPT in Medical Research: Current Status and Future Directions. \u003cem\u003eJ. Multidiscip Healthc.\u003c/em\u003e \u003cb\u003e16\u003c/b\u003e, 1513\u0026ndash;1520 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKhan, R. A., Jawaid, M., Khan, A. R. \u0026amp; Sajjad, M. ChatGPT - Reshaping medical education and clinical management. Pak J Med Sci [Internet]. Feb 16 [cited 2024 Sept 28];39(2). (2023). Available from: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://pjms.org.pk/index.php/pjms/article/view/7653\u003c/span\u003e\u003cspan address=\"https://pjms.org.pk/index.php/pjms/article/view/7653\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOztermeli, A. D. \u0026amp; Oztermeli, A. ChatGPT performance in the medical specialty exam: An observational study. \u003cem\u003eMed. (Baltim).\u003c/em\u003e \u003cb\u003e102\u003c/b\u003e (32), e34673 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSumbal, A., Sumbal, R. \u0026amp; Amir, A. Can ChatGPT-3.5 Pass a Medical Exam? A Systematic Review of ChatGPT\u0026rsquo;s Performance in Academic Testing. \u003cem\u003eJ. Med. Educ. Curric. Dev.\u003c/em\u003e \u003cb\u003e11\u003c/b\u003e, 23821205241238641 (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang, X. et al. ChatGPT Performs on the Chinese National Medical Licensing Examination. \u003cem\u003eJ. Med. Syst.\u003c/em\u003e \u003cb\u003e47\u003c/b\u003e (1), 86 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAljindan, F. K. et al. ChatGPT Conquers the Saudi Medical Licensing Exam: Exploring the Accuracy of Artificial Intelligence in Medical Knowledge Assessment and Implications for Modern Medical Education. \u003cem\u003eCureus\u003c/em\u003e \u003cb\u003e15\u003c/b\u003e (9), e45043 (2023 Sept).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAlessandri Bonetti, M., Giorgino, R., Gallo Afflitto, G., De Lorenzi, F. \u0026amp; Egro, F. M. How Does ChatGPT Perform on the Italian Residency Admission National Exam Compared to 15,869 Medical Graduates? \u003cem\u003eAnn. Biomed. Eng.\u003c/em\u003e \u003cb\u003e52\u003c/b\u003e (4), 745\u0026ndash;749 (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMassey, P. A., Montgomery, C. \u0026amp; Zhang, A. S. Comparison of ChatGPT\u0026ndash;3.5, ChatGPT-4, and Orthopaedic Resident Performance on Orthopaedic Assessment Examinations. \u003cem\u003eJAAOS - J. Am. Acad. Orthop. Surg.\u003c/em\u003e \u003cb\u003e31\u003c/b\u003e (23), 1173 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHuang, Y. et al. Benchmarking ChatGPT-4 on a radiation oncology in-training exam and Red Journal Gray Zone cases: potentials and challenges for ai-assisted medical education and decision making in radiation oncology. \u003cem\u003eFront. Oncol.\u003c/em\u003e \u003cb\u003e13\u003c/b\u003e, 1265024 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMoritz, S., Romeike, B., Stosch, C. \u0026amp; Tolks, D. Generative AI (gAI) in medical education: Chat-GPT and co. \u003cem\u003eGMS J. Med. Educ.\u003c/em\u003e \u003cb\u003e40\u003c/b\u003e (4), Doc54 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAtik, O. Ş. Artificial intelligence: Who must have autonomy the machine or the human? \u003cem\u003eJt. Dis. Relat. Surg.\u003c/em\u003e \u003cb\u003e35\u003c/b\u003e (1), 1\u0026ndash;2 (2024).\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"ChatGPT, Board Examination, Orthopedics, Traumatology, Artificial Intelligence","lastPublishedDoi":"10.21203/rs.3.rs-8464449/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8464449/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e \u003cp\u003eArtificial intelligence (AI) technologies have rapidly expanded into the field of medical education, offering innovative tools for training and assessment.This study aimed to evaluate the performance of the ChatGPT-3.5 language model in the \u0026ldquo;Residency Training Progress Examination\u0026rdquo; (UEGS) and the \u0026ldquo;Competency Examination\u0026rdquo; administered by the Turkish Society of Orthopedics and Traumatology (TOTBID). The objective was to determine whether ChatGPT performs comparably to orthopedic residents and whether it can achieve a passing score in the Competency Exam.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eA total of 2,000 UEGS and 1,000 Competency Exam questions (2012\u0026ndash;2023, excluding 2020) were presented to ChatGPT-3.5 using standardized prompts designed within the Role\u0026ndash;Goals\u0026ndash;Context (RGC) framework. The model\u0026rsquo;s responses were statistically compared with those of orthopedic residents and specialists using the Mann\u0026ndash;Whitney U and Kruskal\u0026ndash;Wallis tests (p\u0026thinsp;\u0026lt;\u0026thinsp;0.05).\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eChatGPT achieved the highest accuracy in the General Orthopedics category (62%) and the lowest in Adult Reconstructive Surgery (40%). It outperformed residents only in the Spine Surgery category (p\u0026thinsp;\u0026lt;\u0026thinsp;0.05). In the Competency Exams, ChatGPT passed four of ten exams.\u003c/p\u003e\u003ch2\u003eConclusion\u003c/h2\u003e \u003cp\u003eChatGPT-3.5 demonstrated limited reliability and accuracy in orthopedic examinations and should be used cautiously as an educational support tool. Future studies involving newer multimodal versions of large language models may clarify their potential role in medical education and assessment.\u003c/p\u003e","manuscriptTitle":"Evaluation of ChatGPT's Performance in Residency Training Progress Exams and Competency Exams in Orthopedics and Traumatology","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-01-12 05:56:50","doi":"10.21203/rs.3.rs-8464449/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"a0949f59-44ae-4567-aa9a-a776a8453c18","owner":[],"postedDate":"January 12th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":60767875,"name":"Health sciences/Health care"},{"id":60767876,"name":"Health sciences/Medical research"}],"tags":[],"updatedAt":"2026-01-29T06:10:42+00:00","versionOfRecord":[],"versionCreatedAt":"2026-01-12 05:56:50","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8464449","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8464449","identity":"rs-8464449","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}