Comparison of ChatGPT 3.5 Turbo and Human Performance in taking the European Board of Ophthalmology Diploma (EBOD) Exam

Comparison of ChatGPT 3.5 Turbo and Human Performance in taking the European Board of Ophthalmology Diploma (EBOD) Exam | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Comparison of ChatGPT 3.5 Turbo and Human Performance in taking the European Board of Ophthalmology Diploma (EBOD) Exam Anna Maino, Jakub Klikowski, Brendan Strong, Wahid Ghaffari, Michał Woźniak, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3894423/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background/Objectives: This paper aims to assess ChatGPT’s performance in answering European Board of Ophthalmology Diploma (EBOD) examination papers and to compare these results to pass benchmarks and candidate results. Methods This cross-sectional study used a sample of previous past exam papers from 2012, 2013, 2020–2023 EBOD examinations. This study analysed ChatGPT’s responses to 392 Multiple Choice Questions (MCQ), each containing 5 true/false statements (1432 statements in total) and 48 Single Best Answer (SBA) questions. Results ChatGPT’s performance for MCQ questions scored on average 64.39%. ChatGPT’s strongest metric performance for MCQ was precision (68.76%). ChatGPT performed best at answering Pathology questions (Grubbs test p < .05). Optics and refraction had the lowest-scoring MCQ performance across all metrics. ChatGPT’s SBA performance averaged 28.43%, with the highest score and strongest performance in precision (29.36%). Pathology SBA questions were consistently the lowest-scoring topic across most metrics. ChatGPT chose option 1 more than other options (p = 0.19). When answering SBAs, human candidates scored higher than ChatGPT in all metric areas measured. Conclusion ChatGPT performed stronger for true/false questions, scoring a pass mark in most instances. Performance was poorer for SBA questions, especially as ChatGPT was more likely to choose the first answer out of four. Our results suggest that ChatGPT’s ability in information retrieval is better than knowledge integration. Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Introduction One of the most important techniques used in artificial intelligence (AI) is neural networks. Their origins date back to 1943 when McCulloch and Pitts proposed the first mathematical model of a neuron ( 1 ). They used the idea presented in Allan Turing's work ( 2 ) and depicted how the brain works as a highly efficient computational system consisting of interconnected simple elements (so-called McCulloch-Pitts neurons). In 1986, Rumelhart et al. proposed a back-propagation algorithm that allowed multilayer neural networks to be trained efficiently and allowed these models to be used to solve real, non-trivial decision tasks ( 3 ). However, only when massive computation was developed did deep neural networks (DNNs) become possible, bringing us closer to achieving a so-called human-level AI, also known as artificial general intelligence. One of the fastest-growing trends is Large Language Models (LLMs), which are DNNs based on so-called transformers and use huge language corpora for learning. These models are dedicated to natural language processing (NLP), making recognizing, translating, predicting, or generating text possible ( 4 ). The most well-known LLM is ChatGPT (Chat Generative Pre-trained Transformer, OpenAI, SanFrancisco, USA), a large language model-based chatbot which can process data into human-like text-based outputs. This increasing exposure to AI has resulted in mixed attitudes. Pessimistic views highlight the potential of AI to automate and replace humans in various fields ( 5 ). Optimists in the medical field suggest that physicians who embrace AI will have more opportunities to benefit from future developments ( 6 ) and more time to develop non-technical skills for patient care. This has prompted various medical colleges including the Royal College of Physicians and Surgeons of Canada to suggest training healthcare professionals about AI ( 7 ). Ophthalmology has been a speciality where AI application has been noted, focusing on diseases such as diabetic retinopathy ( 8 ), macular degeneration ( 9 ), cataracts ( 10 ) and glaucoma ( 11 ). There is significant promise about using AI in ophthalmology due to the significantly high global burden of eye conditions. Interestingly, AI has also been applied further to estimate relevant disease risk factors and refractive errors. It has been proposed that AI can play a key role in ophthalmological screening programs ( 12 ) and optimise cataract surgery by tracking the hardness of the lens nucleus in real-time ( 13 ). Several studies have explored the use of AI in answering practice questions for medical examinations and this is set to increase as AI becomes more widely used ( 14 – 18 ). While ChatGPT cannot be considered a stand-alone reliable source for ophthalmic education due to its limitations in data sources, knowledge cut-off, and lack of critical thinking. However, it offers accessibility, supplemental learning, interactive learning, and collaborative learning ( 19 ). The European Board of Ophthalmology (EBO) is a permanent working group of the Ophthalmology subspecialty section of the European Union of Medical Specialists (UEMS). The EBO is tasked with harmonising the standards of ophthalmology training across Europe. The EBO Diploma (EBOD) examination assesses the knowledge and clinical skills required to deliver high standards of ophthalmic care. The EBOD exam is aimed at eye doctors near the completion of their specialist training. It consists of a written part and a viva voce. Until 2021, the written part consisted of 52 text-based multiple-choice questions (MCQs). Each MCQ contained five statements which had to be independently marked as true or false, for a total of 260 statements per paper. The format changed in 2022, with fewer MCQs (44 questions, for a total of 220 statements) and the addition of 16 Single Best Answers (SBAs). SBA questions offer four options, only one of which is the best answer. We did not administer viva voce questions to ChatGPT as this format lacks standardisation and reproducibility. This study aims to assess ChatGPT performance in answering previous EBOD examination questions and to compare these results to pass benchmarks and candidate results. We also compared ChatGPT performance in answering different question formats (MCQs versus SBAs) and we investigated whether ChatGPT had identifiable biases when answering questions. Methods This cross-sectional study used real past exam papers from 2012, 2013, 2020, 2021, 2022 and 2023 EBOD. The Authors obtained approval from the EBO Executive Committee before using the papers. The Department of Systems and Computer Networks, Wrocław University of Science and Technology, Poland, conducted the study between July and October 2023 using ChatGPT version 3.5 Turbo. We used a brand-new account without previous exposure to EBO exam questions for MCQ. For SBA, we asked ChatGPT the same questions 10 times (original set) then we asked the same questions again but changed the order of the possible answers (randomised set). Quality metrics The performance metrics are calculated based on the so-called confusion matrix, which summarizes the number of correctly and incorrectly classified instances in each class ( 20 ). Let us consider the two-class classification task, e.g., if we would like to distinguish between correct and incorrect decisions. The name of the positive class is used, as a rule, as the class that is more important to us, such as a correctly given answer, and the negative class means the opposite class, such as an incorrectly given answer. The confusion matrix for such a task would then divide answers into True Positive ( TP ), False Negative ( FN ), False Positive ( FP ) and True Negative ( TN ). The most popular metrics measuring classifier performance are: Accuracy=(TP + TN)/(TP + TN + FP + FN) that indicates how often ChatGPT is right overall, Recall = TP/(TP + FN) whether ChatGPT can identify positive predictions from all the positive samples, Precision = TP/(TP + FP) indicates how often positive predictions are correct among the answers indicated as correct by ChatGPT and F1=(2∙Precision∙Recall)/(Precision + Recall) is one of the commonly used aggregate measures, calculated using the first harmonic mean of Recall and Precision ( 21 ). Secondary outcomes Secondary outcomes included exploring whether repeating test papers from different years affected the metric scores and whether ChatGPT was biased towards a particular question category. For the latter, we divided all exam questions according to 12 categories (Optics and Refraction; Strabismus, Paediatric Ophthalmology and Neuro-Ophthalmology; Cornea; Oculoplastics; Glaucoma; Cataract; Retina; Uveitis; General Medicine; Pathology; Pharmacology; Diagnostics and Imaging). Finally, we compared ChatGPT results with candidates’ answers. This study did not require ethics approval due to the sole involvement of data and artificial intelligence without any other research participants. Statistical analysis Data were analysed using MS Excel and GraphPad software (GraphPad QuickCalcs, http://www.graphpad.com/quickcalcs/grubbs1.cfm , accessed November 2023). Continuous data were analysed using linear regression and R-squared values to express the goodness of fit. We also used Grubbs test to identify outliers. Categorical variables were analysed using contingency tables. Results This study analysed ChatGPT responses to 392 MCQs (1432 true/false statements) and 48 SBAs. Multiple choice questions (MCQs) After dividing all answers to MCQ questions in a 2x2 contingency table, we found a statistically significant difference in observed proportions in the matrix vs expected values (p < .001). ChatGPT accuracy in answering MCQ questions was 63.18%, with 60.97% recall and 68.76% precision, giving an F1 score of 64.63%. ChatGPT accuracy did not change significantly, even after sitting several exams (R-squared:0.145) (Fig. 1 ). A similar lack of significant trends was also found for precision scores (R-squared:0.21), recall (R-squared:0.13), and F1 (R-squared:0.16). ChatGPT accuracy is higher for imaging and pathology and lower for Glaucoma and Optics/Refraction questions (Fig. 2 ) but this correlation is weak (R-squared: 0.39) with no significant outliers (Grubbs test p > .05). Similar results were found when using precision, recall and F1 (R-squared 0.29, 0.25 and 0.38 respectively), with pathology scores being significantly higher (Grubbs test p < .05 for precision and recall). Single Best Answers (SBA) ChatGPT selected the first answer in the majority of cases, whether it was correct or not (p = .019), even when having prior knowledge of the questions (Fig. 3 ). ChatGPT accuracy was 28.43% when answering SBA questions, with 28.94% recall and 29.36% precision, giving an F1 score of 26.97% (Fig. 4 ). ChatGPT accuracy for pathology questions was lower (6.67%) compared to other categories, even though the difference was not statistically significant (Grubbs test p > .05, R-squared 0.32). This was followed by pharmacology (13.33%) and cataract surgery (17.5%). Strabismus, paediatric ophthalmology and neuro-ophthalmology were the categories of questions answered with the best accuracy (37.5% correct response rate.) This was followed by uveitis and general medicine with 36.25% and 35% of questions answered correctly. For the other metrics, the highest scores were recorded for oculoplastics questions (precision, recall) and the lowest for pathology questions (F1, recall) even though these values were not statistically significant outliers (Grubbs test p > .05, R-squared precision = 0.16, R-squared recall = 0.45, R-squared F1 = 0.47). Comparison with candidates' performance Our study used question sets that have been administered to candidates, so it was possible to directly compare ChatGPT performance against a large number of candidates sitting the exam. Candidates achieved better scores than ChatGPT for each quality metric, with some overlap as shown in Fig. 5 . Discussion Multiple choice questions (MCQ) Chat GPT3.5 Turbo performance answering MCQ questions was fairly consistent across each exam paper and we did not demonstrate any learning effect. It achieved scores above the minimum pass mark (60%) in 5 exam papers out of 7 and scored very close to the pass mark otherwise. However, it was outperformed by the majority of human candidates. We can extrapolate that ChatGPT 3.5 or later versions are more accurate than older ones. This is in line with other papers as well ( 16 , 22 ). Given that the difficulty and the format of each exam paper are similar, we can assume that variations in performance are due to the way questions are formulated. It has been suggested that ChatGPT performs better on questions that humans found easier ( 23 ) but worse on ambiguous questions. When breaking down the scores for each category, our study shows better performance for MCQs in Pathology and Imaging. The lowest scores were found for Glaucoma and Optics/refraction. A strong performance in Pathology questions was repeated for the other metrics. This is an interesting finding as all Pathology and Imaging questions in our exam papers were text-only. It is widely accepted that current versions of ChatGPT cannot process information presented as graphs and tables even though Deep Learning models have been successfully used to interpret clinical images in Ophthalmology ( 12 ). However, when presented with text-only Pathology and Imaging questions, ChatGPT3.5 Turbo has demonstrated excellent accuracy in determining if a statement is true or false. Single best answer (SBA) questions ChatGPT 3.5 Turbo performance was well below the pass mark for SBA questions and we considered several explanations for this. Firstly, our study showed that this ChatGPT version is biased towards selecting the first answer. To test this, we submitted the same exam paper multiple times but randomly changed the order of the possible answers and the bias became even more obvious. Moreover, our findings are in line with previous studies suggesting that Chat GPT performs better at recalling facts or data rather than interpreting clinical scenarios ( 8 ). It follows that questions that test higher-order knowledge (such as SBAs) challenged ChatGPT more than true/false MCQs. The EBO introduced SBA questions in 2022, so the chatbot has been tested on a relatively smaller number of questions. SBA questions were also designed to test higher-order learning such as discriminating among differential diagnoses, each of them plausible. These questions cannot be answered using a process of elimination. Finally, a possible explanation for this less-than-satisfactory performance could be found in the choice of questions used. Our study is the first to use complete sets of original exam questions. In contrast, previous studies have extracted questions from question banks ( 14 , 15 , 17 ), and self-assessment programmes ( 15 , 16 ) or used sample questions taken from the web ( 18 ). We suggest that these studies’ results have to be interpreted with caution as they might not be a true representation of exam conditions. Panthier and Gatinel, in particular, presented their results as “ChatGPT performance on the French language version of the European Board of Ophthalmology (EBO) examination” ( 17 ). It should be clarified that their work is based on a question bank created in 1998 by French University professors of Ophthalmology but not endorsed by the EBO itself. Secondly, since the questions were formulated in a different language, we cannot automatically extrapolate those findings to exams conducted in English. Most importantly, the authors did not divide their results by question type, jumbling together true/false questions, SBAs and short answer questions making any comparisons challenging. ChatGPT 3.5 Turbo's performance in answering Pathology SBA questions was poor, which is diametrically opposite to its performance with MCQ questions. The chatbot answered questions with better accuracy in the Strabismus, Paediatric Ophthalmology and Neuro-ophthalmology even though it remained well below the pass rate. ChatGPT also performed better in Retina and Uveitis and General Medicine compared to other categories. Previous papers also report low accuracy in answering neuro-ophthalmology and optics questions with better accuracy in answering general medicine questions ( 14 , 15 ). Antaki and colleagues opined that training data in general medicine could be more widely available. In their paper they report a better performance across all categories for ChatGPT Plus and we suggest that this supports the hypothesis that the chatbot version is closely linked to its performance in answering exam papers. The Authors also suggest that some categories are intrinsically more challenging, even for humans, quoting neuro-ophthalmology and pathology as examples ( 15 ). It is therefore interesting to see that ChatGPT3.5 Turbo did relatively better in this type of question in our study. Our study is the first to be conducted on original and complete sets of questions. We compare ChatGPT performance to humans in exam conditions, rather than during the preparation phase. We administered a larger number of questions compared to other studies. Because exam papers are designed to have an equal proportion of easy/intermediate/difficult questions, we can conclude from comparing papers from several years. Finally, the questions are unavailable on the web, so we can rule out that ChatGPT has been trained on these questions. When setting up the study, we used ChatGPT 3.5 Turbo instead of 4.0 because it is more widely available, performed better than the older version ( 24 ) and our questions did not contain images. Conclusion ChatGPT 3.5 Turbo can understand and answer questions specific to the EBO diploma exams, showing an understanding of clinical ophthalmology comparable to a newly qualified specialist. Chatgpt 3.5 Turbo achieves a pass mark for true/false questions in most instances but its performance is poorer for SBA questions, showing that the chatbot can retrieve information better than it can integrate new knowledge. Even in its latest version, ChatGPT does not interpret figures and tables and this is a limiting factor in a visually-rich speciality such as Ophthalmology. It is also possible for ChatGPT to be trained on biased or incorrect data, which could lead to errors. We believe that ChatGPT can be a valuable tool in ophthalmic education, for instance allowing Exam Boards to ensure that their exam papers are easy to understand and are pitched at the right level of complexity. Further research is needed to explore ChatGPT's ability to generate exam questions or provide feedback to trainees. However, judging from the pace of evolution of the chatbot, that moment could be just around the corner. The Authors declare that there are no competing financial interests and that they did not receive any funding/sponsorships for the work presented in this article. Declarations The Authors declare that there are no competing financial interests and that they did not receive any funding/sponsorships for the work presented in this article. References McCulloch WS, Pitts W. A logical calculus of the ideas immanent in nervous activity. Bull Math Biophys. 1943 Dec;5(4):115–33. Turing AM. On Computable Numbers, with an Application to the Entscheidungsproblem. Proceedings of the London Mathematical Society. 1937;s2-42(1):230–65. Rumelhart DE, Hinton GE, Williams RJ. Learning representations by back-propagating errors. Nature. 1986 Oct;323(6088):533–6. Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P et al. Language models are few-shot learners. In: Advances in neural information processing systems. 2020; 1877–901. Mousavi Baigi SF, Sarbaz M, Ghaddaripouri K, Ghaddaripouri M, Mousavi AS, Kimiafar K. Attitudes, knowledge, and skills towards artificial intelligence among healthcare students: A systematic review. Health Science Reports. 2023; 6(3):e1138. Johnston SC. Anticipating and Training the Physician of the Future. Academic Medicine. 2018;93(8):1105-6. Reznick RK, Harris K, Horsley T, Hassani MS. Task Force Report on Artificial Intelligence and Emerging Digital Technologies. Royal College of Physicians and Surgeons of Canada. 2020;(February). Ting DSW, Cheung CYL, Lim G, Tan GSW, Quang ND, Gan A, et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA - Journal of the American Medical Association. 2017;318(22):2211-23. Burlina PM, Joshi N, Pekala M, Pacheco KD, Freund DE, Bressler NM. Automated grading of age-related macular degeneration from color fundus images using deep convolutional neural networks. JAMA Ophthalmol. 2017;135(11):1170-6. Wu X, Huang Y, Liu Z, Lai W, Long E, Zhang K, et al. Universal artificial intelligence platform for collaborative management of cataracts. British Journal of Ophthalmology. 2019;103(11):1553-60. Li Z, He Y, Keel S, Meng W, Chang RT, He M. Efficacy of a Deep Learning System for Detecting Glaucomatous Optic Neuropathy Based on Color Fundus Photographs. Ophthalmology. 2018;125(8):1199-1206. Ting DSW, Pasquale LR, Peng L, Campbell JP, Lee AY, Raman R, et al. Artificial intelligence and deep learning in ophthalmology. British Journal of Ophthalmology. 2019; 103(2):167-175. Tian S, Yin XC, Wang Z Bin, Zhou F, Hao HW. A VidEo-Based Intelligent Recognition and Decision System for the Phacoemulsification Cataract Surgery. Comput Math Methods Med. 2015: 202934. Mihalache A, Popovic MM, Muni RH. Performance of an Artificial Intelligence Chatbot in Ophthalmic Knowledge Assessment. JAMA Ophthalmol. 2023;141(6):589-597. Antaki F, Touma S, Milad D, El-Khoury J, Duval R. Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of Its Successes and Shortcomings. Ophthalmology Science. 2023;3(4):100324. Lin JC, Younessi DN, Kurapati SS, Tang OY, Scott IU. Comparison of GPT-3.5, GPT-4, and human user performance on a practice ophthalmology written examination. Eye (Basingstoke). 2023; 37(17):3694-5. Panthier C, Gatinel D. Success of ChatGPT, an AI language model, in taking the French language version of the European Board of Ophthalmology examination: A novel approach to medical knowledge assessment. J Fr Ophtalmol. 2023;46(7):706-11. Fowler T, Pullen S, Birkett L. Performance of ChatGPT and Bard on the official part 1 FRCOphth practice questions. British Journal of Ophthalmology. 2023 Nov 6;bjo-2023-324091. Gurnani B, Kaur K. Leveraging ChatGPT for ophthalmic education: A critical appraisal. Eur J Ophthalmol. 2023 Nov 16:11206721231215862. Japkowicz N, Shah M. Evaluating Learning Algorithms. Cambridge University Press; 2011.pp 100-106. Stapor K, Ksieniewicz P, García S, Woźniak M. How to design the fair experimental classifier evaluation. Appl Soft Comput. 2021 Jun;104:107219. Taloni A, Borselli M, Scarsi V, Rossi C, Coco G, Scorcia V, et al. Comparative performance of humans versus GPT-4.0 and GPT-3.5 in the self-assessment program of American Academy of Ophthalmology. Sci Rep. 2023 Oct 29;13(1):18562. Moshirfar M, Altaf AW, Stoakes IM, Tuttle JJ, Hoopes PC. Artificial Intelligence in Ophthalmology: A Comparative Analysis of GPT-3.5, GPT-4, and Human Expertise in Answering StatPearls Questions. Cureus. 2023; 5(6):e40822. Thirunavukarasu AJ. ChatGPT cannot pass FRCOphth examinations: implications for ophthalmology and large language model artificial intelligence. EyeNews. 2023; accessed 17 November 2023. Additional Declarations There is no conflict of interest Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3894423","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":279037514,"identity":"8a0c7467-c4c7-4934-97e0-bb1468bcb40e","order_by":0,"name":"Anna Maino","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA70lEQVRIie2RsQrCMBCGLwhxCbi24EMEhGqh6KukCE7VxUVwUChkqruP0alziqCL1jVSBydnHQQnMdHBLe3okG84Lpd85A8BsFj+EKwL4kAAGgJmakH1YGlQWh+l0ApmsGc1FFdvopduCa2n0HKV31DWb3ebySPNnzDptgR21yblXAwdlA2JnxwyKRhM/TXDbmpSZESV0iBUjrPyxiBMJWD3YlAGMuo8EV8oJbrqW8L0KMyKOuk5iG+0gr+KqA7m9UK+I3S/9aQYOSpYGPvG56tg8s7nA7qLVbAgUME2+SkxKB/Yr3Wg4lcsFovFUoc3Jx5XZOPfRmkAAAAASUVORK5CYII=","orcid":"https://orcid.org/0000-0003-1758-4789","institution":"Manchester Royal Eye Hospital","correspondingAuthor":true,"prefix":"","firstName":"Anna","middleName":"","lastName":"Maino","suffix":""},{"id":279037515,"identity":"eb497b06-cada-4df3-851c-466d63a99991","order_by":1,"name":"Jakub Klikowski","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Jakub","middleName":"","lastName":"Klikowski","suffix":""},{"id":279037516,"identity":"d66d9c8c-274c-4fad-9066-4b5cc57abf86","order_by":2,"name":"Brendan Strong","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Brendan","middleName":"","lastName":"Strong","suffix":""},{"id":279037517,"identity":"ee08e97e-9241-4a12-8d8d-5c3683c18b7b","order_by":3,"name":"Wahid Ghaffari","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Wahid","middleName":"","lastName":"Ghaffari","suffix":""},{"id":279037518,"identity":"c2ec5e9a-d3ba-4016-8857-f44ebd54ae15","order_by":4,"name":"Michał Woźniak","email":"","orcid":"https://orcid.org/0000-0003-0146-4205","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Michał","middleName":"","lastName":"Woźniak","suffix":""},{"id":279037519,"identity":"3e42b20c-454e-4e7b-8cd7-a20f41bc253d","order_by":5,"name":"Tristan BOURCIER","email":"","orcid":"","institution":"Faculté de Médecine, Strasbourg, FRANCE","correspondingAuthor":false,"prefix":"","firstName":"Tristan","middleName":"","lastName":"BOURCIER","suffix":""},{"id":279037520,"identity":"ed6e0c95-c82e-47dc-a40b-ed056c542be1","order_by":6,"name":"Andrzej Grzybowski","email":"","orcid":"https://orcid.org/0000-0002-3724-2391","institution":"University of Warmia and Mazury","correspondingAuthor":false,"prefix":"","firstName":"Andrzej","middleName":"","lastName":"Grzybowski","suffix":""}],"badges":[],"createdAt":"2024-01-24 14:56:32","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3894423/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3894423/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":52737535,"identity":"1de59d30-2546-41b0-a2cf-628ac8a03d3c","added_by":"auto","created_at":"2024-03-15 06:54:43","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":35446,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eAccuracy scores for each exam paper. SAMP indicates the last bank of questions (“sample” set)\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-3894423/v1/662f9af67f0fb222e87b5477.png"},{"id":52737536,"identity":"419da3a0-c941-4a87-a2ad-70c56c77ae8c","added_by":"auto","created_at":"2024-03-15 06:54:43","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":566438,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eMCQ scores divided by categories.\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-3894423/v1/4fb55aca201f8e3c17fe7812.png"},{"id":52738142,"identity":"dc7cd0b9-9c41-4f89-b96e-4e58db691f1e","added_by":"auto","created_at":"2024-03-15 07:02:43","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":28176,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eConfusion matrix for randomised SBA questions.\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-3894423/v1/b627f35cdfe82ed6e48f868d.png"},{"id":52737538,"identity":"83e7a84a-49de-4b77-9476-648201240ddb","added_by":"auto","created_at":"2024-03-15 06:54:43","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":434166,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eSBA scores divided by categories.\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-3894423/v1/34a0feada89b8bf48ecc8301.png"},{"id":52737539,"identity":"72285acd-ed8c-4a83-8829-a9144261974d","added_by":"auto","created_at":"2024-03-15 06:54:43","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":49730,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eCandidate performance metrics.\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-3894423/v1/8d1e094ad813ef157fabcfc2.png"},{"id":70342325,"identity":"f605494e-e790-430b-b1e8-60e230e187d1","added_by":"auto","created_at":"2024-12-02 10:19:21","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1805861,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3894423/v1/14f59cce-be76-4a64-a6a7-1a1bc49e3bf1.pdf"}],"financialInterests":"There is no conflict of interest","formattedTitle":"Comparison of ChatGPT 3.5 Turbo and Human Performance in taking the European Board of Ophthalmology Diploma (EBOD) Exam","fulltext":[{"header":"Introduction","content":"\u003cp\u003eOne of the most important techniques used in artificial intelligence (AI) is neural networks. Their origins date back to 1943 when McCulloch and Pitts proposed the first mathematical model of a neuron (\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e). They used the idea presented in Allan Turing's work (\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e) and depicted how the brain works as a highly efficient computational system consisting of interconnected simple elements (so-called McCulloch-Pitts neurons).\u003c/p\u003e \u003cp\u003eIn 1986, Rumelhart et al. proposed a back-propagation algorithm that allowed multilayer neural networks to be trained efficiently and allowed these models to be used to solve real, non-trivial decision tasks (\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e). However, only when massive computation was developed did deep neural networks (DNNs) become possible, bringing us closer to achieving a so-called human-level AI, also known as artificial general intelligence.\u003c/p\u003e \u003cp\u003eOne of the fastest-growing trends is Large Language Models (LLMs), which are DNNs based on so-called transformers and use huge language corpora for learning. These models are dedicated to natural language processing (NLP), making recognizing, translating, predicting, or generating text possible (\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThe most well-known LLM is ChatGPT (Chat Generative Pre-trained Transformer, OpenAI, SanFrancisco, USA), a large language model-based chatbot which can process data into human-like text-based outputs.\u003c/p\u003e \u003cp\u003eThis increasing exposure to AI has resulted in mixed attitudes. Pessimistic views highlight the potential of AI to automate and replace humans in various fields (\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e). Optimists in the medical field suggest that physicians who embrace AI will have more opportunities to benefit from future developments (\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e) and more time to develop non-technical skills for patient care. This has prompted various medical colleges including the Royal College of Physicians and Surgeons of Canada to suggest training healthcare professionals about AI (\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eOphthalmology has been a speciality where AI application has been noted, focusing on diseases such as diabetic retinopathy (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e), macular degeneration (\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e), cataracts (\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e) and glaucoma (\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e). There is significant promise about using AI in ophthalmology due to the significantly high global burden of eye conditions. Interestingly, AI has also been applied further to estimate relevant disease risk factors and refractive errors. It has been proposed that AI can play a key role in ophthalmological screening programs (\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e) and optimise cataract surgery by tracking the hardness of the lens nucleus in real-time (\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eSeveral studies have explored the use of AI in answering practice questions for medical examinations and this is set to increase as AI becomes more widely used (\u003cspan additionalcitationids=\"CR15 CR16 CR17\" citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e). While ChatGPT cannot be considered a stand-alone reliable source for ophthalmic education due to its limitations in data sources, knowledge cut-off, and lack of critical thinking. However, it offers accessibility, supplemental learning, interactive learning, and collaborative learning (\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThe European Board of Ophthalmology (EBO) is a permanent working group of the Ophthalmology subspecialty section of the European Union of Medical Specialists (UEMS). The EBO is tasked with harmonising the standards of ophthalmology training across Europe. The EBO Diploma (EBOD) examination assesses the knowledge and clinical skills required to deliver high standards of ophthalmic care. The EBOD exam is aimed at eye doctors near the completion of their specialist training. It consists of a written part and a viva voce. Until 2021, the written part consisted of 52 text-based multiple-choice questions (MCQs). Each MCQ contained five statements which had to be independently marked as true or false, for a total of 260 statements per paper. The format changed in 2022, with fewer MCQs (44 questions, for a total of 220 statements) and the addition of 16 Single Best Answers (SBAs). SBA questions offer four options, only one of which is the best answer. We did not administer viva voce questions to ChatGPT as this format lacks standardisation and reproducibility.\u003c/p\u003e \u003cp\u003eThis study aims to assess ChatGPT performance in answering previous EBOD examination questions and to compare these results to pass benchmarks and candidate results. We also compared ChatGPT performance in answering different question formats (MCQs versus SBAs) and we investigated whether ChatGPT had identifiable biases when answering questions.\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003eThis cross-sectional study used real past exam papers from 2012, 2013, 2020, 2021, 2022 and 2023 EBOD. The Authors obtained approval from the EBO Executive Committee before using the papers. The Department of Systems and Computer Networks, Wrocław University of Science and Technology, Poland, conducted the study between July and October 2023 using ChatGPT version 3.5 Turbo.\u003c/p\u003e \u003cp\u003eWe used a brand-new account without previous exposure to EBO exam questions for MCQ. For SBA, we asked ChatGPT the same questions 10 times (original set) then we asked the same questions again but changed the order of the possible answers (randomised set).\u003c/p\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eQuality metrics\u003c/h2\u003e \u003cp\u003eThe performance metrics are calculated based on the so-called confusion matrix, which summarizes the number of correctly and incorrectly classified instances in each class (\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e). Let us consider the two-class classification task, e.g., if we would like to distinguish between correct and incorrect decisions. The name of the \u003cem\u003epositive class\u003c/em\u003e is used, as a rule, as the class that is more important to us, such as a correctly given answer, and the negative class means the opposite class, such as an incorrectly given answer. The confusion matrix for such a task would then divide answers into True Positive (\u003cem\u003eTP\u003c/em\u003e), False Negative (\u003cem\u003eFN\u003c/em\u003e), False Positive (\u003cem\u003eFP\u003c/em\u003e) and True Negative (\u003cem\u003eTN\u003c/em\u003e). The most popular metrics measuring classifier performance are:\u003c/p\u003e \u003cp\u003eAccuracy=(TP\u0026thinsp;+\u0026thinsp;TN)/(TP\u0026thinsp;+\u0026thinsp;TN\u0026thinsp;+\u0026thinsp;FP\u0026thinsp;+\u0026thinsp;FN)\u003c/p\u003e \u003cp\u003ethat indicates how often ChatGPT is right overall,\u003c/p\u003e \u003cp\u003eRecall\u0026thinsp;=\u0026thinsp;TP/(TP\u0026thinsp;+\u0026thinsp;FN)\u003c/p\u003e \u003cp\u003ewhether ChatGPT can identify positive predictions from all the positive samples,\u003c/p\u003e \u003cp\u003ePrecision\u0026thinsp;=\u0026thinsp;TP/(TP\u0026thinsp;+\u0026thinsp;FP)\u003c/p\u003e \u003cp\u003eindicates how often positive predictions are correct among the answers indicated as correct by ChatGPT and\u003c/p\u003e \u003cp\u003eF1=(2∙Precision∙Recall)/(Precision\u0026thinsp;+\u0026thinsp;Recall)\u003c/p\u003e \u003cp\u003eis one of the commonly used aggregate measures, calculated using the first harmonic mean of \u003cem\u003eRecall\u003c/em\u003e and \u003cem\u003ePrecision\u003c/em\u003e (\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003eSecondary outcomes\u003c/h2\u003e \u003cp\u003eSecondary outcomes included exploring whether repeating test papers from different years affected the metric scores and whether ChatGPT was biased towards a particular question category. For the latter, we divided all exam questions according to 12 categories (Optics and Refraction; Strabismus, Paediatric Ophthalmology and Neuro-Ophthalmology; Cornea; Oculoplastics; Glaucoma; Cataract; Retina; Uveitis; General Medicine; Pathology; Pharmacology; Diagnostics and Imaging). Finally, we compared ChatGPT results with candidates\u0026rsquo; answers.\u003c/p\u003e \u003cp\u003eThis study did not require ethics approval due to the sole involvement of data and artificial intelligence without any other research participants.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003eStatistical analysis\u003c/h2\u003e \u003cp\u003eData were analysed using MS Excel and GraphPad software (GraphPad QuickCalcs, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://www.graphpad.com/quickcalcs/grubbs1.cfm\u003c/span\u003e\u003cspan address=\"http://www.graphpad.com/quickcalcs/grubbs1.cfm\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, accessed November 2023). Continuous data were analysed using linear regression and R-squared values to express the goodness of fit. We also used Grubbs test to identify outliers. Categorical variables were analysed using contingency tables.\u003c/p\u003e \u003c/div\u003e"},{"header":"Results","content":"\u003cp\u003eThis study analysed ChatGPT responses to 392 MCQs (1432 true/false statements) and 48 SBAs.\u003c/p\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003eMultiple choice questions (MCQs)\u003c/h2\u003e \u003cp\u003eAfter dividing all answers to MCQ questions in a 2x2 contingency table, we found a statistically significant difference in observed proportions in the matrix vs expected values (p\u0026thinsp;\u0026lt;\u0026thinsp;.001). ChatGPT accuracy in answering MCQ questions was 63.18%, with 60.97% recall and 68.76% precision, giving an F1 score of 64.63%.\u003c/p\u003e \u003cp\u003eChatGPT accuracy did not change significantly, even after sitting several exams (R-squared:0.145) (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). A similar lack of significant trends was also found for precision scores (R-squared:0.21), recall (R-squared:0.13), and F1 (R-squared:0.16).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eChatGPT accuracy is higher for imaging and pathology and lower for Glaucoma and Optics/Refraction questions (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e) but this correlation is weak (R-squared: 0.39) with no significant outliers (Grubbs test p\u0026thinsp;\u0026gt;\u0026thinsp;.05). Similar results were found when using precision, recall and F1 (R-squared 0.29, 0.25 and 0.38 respectively), with pathology scores being significantly higher (Grubbs test p\u0026thinsp;\u0026lt;\u0026thinsp;.05 for precision and recall).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eSingle Best Answers (SBA)\u003c/h2\u003e \u003cp\u003eChatGPT selected the first answer in the majority of cases, whether it was correct or not (p\u0026thinsp;=\u0026thinsp;.019), even when having prior knowledge of the questions (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eChatGPT accuracy was 28.43% when answering SBA questions, with 28.94% recall and 29.36% precision, giving an F1 score of 26.97% (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eChatGPT accuracy for pathology questions was lower (6.67%) compared to other categories, even though the difference was not statistically significant (Grubbs test p\u0026thinsp;\u0026gt;\u0026thinsp;.05, R-squared 0.32). This was followed by pharmacology (13.33%) and cataract surgery (17.5%).\u003c/p\u003e \u003cp\u003eStrabismus, paediatric ophthalmology and neuro-ophthalmology were the categories of questions answered with the best accuracy (37.5% correct response rate.) This was followed by uveitis and general medicine with 36.25% and 35% of questions answered correctly.\u003c/p\u003e \u003cp\u003eFor the other metrics, the highest scores were recorded for oculoplastics questions (precision, recall) and the lowest for pathology questions (F1, recall) even though these values were not statistically significant outliers (Grubbs test p\u0026thinsp;\u0026gt;\u0026thinsp;.05, R-squared precision\u0026thinsp;=\u0026thinsp;0.16, R-squared recall\u0026thinsp;=\u0026thinsp;0.45, R-squared F1\u0026thinsp;=\u0026thinsp;0.47).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003eComparison with candidates' performance\u003c/h2\u003e \u003cp\u003eOur study used question sets that have been administered to candidates, so it was possible to directly compare ChatGPT performance against a large number of candidates sitting the exam. Candidates achieved better scores than ChatGPT for each quality metric, with some overlap as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"Discussion","content":"\u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003eMultiple choice questions (MCQ)\u003c/h2\u003e \u003cp\u003eChat GPT3.5 Turbo performance answering MCQ questions was fairly consistent across each exam paper and we did not demonstrate any learning effect. It achieved scores above the minimum pass mark (60%) in 5 exam papers out of 7 and scored very close to the pass mark otherwise. However, it was outperformed by the majority of human candidates. We can extrapolate that ChatGPT 3.5 or later versions are more accurate than older ones. This is in line with other papers as well (\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e, \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eGiven that the difficulty and the format of each exam paper are similar, we can assume that variations in performance are due to the way questions are formulated. It has been suggested that ChatGPT performs better on questions that humans found easier (\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e) but worse on ambiguous questions.\u003c/p\u003e \u003cp\u003eWhen breaking down the scores for each category, our study shows better performance for MCQs in Pathology and Imaging. The lowest scores were found for Glaucoma and Optics/refraction. A strong performance in Pathology questions was repeated for the other metrics. This is an interesting finding as all Pathology and Imaging questions in our exam papers were text-only. It is widely accepted that current versions of ChatGPT cannot process information presented as graphs and tables even though Deep Learning models have been successfully used to interpret clinical images in Ophthalmology (\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e). However, when presented with text-only Pathology and Imaging questions, ChatGPT3.5 Turbo has demonstrated excellent accuracy in determining if a statement is true or false.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003eSingle best answer (SBA) questions\u003c/h2\u003e \u003cp\u003eChatGPT 3.5 Turbo performance was well below the pass mark for SBA questions and we considered several explanations for this. Firstly, our study showed that this ChatGPT version is biased towards selecting the first answer. To test this, we submitted the same exam paper multiple times but randomly changed the order of the possible answers and the bias became even more obvious. Moreover, our findings are in line with previous studies suggesting that Chat GPT performs better at recalling facts or data rather than interpreting clinical scenarios (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e). It follows that questions that test higher-order knowledge (such as SBAs) challenged ChatGPT more than true/false MCQs. The EBO introduced SBA questions in 2022, so the chatbot has been tested on a relatively smaller number of questions. SBA questions were also designed to test higher-order learning such as discriminating among differential diagnoses, each of them plausible. These questions cannot be answered using a process of elimination. Finally, a possible explanation for this less-than-satisfactory performance could be found in the choice of questions used. Our study is the first to use complete sets of original exam questions. In contrast, previous studies have extracted questions from question banks (\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e), and self-assessment programmes (\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e, \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e) or used sample questions taken from the web (\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e). We suggest that these studies\u0026rsquo; results have to be interpreted with caution as they might not be a true representation of exam conditions. Panthier and Gatinel, in particular, presented their results as \u0026ldquo;ChatGPT performance on the French language version of the European Board of Ophthalmology (EBO) examination\u0026rdquo; (\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e). It should be clarified that their work is based on a question bank created in 1998 by French University professors of Ophthalmology but not endorsed by the EBO itself. Secondly, since the questions were formulated in a different language, we cannot automatically extrapolate those findings to exams conducted in English.\u003c/p\u003e \u003cp\u003eMost importantly, the authors did not divide their results by question type, jumbling together true/false questions, SBAs and short answer questions making any comparisons challenging.\u003c/p\u003e \u003cp\u003eChatGPT 3.5 Turbo's performance in answering Pathology SBA questions was poor, which is diametrically opposite to its performance with MCQ questions. The chatbot answered questions with better accuracy in the Strabismus, Paediatric Ophthalmology and Neuro-ophthalmology even though it remained well below the pass rate. ChatGPT also performed better in Retina and Uveitis and General Medicine compared to other categories. Previous papers also report low accuracy in answering neuro-ophthalmology and optics questions with better accuracy in answering general medicine questions (\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eAntaki and colleagues opined that training data in general medicine could be more widely available. In their paper they report a better performance across all categories for ChatGPT Plus and we suggest that this supports the hypothesis that the chatbot version is closely linked to its performance in answering exam papers. The Authors also suggest that some categories are intrinsically more challenging, even for humans, quoting neuro-ophthalmology and pathology as examples (\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e). It is therefore interesting to see that ChatGPT3.5 Turbo did relatively better in this type of question in our study.\u003c/p\u003e \u003cp\u003eOur study is the first to be conducted on original and complete sets of questions. We compare ChatGPT performance to humans in exam conditions, rather than during the preparation phase. We administered a larger number of questions compared to other studies. Because exam papers are designed to have an equal proportion of easy/intermediate/difficult questions, we can conclude from comparing papers from several years. Finally, the questions are unavailable on the web, so we can rule out that ChatGPT has been trained on these questions.\u003c/p\u003e \u003cp\u003eWhen setting up the study, we used ChatGPT 3.5 Turbo instead of 4.0 because it is more widely available, performed better than the older version (\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e) and our questions did not contain images.\u003c/p\u003e \u003c/div\u003e"},{"header":"Conclusion","content":"\u003cp\u003eChatGPT 3.5 Turbo can understand and answer questions specific to the EBO diploma exams, showing an understanding of clinical ophthalmology comparable to a newly qualified specialist. Chatgpt 3.5 Turbo achieves a pass mark for true/false questions in most instances but its performance is poorer for SBA questions, showing that the chatbot can retrieve information better than it can integrate new knowledge. Even in its latest version, ChatGPT does not interpret figures and tables and this is a limiting factor in a visually-rich speciality such as Ophthalmology. It is also possible for ChatGPT to be trained on biased or incorrect data, which could lead to errors.\u003c/p\u003e \u003cp\u003eWe believe that ChatGPT can be a valuable tool in ophthalmic education, for instance allowing Exam Boards to ensure that their exam papers are easy to understand and are pitched at the right level of complexity. Further research is needed to explore ChatGPT's ability to generate exam questions or provide feedback to trainees. However, judging from the pace of evolution of the chatbot, that moment could be just around the corner.\u003c/p\u003e \u003cp\u003e \u003cem\u003eThe Authors declare that there are no competing financial interests and that they did not receive any funding/sponsorships for the work presented in this article.\u003c/em\u003e \u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cem\u003eThe\u0026nbsp;\u003c/em\u003e\u003cem\u003eAuthors declare that there are no competing financial interests and that they did not receive any funding/sponsorships for the work presented in this article.\u003c/em\u003e\u003c/p\u003e\n"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eMcCulloch WS, Pitts W. A logical calculus of the ideas immanent in nervous activity. Bull Math Biophys. 1943 Dec;5(4):115\u0026ndash;33.\u003c/li\u003e\n\u003cli\u003eTuring AM. On Computable Numbers, with an Application to the Entscheidungsproblem. Proceedings of the London Mathematical Society. 1937;s2-42(1):230\u0026ndash;65.\u003c/li\u003e\n\u003cli\u003eRumelhart DE, Hinton GE, Williams RJ. Learning representations by back-propagating errors. Nature. 1986 Oct;323(6088):533\u0026ndash;6.\u003c/li\u003e\n\u003cli\u003eBrown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P et al. Language models are few-shot learners. In: Advances in neural information processing systems. 2020; 1877\u0026ndash;901.\u003c/li\u003e\n\u003cli\u003eMousavi Baigi SF, Sarbaz M, Ghaddaripouri K, Ghaddaripouri M, Mousavi AS, Kimiafar K. Attitudes, knowledge, and skills towards artificial intelligence among healthcare students: A systematic review. Health Science Reports. 2023; 6(3):e1138.\u003c/li\u003e\n\u003cli\u003eJohnston SC. Anticipating and Training the Physician of the Future. Academic Medicine. 2018;93(8):1105-6.\u003c/li\u003e\n\u003cli\u003eReznick RK, Harris K, Horsley T, Hassani MS. Task Force Report on Artificial Intelligence and Emerging Digital Technologies. Royal College of Physicians and Surgeons of Canada. 2020;(February).\u003c/li\u003e\n\u003cli\u003eTing DSW, Cheung CYL, Lim G, Tan GSW, Quang ND, Gan A, et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA - Journal of the American Medical Association. 2017;318(22):2211-23.\u003c/li\u003e\n\u003cli\u003eBurlina PM, Joshi N, Pekala M, Pacheco KD, Freund DE, Bressler NM. Automated grading of age-related macular degeneration from color fundus images using deep convolutional neural networks. JAMA Ophthalmol. 2017;135(11):1170-6.\u003c/li\u003e\n\u003cli\u003eWu X, Huang Y, Liu Z, Lai W, Long E, Zhang K, et al. Universal artificial intelligence platform for collaborative management of cataracts. British Journal of Ophthalmology. 2019;103(11):1553-60.\u003c/li\u003e\n\u003cli\u003eLi Z, He Y, Keel S, Meng W, Chang RT, He M. Efficacy of a Deep Learning System for Detecting Glaucomatous Optic Neuropathy Based on Color Fundus Photographs. Ophthalmology. 2018;125(8):1199-1206.\u003c/li\u003e\n\u003cli\u003eTing DSW, Pasquale LR, Peng L, Campbell JP, Lee AY, Raman R, et al. Artificial intelligence and deep learning in ophthalmology. British Journal of Ophthalmology. 2019; 103(2):167-175.\u003c/li\u003e\n\u003cli\u003eTian S, Yin XC, Wang Z Bin, Zhou F, Hao HW. A VidEo-Based Intelligent Recognition and Decision System for the Phacoemulsification Cataract Surgery. Comput Math Methods Med. 2015: 202934.\u003c/li\u003e\n\u003cli\u003eMihalache A, Popovic MM, Muni RH. Performance of an Artificial Intelligence Chatbot in Ophthalmic Knowledge Assessment. JAMA Ophthalmol. 2023;141(6):589-597.\u003c/li\u003e\n\u003cli\u003eAntaki F, Touma S, Milad D, El-Khoury J, Duval R. Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of Its Successes and Shortcomings. Ophthalmology Science. 2023;3(4):100324.\u003c/li\u003e\n\u003cli\u003eLin JC, Younessi DN, Kurapati SS, Tang OY, Scott IU. Comparison of GPT-3.5, GPT-4, and human user performance on a practice ophthalmology written examination. Eye (Basingstoke). 2023; 37(17):3694-5. \u003c/li\u003e\n\u003cli\u003ePanthier C, Gatinel D. Success of ChatGPT, an AI language model, in taking the French language version of the European Board of Ophthalmology examination: A novel approach to medical knowledge assessment. J Fr Ophtalmol. 2023;46(7):706-11.\u003c/li\u003e\n\u003cli\u003eFowler T, Pullen S, Birkett L. Performance of ChatGPT and Bard on the official part 1 FRCOphth practice questions. British Journal of Ophthalmology. 2023 Nov 6;bjo-2023-324091.\u003c/li\u003e\n\u003cli\u003eGurnani B, Kaur K. Leveraging ChatGPT for ophthalmic education: A critical appraisal. Eur J Ophthalmol. 2023 Nov 16:11206721231215862.\u003c/li\u003e\n\u003cli\u003eJapkowicz N, Shah M. Evaluating Learning Algorithms. Cambridge University Press; 2011.pp 100-106.\u003c/li\u003e\n\u003cli\u003eStapor K, Ksieniewicz P, Garc\u0026iacute;a S, Woźniak M. How to design the fair experimental classifier evaluation. Appl Soft Comput. 2021 Jun;104:107219.\u003c/li\u003e\n\u003cli\u003eTaloni A, Borselli M, Scarsi V, Rossi C, Coco G, Scorcia V, et al. Comparative performance of humans versus GPT-4.0 and GPT-3.5 in the self-assessment program of American Academy of Ophthalmology. Sci Rep. 2023 Oct 29;13(1):18562.\u003c/li\u003e\n\u003cli\u003eMoshirfar M, Altaf AW, Stoakes IM, Tuttle JJ, Hoopes PC. Artificial Intelligence in Ophthalmology: A Comparative Analysis of GPT-3.5, GPT-4, and Human Expertise in Answering StatPearls Questions. Cureus. 2023; 5(6):e40822.\u003c/li\u003e\n\u003cli\u003eThirunavukarasu AJ. ChatGPT cannot pass FRCOphth examinations: implications for ophthalmology and large language model artificial intelligence. EyeNews. 2023; accessed 17 November 2023.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-3894423/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3894423/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground/Objectives:\u003c/h2\u003e \u003cp\u003eThis paper aims to assess ChatGPT\u0026rsquo;s performance in answering European Board of Ophthalmology Diploma (EBOD) examination papers and to compare these results to pass benchmarks and candidate results.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eThis cross-sectional study used a sample of previous past exam papers from 2012, 2013, 2020\u0026ndash;2023 EBOD examinations. This study analysed ChatGPT\u0026rsquo;s responses to 392 Multiple Choice Questions (MCQ), each containing 5 true/false statements (1432 statements in total) and 48 Single Best Answer (SBA) questions.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eChatGPT\u0026rsquo;s performance for MCQ questions scored on average 64.39%. ChatGPT\u0026rsquo;s strongest metric performance for MCQ was precision (68.76%). ChatGPT performed best at answering Pathology questions (Grubbs test p\u0026thinsp;\u0026lt;\u0026thinsp;.05). Optics and refraction had the lowest-scoring MCQ performance across all metrics. ChatGPT\u0026rsquo;s SBA performance averaged 28.43%, with the highest score and strongest performance in precision (29.36%). Pathology SBA questions were consistently the lowest-scoring topic across most metrics. ChatGPT chose option 1 more than other options (p\u0026thinsp;=\u0026thinsp;0.19). When answering SBAs, human candidates scored higher than ChatGPT in all metric areas measured.\u003c/p\u003e\u003ch2\u003eConclusion\u003c/h2\u003e \u003cp\u003eChatGPT performed stronger for true/false questions, scoring a pass mark in most instances. Performance was poorer for SBA questions, especially as ChatGPT was more likely to choose the first answer out of four. Our results suggest that ChatGPT\u0026rsquo;s ability in information retrieval is better than knowledge integration.\u003c/p\u003e","manuscriptTitle":"Comparison of ChatGPT 3.5 Turbo and Human Performance in taking the European Board of Ophthalmology Diploma (EBOD) Exam","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-03-15 06:54:39","doi":"10.21203/rs.3.rs-3894423/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"849849f9-822d-4554-a370-e9e071ef9b76","owner":[],"postedDate":"March 15th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-12-02T10:11:13+00:00","versionOfRecord":[],"versionCreatedAt":"2024-03-15 06:54:39","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-3894423","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3894423","identity":"rs-3894423","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}