Are Large Language Models Ready for Specialty-Level Periodontology? A Comparative Evaluation Across Question Types and Difficulty Strata

Are Large Language Models Ready for Specialty-Level Periodontology? A Comparative Evaluation Across Question Types and Difficulty Strata | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Are Large Language Models Ready for Specialty-Level Periodontology? A Comparative Evaluation Across Question Types and Difficulty Strata Helmi Mostafa Abdaljabbar Khatib This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9468440/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background Artificial intelligence (AI), particularly large language models (LLMs), has emerged as a promising tool in healthcare, with potential applications in clinical decision support and dental education. Despite increasing interest, evidence regarding the performance of LLMs in periodontology—especially in clinically oriented, scenario-based assessments—remains limited. This study aimed to evaluate and compare the accuracy of multiple LLMs in answering knowledge-based and scenario-based multiple-choice questions (MCQs) in periodontology across different difficulty levels. Methods A total of 100 periodontology MCQs were selected from validated academic sources and divided into two categories: knowledge-based questions (n = 50) and scenario-based questions (n = 50). Each category was further stratified into easy and moderate–difficult levels (25 questions each) based on expert consensus. Four publicly available LLMs (GPT-4o, Gemini 1.5 Flash, DeepSeek-V3, and Microsoft Copilot) were evaluated using a standardized prompting framework. Model responses were assessed for accuracy against verified answer keys. Statistical analysis was performed using Pearson’s Chi-square test, with significance set at p < 0.05. Results Overall accuracy ranged from 63% to 71%, with Gemini achieving the highest overall performance (71%), followed by GPT (70%), DeepSeek (65%), and Copilot (63%), without statistically significant differences (p = 0.80). All models demonstrated higher accuracy in scenario-based MCQs compared to knowledge-based questions, with statistically significant improvements observed for GPT (p < 0.001), Gemini (p = 0.014), and DeepSeek (p = 0.022). Accuracy decreased with increasing question difficulty, with significant performance declines observed for Gemini (p = 0.015) and Copilot (p = 0.022), while GPT and DeepSeek showed more stable performance. Conclusions LLMs demonstrate baseline competency in periodontology and show improved performance in context-rich, scenario-based questions. However, their accuracy remains variable and task-dependent, particularly under increasing difficulty. While these models may serve as useful adjuncts in dental education and clinical support, they are not yet reliable as standalone tools for clinical decision-making. Artificial intelligence Large language models Periodontology Dental education Clinical reasoning Multiple-choice questions Figures Figure 1 1. Introduction Artificial intelligence (AI) has emerged as a transformative force in modern technology, aiming to develop systems capable of performing tasks that traditionally require human intelligence [ 1 ]. In recent decades, the convergence of big data, increased computational power, and sophisticated AI algorithms has begun to permeate and simplify numerous aspects of daily life [ 2 ]. Within the health sciences, this technological revolution is particularly pronounced, driving significant changes in medicine and dentistry. The potential of AI is becoming increasingly evident across a spectrum of applications, including diagnostic processes, treatment planning, and health professions education [ 3 ]. By processing extensive datasets with remarkable speed, AI models can enable objective and comprehensive analyses—from clinical findings to histopathological features—that hold the potential to refine treatment methodologies and improve prognostic outcomes [ 4 ]. Large Language Models (LLMs) represent a specific and rapidly advancing subset of generative AI. These sophisticated systems simulate human language processing through deep learning and neural networks, trained on vast text corpora comprising books, scientific articles, and websites [ 5 ]. In the medical and dental fields, LLMs are being explored for a diverse range of tasks, including clinical decision support, patient education, telemedicine, administrative workflow optimization, and the enhancement of educational processes [ 6 – 9 ]. More recent studies have specifically evaluated their accuracy and consistency in answering domain-specific questions in oral pathology, endodontics and pediatric dentistry, revealing both promising capabilities and notable limitations in reliability and depth of reasoning [ 10 – 14 ]. The field of periodontology, which focuses on the diagnosis and treatment of chronic inflammatory diseases like periodontitis, stands to benefit significantly from such advancements. Periodontitis, if not managed effectively, leads to the destruction of the tooth-supporting apparatus and is a major cause of tooth loss [ 15 ]. Its profound links to systemic conditions such as cardiovascular disease and diabetes further underscore the critical importance of early and accurate diagnosis and treatment planning [ 16 ].However, diagnosis often relies heavily on a clinician's expertise and subjective interpretation of clinical and radiographic findings, introducing potential variability [ 17 ]. This variability highlights an area where artificial intelligence (AI) could provide meaningful support. In this context, AI technologies—particularly LLMs—could serve as powerful adjuncts to clinicians by analyzing patient histories, synthesizing complex clinical information, proposing differential diagnoses, and reinforcing evidence-based decision-making [ 18 – 20 ]. Despite this promise, the integration of AI in dentistry remains in its nascent stages. While many AI models demonstrate high precision in controlled, experimental settings, they face significant challenges in terms of clinical accuracy, reliability, and generalizability [ 21 , 22 ]. However, there is very limited evidence, and in some contexts virtually none, regarding the performance of LLMs on multiple-choice questions specifically focused on periodontics, particularly when assessed using clinically nuanced, case-based scenarios. This study was designed to address this critical gap. Our primary aim was to evaluate and compare the performance of several freely accessible large language models—including GPT, Gemini, DeepSeek, and Copilot—in answering a set of 100 multiple-choice questions in periodontology. The question set was carefully curated and stratified by difficulty (easy, medium-hard) by expert dentists, comprising 50 general knowledge questions and 50 complex clinical scenarios. This difficulty-stratified, comparative approach. This difficulty-based design enables a clear comparison of model performance across varying levels of question complexity, providing practical insight into their current capabilities in periodontology. 2. Methodology This study evaluated the diagnostic accuracy of four large language models (LLMs) in answering expert-level multiple-choice questions (MCQs) in periodontics. Two independent question sets, comprising a total of 100 MCQs, were used for the assessment. 2.1 Question Sources and Preparation Two distinct MCQ datasets were utilized: 1. Knowledge-based MCQs (n = 50) 2. Scenario-based MCQs (S-MCQs) (n = 50) The content domains were guided by widely used periodontal references and board-style examination resources. To ensure transparency and address copyright considerations, all questions were independently paraphrased, structurally modified, and revalidated by the authors. A representative subset of the questions and answers is provided in the Appendix to support reproducibility. All questions were designed for advanced dental learners and clinicians, reflecting expert-level knowledge and clinical reasoning in periodontology. 2.2 Expert Review and Difficulty Classification To ensure methodological rigor, three experienced dentists, independent of the study author, reviewed all questions in a blinded manner. Each evaluator verified the correctness of the answer options to ensure content accuracy. Following independent verification, the evaluators convened to discuss any discrepancies and reach consensus through structured deliberation. Only after agreement on answer validity was achieved were the questions classified according to difficulty level. Items were subsequently categorized into two difficulty groups based on collective expert judgment: · Easy (n = 25 per set) · Moderate–Difficult (n = 25 per set) The final classification was determined through consensus agreement among the three evaluators to minimize subjective bias and enhance reliability. 2.3 Large Language Models Evaluated The evaluation was conducted using the most recent publicly available free versions of each model at the time of testing (2026), accessed through their official web interfaces under default settings (no parameter modification): · GPT: GPT-4o (free-tier version available via the ChatGPT platform by OpenAI) · Gemini: Gemini 1.5 Flash provided by Google · DeepSeek: DeepSeek-V3 by DeepSeek · Copilot: Microsoft Copilot powered by GPT-4o and provided by Microsoft All models were tested under default inference configurations without temperature adjustment or system-level customization. 2.4 Prompting Strategy The prompting framework followed a principled instruction design approach to ensure consistency across models. Each model received the following standardized prompt. This approach was implemented in accordance with previously published guidelines on principled prompting for large language models [23], which emphasize that carefully structured instructions enhance response consistency and reliability. Your task is to answer the following multiple-choice question. You must respond with only the letter of the correct option (A, B, C, or D). Please number your answer according to the question order. Do not provide explanations, justifications, or reasoning. The intended audience is domain experts in dentistry. Each 50-question set was submitted as a single batch prompt , rather than sequential single-question entries. This approach was adopted to: 1. Maintain identical contextual conditions across items. 2. Avoid inter-session variability. 3. Minimize potential memory carryover bias. 4. Ensure standardized evaluation across models. No follow-up questions, clarifications, or feedback were provided after model responses in order to prevent iterative learning bias. 2.5 Outcome Measures Model responses were classified as correct or incorrect based on the verified official answer keys. Primary outcome measures included: · Accuracy (%) for Easy questions · Accuracy (%) for Moderate–Difficult questions · Overall accuracy (%) · performance change (%) · Performance Drop (%) Statistical significance between difficulty levels was assessed using Pearson’s Chi-square test , with significance set at p < 0.05. 2.6 Appendix A An Appendix accompanies this manuscript, containing the full set of questions with their correct answers, to facilitate transparency, reproducibility, and verification of the study procedures. 3. Results A total of 100 multiple-choice questions were evaluated, including 50 knowledge-based MCQs and 50 scenario-based MCQs (S-MCQs). Model performance varied across question types and difficulty levels as shown in Fig. 1 and detailed in Table 1. In knowledge-based MCQs, overall accuracy ranged from 54% to 60%, with Gemini achieving the highest accuracy (60%), followed by GPT, DeepSeek, and Copilot (54% each). No statistically significant differences were observed between models (χ² = 0.36, p = 0.95). In scenario-based MCQs, accuracy improved across all models, ranging from 72% to 86%. GPT demonstrated the highest accuracy (86%), followed by Gemini (82%), DeepSeek (76%), and Copilot (72%). However, inter-model differences did not reach statistical significance (χ² = 2.77, p = 0.43). When total performance across all 100 questions was analyzed, overall accuracy ranged between 63% and 71%, with Gemini achieving the highest overall accuracy (71%), followed by GPT (70%), DeepSeek (65%), and Copilot (63%). These differences were not statistically significant (χ² = 0.99, p = 0.80). Table1: Comparison of the accuracy rates of LLM responses based on different question type MCQs (Knowledge-Based) True n (%) False n (%) Test Statistics p* GPT 27 (54%) 23 (46%) Gemini 30 (60%) 20 (40%) DeepSeek 27 (54%) 23 (46%) Copilot 27 (54%) 23 (46%) 0.36 0.95 S-MCQs (Scenario-Based) GPT 43 (86%) 7 (14%) Gemini 41 (82%) 9 (18%) DeepSeek 38 (76%) 12 (24%) Copilot 36 (72%) 14 (28%) 2.77 0.43 Total Questions (n = 100) GPT 70 (70%) 30 (30%) Gemini 71 (71%) 29 (29%) DeepSeek 65 (65%) 35 (35%) Copilot 63 (63%) 37 (37%) 0.99 0.80 Table 2 summarizes the performance changes of the LLMs across question types. All models demonstrated higher accuracy on scenario-based MCQs (S-MCQs) compared to standard MCQs, with GPT showing the largest improvement (+59.3%), followed by DeepSeek (+40.7%), Gemini (+36.7%), and Copilot (+33.3%). The χ² analysis indicated statistically significant performance gains for GPT (p < 0.001), Gemini (p = 0.014), and DeepSeek (p = 0.022), whereas the increase for Copilot did not reach significance (p = 0.058). Effect sizes (Φ) ranged from 0.19 to 0.35, reflecting small to moderate associations between question type and model accuracy. Table2: Comparison of performance change experienced by LLMs across question type Model MCQs True n (%) S-MCQs True n (%) Percent Change Test Statistics (χ²) p ES (Φ) 95% CI for Φ GPT 27 (54%) 43 (86%) +59.3% 12.24 <0.001 0.35 [0.16–0.51] False 23 (46%) 7 (14%) Gemini 30 (60%) 41 (82%) +36.7% 5.98 0.014 0.24 [0.05–0.41] False 20 (40%) 9 (18%) DeepSeek 27 (54%) 38 (76%) +40.7% 5.21 0.022 0.23 [0.04–0.40] False 23 (46%) 12 (24%) Copilot 27 (54%) 36 (72%) +33.3% 3.59 0.058 0.19 [0.00–0.37] False 23 (46%) 14 (28%) Performance analysis according to question difficulty (easy vs. moderate–difficult) demonstrated a decline in accuracy with increasing complexity in most models (Table 3). Statistically significant reductions were observed for Gemini (χ² = 5.88, p = 0.015) and Copilot (χ² = 5.26, p = 0.022), whereas GPT and DeepSeek did not show significant performance differences across difficulty levels (p > 0.05). Table3: Performance of LLMs According to Question Difficulty Model Easy n (%) Moderate–Difficult n (%) Performance Drop (%) χ² p-value Φ 95% CI for Φ GPT 38/50 (76%) 32/50 (64%) −12% 1.74 0.19 0.13 [0.00–0.33] Gemini 41/50 (82%) 30/50 (60%) −22% 5.88 0.015 0.24 [0.05–0.42] DeepSeek 34/50 (68%) 30/50 (60%) −8% 0.69 0.41 0.08 [0.00–0.27] Copilot 37/50 (74%) 26/50 (52%) −22% 5.26 0.022 0.23 [0.04–0.40] Overall, while numerical differences in accuracy were observed across models and question types, most inter-model comparisons did not demonstrate statistical significance. 4. Discussion This study provides a difficulty-stratified, comparative evaluation of four prominent large language models—GPT, Gemini, DeepSeek, and Copilot—on a curated set of 100 periodontology multiple-choice questions. The principal findings reveal a nuanced landscape of AI readiness for this specialty. While no single model demonstrated statistically significant overall superiority, their performance varied markedly across question types and difficulty levels. A key observation was the consistent and, for some models, statistically significant improvement in accuracy when addressing complex, scenario-based clinical questions compared to basic knowledge recall. However, this capability was tempered by a general decline in performance as question difficulty increased, a trend that reached statistical significance for two of the four models. These findings suggest that, although LLMs demonstrate baseline competency in periodontology, their readiness for unsupervised academic or clinical use remains limited and task-dependent. While they may serve as useful adjuncts, they cannot replace the nuanced, experience-based judgment of a trained periodontist. The overall accuracy rates observed in this study, ranging from 63% to 71%, align with the broader body of literature evaluating LLMs in dental and medical education. Previous research has reported a wide spectrum of accuracy, from 42.5% to over 86%, depending on the model, question format, and subject domain [24-27]. The performance of the models in our study falls comfortably within this range, reinforcing the notion that LLMs have a baseline capability in specialized biomedical fields. The lack of a statistically significant difference between models in overall performance, as well as within the knowledge-based and scenario-based subsets, suggests a convergence in the capabilities of leading freely accessible LLMs for general periodontology knowledge. This finding is noteworthy, as it implies that for broad, curriculum-based queries, the choice of model may be less critical than other factors, such as accessibility or specific features. This contrasts with some studies that have found a clear numerical, and sometimes statistical, advantage for models like ChatGPT-4o [28], though our study did not include this specific variant. A particularly noteworthy finding of this study was the consistent and, for three of the four models, statistically significant improvement in accuracy when transitioning from basic knowledge MCQs to scenario-based MCQs (S-MCQs). GPT demonstrated the most substantial gain, with a 59.3% increase in accuracy (from 54% to 86%), followed by DeepSeek (+40.7%, p = 0.022) and Gemini (+36.7%, p = 0.014). The performance gain for Copilot (+33.3%) approached but did not reach statistical significance (p = 0.058). This finding appears to contrast with a recent comprehensive evaluation by Urda-Cîmpean et al.[29], who reported that four leading LLMs performed better on knowledge-based questions than on clinical reasoning tasks. However, the superior performance of LLMs on scenario-based questions in our study likely reflects methodological differences—particularly in domain structure, standardized question design, and lower contextual ambiguity—rather than a true contradiction of prior findings. The difficulty-stratified analysis, a core strength of this study, revealed that this clinical reasoning capacity has its limits. All models exhibited a decline in accuracy when moving from easy to moderate-difficult questions, with the performance drop being statistically significant for Gemini and Copilot. The fact that GPT and DeepSeek maintained their performance levels across difficulty tiers, without a statistically significant drop, may hint at more robust reasoning architectures or fine-tuning strategies that better equip them to handle cognitive complexity. This study has several limitations that warrant consideration. First, the question set, while expertly curated and stratified, may not fully represent the breadth and depth of the periodontology field. Second, the analysis was limited to text-based queries and did not incorporate radiographic images or other visual data, which are integral to periodontal diagnosis and treatment planning. Third, the proprietary nature of the models means their training data and internal parameters are opaque. The observed performance differences could be attributed to factors like training data cutoff dates, which are beyond our control and can change without notice. Finally, as with any study using a specific snapshot of AI models, the rapid pace of development means that these results are a point-in-time assessment, and future iterations may perform differently. In conclusion, this study demonstrates that current large language models possess a functional, though not flawless, grasp of periodontology. Their ability to reason through clinical scenarios is particularly promising, yet their susceptibility to error on more complex and knowledge-based questions necessitates a cautious approach to their integration. For educators and clinicians, the take-home message is clear: LLMs are powerful assistants, but they are not autonomous experts. Their role should be to augment, not supplant, the critical thinking and expertise of the periodontist. 5. Conclusion This study provides a difficulty-stratified evaluation of four contemporary large language models—GPT, Gemini, DeepSeek, and Copilot—in the domain of periodontology, demonstrating moderate overall accuracy and context-dependent performance. Notably, the improved outcomes observed in scenario-based questions suggest a developing capacity for structured clinical reasoning, particularly within well-defined and standardized contexts. However, the decline in performance with increasing question difficulty, alongside the absence of statistically significant superiority among models, underscores important limitations in reliability and higher-order reasoning. These findings indicate that, while LLMs hold potential as supportive tools in dental education and clinical decision-making, their current performance remains insufficient for independent application. Declarations Declaration of generative AI and AI-assisted technologies in the manuscript preparation process During the preparation of this work, the authors used ChatGPT to assist in language refinement and academic phrasing. All content was critically reviewed, edited, and validated by the authors, who take full responsibility for the final manuscript. References Alowais SA et al (2023) Revolutionizing healthcare: the role of artificial intelligence in clinical practice. BMC Med Educ 23(1):689 Ding H et al (2023) Artificial intelligence in dentistry—A review. Front Dent Med 4:1085251 Bindra S, Jain R (2024) Artificial intelligence in medical science: a review. Ir J Med Sci (1971-) 193(3):1419–1429 Araújo ALD et al (2023) Machine learning concepts applied to oral pathology and oral medicine: a convolutional neural networks' approach. J Oral Pathol Med 52(2):109–118 Kumar P (2024) Large language models (LLMs): survey, technical frameworks, and future challenges. Artif Intell Rev 57(10):260 Kung TH et al (2023) Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLoS Digit health 2(2):e0000198 Eggmann F et al (2023) Implications of large language models such as ChatGPT for dental medicine. J Esthetic Restor Dentistry 35(7):1098–1102 Puleio F et al (2024) Clinical, research, and educational applications of ChatGPT in dentistry: a narrative review. Appl Sci 14(23):10802 Aminoshariae A, Kulild J, Nagendrababu V (2021) Artificial intelligence in endodontics: current applications and future directions. J Endod 47(9):1352–1357 Yilmaz BE, Gokkurt BN, Yilmaz, Ozbey F (2025) Artificial intelligence performance in answering multiple-choice oral pathology questions: a comparative analysis. BMC Oral Health 25(1):573 Suárez A et al (2024) Unveiling the ChatGPT phenomenon: evaluating the consistency and accuracy of endodontic question answers. Int Endod J 57(1):108–113 Rokhshad R et al (2024) Accuracy and consistency of chatbots versus clinicians for answering pediatric dentistry questions: A pilot study. J Dent 144:104938 Jung YS et al (2024) Evaluating the accuracy of artificial intelligence-based chatbots on pediatric dentistry questions in the Korean national dental board exam. J Korean Acad Pediatr Dentistry 51(3):299–309 Çeki̇ç EC, Tavşan O (2025) Evaluating large language models using national endodontic specialty examination questions: are they ready for real-world dentistry? BMC Med Educ 25(1):1308 Dentino A et al (2000) Principles of periodontology. Periodontology., 2013. 61(1): pp. 16–53 Bourgeois D et al (2019) Periodontal pathogens as risk factors of cardiovascular diseases, diabetes, rheumatoid arthritis, cancer, and chronic obstructive pulmonary disease—Is there cause for consideration? Microorganisms 7(10):424 Petersson AR et al (1984) Observer variations in the interpretation of periapical osseous structures: A comparison between xeroradiography and conventional radiography. J Endod 10(5):205–209 Farhadi Nia M, Ahmadi M, Irankhah E (2025) Transforming dental diagnostics with artificial intelligence: advanced integration of ChatGPT and large language models for patient care. Front Dent Med 5:1456208 Özbay Y, Erdoğan D, Dinçer GA (2025) Evaluation of the performance of large language models in clinical decision-making in endodontics. BMC Oral Health 25(1):648 Dashti M et al (2024) Performance of ChatGPT 3.5 and 4 on US dental examinations: the INBDE, ADAT, and DAT. Imaging Sci dentistry 54(3):271 Katne T et al (2019) Artificial intelligence: demystifying dentistry–the future and beyond. Int J Contemp Med Surg Radiol 4(4):D6–D9 Banerjee M et al (2024) The Prospect of Artificial Intelligence in Dentistry. Med Res Its Appl Vol 6:136–146 Bsharat SM, Myrzakhan A, Shen Z (2023) Principled instructions are all you need for questioning llama-1/2, gpt-3.5/4. arXiv preprint arXiv:2312.16171 Danesh A et al (2023) The performance of artificial intelligence language models in board-style dental knowledge assessment: A preliminary study on ChatGPT. J Am Dent Assoc 154(11):970–974 Quah B et al (2024) Performance of large language models in oral and maxillofacial surgery examinations. Int J Oral Maxillofac Surg 53(10):881–886 Chau RCW et al (2024) Performance of Generative Artificial Intelligence in Dental Licensing Examinations. Int Dent J 74(3):616–621 Tassoker M (2025) ChatGPT-4 Omni's superiority in answering multiple-choice oral radiology questions. BMC Oral Health 25(1):173 Künzle P, Paris S (2024) Performance of large language artificial intelligence models on solving restorative dentistry and endodontics student assessments. Clin Oral Investig 28(11):575 Urda-Cîmpean AE et al (2025) Assessing the Accuracy of Diagnostic Capabilities of Large Language Models. Diagnostics 15(13):1657 Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9468440","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":626073306,"identity":"82bbc7bb-f6e4-4b62-92c4-4d5baea98280","order_by":0,"name":"Helmi Mostafa Abdaljabbar Khatib","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA6klEQVRIiWNgGAWjYBACA2bmBoYEhoQEBgbGNoYPQBE2doJaGKFa2BjbGGeAtDAT0sIA1MIA1gJUzAMSIqTFnJ2xdcMDhrQ8/vnNbY9tfm2T52NmYPzwMQe3FstmxrYbCQw5xRLHGNuNc/tuG7YxMzBLztyGx2GHwVoqEhuOMbZJ5/bcZgRqYWPmJUbLfJAWy57b9sRqyUncANLC8ON2IpFaDNISNx5LbDfsbbid3MbM2IzfL+cPH7v5oyI5cd7h488e/Phz23Z+e/PBDx/xaIFqhNLABAAiGwipRwZ/SFE8CkbBKBgFIwUAACf0VEehiFocAAAAAElFTkSuQmCC","orcid":"","institution":"An-Najah National University","correspondingAuthor":true,"prefix":"","firstName":"Helmi","middleName":"Mostafa Abdaljabbar","lastName":"Khatib","suffix":""}],"badges":[],"createdAt":"2026-04-20 07:37:46","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-9468440/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9468440/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":107400988,"identity":"627056b6-b69d-43c1-bffc-27703e22328e","added_by":"auto","created_at":"2026-04-21 07:37:38","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":38961,"visible":true,"origin":"","legend":"\u003cp\u003eAccuracy Distribution of LLM Responses Across Different Question Types. \u003cem\u003eMCQs\u003c/em\u003e Knowledge-Based Multiple-Choice Questions, \u003cem\u003eS-MCQs\u003c/em\u003e Scenario-Based Multiple-Choice Questions, \u003cem\u003eLLM\u003c/em\u003e Large Language Model.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-9468440/v1/d0a8c746a21b472bd5f1b022.png"},{"id":107489539,"identity":"93fae306-7b59-469d-9b4e-f88623dec248","added_by":"auto","created_at":"2026-04-22 02:48:04","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":531907,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9468440/v1/6268c86c-b229-42f2-bfa1-1d230ff2926b.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003e\u003cstrong\u003eAre Large Language Models Ready for Specialty-Level Periodontology? A Comparative Evaluation Across Question Types and Difficulty Strata\u003c/strong\u003e\u003c/p\u003e","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eArtificial intelligence (AI) has emerged as a transformative force in modern technology, aiming to develop systems capable of performing tasks that traditionally require human intelligence [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. In recent decades, the convergence of big data, increased computational power, and sophisticated AI algorithms has begun to permeate and simplify numerous aspects of daily life [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. Within the health sciences, this technological revolution is particularly pronounced, driving significant changes in medicine and dentistry. The potential of AI is becoming increasingly evident across a spectrum of applications, including diagnostic processes, treatment planning, and health professions education [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. By processing extensive datasets with remarkable speed, AI models can enable objective and comprehensive analyses\u0026mdash;from clinical findings to histopathological features\u0026mdash;that hold the potential to refine treatment methodologies and improve prognostic outcomes [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eLarge Language Models (LLMs) represent a specific and rapidly advancing subset of generative AI. These sophisticated systems simulate human language processing through deep learning and neural networks, trained on vast text corpora comprising books, scientific articles, and websites [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. In the medical and dental fields, LLMs are being explored for a diverse range of tasks, including clinical decision support, patient education, telemedicine, administrative workflow optimization, and the enhancement of educational processes [\u003cspan additionalcitationids=\"CR7 CR8\" citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. More recent studies have specifically evaluated their accuracy and consistency in answering domain-specific questions in oral pathology, endodontics and pediatric dentistry, revealing both promising capabilities and notable limitations in reliability and depth of reasoning [\u003cspan additionalcitationids=\"CR11 CR12 CR13\" citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe field of periodontology, which focuses on the diagnosis and treatment of chronic inflammatory diseases like periodontitis, stands to benefit significantly from such advancements. Periodontitis, if not managed effectively, leads to the destruction of the tooth-supporting apparatus and is a major cause of tooth loss [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. Its profound links to systemic conditions such as cardiovascular disease and diabetes further underscore the critical importance of early and accurate diagnosis and treatment planning [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e].However, diagnosis often relies heavily on a clinician's expertise and subjective interpretation of clinical and radiographic findings, introducing potential variability [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. This variability highlights an area where artificial intelligence (AI) could provide meaningful support.\u003c/p\u003e \u003cp\u003eIn this context, AI technologies\u0026mdash;particularly LLMs\u0026mdash;could serve as powerful adjuncts to clinicians by analyzing patient histories, synthesizing complex clinical information, proposing differential diagnoses, and reinforcing evidence-based decision-making [\u003cspan additionalcitationids=\"CR19\" citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eDespite this promise, the integration of AI in dentistry remains in its nascent stages. While many AI models demonstrate high precision in controlled, experimental settings, they face significant challenges in terms of clinical accuracy, reliability, and generalizability [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e, \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eHowever, there is very limited evidence, and in some contexts virtually none, regarding the performance of LLMs on multiple-choice questions specifically focused on periodontics, particularly when assessed using clinically nuanced, case-based scenarios.\u003c/p\u003e \u003cp\u003eThis study was designed to address this critical gap. Our primary aim was to evaluate and compare the performance of several freely accessible large language models\u0026mdash;including GPT, Gemini, DeepSeek, and Copilot\u0026mdash;in answering a set of 100 multiple-choice questions in periodontology. The question set was carefully curated and stratified by difficulty (easy, medium-hard) by expert dentists, comprising 50 general knowledge questions and 50 complex clinical scenarios. This difficulty-stratified, comparative approach. This difficulty-based design enables a clear comparison of model performance across varying levels of question complexity, providing practical insight into their current capabilities in periodontology.\u003c/p\u003e"},{"header":"2. Methodology","content":"\u003cp\u003eThis study evaluated the diagnostic accuracy of four large language models (LLMs) in answering expert-level multiple-choice questions (MCQs) in periodontics. Two independent question sets, comprising a total of 100 MCQs, were used for the assessment.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.1 Question Sources and Preparation\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTwo distinct MCQ datasets were utilized:\u003c/p\u003e\n\u003cp\u003e1.\u0026nbsp; \u0026nbsp;\u0026nbsp;\u003cstrong\u003eKnowledge-based MCQs (n = 50)\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e2.\u0026nbsp; \u0026nbsp;\u0026nbsp;\u003cstrong\u003eScenario-based MCQs (S-MCQs) (n = 50)\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe content domains were guided by widely used periodontal references and board-style examination resources. To ensure transparency and address copyright considerations, all questions were independently paraphrased, structurally modified, and revalidated by the authors. A representative subset of the questions and answers is provided in the Appendix to support reproducibility.\u003c/p\u003e\n\u003cp\u003eAll questions were designed for advanced dental learners and clinicians, reflecting expert-level knowledge and clinical reasoning in periodontology.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.2 Expert Review and Difficulty Classification\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTo ensure methodological rigor, three experienced dentists, independent of the study author, reviewed all questions in a blinded manner. Each evaluator verified the correctness of the answer options to ensure content accuracy.\u003c/p\u003e\n\u003cp\u003eFollowing independent verification, the evaluators convened to discuss any discrepancies and reach consensus through structured deliberation. Only after agreement on answer validity was achieved were the questions classified according to difficulty level. Items were subsequently categorized into two difficulty groups based on collective expert judgment:\u003c/p\u003e\n\u003cp\u003e\u0026middot; \u003cstrong\u003eEasy (n = 25 per set)\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u0026middot; \u003cstrong\u003eModerate\u0026ndash;Difficult (n = 25 per set)\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe final classification was determined through consensus agreement among the three evaluators to minimize subjective bias and enhance reliability.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.3 Large Language Models Evaluated\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe evaluation was conducted using the most recent publicly available free versions of each model at the time of testing (2026), accessed through their official web interfaces under default settings (no parameter modification):\u003c/p\u003e\n\u003cp\u003e\u0026middot; \u003cstrong\u003eGPT:\u003c/strong\u003e GPT-4o (free-tier version available via the ChatGPT platform by OpenAI)\u003c/p\u003e\n\u003cp\u003e\u0026middot; \u003cstrong\u003eGemini:\u003c/strong\u003e Gemini 1.5 Flash provided by Google\u003c/p\u003e\n\u003cp\u003e\u0026middot; \u003cstrong\u003eDeepSeek:\u003c/strong\u003e DeepSeek-V3 by DeepSeek\u003c/p\u003e\n\u003cp\u003e\u0026middot; \u003cstrong\u003eCopilot:\u003c/strong\u003e Microsoft Copilot powered by GPT-4o and provided by Microsoft\u003c/p\u003e\n\u003cp\u003eAll models were tested under default inference configurations without temperature adjustment or system-level customization.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.4 Prompting Strategy\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe prompting framework followed a principled instruction design approach to ensure consistency across models.\u003c/p\u003e\n\u003cp\u003eEach model received the following standardized prompt. This approach was implemented in accordance with previously published guidelines on principled prompting for large language models [23], which emphasize that carefully structured instructions enhance response consistency and reliability.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eYour task is to answer the following multiple-choice question. You must respond with only the letter of the correct option (A, B, C, or D). Please number your answer according to the question order. Do not provide explanations, justifications, or reasoning. The intended audience is domain experts in dentistry.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eEach 50-question set was submitted as a \u003cstrong\u003esingle batch prompt\u003c/strong\u003e, rather than sequential single-question entries. This approach was adopted to:\u003c/p\u003e\n\u003cp\u003e1.\u0026nbsp; \u0026nbsp;\u0026nbsp;Maintain identical contextual conditions across items.\u003c/p\u003e\n\u003cp\u003e2.\u0026nbsp; \u0026nbsp;\u0026nbsp;Avoid inter-session variability.\u003c/p\u003e\n\u003cp\u003e3.\u0026nbsp; \u0026nbsp;\u0026nbsp;Minimize potential memory carryover bias.\u003c/p\u003e\n\u003cp\u003e4.\u0026nbsp; \u0026nbsp;\u0026nbsp;Ensure standardized evaluation across models.\u003c/p\u003e\n\u003cp\u003eNo follow-up questions, clarifications, or feedback were provided after model responses in order to prevent iterative learning bias.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.5 Outcome Measures\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eModel responses were classified as \u003cstrong\u003ecorrect\u003c/strong\u003e or \u003cstrong\u003eincorrect\u003c/strong\u003e based on the verified official answer keys.\u003c/p\u003e\n\u003cp\u003ePrimary outcome measures included:\u003c/p\u003e\n\u003cp\u003e\u0026middot; Accuracy (%) for Easy questions\u003c/p\u003e\n\u003cp\u003e\u0026middot; Accuracy (%) for Moderate\u0026ndash;Difficult questions\u003c/p\u003e\n\u003cp\u003e\u0026middot; Overall accuracy (%)\u003c/p\u003e\n\u003cp\u003e\u0026middot; performance change\u003cspan dir=\"RTL\"\u003e\u0026nbsp;\u003c/span\u003e(%)\u003c/p\u003e\n\u003cp\u003e\u0026middot; Performance Drop (%)\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eStatistical significance between difficulty levels was assessed using \u003cstrong\u003ePearson\u0026rsquo;s Chi-square test\u003c/strong\u003e, with significance set at \u003cem\u003ep\u003c/em\u003e \u0026lt; 0.05.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.6 Appendix A\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAn Appendix accompanies this manuscript, containing the full set of questions with their correct answers, to facilitate transparency, reproducibility, and verification of the study procedures.\u003c/p\u003e"},{"header":"3. Results","content":"\u003cp\u003eA total of 100 multiple-choice questions were evaluated, including 50 knowledge-based MCQs and 50 scenario-based MCQs (S-MCQs). Model performance varied across question types and difficulty levels as shown in Fig. 1 and detailed in Table 1.\u003c/p\u003e\n\u003cp\u003eIn knowledge-based MCQs, overall accuracy ranged from 54% to 60%, with Gemini achieving the highest accuracy (60%), followed by GPT, DeepSeek, and Copilot (54% each). No statistically significant differences were observed between models (\u0026chi;\u0026sup2; = 0.36, p = 0.95).\u003c/p\u003e\n\u003cp\u003eIn scenario-based MCQs, accuracy improved across all models, ranging from 72% to 86%. GPT demonstrated the highest accuracy (86%), followed by Gemini (82%), DeepSeek (76%), and Copilot (72%). However, inter-model differences did not reach statistical significance (\u0026chi;\u0026sup2; = 2.77, p = 0.43). When total performance across all 100 questions was analyzed, overall accuracy ranged between 63% and 71%, with Gemini achieving the highest overall accuracy (71%), followed by GPT (70%), DeepSeek (65%), and Copilot (63%). These differences were not statistically significant (\u0026chi;\u0026sup2; = 0.99, p = 0.80).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable1:\u003c/strong\u003e Comparison of the accuracy rates of LLM responses based on different question type\u003c/p\u003e\n\u003ctable border=\"0\" cellspacing=\"3\" cellpadding=\"0\"\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eMCQs (Knowledge-Based)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eTrue n (%)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eFalse n (%)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eTest Statistics\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003ep*\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eGPT\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e27 (54%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e23 (46%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eGemini\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e30 (60%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e20 (40%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eDeepSeek\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e27 (54%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e23 (46%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eCopilot\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e27 (54%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e23 (46%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003e0.36\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003e0.95\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eS-MCQs (Scenario-Based)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eGPT\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e43 (86%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e7 (14%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eGemini\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e41 (82%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e9 (18%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eDeepSeek\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e38 (76%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e12 (24%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eCopilot\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e36 (72%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e14 (28%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003e2.77\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003e0.43\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eTotal Questions (n = 100)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eGPT\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e70 (70%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e30 (30%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eGemini\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e71 (71%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e29 (29%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eDeepSeek\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e65 (65%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e35 (35%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eCopilot\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e63 (63%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e37 (37%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003e0.99\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003e0.80\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eTable 2 summarizes the performance changes of the LLMs across question types. All models demonstrated higher accuracy on scenario-based MCQs (S-MCQs) compared to standard MCQs, with GPT showing the largest improvement (+59.3%), followed by DeepSeek (+40.7%), Gemini (+36.7%), and Copilot (+33.3%). The \u0026chi;\u0026sup2; analysis indicated statistically significant performance gains for GPT (p \u0026lt; 0.001), Gemini (p = 0.014), and DeepSeek (p = 0.022), whereas the increase for Copilot did not reach significance (p = 0.058). Effect sizes (\u0026Phi;) ranged from 0.19 to 0.35, reflecting small to moderate associations between question type and model accuracy.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable2:\u0026nbsp;\u003c/strong\u003eComparison of performance change experienced by LLMs across question type\u003c/p\u003e\n\u003ctable border=\"0\" cellspacing=\"3\" cellpadding=\"0\"\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eModel\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eMCQs True n (%)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eS-MCQs True n (%)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003ePercent Change\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eTest Statistics (\u0026chi;\u0026sup2;)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003ep\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eES (\u0026Phi;)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003e95% CI for \u0026Phi;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eGPT\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e27 (54%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e43 (86%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e+59.3%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e12.24\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026lt;0.001\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.35\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e[0.16\u0026ndash;0.51]\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eFalse\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e23 (46%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e7 (14%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eGemini\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e30 (60%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e41 (82%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e+36.7%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e5.98\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.014\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.24\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e[0.05\u0026ndash;0.41]\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eFalse\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e20 (40%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e9 (18%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eDeepSeek\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e27 (54%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e38 (76%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e+40.7%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e5.21\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.022\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.23\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e[0.04\u0026ndash;0.40]\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eFalse\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e23 (46%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e12 (24%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eCopilot\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e27 (54%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e36 (72%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e+33.3%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e3.59\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.058\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.19\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e[0.00\u0026ndash;0.37]\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eFalse\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e23 (46%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e14 (28%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003ePerformance analysis according to question difficulty (easy vs. moderate\u0026ndash;difficult) demonstrated a decline in accuracy with increasing complexity in most models (Table 3). Statistically significant reductions were observed for Gemini (\u0026chi;\u0026sup2; = 5.88, p = 0.015) and Copilot (\u0026chi;\u0026sup2; = 5.26, p = 0.022), whereas GPT and DeepSeek did not show significant performance differences across difficulty levels (p \u0026gt; 0.05).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable3:\u003c/strong\u003e Performance of LLMs According to Question Difficulty\u003c/p\u003e\n\u003ctable border=\"0\" cellspacing=\"3\" cellpadding=\"0\" class=\"fr-table-selection-hover\"\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eModel\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eEasy n (%)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eModerate\u0026ndash;Difficult n (%)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003ePerformance Drop (%)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026chi;\u0026sup2;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003ep-value\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026Phi;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003e95% CI for \u0026Phi;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eGPT\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e38/50 (76%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e32/50 (64%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026minus;12%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.74\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.19\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.13\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e[0.00\u0026ndash;0.33]\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eGemini\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e41/50 (82%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e30/50 (60%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026minus;22%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e5.88\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.015\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.24\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e[0.05\u0026ndash;0.42]\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eDeepSeek\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e34/50 (68%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e30/50 (60%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026minus;8%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.69\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.41\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.08\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e[0.00\u0026ndash;0.27]\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eCopilot\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e37/50 (74%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e26/50 (52%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026minus;22%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e5.26\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.022\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.23\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e[0.04\u0026ndash;0.40]\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eOverall, while numerical differences in accuracy were observed across models and question types, most inter-model comparisons did not demonstrate statistical significance.\u003c/p\u003e"},{"header":"4. Discussion","content":"\u003cp\u003eThis study provides a difficulty-stratified, comparative evaluation of four prominent large language models\u0026mdash;GPT, Gemini, DeepSeek, and Copilot\u0026mdash;on a curated set of 100 periodontology multiple-choice questions. The principal findings reveal a nuanced landscape of AI readiness for this specialty. While no single model demonstrated statistically significant overall superiority, their performance varied markedly across question types and difficulty levels. A key observation was the consistent and, for some models, statistically significant improvement in accuracy when addressing complex, scenario-based clinical questions compared to basic knowledge recall. However, this capability was tempered by a general decline in performance as question difficulty increased, a trend that reached statistical significance for two of the four models.\u0026nbsp;These findings suggest that, although LLMs demonstrate baseline competency in periodontology, their readiness for unsupervised academic or clinical use remains limited and task-dependent. While they may serve as useful adjuncts, they cannot replace the nuanced, experience-based judgment of a trained periodontist.\u003c/p\u003e\n\u003cp\u003eThe overall accuracy rates observed in this study, ranging from 63% to 71%, align with the broader body of literature evaluating LLMs in dental and medical education. Previous research has reported a wide spectrum of accuracy, from 42.5% to over 86%, depending on the model, question format, and subject domain [24-27]. The performance of the models in our study falls comfortably within this range, reinforcing the notion that LLMs have a baseline capability in specialized biomedical fields. The lack of a statistically significant difference between models in overall performance, as well as within the knowledge-based and scenario-based subsets, suggests a convergence in the capabilities of leading freely accessible LLMs for general periodontology knowledge. This finding is noteworthy, as it implies that for broad, curriculum-based queries, the choice of model may be less critical than other factors, such as accessibility or specific features. This contrasts with some studies that have found a clear numerical, and sometimes statistical, advantage for models like ChatGPT-4o [28], though our study did not include this specific variant.\u003c/p\u003e\n\u003cp\u003eA particularly noteworthy finding of this study was the consistent and, for three of the four models, statistically significant improvement in accuracy when transitioning from basic knowledge MCQs to scenario-based MCQs (S-MCQs). GPT demonstrated the most substantial gain, with a 59.3% increase in accuracy (from 54% to 86%), followed by DeepSeek (+40.7%, p = 0.022) and Gemini (+36.7%, p = 0.014). The performance gain for Copilot (+33.3%) approached but did not reach statistical significance (p = 0.058).\u003c/p\u003e\n\u003cp\u003eThis finding appears to contrast with a recent comprehensive evaluation by Urda-C\u0026icirc;mpean et al.[29], who reported that four leading LLMs performed better on knowledge-based questions than on clinical reasoning tasks. However, the superior performance of LLMs on scenario-based questions in our study likely reflects methodological differences\u0026mdash;particularly in domain structure, standardized question design, and lower contextual ambiguity\u0026mdash;rather than a true contradiction of prior findings.\u003c/p\u003e\n\u003cp\u003eThe difficulty-stratified analysis, a core strength of this study, revealed that this clinical reasoning capacity has its limits. All models exhibited a decline in accuracy when moving from easy to moderate-difficult questions, with the performance drop being statistically significant for Gemini and Copilot. The fact that GPT and DeepSeek maintained their performance levels across difficulty tiers, without a statistically significant drop, may hint at more robust reasoning architectures or fine-tuning strategies that better equip them to handle cognitive complexity.\u003c/p\u003e\n\u003cp\u003eThis study has several limitations that warrant consideration. First, the question set, while expertly curated and stratified, may not fully represent the breadth and depth of the periodontology field. Second, the analysis was limited to text-based queries and did not incorporate radiographic images or other visual data, which are integral to periodontal diagnosis and treatment planning. Third, the proprietary nature of the models means their training data and internal parameters are opaque. The observed performance differences could be attributed to factors like training data cutoff dates, which are beyond our control and can change without notice. Finally, as with any study using a specific snapshot of AI models, the rapid pace of development means that these results are a point-in-time assessment, and future iterations may perform differently.\u003c/p\u003e\n\u003cp\u003eIn conclusion, this study demonstrates that current large language models possess a functional, though not flawless, grasp of periodontology. Their ability to reason through clinical scenarios is particularly promising, yet their susceptibility to error on more complex and knowledge-based questions necessitates a cautious approach to their integration. For educators and clinicians, the take-home message is clear: LLMs are powerful assistants, but they are not autonomous experts. Their role should be to augment, not supplant, the critical thinking and expertise of the periodontist.\u003c/p\u003e"},{"header":"5. Conclusion","content":"\u003cp\u003eThis study provides a difficulty-stratified evaluation of four contemporary large language models\u0026mdash;GPT, Gemini, DeepSeek, and Copilot\u0026mdash;in the domain of periodontology, demonstrating moderate overall accuracy and context-dependent performance. Notably, the improved outcomes observed in scenario-based questions suggest a developing capacity for structured clinical reasoning, particularly within well-defined and standardized contexts.\u003c/p\u003e\n\u003cp\u003eHowever, the decline in performance with increasing question difficulty, alongside the absence of statistically significant superiority among models, underscores important limitations in reliability and higher-order reasoning. These findings indicate that, while LLMs hold potential as supportive tools in dental education and clinical decision-making, their current performance remains insufficient for independent application.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eDeclaration of generative AI and AI-assisted technologies in the manuscript preparation process\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eDuring the preparation of this work, the authors used ChatGPT to assist in language refinement and academic phrasing. All content was critically reviewed, edited, and validated by the authors, who take full responsibility for the final manuscript.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eAlowais SA et al (2023) Revolutionizing healthcare: the role of artificial intelligence in clinical practice. BMC Med Educ 23(1):689\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDing H et al (2023) Artificial intelligence in dentistry\u0026mdash;A review. Front Dent Med 4:1085251\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBindra S, Jain R (2024) Artificial intelligence in medical science: a review. Ir J Med Sci (1971-) 193(3):1419\u0026ndash;1429\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAra\u0026uacute;jo ALD et al (2023) Machine learning concepts applied to oral pathology and oral medicine: a convolutional neural networks' approach. J Oral Pathol Med 52(2):109\u0026ndash;118\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKumar P (2024) Large language models (LLMs): survey, technical frameworks, and future challenges. Artif Intell Rev 57(10):260\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKung TH et al (2023) Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLoS Digit health 2(2):e0000198\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eEggmann F et al (2023) Implications of large language models such as ChatGPT for dental medicine. J Esthetic Restor Dentistry 35(7):1098\u0026ndash;1102\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePuleio F et al (2024) Clinical, research, and educational applications of ChatGPT in dentistry: a narrative review. Appl Sci 14(23):10802\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAminoshariae A, Kulild J, Nagendrababu V (2021) Artificial intelligence in endodontics: current applications and future directions. J Endod 47(9):1352\u0026ndash;1357\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYilmaz BE, Gokkurt BN, Yilmaz, Ozbey F (2025) Artificial intelligence performance in answering multiple-choice oral pathology questions: a comparative analysis. BMC Oral Health 25(1):573\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSu\u0026aacute;rez A et al (2024) Unveiling the ChatGPT phenomenon: evaluating the consistency and accuracy of endodontic question answers. Int Endod J 57(1):108\u0026ndash;113\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRokhshad R et al (2024) Accuracy and consistency of chatbots versus clinicians for answering pediatric dentistry questions: A pilot study. J Dent 144:104938\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJung YS et al (2024) Evaluating the accuracy of artificial intelligence-based chatbots on pediatric dentistry questions in the Korean national dental board exam. J Korean Acad Pediatr Dentistry 51(3):299\u0026ndash;309\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003e\u0026Ccedil;eki̇\u0026ccedil; EC, Tavşan O (2025) Evaluating large language models using national endodontic specialty examination questions: are they ready for real-world dentistry? BMC Med Educ 25(1):1308\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDentino A et al (2000) \u003cem\u003ePrinciples of periodontology.\u003c/em\u003e Periodontology., 2013. 61(1): pp. 16\u0026ndash;53\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBourgeois D et al (2019) Periodontal pathogens as risk factors of cardiovascular diseases, diabetes, rheumatoid arthritis, cancer, and chronic obstructive pulmonary disease\u0026mdash;Is there cause for consideration? Microorganisms 7(10):424\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePetersson AR et al (1984) Observer variations in the interpretation of periapical osseous structures: A comparison between xeroradiography and conventional radiography. J Endod 10(5):205\u0026ndash;209\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFarhadi Nia M, Ahmadi M, Irankhah E (2025) Transforming dental diagnostics with artificial intelligence: advanced integration of ChatGPT and large language models for patient care. Front Dent Med 5:1456208\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003e\u0026Ouml;zbay Y, Erdoğan D, Din\u0026ccedil;er GA (2025) Evaluation of the performance of large language models in clinical decision-making in endodontics. BMC Oral Health 25(1):648\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDashti M et al (2024) Performance of ChatGPT 3.5 and 4 on US dental examinations: the INBDE, ADAT, and DAT. Imaging Sci dentistry 54(3):271\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKatne T et al (2019) Artificial intelligence: demystifying dentistry\u0026ndash;the future and beyond. Int J Contemp Med Surg Radiol 4(4):D6\u0026ndash;D9\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBanerjee M et al (2024) The Prospect of Artificial Intelligence in Dentistry. Med Res Its Appl Vol 6:136\u0026ndash;146\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBsharat SM, Myrzakhan A, Shen Z (2023) \u003cem\u003ePrincipled instructions are all you need for questioning llama-1/2, gpt-3.5/4.\u003c/em\u003e arXiv preprint arXiv:2312.16171\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDanesh A et al (2023) The performance of artificial intelligence language models in board-style dental knowledge assessment: A preliminary study on ChatGPT. J Am Dent Assoc 154(11):970\u0026ndash;974\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eQuah B et al (2024) Performance of large language models in oral and maxillofacial surgery examinations. Int J Oral Maxillofac Surg 53(10):881\u0026ndash;886\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChau RCW et al (2024) Performance of Generative Artificial Intelligence in Dental Licensing Examinations. Int Dent J 74(3):616\u0026ndash;621\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTassoker M (2025) ChatGPT-4 Omni's superiority in answering multiple-choice oral radiology questions. BMC Oral Health 25(1):173\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eK\u0026uuml;nzle P, Paris S (2024) Performance of large language artificial intelligence models on solving restorative dentistry and endodontics student assessments. Clin Oral Investig 28(11):575\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eUrda-C\u0026icirc;mpean AE et al (2025) Assessing the Accuracy of Diagnostic Capabilities of Large Language Models. Diagnostics 15(13):1657\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Artificial intelligence, Large language models, Periodontology, Dental education, Clinical reasoning, Multiple-choice questions","lastPublishedDoi":"10.21203/rs.3.rs-9468440/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9468440/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e \u003cp\u003eArtificial intelligence (AI), particularly large language models (LLMs), has emerged as a promising tool in healthcare, with potential applications in clinical decision support and dental education. Despite increasing interest, evidence regarding the performance of LLMs in periodontology\u0026mdash;especially in clinically oriented, scenario-based assessments\u0026mdash;remains limited. This study aimed to evaluate and compare the accuracy of multiple LLMs in answering knowledge-based and scenario-based multiple-choice questions (MCQs) in periodontology across different difficulty levels.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eA total of 100 periodontology MCQs were selected from validated academic sources and divided into two categories: knowledge-based questions (n\u0026thinsp;=\u0026thinsp;50) and scenario-based questions (n\u0026thinsp;=\u0026thinsp;50). Each category was further stratified into easy and moderate\u0026ndash;difficult levels (25 questions each) based on expert consensus. Four publicly available LLMs (GPT-4o, Gemini 1.5 Flash, DeepSeek-V3, and Microsoft Copilot) were evaluated using a standardized prompting framework. Model responses were assessed for accuracy against verified answer keys. Statistical analysis was performed using Pearson\u0026rsquo;s Chi-square test, with significance set at p\u0026thinsp;\u0026lt;\u0026thinsp;0.05.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eOverall accuracy ranged from 63% to 71%, with Gemini achieving the highest overall performance (71%), followed by GPT (70%), DeepSeek (65%), and Copilot (63%), without statistically significant differences (p\u0026thinsp;=\u0026thinsp;0.80). All models demonstrated higher accuracy in scenario-based MCQs compared to knowledge-based questions, with statistically significant improvements observed for GPT (p\u0026thinsp;\u0026lt;\u0026thinsp;0.001), Gemini (p\u0026thinsp;=\u0026thinsp;0.014), and DeepSeek (p\u0026thinsp;=\u0026thinsp;0.022). Accuracy decreased with increasing question difficulty, with significant performance declines observed for Gemini (p\u0026thinsp;=\u0026thinsp;0.015) and Copilot (p\u0026thinsp;=\u0026thinsp;0.022), while GPT and DeepSeek showed more stable performance.\u003c/p\u003e\u003ch2\u003eConclusions\u003c/h2\u003e \u003cp\u003eLLMs demonstrate baseline competency in periodontology and show improved performance in context-rich, scenario-based questions. However, their accuracy remains variable and task-dependent, particularly under increasing difficulty. While these models may serve as useful adjuncts in dental education and clinical support, they are not yet reliable as standalone tools for clinical decision-making.\u003c/p\u003e","manuscriptTitle":"Are Large Language Models Ready for Specialty-Level Periodontology? A Comparative Evaluation Across Question Types and Difficulty Strata","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-21 07:37:35","doi":"10.21203/rs.3.rs-9468440/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"3fe49deb-436e-46f1-8900-92c35b9c7f7d","owner":[],"postedDate":"April 21st, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-04-21T07:37:35+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-21 07:37:35","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9468440","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9468440","identity":"rs-9468440","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}