{"paper_id":"3452be73-5f2d-4e78-b718-bba43249ecbd","body_text":"A Comparative Performance Analysis of AI-Assisted Language Models in Preoperative Patient Education for Mitral Valve Surgery | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A Comparative Performance Analysis of AI-Assisted Language Models in Preoperative Patient Education for Mitral Valve Surgery Banu Bahriye Akdağ, Mehmet Şenel Bademci, İhsan Peker, Okay Güven Karaca, and 6 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6965764/v1 This work is licensed under a CC BY 4.0 License Status: Under Revision Version 1 posted 12 You are reading this latest preprint version Abstract Background Currently, large language models (LLMs) supported by artificial intelligence (AI) are increasingly being utilized in patient education and information delivery within healthcare services. The aim of this study was to perform a comparative analysis of five different LLMs ( i.e. , ChatGPT-4o, Claude 3.7 Sonnet, Gemini 2.5 Pro Preview, DeepSeek-V3, and Microsoft Copilot) in terms of accuracy, completeness, and readability, based on their responses to frequently asked questions in preoperative patient education for mitral valve surgery (MVS). Methods A standardized questionnaire comprising seven frequently asked questions by patients prior to MVS was developed. These questions were presented to each LLM in an identical manner. The responses were evaluated by two academic experts in cardiac surgery using structured assessment criteria across three main dimensions: accuracy, completeness, and readability. For the readability analysis, the Simplified Measure of Gobbledygook (SMOG) Index and the Flesch-Kincaid Reading Ease (FRE) scale were utilized. Results The ChatGPT-4o and Gemini models received statistically significantly higher scores in terms of accuracy and completeness (p < 0.05), while the Claude 3.7 Sonnet model achieved the highest readability scores (p < 0.001). This model provided reader-friendly content using simpler and more comprehensible sentence structures. The Gemini and DeepSeek models demonstrated moderate performance, whereas the Microsoft Copilot model showed limitations in semantic coherence and medical specificity. Some models were found to provide misleading or incomplete information regarding surgical risks, the postoperative course, and potential complications. Conclusions The LLMs represent valuable supplementary tools in patient education processes. However, their implementation in clinical practice must be carefully evaluated, particularly with regard to accuracy and completeness. This study highlights the potential applicability of ChatGPT-4o and Claude models for preoperative patient education in MVS, while emphasizing that all LLMs should be used under the supervision and guidance of healthcare professionals. For LLMs to be reliably utilized in the medical field, improvement in medical accuracy and standardization are essential. Large Language Models Artificial Intelligence Mitral Valve Surgery Patient Education Accuracy Readability Figures Figure 1 Figure 2 1. Introduction Mitral valve diseases are among the most common valvular heart pathologies, leading to significant clinical outcomes. These conditions are classified into two main clinical forms: mitral stenosis and mitral regurgitation. Over time, they may impair left ventricular function and predispose patients to heart failure, atrial fibrillation, and thromboembolic events [ 1 ]. In symptomatic patients, medical therapy provides limited benefits, and surgical intervention often offers a more definitive solution. Currently, mitral valve repair is usually preferred over replacement, particularly in degenerative mitral regurgitation, owing to its certain advantages in preserving valve function, maintaining left ventricular geometry, and improving survival. Furthermore, advancements in surgical techniques and the widespread adoption of minimally invasive approaches have shortened postoperative recovery times and reduced complication rates [ 2 ]. The decision to undergo MVS involves not only anatomical and clinical factors, but also the patient’s preferences, lifestyle, expectations, and understanding of potential outcomes. Comprehensive patient education and information-sharing processes led by heart teams have been shown to improve patient engagement and positively influence satisfaction with care [ 2 ]. The success of MVS heavily relies on effective patient education, which is crucial for treatment adherence and complication reduction. Clearly communicating the differences between valve repair and replacement and, if replacement is necessary, between bioprosthetic and mechanical valves, as well as the potential risks and benefits, supports patients in actively participating in informed decision-making. Current guidelines recommend a patient-centered approach as a cornerstone of surgical success and emphasize the ethical and clinical importance of conducting the informed consent process with great care [ 1 , 2 ]. With advancing technology, large language models (LLMs) supported by artificial intelligence (AI) led initially by ChatGPT have emerged as tools to meet patients' informational needs regarding surgical treatments. In subsequent years, numerous other models have also entered widespread use, including ChatGPT-4o by OpenAI, Claude 3.7 Sonnet by Anthropic, Gemini 2.5 Pro Preview by Google, Microsoft Copilot, and DeepSeek-V3. These models can provide rapid, accessible, and readable basic-level information on topics such as preoperative procedures, surgical risks, alternative treatments, and the recovery process. Recent studies have shown that certain LLMs may deliver incomplete or misleading information regarding surgical risks and postoperative recovery [ 3 ]. Nonetheless, their ability to communicate information with high readability and in non-technical language represents a significant advantage, particularly for individuals with low health literacy. Currently, healthcare professionals view these technologies as supportive tools in patient education; however, it is strongly emphasized that final medical decisions must always be made by clinical experts. In recent years, AI-based health communication has increasingly become a complementary component of patient-centered care in modern medicine. Artificial intelligence-supported LLMs are also being utilized for candidates undergoing cardiac surgery [ 4 ]. In particularly complex procedures such as MVS, these models can facilitate rapid access to information, enhance physician–patient communication, and support patients’ psychological preparedness for the surgical process [ 5 ]. However, despite their growing use, there is limited research comparing the accuracy, completeness, and readability of the information these models provide specifically for MVS patient education. This gap makes it challenging to determine which models offer the most reliable and accessible information to support patient understanding and decision-making. In the present study, we, therefore, aimed to evaluate and compare the accuracy, completeness, and readability of information provided by different LLMs for patient education and counseling in patients for MVS. 2. Materials and Methods 2.1. Study design and study sample The seven open-ended questions used to collect the data were derived from a detailed operative process description, which was not individually designed to maintain objectivity. Instead, the content was obtained from the patient information section of The Society for Cardiothoracic Surgery in Great Britain & Ireland website ( https://scts.org/patients/heart/procedures/20/mitral_valve_surgery ). These questions were simultaneously presented in April 2025 to five different LLMs—ChatGPT-4o by OpenAI, Claude 3.7 Sonnet by Anthropic, Gemini 2.5 Pro Preview by Google, Microsoft Copilot, and DeepSeek-V3 and their responses were collected for evaluation. 2.2. Selection of LLM Language models which were freely or partially freely accessible to any user were selected for the study. Additionally, the selected models were chosen based on their popularity and accessibility within the healthcare sector. 2.2.1. Evaluation Process Two experts with at least 10 years of experience in MVS independently evaluated the data in terms of accuracy and completeness using a five-point Likert scale to maintain objectivity. For the accuracy evaluation, current guidelines on MVS and the clinical expertise of specialists in the field were used as primary reference sources [ 1 , 2 ]. After the initial independent evaluations, the evaluators met to reach a consensus and determine a final, unified score. To assess inter-rater reliability, the intraclass correlation coefficient (ICC) was calculated [ 6 ], and the result was found to be 0.85, indicating high reliability. 2.2.2. Evaluation Criteria Accuracy Evaluation : A five-point Likert scale was used to evaluate the accuracy of the information [ 7 ]: 5 points = Completely accurate, 4 points = More accurate than inaccurate, 3 points = Accuracy and inaccuracy are roughly equal, 2 points = More inaccurate than accurate, 1 point = Completely inaccurate. Completeness Evaluation : The completeness of the information was also evaluated using a five-point Likert scale [ 7 ]: 5 = Completely sufficient (Fully covers the topic/No missing information), 4 = Mostly sufficient (Covers the topic/Minor omissions), 3 = Moderately sufficient (Relevant to the topic/Some ambiguities), 2 = Mostly insufficient (Partially related to the topic/Significant omissions), 1 = Very insufficient (Unrelated to the topic/Overly basic or nonsensical). Readability Evaluation To evaluate the readability of the responses, the Simplified Measure of Gobbledygook (SMOG) Index [ 7 ] and the Flesch-Kincaid Reading Ease (FRE) scale were used [ 8 ]. 2.3. Statistical Analysis Statistical analysis was performed using the SPSS for Windows version 23.0 software (IBM Corp., Armonk, NY, USA). Comparisons of the LLMs in terms of accuracy and completeness were conducted using the Kruskal-Wallis test, followed by post-hoc Dunn-Bonferroni tests after detecting overall significance. The normality of the readability scores was checked using the Shapiro-Wilk test. Continuous variables were presented in mean ± standard deviation (SD) or median (min-max), while categorical variables were presented in number and frequency. The readability scores which met the normality assumption were compared using one-way analysis of variance (ANOVA). Subsequent subgroup analyses following ANOVA were performed using the Bonferroni corrections. A p value of < 0.05 was considered statistically significant. 3. Results The median accuracy scores of the LLMs were 5 (range, 5 to 5) for ChatGPT 4o, 4 (range, 4 to 4) for Claude 3.7 Sonnet, 5 (range, 4 to 5) for DeepSeek-V3, 5 (range, 5 to 5) for Gemini 2.5 Pro Preview, and 4 (range, 4 to 4) for Microsoft Copilot. A statistically significant difference was found among the LLMs in terms of accuracy scores (p < 0.001).(Table 1 ) In the post-hoc analyses, the accuracy level of ChatGPT 4o was found to be significantly higher than that of Claude 3.7 Sonnet (5 & 4; p = 0.002) and Microsoft Copilot (5 & 4; p = 0.002). Similarly, Gemini 2.5 Pro Preview had a higher accuracy level compared to Claude (5 & 4; p = 0.002) and Microsoft Copilot (5 & 4; p = 0.002). Subgroup analyses revealed no statistically significant differences among the LLMs (p > 0.05).(Fig. 1 ) Table 1 Comparative Performance of Large Language Models in Accuracy, Completeness, and Readability Metrics LLM Model Accuracy Completeness SMOG FRE ChatGPT 4o 5 (5–5) 4 (4–4) 12.24 ± 0.69 9.04 0.55 Claude 3.7 Sonnet 4 (4–4) 3 (3–3) 10.90 ± 0.54 8 ± 0.41 DeepSeek-V3 5 (4–5) 4 (3–4) 12.61 ± 0.72 9.37 ± 0.53 Gemini 2.5 Pro Preview 5 (5–5) 5 (5–5) 12.41 ± 0.64 9.17 0.48 Microsoft Copilot 4 (4–4) 3 (3–4) 11.84 ± 0.66 8.77 ± 0.49 p-value < 0.001 < 0.001 < 0.001 < 0.001 Data are given in mean ± SD or median (min-max), unless otherwise stated. LLM: large language model; SMOG: Simplified Measure of Gobbledygook; FRE: Flesch-Kincaid Reading Ease. The median completeness scores of the LLMs were 4 (range, 4 to 4) for ChatGPT 4o, 3 (range, 3 to 3) for Claude 3.7 Sonnet, 4 (range, 3 to 4) for DeepSeek-V3, 5 (range, 5 to 5) for Gemini 2.5 Pro Preview, and 3 (range, 3 to 4) for Microsoft Copilot.(Table 1 ) There was a statistically significant difference among the LLMs regarding completeness scores (p < 0.001). In the post-hoc analyses, the completeness score of Gemini 2.5 Pro Preview was found to be significantly higher than that of Claude 3.7 Sonnet (5 & 3; p = 0.000) and Microsoft Copilot (5 & 3; p < 0.001). Subgroup analyses revealed no statistically significant differences among the LLMs (p > 0.05).(Fig. 1 ) Considering the SMOG scores of the LLMs, the mean score was found to be 12.24 ± 0.69 for ChatGPT 4o, 10.90 ± 0.54 for Claude 3.7 Sonnet, 12.61 ± 0.72 for DeepSeek-V3, 12.41 ± 0.64 for Gemini 2.5 Pro Preview, and 11.84 ± 0.66 for Microsoft Copilot, indicating a statistically significant difference among the LLMs (p < 0.001).(Table 1 ) In the post-hoc analyses, the mean SMOG score of Claude 3.7 Sonnet was found to be significantly lower than that of ChatGPT 4o (10.90 & 12.24; p = 0.006), DeepSeek-V3 (10.90 & 12.61; p < 0.001), and Gemini 2.5 Pro Preview (10.90 & 12.41; p = 0.001). Subgroup analyses revealed no statistically significant differences among the LLMs (p > 0.05).(Fig. 2 ) The mean FRE scores of the LLMs were 9.04 ± 0.55 for ChatGPT 4o, 8.00 ± 0.41 for Claude 3.7 Sonnet, 9.37 ± 0.53 for DeepSeek-V3, 9.17 ± 0.48 for Gemini 2.5 Pro Preview, and 8.77 ± 0.49 for Microsoft Copilot. There was a statistically significant difference in the FRE scores among the LLMs (p < 0.001). In the post-hoc analyses, the mean FRE score of Claude 3.7 Sonnet was found to be significantly lower than that of ChatGPT 4o (8 & 9.04; p = 0.004), DeepSeek-V3 (8 & 9.37; p < 0.001), and Gemini 2.5 Pro Preview (8 & 9.17; p = 0.001). Subgroup analyses revealed no statistically significant differences among the LLMs (p > 0.05).(Fig. 2 ) 4. Discussion Despite the increasing integration of LLMs into healthcare communication, particularly for patient education, there remains a significant gap in comparative evaluations of these tools in the context of complex surgical procedures. Most existing studies have focused on the performance of individual LLMs or their application in general health information dissemination. However, comprehensive analyses assessing the accuracy, completeness, and readability of multiple, state-of-the-art LLMs in delivering patient-centered information for specific and high-risk interventions, such as MVS, are still limited. In the present study, we evaluated the accuracy, completeness, and readability levels of five LLMs ( i.e. , ChatGPT 4o, Claude 3.7 Sonnet, DeepSeek-V3, Gemini 2.5 Pro Preview, and Microsoft Copilot), used for patient education and information regarding MVS. Notably, the ChatGPT 4o and Gemini 2.5 Pro Preview models were found to be statistically significantly superior to Claude 3.7 Sonnet and Microsoft Copilot in terms of both accuracy and completeness. Considering the readability of responses generated by the LLMs, the Claude model yielded statistically significantly more readable responses compared to ChatGPT 4o, DeepSeek-V3, and Gemini 2.5 Pro Preview. In recent years, there has been a growing number of studies examining the accuracy of data generated by LLMs, particularly in the context of patient education. Lee et al. [ 9 ] showed that the ChatGPT model yielded higher accuracy in their study on atrial fibrillation patient education. Similarly, other studies have demonstrated that the ChatGPT model provides clinically accurate information at a high level [ 10 ]. In a related study by Zhao et al. [ 11 ], which focused on developing educational materials for patients with lower back pain, the Gemini 2.5 Pro Preview model achieved similarly high levels of accuracy and completeness as ChatGPT 4o. The superior performance of the ChatGPT 4o and Gemini 2.5 Pro Preview models in terms of accuracy and completeness can primarily be attributed to their use of Retrieval-Augmented Generation (RAG) architecture [ 12 ]. The RAG-based models are capable of accessing up-to-date external sources before generating responses, thereby enabling the production of more accurate and comprehensive content [ 12 ]. In contrast, conventional LLMs only reflect information available in their training datasets and are usually limited in providing sources for the information they present. In their study evaluating anterior cruciate ligament (ACL) injuries, Woo et al. [ 13 ] showed that RAG-based LLMs significantly improved accuracy rates. In the aforementioned study, the implementation of RAG resulted in an average increase of 39.7% in model accuracy. The RAG framework has proven to be a valuable tool in enhancing the accuracy of patient education materials, offering a clear advantage over traditional LLMs [ 13 ]. One of the main concerns regarding the use of LLMs in patient education and information delivery is their potential to generate inaccurate, nonsensical, or faulty content based on outdated or irrelevant model training data [ 14 ]. Although this issue is less common in RAG-based LLMs, to further eliminate this risk in our study, all model outputs were evaluated by two experienced surgeons specialized in the field. Another critical aspect of the data obtained from LLMs is ensuring ease of readability and comprehension. Review of the recent literature reveals numerous studies presenting varying perspectives on this issue [ 15 , 16 ]. The coexistence of plain language and medical accuracy is of utmost importance for patient safety. In our study, the Claude model demonstrated the highest performance in terms of readability compared to the other models. In a study conducted by Zhang et al. [ 17 ] on inflammatory bowel disease, readability scores were found to be higher for the Claude 3.7 Sonnet model than for ChatGPT-4o and Gemini 2.5 Pro Preview. This finding highlights Claude's success in generating user-friendly and more comprehensible texts. This advantage stems from the model's ability to use shorter sentences and less complex vocabulary, which enhances the readability of the generated content, particularly in medical contexts where such language improves overall understanding [ 16 ]. Furthermore, the consistency and efficiency of Claude 3.7 Sonnet's responses also positively influenced its readability scores. As highlighted in the study by Jin et al. [ 18 ], fast and internally consistent responses help structure texts more effectively, contributing to better comprehension. Furthermore, there are also ethical concerns regarding the use of LLMs in patient education. Patient privacy and data security are of paramount importance in this context. Most LLMs utilize large amounts of data during their training processes, which increases the risk of misuse of patients' personal health information [ 19 ]. Additionally, there is a risk that patients may accept information generated by the model without questioning it (overtrust), potentially replacing professional medical advice. Therefore, the use of LLMs in patient education should be conducted under strict supervision, transparency, and ethical guidelines. 4.1. Clinical Implications By improving the clarity and accessibility of medical information, LLMs can contribute to increased health literacy, informed decision-making, and greater patient satisfaction. These models can be particularly beneficial in busy clinical environments where time constraints may limit detailed one-on-one education. However, integration of LLMs into routine practice should be accompanied by proper oversight to ensure the accuracy and appropriateness of content delivered to patients. Future implementation strategies should also consider patient-specific factors such as age, educational background, and digital literacy to optimize the impact of LLM-assisted education. Ultimately, these models may serve as valuable adjuncts in shared decision-making and personalized patient communication frameworks. 4.2. Limitations Although the independent and double-blind evaluation of the data by two expert surgeons strengthens the assessment method and analysis, this study has certain limitations that should be acknowledged. First, the responses were evaluated by two experts, while human judgment inherently carries some level of subjectivity. Although this was mitigated by using structured criteria, interrater reliability is still a potential concern. Second, the evaluation was conducted only on responses in English; model performances in other languages were unable to be assessed. Third, the examined topic was limited to MVS, and model performance may vary for other diseases; therefore, making a general characterization of LLMs may not always be fully accurate. In addition, the study analyzed specific versions of the LLMs at a single time point. As LLMs are updated frequently, their performance could change over time, potentially altering accuracy, completeness, and readability outcomes in future assessments. Fourth, cultural differences among patients using LLMs may exist, and behavioral outcomes such as patient experience and satisfaction were not included in the evaluation, which can be deemed as another limitation. Finally, although the questions were open-ended, the small number of questions may also limit the generalizability of the results. Differences in future versions of LLMs may also affect the outcomes. 5. Conclusion In conclusion, the contribution of AI-supported LLMs in everyday clinical practice for patient education and information is increasing day by day in the healthcare field. Our study results showed that ChatGPT 4.0 and Gemini 2.5 Pro Preview models had a higher level of performance in terms of accuracy and completeness regarding patient information related to MVS, while Claude 3.7 Sonnet performed better in terms of readability. Taken together, the use of LLMs enhances health literacy among patients and makes patient education resources more accessible and understandable. Future studies incorporating patient-centered evaluations and considering variables such as patients' age and educational status are warranted to draw more comprehensive data on this subject. Declarations Ethics approval and consent to participate This study did not involve human participants, human data, or human tissue. The research protocol was reviewed and approved by the Ethics Committee of University of Health Sciences Turkey, Izmir Faculty of Medicine, Izmir City Hospital (Approval Number: 2025/368, Date: July 23, 2025). The study was conducted in accordance with the ethical standards of the institutional research committee and with the Declaration of Helsinki (1964) and its subsequent amendments. Informed consent was obtained from all participants involved in the study. Consent for publication All the authors have given their consent for publication. Availability of data and materials The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions. Competing interests The authors declare that they have no competing interest. Funding This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. Authors’ contributions Methodology:B.B.A, M.S.B and O.G.K ; validation: I.P, C.K and B.O ; formal analysis: H.D, M.C ; investigation: I.O, H.O.S ; writing—original draft: B.B.A, M.S.B ; writing—review and editing: B.B.A, M.S.B and O.G.K. All authors read and approved the final manuscript. Acknowledgements Not applicable. Author details 1 Department of Cardiovascular Surgery, Izmir Faculty of Medicine , University of Health Sciences, Izmir City Hospital, 35530 Izmir, Turkey; [email protected] (B.B.A.), [email protected] (M.S.B.) 2 Department of Cardiovascular Surgery, Izmir City Hospital, 35530 Izmir, Turkey; [email protected] (O.G.K.), [email protected] (I.P.), [email protected] (C.K.) 3 Department of Cardiovascular Surgery, Faculty of Medicine, Near East University, Nicosia 99138, Cyprus; [email protected] (B.O.) 4 Department of Cardiovascular Surgery, Konya City Hospital, 42020 Konya, Turkey; [email protected] (H.D.) 5 Department of Family Medicine, Izmir City Hospital, 35530 Izmir, Turkey; [email protected] (M.C). 6 Department of Cardiovascular Surgery, Izmir Katip Çelebi University, Atatürk Training and Research Hospital, 35360 Izmir, Turkey; [email protected] (I.O.), [email protected] (H.O.S.) References Otto CM, Nishimura RA, Bonow RO, et al. 2020 ACC/AHA Guideline for the Management of Patients With Valvular Heart Disease: Executive Summary: A Report of the American College of Cardiology/American Heart Association Joint Committee on Clinical Practice Guidelines [published correction appears in Circulation. 2021 Feb 2;143(5):e228. doi: 10.1161/CIR.0000000000000960.] [published correction appears in Circulation. 2021 Mar 9;143(10):e784. Circulation. 2021;143(5):e35-e71. doi:10.1161/CIR.0000000000000932. Vahanian A, Beyersdorf F, Praz F, et al. 2021 ESC/EACTS Guidelines for the management of valvular heart disease: Developed by the Task Force for the management of valvular heart disease of the European Society of Cardiology (ESC) and the European Association for Cardio-Thoracic Surgery (EACTS). Rev Esp Cardiol (Engl Ed). 2022;75(6):524 Labkoff S, Oladimeji B, Kannry J, et al. Toward a responsible future: recommendations for AI-enabled clinical decision support. J Am Med Inform Assoc. 2024;31(11):2730-2739. Mehta R, Reitz JG, Venna A, et al. Navigating the future of pediatric cardiovascular surgery: Insights and innovation powered by Chat Generative Pre-Trained Transformer (ChatGPT). J Thorac Cardiovasc Surg. Published online February 1, 2025. Aydin S, Karabacak M, Vlachos V, Margetis K. Large language models in patient education: a scoping review of applications in medicine. Front Med (Lausanne). 2024;11:1477898. Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research [published correction appears in J Chiropr Med. 2017 Dec;16(4):346. doi: 10.1016/j.jcm.2017.10.001.]. J Chiropr Med. 2016;15(2):155-163 Zheng C, Ye H, Guo J, et al. Development and evaluation of a large language model of ophthalmology in Chinese. Br J Ophthalmol. 2024;108(10):1390-1397. Rahimli Ocakoglu S, Coskun B. The Emerging Role of AI in Patient Education: A Comparative Analysis of LLM Accuracy for Pelvic Organ Prolapse. Med Princ Pract. Published online March 25, 2024. Lee TJ, Campbell DJ, Rao AK, et al. Evaluating ChatGPT Responses on Atrial Fibrillation for Patient Education. Cureus. 2024;16(6):e61680. Rao A, Pang M, Kim J, et al. Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study. J Med Internet Res. 2023;25:e48659. Published 2023 Aug 22. Zhao, Yi-Fei, et al. \"Generative AI Is Not Ready for Clinical Use in Patient Education for Lower Back Pain Patients, Even With Retrieval-Augmented Generation.\" arXiv preprint arXiv:2409.15260 (2024). Ge J, Sun S, Owens J, et al. Development of a liver disease-specific large language model chat interface using retrieval-augmented generation. Hepatology. 2024;80(5):1158-1168. Woo JJ, Yang AJ, Olsen RJ, et al. Custom Large Language Models Improve Accuracy: Comparing Retrieval Augmented Generation and Artificial Intelligence Agents to Noncustom Models for Evidence-Based Medicine. Arthroscopy. 2025;41(3):565-573.e6. Azamfirei R, Kudchadkar SR, Fackler J. Large language models and the perils of their hallucinations. Crit Care. 2023;27(1):120. Published 2023 Mar 21. Hanci V, Ergün B, Gül S, Uzun Ö, Erdemir I, Hanci FB. Assessment of readability, reliability, and quality of ChatGPT®, BARD®, Gemini®, Copilot®, Perplexity® responses on palliative care. Medicine (Baltimore). 2024;103(33):e39305. Nian PP, Saleet J, Magruder M, et al. ChatGPT as a Source of Patient Information for Lumbar Spinal Fusion and Laminectomy: A Comparative Analysis Against Google Web Search. Clin Spine Surg. 2024;37(10):E394-E403. Zhang Y, Wan XH, Kong QZ, et al. Evaluating large language models as patient education tools for inflammatory bowel disease: A comparative study. World J Gastroenterol. 2025;31(6):102090. Jin H, Guo J, Lin Q, Wu S, Hu W, Li X. Comparative study of Claude 3.5-Sonnet and human physicians in generating discharge summaries for patients with renal insufficiency: assessment of efficiency, accuracy, and quality. Front Digit Health. 2024;6:1456911. Published 2024 Dec 5. Zhui L, Fenghe L, Xuehu W, Qining F, Wei R. Ethical Considerations and Fundamental Principles of Large Language Models in Medical Education: Viewpoint. J Med Internet Res. 2024;26:e60083. Additional Declarations No competing interests reported. Supplementary Files MitralValveSurgerySCTSquestions.pdf Cite Share Download PDF Status: Under Revision Version 1 posted Editorial decision: Revision requested 07 Jan, 2026 Reviewers agreed at journal 04 Jan, 2026 Reviews received at journal 03 Jan, 2026 Reviewers agreed at journal 03 Jan, 2026 Reviews received at journal 02 Jan, 2026 Reviewers agreed at journal 01 Jan, 2026 Reviewers agreed at journal 10 Sep, 2025 Reviewers invited by journal 29 Aug, 2025 Editor assigned by journal 28 Aug, 2025 Editor invited by journal 04 Aug, 2025 Submission checks completed at journal 01 Aug, 2025 First submitted to journal 01 Aug, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {\"props\":{\"pageProps\":{\"initialData\":{\"identity\":\"rs-6965764\",\"acceptedTermsAndConditions\":true,\"allowDirectSubmit\":false,\"archivedVersions\":[],\"articleType\":\"Research Article\",\"associatedPublications\":[],\"authors\":[{\"id\":509081170,\"identity\":\"19f6ec4c-7fe3-4183-9e8d-c85723ec49f0\",\"order_by\":0,\"name\":\"Banu Bahriye Akdağ\",\"email\":\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA2ElEQVRIiWNgGAWjYBACAzjrAPMBICkhQ5yWA2CCLQGkhYcULTxgNmEt5uxnj0l/bLsjz3f8zOdXN2oseBjYDx/dgE+LZU9emsTBtmeGM8/kbrPOOQZ0GE9a2g28DjuQYwbUcphxw4HcbcY5bEAtEjxm+LWcfwPWYr/h/Jtnxjn/iNFyA2JL4oYbOcyPc9uI0GI5442xxZlzh5Nn3nhmxpzbJ8HDRsgv5vw5hjcqyg7b9p1Pfvw551udHD/74WN4tSADNgkwSaxyEGD+QIrqUTAKRsEoGDkAAFuJTxaMTRWrAAAAAElFTkSuQmCC\",\"orcid\":\"\",\"institution\":\"Izmir Faculty of Medicine, University of Health Sciences, Izmir City Hospital\",\"correspondingAuthor\":true,\"prefix\":\"\",\"firstName\":\"Banu\",\"middleName\":\"Bahriye\",\"lastName\":\"Akdağ\",\"suffix\":\"\"},{\"id\":509081171,\"identity\":\"662464e5-9177-496b-98bc-6e4762495dc7\",\"order_by\":1,\"name\":\"Mehmet Şenel Bademci\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Izmir Faculty of Medicine, University of Health Sciences, Izmir City Hospital\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Mehmet\",\"middleName\":\"Şenel\",\"lastName\":\"Bademci\",\"suffix\":\"\"},{\"id\":509081172,\"identity\":\"720cce9f-0c78-49bf-b7a3-aea79aa1d4f9\",\"order_by\":2,\"name\":\"İhsan Peker\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Izmir Faculty of Medicine, University of Health Sciences, Izmir City Hospital\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"İhsan\",\"middleName\":\"\",\"lastName\":\"Peker\",\"suffix\":\"\"},{\"id\":509081173,\"identity\":\"4cdcbb4f-93e4-441a-bf9c-c464fad07226\",\"order_by\":3,\"name\":\"Okay Güven Karaca\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Izmir Faculty of Medicine, University of Health Sciences, Izmir City Hospital\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Okay\",\"middleName\":\"Güven\",\"lastName\":\"Karaca\",\"suffix\":\"\"},{\"id\":509081174,\"identity\":\"510d3d8b-1d97-443f-91a8-6d8509f3a5f3\",\"order_by\":4,\"name\":\"Çağrı Kandemir\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Izmir Faculty of Medicine, University of Health Sciences, Izmir City Hospital\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Çağrı\",\"middleName\":\"\",\"lastName\":\"Kandemir\",\"suffix\":\"\"},{\"id\":509081175,\"identity\":\"a1201832-36d3-42cc-86c9-3d8b08234e46\",\"order_by\":5,\"name\":\"Barçın Özcem\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Faculty of Medicine, Near East University, Nicosia\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Barçın\",\"middleName\":\"\",\"lastName\":\"Özcem\",\"suffix\":\"\"},{\"id\":509081176,\"identity\":\"10745a35-6b0a-4d81-88df-adf49eee91c0\",\"order_by\":6,\"name\":\"Hüseyin Durmaz\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Konya City Hospital\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Hüseyin\",\"middleName\":\"\",\"lastName\":\"Durmaz\",\"suffix\":\"\"},{\"id\":509081177,\"identity\":\"b972e2a9-d0f0-43d2-b53b-08e6b6bad9b3\",\"order_by\":7,\"name\":\"Meryem Çakır\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Izmir City Hospital\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Meryem\",\"middleName\":\"\",\"lastName\":\"Çakır\",\"suffix\":\"\"},{\"id\":509081178,\"identity\":\"da029a72-a217-47c6-964b-a954b9f42b4a\",\"order_by\":8,\"name\":\"İrem Özçetin\",\"email\":\"\",\"orcid\":\"\",\"institution\":\", Izmir Katip Çelebi University, Atatürk Training and Research Hospital\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"İrem\",\"middleName\":\"\",\"lastName\":\"Özçetin\",\"suffix\":\"\"},{\"id\":509081179,\"identity\":\"d2ce30a9-5ffe-4600-8aea-f8199279d788\",\"order_by\":9,\"name\":\"Hidayet Onur Selçuk\",\"email\":\"\",\"orcid\":\"\",\"institution\":\", Izmir Katip Çelebi University, Atatürk Training and Research Hospital\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Hidayet\",\"middleName\":\"Onur\",\"lastName\":\"Selçuk\",\"suffix\":\"\"}],\"badges\":[],\"createdAt\":\"2025-06-24 12:38:09\",\"currentVersionCode\":1,\"declarations\":\"\",\"doi\":\"10.21203/rs.3.rs-6965764/v1\",\"doiUrl\":\"https://doi.org/10.21203/rs.3.rs-6965764/v1\",\"draftVersion\":[],\"editorialEvents\":[],\"editorialNote\":\"\",\"failedWorkflow\":false,\"files\":[{\"id\":90896943,\"identity\":\"5583fbc8-26d5-4281-bb33-e801c4bab3ca\",\"added_by\":\"auto\",\"created_at\":\"2025-09-09 11:38:59\",\"extension\":\"png\",\"order_by\":1,\"title\":\"Figure 1\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":59075,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eComparison of accuracy and completeness scores across large language models (LLMs) in patient education on mitral valve surgery.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"1.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-6965764/v1/d64808690544d6905279a1cf.png\"},{\"id\":90894637,\"identity\":\"ef3aca4e-5e2d-4e97-842c-44c37e9c451c\",\"added_by\":\"auto\",\"created_at\":\"2025-09-09 11:30:59\",\"extension\":\"png\",\"order_by\":2,\"title\":\"Figure 2\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":53732,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eComparison of readability scores (SMOG and FRE) across LLMs in patient education on mitral valve surgery.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"2.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-6965764/v1/c4c5ad4fc3f54f51629d4415.png\"},{\"id\":90897639,\"identity\":\"38198d6a-09ed-4549-bcbd-094b84abeeb1\",\"added_by\":\"auto\",\"created_at\":\"2025-09-09 11:47:04\",\"extension\":\"pdf\",\"order_by\":0,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"manuscript-pdf\",\"size\":758989,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"manuscript.pdf\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-6965764/v1/39aa0744-14d3-476c-8dc9-6614d0d0169f.pdf\"},{\"id\":90896946,\"identity\":\"4b1cf1fd-5c1c-4db5-8e7b-8892ce6e6d39\",\"added_by\":\"auto\",\"created_at\":\"2025-09-09 11:38:59\",\"extension\":\"pdf\",\"order_by\":0,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"supplement\",\"size\":311643,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"MitralValveSurgerySCTSquestions.pdf\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-6965764/v1/b544dbec94ee8e265935bc41.pdf\"}],\"financialInterests\":\"No competing interests reported.\",\"formattedTitle\":\"A Comparative Performance Analysis of AI-Assisted Language Models in Preoperative Patient Education for Mitral Valve Surgery\",\"fulltext\":[{\"header\":\"1. Introduction\",\"content\":\"\\u003cp\\u003eMitral valve diseases are among the most common valvular heart pathologies, leading to significant clinical outcomes. These conditions are classified into two main clinical forms: mitral stenosis and mitral regurgitation. Over time, they may impair left ventricular function and predispose patients to heart failure, atrial fibrillation, and thromboembolic events [\\u003cspan citationid=\\\"CR1\\\" class=\\\"CitationRef\\\"\\u003e1\\u003c/span\\u003e]. In symptomatic patients, medical therapy provides limited benefits, and surgical intervention often offers a more definitive solution. Currently, mitral valve repair is usually preferred over replacement, particularly in degenerative mitral regurgitation, owing to its certain advantages in preserving valve function, maintaining left ventricular geometry, and improving survival. Furthermore, advancements in surgical techniques and the widespread adoption of minimally invasive approaches have shortened postoperative recovery times and reduced complication rates [\\u003cspan citationid=\\\"CR2\\\" class=\\\"CitationRef\\\"\\u003e2\\u003c/span\\u003e].\\u003c/p\\u003e\\u003cp\\u003eThe decision to undergo MVS involves not only anatomical and clinical factors, but also the patient\\u0026rsquo;s preferences, lifestyle, expectations, and understanding of potential outcomes. Comprehensive patient education and information-sharing processes led by heart teams have been shown to improve patient engagement and positively influence satisfaction with care [\\u003cspan citationid=\\\"CR2\\\" class=\\\"CitationRef\\\"\\u003e2\\u003c/span\\u003e].\\u003c/p\\u003e\\u003cp\\u003eThe success of MVS heavily relies on effective patient education, which is crucial for treatment adherence and complication reduction. Clearly communicating the differences between valve repair and replacement and, if replacement is necessary, between bioprosthetic and mechanical valves, as well as the potential risks and benefits, supports patients in actively participating in informed decision-making. Current guidelines recommend a patient-centered approach as a cornerstone of surgical success and emphasize the ethical and clinical importance of conducting the informed consent process with great care [\\u003cspan citationid=\\\"CR1\\\" class=\\\"CitationRef\\\"\\u003e1\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR2\\\" class=\\\"CitationRef\\\"\\u003e2\\u003c/span\\u003e].\\u003c/p\\u003e\\u003cp\\u003eWith advancing technology, large language models (LLMs) supported by artificial intelligence (AI) led initially by ChatGPT have emerged as tools to meet patients' informational needs regarding surgical treatments. In subsequent years, numerous other models have also entered widespread use, including ChatGPT-4o by OpenAI, Claude 3.7 Sonnet by Anthropic, Gemini 2.5 Pro Preview by Google, Microsoft Copilot, and DeepSeek-V3. These models can provide rapid, accessible, and readable basic-level information on topics such as preoperative procedures, surgical risks, alternative treatments, and the recovery process.\\u003c/p\\u003e\\u003cp\\u003eRecent studies have shown that certain LLMs may deliver incomplete or misleading information regarding surgical risks and postoperative recovery [\\u003cspan citationid=\\\"CR3\\\" class=\\\"CitationRef\\\"\\u003e3\\u003c/span\\u003e]. Nonetheless, their ability to communicate information with high readability and in non-technical language represents a significant advantage, particularly for individuals with low health literacy.\\u003c/p\\u003e\\u003cp\\u003eCurrently, healthcare professionals view these technologies as supportive tools in patient education; however, it is strongly emphasized that final medical decisions must always be made by clinical experts. In recent years, AI-based health communication has increasingly become a complementary component of patient-centered care in modern medicine.\\u003c/p\\u003e\\u003cp\\u003eArtificial intelligence-supported LLMs are also being utilized for candidates undergoing cardiac surgery [\\u003cspan citationid=\\\"CR4\\\" class=\\\"CitationRef\\\"\\u003e4\\u003c/span\\u003e]. In particularly complex procedures such as MVS, these models can facilitate rapid access to information, enhance physician\\u0026ndash;patient communication, and support patients\\u0026rsquo; psychological preparedness for the surgical process [\\u003cspan citationid=\\\"CR5\\\" class=\\\"CitationRef\\\"\\u003e5\\u003c/span\\u003e]. However, despite their growing use, there is limited research comparing the accuracy, completeness, and readability of the information these models provide specifically for MVS patient education. This gap makes it challenging to determine which models offer the most reliable and accessible information to support patient understanding and decision-making. In the present study, we, therefore, aimed to evaluate and compare the accuracy, completeness, and readability of information provided by different LLMs for patient education and counseling in patients for MVS.\\u003c/p\\u003e\"},{\"header\":\"2. Materials and Methods\",\"content\":\"\\u003cdiv id=\\\"Sec3\\\" class=\\\"Section2\\\"\\u003e\\u003ch2\\u003e2.1. Study design and study sample\\u003c/h2\\u003e\\u003cp\\u003eThe seven open-ended questions used to collect the data were derived from a detailed operative process description, which was not individually designed to maintain objectivity. Instead, the content was obtained from the patient information section of The Society for Cardiothoracic Surgery in Great Britain \\u0026amp; Ireland website (\\u003cspan class=\\\"ExternalRef\\\"\\u003e\\u003cspan class=\\\"RefSource\\\"\\u003ehttps://scts.org/patients/heart/procedures/20/mitral_valve_surgery\\u003c/span\\u003e\\u003cspan address=\\\"https://scts.org/patients/heart/procedures/20/mitral_valve_surgery\\\" targettype=\\\"URL\\\" class=\\\"RefTarget\\\"\\u003e\\u003c/span\\u003e\\u003c/span\\u003e). These questions were simultaneously presented in April 2025 to five different LLMs\\u0026mdash;ChatGPT-4o by OpenAI, Claude 3.7 Sonnet by Anthropic, Gemini 2.5 Pro Preview by Google, Microsoft Copilot, and DeepSeek-V3 and their responses were collected for evaluation.\\u003c/p\\u003e\\u003c/div\\u003e\\u003cdiv id=\\\"Sec4\\\" class=\\\"Section2\\\"\\u003e\\u003ch2\\u003e2.2. Selection of LLM\\u003c/h2\\u003e\\u003cp\\u003eLanguage models which were freely or partially freely accessible to any user were selected for the study. Additionally, the selected models were chosen based on their popularity and accessibility within the healthcare sector.\\u003c/p\\u003e\\u003cdiv id=\\\"Sec5\\\" class=\\\"Section3\\\"\\u003e\\u003ch2\\u003e2.2.1. Evaluation Process\\u003c/h2\\u003e\\u003cp\\u003eTwo experts with at least 10 years of experience in MVS independently evaluated the data in terms of accuracy and completeness using a five-point Likert scale to maintain objectivity. For the accuracy evaluation, current guidelines on MVS and the clinical expertise of specialists in the field were used as primary reference sources [\\u003cspan citationid=\\\"CR1\\\" class=\\\"CitationRef\\\"\\u003e1\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR2\\\" class=\\\"CitationRef\\\"\\u003e2\\u003c/span\\u003e]. After the initial independent evaluations, the evaluators met to reach a consensus and determine a final, unified score. To assess inter-rater reliability, the intraclass correlation coefficient (ICC) was calculated [\\u003cspan citationid=\\\"CR6\\\" class=\\\"CitationRef\\\"\\u003e6\\u003c/span\\u003e], and the result was found to be 0.85, indicating high reliability.\\u003c/p\\u003e\\u003c/div\\u003e\\u003cdiv id=\\\"Sec6\\\" class=\\\"Section3\\\"\\u003e\\u003ch2\\u003e2.2.2. Evaluation Criteria\\u003c/h2\\u003e\\u003cp\\u003e\\u003cb\\u003eAccuracy Evaluation\\u003c/b\\u003e: A five-point Likert scale was used to evaluate the accuracy of the information [\\u003cspan citationid=\\\"CR7\\\" class=\\\"CitationRef\\\"\\u003e7\\u003c/span\\u003e]: 5 points\\u0026thinsp;=\\u0026thinsp;Completely accurate, 4 points\\u0026thinsp;=\\u0026thinsp;More accurate than inaccurate, 3 points\\u0026thinsp;=\\u0026thinsp;Accuracy and inaccuracy are roughly equal, 2 points\\u0026thinsp;=\\u0026thinsp;More inaccurate than accurate, 1 point\\u0026thinsp;=\\u0026thinsp;Completely inaccurate.\\u003c/p\\u003e\\u003cp\\u003e\\u003cb\\u003eCompleteness Evaluation\\u003c/b\\u003e: The completeness of the information was also evaluated using a five-point Likert scale [\\u003cspan citationid=\\\"CR7\\\" class=\\\"CitationRef\\\"\\u003e7\\u003c/span\\u003e]: 5\\u0026thinsp;=\\u0026thinsp;Completely sufficient (Fully covers the topic/No missing information), 4\\u0026thinsp;=\\u0026thinsp;Mostly sufficient (Covers the topic/Minor omissions), 3\\u0026thinsp;=\\u0026thinsp;Moderately sufficient (Relevant to the topic/Some ambiguities), 2\\u0026thinsp;=\\u0026thinsp;Mostly insufficient (Partially related to the topic/Significant omissions), 1\\u0026thinsp;=\\u0026thinsp;Very insufficient (Unrelated to the topic/Overly basic or nonsensical).\\u003c/p\\u003e\\u003cp\\u003e\\u003cstrong\\u003eReadability Evaluation\\u003c/strong\\u003e\\u003cp\\u003eTo evaluate the readability of the responses, the Simplified Measure of Gobbledygook (SMOG) Index [\\u003cspan citationid=\\\"CR7\\\" class=\\\"CitationRef\\\"\\u003e7\\u003c/span\\u003e] and the Flesch-Kincaid Reading Ease (FRE) scale were used [\\u003cspan citationid=\\\"CR8\\\" class=\\\"CitationRef\\\"\\u003e8\\u003c/span\\u003e].\\u003c/p\\u003e\\u003c/p\\u003e\\u003c/div\\u003e\\u003c/div\\u003e\\u003cdiv id=\\\"Sec7\\\" class=\\\"Section2\\\"\\u003e\\u003ch2\\u003e2.3. Statistical Analysis\\u003c/h2\\u003e\\u003cp\\u003eStatistical analysis was performed using the SPSS for Windows version 23.0 software (IBM Corp., Armonk, NY, USA). Comparisons of the LLMs in terms of accuracy and completeness were conducted using the Kruskal-Wallis test, followed by \\u003cem\\u003epost-hoc\\u003c/em\\u003e Dunn-Bonferroni tests after detecting overall significance. The normality of the readability scores was checked using the Shapiro-Wilk test. Continuous variables were presented in mean\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;standard deviation (SD) or median (min-max), while categorical variables were presented in number and frequency. The readability scores which met the normality assumption were compared using one-way analysis of variance (ANOVA). Subsequent subgroup analyses following ANOVA were performed using the Bonferroni corrections. A \\u003cem\\u003ep\\u003c/em\\u003e value of \\u0026lt;\\u0026thinsp;0.05 was considered statistically significant.\\u003c/p\\u003e\\u003c/div\\u003e\"},{\"header\":\"3. Results\",\"content\":\"\\u003cp\\u003eThe median accuracy scores of the LLMs were 5 (range, 5 to 5) for ChatGPT 4o, 4 (range, 4 to 4) for Claude 3.7 Sonnet, 5 (range, 4 to 5) for DeepSeek-V3, 5 (range, 5 to 5) for Gemini 2.5 Pro Preview, and 4 (range, 4 to 4) for Microsoft Copilot. A statistically significant difference was found among the LLMs in terms of accuracy scores (p\\u0026thinsp;\\u0026lt;\\u0026thinsp;0.001).(Table \\u003cspan refid=\\\"Tab1\\\" class=\\\"InternalRef\\\"\\u003e1\\u003c/span\\u003e) In the \\u003cem\\u003epost-hoc\\u003c/em\\u003e analyses, the accuracy level of ChatGPT 4o was found to be significantly higher than that of Claude 3.7 Sonnet (5 \\u0026amp; 4; p\\u0026thinsp;=\\u0026thinsp;0.002) and Microsoft Copilot (5 \\u0026amp; 4; p\\u0026thinsp;=\\u0026thinsp;0.002). Similarly, Gemini 2.5 Pro Preview had a higher accuracy level compared to Claude (5 \\u0026amp; 4; p\\u0026thinsp;=\\u0026thinsp;0.002) and Microsoft Copilot (5 \\u0026amp; 4; p\\u0026thinsp;=\\u0026thinsp;0.002). Subgroup analyses revealed no statistically significant differences among the LLMs (p\\u0026thinsp;\\u0026gt;\\u0026thinsp;0.05).(Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig1\\\" class=\\\"InternalRef\\\"\\u003e1\\u003c/span\\u003e)\\u003c/p\\u003e\\u003cp\\u003e\\u003cdiv class=\\\"gridtable\\\"\\u003e\\u003ctable float=\\\"Yes\\\" id=\\\"Tab1\\\" border=\\\"1\\\"\\u003e\\u003ccaption language=\\\"En\\\"\\u003e\\u003cdiv class=\\\"CaptionNumber\\\"\\u003eTable 1\\u003c/div\\u003e\\u003cdiv class=\\\"CaptionContent\\\"\\u003e\\u003cp\\u003eComparative Performance of Large Language Models in Accuracy, Completeness, and Readability Metrics\\u003c/p\\u003e\\u003c/div\\u003e\\u003c/caption\\u003e\\u003ccolgroup cols=\\\"5\\\"\\u003e\\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c1\\\" colnum=\\\"1\\\"\\u003e\\u003c/div\\u003e\\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c2\\\" colnum=\\\"2\\\"\\u003e\\u003c/div\\u003e\\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c3\\\" colnum=\\\"3\\\"\\u003e\\u003c/div\\u003e\\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c4\\\" colnum=\\\"4\\\"\\u003e\\u003c/div\\u003e\\u003cdiv align=\\\"char\\\" char=\\\".\\\" class=\\\"colspec\\\" colname=\\\"c5\\\" colnum=\\\"5\\\"\\u003e\\u003c/div\\u003e\\u003cthead\\u003e\\u003ctr\\u003e\\u003cth align=\\\"left\\\" colname=\\\"c1\\\"\\u003e\\u003cp\\u003eLLM Model\\u003c/p\\u003e\\u003c/th\\u003e\\u003cth align=\\\"left\\\" colname=\\\"c2\\\"\\u003e\\u003cp\\u003eAccuracy\\u003c/p\\u003e\\u003c/th\\u003e\\u003cth align=\\\"left\\\" colname=\\\"c3\\\"\\u003e\\u003cp\\u003eCompleteness\\u003c/p\\u003e\\u003c/th\\u003e\\u003cth align=\\\"left\\\" colname=\\\"c4\\\"\\u003e\\u003cp\\u003eSMOG\\u003c/p\\u003e\\u003c/th\\u003e\\u003cth align=\\\"left\\\" colname=\\\"c5\\\"\\u003e\\u003cp\\u003eFRE\\u003c/p\\u003e\\u003c/th\\u003e\\u003c/tr\\u003e\\u003c/thead\\u003e\\u003ctbody\\u003e\\u003ctr\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e\\u003cp\\u003e\\u003cb\\u003eChatGPT 4o\\u003c/b\\u003e\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e\\u003cp\\u003e5 (5\\u0026ndash;5)\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e\\u003cp\\u003e4 (4\\u0026ndash;4)\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e\\u003cp\\u003e12.24\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;0.69\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e\\u003cp\\u003e9.04 0.55\\u003c/p\\u003e\\u003c/td\\u003e\\u003c/tr\\u003e\\u003ctr\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e\\u003cp\\u003e\\u003cb\\u003eClaude 3.7 Sonnet\\u003c/b\\u003e\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e\\u003cp\\u003e4 (4\\u0026ndash;4)\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e\\u003cp\\u003e3 (3\\u0026ndash;3)\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e\\u003cp\\u003e10.90\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;0.54\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e\\u003cp\\u003e8 \\u0026plusmn;\\u0026thinsp;0.41\\u003c/p\\u003e\\u003c/td\\u003e\\u003c/tr\\u003e\\u003ctr\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e\\u003cp\\u003e\\u003cb\\u003eDeepSeek-V3\\u003c/b\\u003e\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e\\u003cp\\u003e5 (4\\u0026ndash;5)\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e\\u003cp\\u003e4 (3\\u0026ndash;4)\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e\\u003cp\\u003e12.61\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;0.72\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e\\u003cp\\u003e9.37\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;0.53\\u003c/p\\u003e\\u003c/td\\u003e\\u003c/tr\\u003e\\u003ctr\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e\\u003cp\\u003e\\u003cb\\u003eGemini 2.5 Pro Preview\\u003c/b\\u003e\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e\\u003cp\\u003e5 (5\\u0026ndash;5)\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e\\u003cp\\u003e5 (5\\u0026ndash;5)\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e\\u003cp\\u003e12.41\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;0.64\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e\\u003cp\\u003e9.17 0.48\\u003c/p\\u003e\\u003c/td\\u003e\\u003c/tr\\u003e\\u003ctr\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e\\u003cp\\u003e\\u003cb\\u003eMicrosoft Copilot\\u003c/b\\u003e\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e\\u003cp\\u003e4 (4\\u0026ndash;4)\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e\\u003cp\\u003e3 (3\\u0026ndash;4)\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e\\u003cp\\u003e11.84\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;0.66\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e\\u003cp\\u003e8.77\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;0.49\\u003c/p\\u003e\\u003c/td\\u003e\\u003c/tr\\u003e\\u003ctr\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e\\u003cp\\u003e\\u003cb\\u003ep-value\\u003c/b\\u003e\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e\\u003cp\\u003e\\u003cb\\u003e\\u0026lt;\\u0026thinsp;0.001\\u003c/b\\u003e\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e\\u003cp\\u003e\\u003cb\\u003e\\u0026lt;\\u0026thinsp;0.001\\u003c/b\\u003e\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e\\u003cp\\u003e\\u003cb\\u003e\\u0026lt;\\u0026thinsp;0.001\\u003c/b\\u003e\\u003c/p\\u003e\\u003c/td\\u003e\\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e\\u003cp\\u003e\\u003cb\\u003e\\u0026lt;\\u0026thinsp;0.001\\u003c/b\\u003e\\u003c/p\\u003e\\u003c/td\\u003e\\u003c/tr\\u003e\\u003c/tbody\\u003e\\u003c/colgroup\\u003e\\u003c/table\\u003e\\u003c/div\\u003e\\u003c/p\\u003e\\u003cp\\u003e\\u003cem\\u003eData are given in mean\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;SD or median (min-max), unless otherwise stated. LLM: large language model; SMOG: Simplified Measure of Gobbledygook; FRE: Flesch-Kincaid Reading Ease.\\u003c/em\\u003e\\u003c/p\\u003e\\u003cp\\u003eThe median completeness scores of the LLMs were 4 (range, 4 to 4) for ChatGPT 4o, 3 (range, 3 to 3) for Claude 3.7 Sonnet, 4 (range, 3 to 4) for DeepSeek-V3, 5 (range, 5 to 5) for Gemini 2.5 Pro Preview, and 3 (range, 3 to 4) for Microsoft Copilot.(Table \\u003cspan refid=\\\"Tab1\\\" class=\\\"InternalRef\\\"\\u003e1\\u003c/span\\u003e) There was a statistically significant difference among the LLMs regarding completeness scores (p\\u0026thinsp;\\u0026lt;\\u0026thinsp;0.001). In the \\u003cem\\u003epost-hoc\\u003c/em\\u003e analyses, the completeness score of Gemini 2.5 Pro Preview was found to be significantly higher than that of Claude 3.7 Sonnet (5 \\u0026amp; 3; p\\u0026thinsp;=\\u0026thinsp;0.000) and Microsoft Copilot (5 \\u0026amp; 3; p\\u0026thinsp;\\u0026lt;\\u0026thinsp;0.001). Subgroup analyses revealed no statistically significant differences among the LLMs (p\\u0026thinsp;\\u0026gt;\\u0026thinsp;0.05).(Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig1\\\" class=\\\"InternalRef\\\"\\u003e1\\u003c/span\\u003e)\\u003c/p\\u003e\\u003cp\\u003e\\u003c/p\\u003e\\u003cp\\u003eConsidering the SMOG scores of the LLMs, the mean score was found to be 12.24\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;0.69 for ChatGPT 4o, 10.90\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;0.54 for Claude 3.7 Sonnet, 12.61\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;0.72 for DeepSeek-V3, 12.41\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;0.64 for Gemini 2.5 Pro Preview, and 11.84\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;0.66 for Microsoft Copilot, indicating a statistically significant difference among the LLMs (p\\u0026thinsp;\\u0026lt;\\u0026thinsp;0.001).(Table \\u003cspan refid=\\\"Tab1\\\" class=\\\"InternalRef\\\"\\u003e1\\u003c/span\\u003e) In the \\u003cem\\u003epost-hoc\\u003c/em\\u003e analyses, the mean SMOG score of Claude 3.7 Sonnet was found to be significantly lower than that of ChatGPT 4o (10.90 \\u0026amp; 12.24; p\\u0026thinsp;=\\u0026thinsp;0.006), DeepSeek-V3 (10.90 \\u0026amp; 12.61; p\\u0026thinsp;\\u0026lt;\\u0026thinsp;0.001), and Gemini 2.5 Pro Preview (10.90 \\u0026amp; 12.41; p\\u0026thinsp;=\\u0026thinsp;0.001). Subgroup analyses revealed no statistically significant differences among the LLMs (p\\u0026thinsp;\\u0026gt;\\u0026thinsp;0.05).(Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig2\\\" class=\\\"InternalRef\\\"\\u003e2\\u003c/span\\u003e)\\u003c/p\\u003e\\u003cp\\u003e\\u003c/p\\u003e\\u003cp\\u003eThe mean FRE scores of the LLMs were 9.04\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;0.55 for ChatGPT 4o, 8.00\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;0.41 for Claude 3.7 Sonnet, 9.37\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;0.53 for DeepSeek-V3, 9.17\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;0.48 for Gemini 2.5 Pro Preview, and 8.77\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;0.49 for Microsoft Copilot. There was a statistically significant difference in the FRE scores among the LLMs (p\\u0026thinsp;\\u0026lt;\\u0026thinsp;0.001). In the \\u003cem\\u003epost-hoc\\u003c/em\\u003e analyses, the mean FRE score of Claude 3.7 Sonnet was found to be significantly lower than that of ChatGPT 4o (8 \\u0026amp; 9.04; p\\u0026thinsp;=\\u0026thinsp;0.004), DeepSeek-V3 (8 \\u0026amp; 9.37; p\\u0026thinsp;\\u0026lt;\\u0026thinsp;0.001), and Gemini 2.5 Pro Preview (8 \\u0026amp; 9.17; p\\u0026thinsp;=\\u0026thinsp;0.001). Subgroup analyses revealed no statistically significant differences among the LLMs (p\\u0026thinsp;\\u0026gt;\\u0026thinsp;0.05).(Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig2\\\" class=\\\"InternalRef\\\"\\u003e2\\u003c/span\\u003e)\\u003c/p\\u003e\"},{\"header\":\"4. Discussion\",\"content\":\"\\u003cp\\u003eDespite the increasing integration of LLMs into healthcare communication, particularly for patient education, there remains a significant gap in comparative evaluations of these tools in the context of complex surgical procedures. Most existing studies have focused on the performance of individual LLMs or their application in general health information dissemination. However, comprehensive analyses assessing the accuracy, completeness, and readability of multiple, state-of-the-art LLMs in delivering patient-centered information for specific and high-risk interventions, such as MVS, are still limited. In the present study, we evaluated the accuracy, completeness, and readability levels of five LLMs (\\u003cem\\u003ei.e.\\u003c/em\\u003e, ChatGPT 4o, Claude 3.7 Sonnet, DeepSeek-V3, Gemini 2.5 Pro Preview, and Microsoft Copilot), used for patient education and information regarding MVS. Notably, the ChatGPT 4o and Gemini 2.5 Pro Preview models were found to be statistically significantly superior to Claude 3.7 Sonnet and Microsoft Copilot in terms of both accuracy and completeness. Considering the readability of responses generated by the LLMs, the Claude model yielded statistically significantly more readable responses compared to ChatGPT 4o, DeepSeek-V3, and Gemini 2.5 Pro Preview.\\u003c/p\\u003e\\u003cp\\u003eIn recent years, there has been a growing number of studies examining the accuracy of data generated by LLMs, particularly in the context of patient education. Lee et al. [\\u003cspan citationid=\\\"CR9\\\" class=\\\"CitationRef\\\"\\u003e9\\u003c/span\\u003e] showed that the ChatGPT model yielded higher accuracy in their study on atrial fibrillation patient education. Similarly, other studies have demonstrated that the ChatGPT model provides clinically accurate information at a high level [\\u003cspan citationid=\\\"CR10\\\" class=\\\"CitationRef\\\"\\u003e10\\u003c/span\\u003e]. In a related study by Zhao et al. [\\u003cspan citationid=\\\"CR11\\\" class=\\\"CitationRef\\\"\\u003e11\\u003c/span\\u003e], which focused on developing educational materials for patients with lower back pain, the Gemini 2.5 Pro Preview model achieved similarly high levels of accuracy and completeness as ChatGPT 4o.\\u003c/p\\u003e\\u003cp\\u003eThe superior performance of the ChatGPT 4o and Gemini 2.5 Pro Preview models in terms of accuracy and completeness can primarily be attributed to their use of Retrieval-Augmented Generation (RAG) architecture [\\u003cspan citationid=\\\"CR12\\\" class=\\\"CitationRef\\\"\\u003e12\\u003c/span\\u003e]. The RAG-based models are capable of accessing up-to-date external sources before generating responses, thereby enabling the production of more accurate and comprehensive content [\\u003cspan citationid=\\\"CR12\\\" class=\\\"CitationRef\\\"\\u003e12\\u003c/span\\u003e]. In contrast, conventional LLMs only reflect information available in their training datasets and are usually limited in providing sources for the information they present.\\u003c/p\\u003e\\u003cp\\u003eIn their study evaluating anterior cruciate ligament (ACL) injuries, Woo et al. [\\u003cspan citationid=\\\"CR13\\\" class=\\\"CitationRef\\\"\\u003e13\\u003c/span\\u003e] showed that RAG-based LLMs significantly improved accuracy rates. In the aforementioned study, the implementation of RAG resulted in an average increase of 39.7% in model accuracy. The RAG framework has proven to be a valuable tool in enhancing the accuracy of patient education materials, offering a clear advantage over traditional LLMs [\\u003cspan citationid=\\\"CR13\\\" class=\\\"CitationRef\\\"\\u003e13\\u003c/span\\u003e]. One of the main concerns regarding the use of LLMs in patient education and information delivery is their potential to generate inaccurate, nonsensical, or faulty content based on outdated or irrelevant model training data [\\u003cspan citationid=\\\"CR14\\\" class=\\\"CitationRef\\\"\\u003e14\\u003c/span\\u003e]. Although this issue is less common in RAG-based LLMs, to further eliminate this risk in our study, all model outputs were evaluated by two experienced surgeons specialized in the field.\\u003c/p\\u003e\\u003cp\\u003eAnother critical aspect of the data obtained from LLMs is ensuring ease of readability and comprehension. Review of the recent literature reveals numerous studies presenting varying perspectives on this issue [\\u003cspan citationid=\\\"CR15\\\" class=\\\"CitationRef\\\"\\u003e15\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR16\\\" class=\\\"CitationRef\\\"\\u003e16\\u003c/span\\u003e]. The coexistence of plain language and medical accuracy is of utmost importance for patient safety. In our study, the Claude model demonstrated the highest performance in terms of readability compared to the other models. In a study conducted by Zhang et al. [\\u003cspan citationid=\\\"CR17\\\" class=\\\"CitationRef\\\"\\u003e17\\u003c/span\\u003e] on inflammatory bowel disease, readability scores were found to be higher for the Claude 3.7 Sonnet model than for ChatGPT-4o and Gemini 2.5 Pro Preview. This finding highlights Claude's success in generating user-friendly and more comprehensible texts. This advantage stems from the model's ability to use shorter sentences and less complex vocabulary, which enhances the readability of the generated content, particularly in medical contexts where such language improves overall understanding [\\u003cspan citationid=\\\"CR16\\\" class=\\\"CitationRef\\\"\\u003e16\\u003c/span\\u003e]. Furthermore, the consistency and efficiency of Claude 3.7 Sonnet's responses also positively influenced its readability scores. As highlighted in the study by Jin et al. [\\u003cspan citationid=\\\"CR18\\\" class=\\\"CitationRef\\\"\\u003e18\\u003c/span\\u003e], fast and internally consistent responses help structure texts more effectively, contributing to better comprehension.\\u003c/p\\u003e\\u003cp\\u003eFurthermore, there are also ethical concerns regarding the use of LLMs in patient education. Patient privacy and data security are of paramount importance in this context. Most LLMs utilize large amounts of data during their training processes, which increases the risk of misuse of patients' personal health information [\\u003cspan citationid=\\\"CR19\\\" class=\\\"CitationRef\\\"\\u003e19\\u003c/span\\u003e]. Additionally, there is a risk that patients may accept information generated by the model without questioning it (overtrust), potentially replacing professional medical advice. Therefore, the use of LLMs in patient education should be conducted under strict supervision, transparency, and ethical guidelines.\\u003c/p\\u003e\\u003cdiv id=\\\"Sec10\\\" class=\\\"Section2\\\"\\u003e\\u003ch2\\u003e4.1. Clinical Implications\\u003c/h2\\u003e\\u003cp\\u003eBy improving the clarity and accessibility of medical information, LLMs can contribute to increased health literacy, informed decision-making, and greater patient satisfaction. These models can be particularly beneficial in busy clinical environments where time constraints may limit detailed one-on-one education. However, integration of LLMs into routine practice should be accompanied by proper oversight to ensure the accuracy and appropriateness of content delivered to patients. Future implementation strategies should also consider patient-specific factors such as age, educational background, and digital literacy to optimize the impact of LLM-assisted education. Ultimately, these models may serve as valuable adjuncts in shared decision-making and personalized patient communication frameworks.\\u003c/p\\u003e\\u003c/div\\u003e\\u003cdiv id=\\\"Sec11\\\" class=\\\"Section2\\\"\\u003e\\u003ch2\\u003e4.2. Limitations\\u003c/h2\\u003e\\u003cp\\u003eAlthough the independent and double-blind evaluation of the data by two expert surgeons strengthens the assessment method and analysis, this study has certain limitations that should be acknowledged. First, the responses were evaluated by two experts, while human judgment inherently carries some level of subjectivity. Although this was mitigated by using structured criteria, interrater reliability is still a potential concern. Second, the evaluation was conducted only on responses in English; model performances in other languages were unable to be assessed. Third, the examined topic was limited to MVS, and model performance may vary for other diseases; therefore, making a general characterization of LLMs may not always be fully accurate. In addition, the study analyzed specific versions of the LLMs at a single time point. As LLMs are updated frequently, their performance could change over time, potentially altering accuracy, completeness, and readability outcomes in future assessments. Fourth, cultural differences among patients using LLMs may exist, and behavioral outcomes such as patient experience and satisfaction were not included in the evaluation, which can be deemed as another limitation. Finally, although the questions were open-ended, the small number of questions may also limit the generalizability of the results. Differences in future versions of LLMs may also affect the outcomes.\\u003c/p\\u003e\\u003c/div\\u003e\"},{\"header\":\"5. Conclusion\",\"content\":\"\\u003cp\\u003eIn conclusion, the contribution of AI-supported LLMs in everyday clinical practice for patient education and information is increasing day by day in the healthcare field. Our study results showed that ChatGPT 4.0 and Gemini 2.5 Pro Preview models had a higher level of performance in terms of accuracy and completeness regarding patient information related to MVS, while Claude 3.7 Sonnet performed better in terms of readability. Taken together, the use of LLMs enhances health literacy among patients and makes patient education resources more accessible and understandable. Future studies incorporating patient-centered evaluations and considering variables such as patients' age and educational status are warranted to draw more comprehensive data on this subject.\\u003c/p\\u003e\"},{\"header\":\"Declarations\",\"content\":\"\\u003cp\\u003e\\u003cstrong\\u003eEthics approval and consent to participate\\u003c/strong\\u003e\\u003c/p\\u003e\\n\\u003cp\\u003eThis study did not involve human participants, human data, or human tissue. The research protocol was reviewed and approved by the Ethics Committee of \\u0026nbsp;University of Health Sciences Turkey, Izmir Faculty of Medicine, Izmir City Hospital (Approval Number: 2025/368, Date: July 23, 2025). The study was conducted in accordance with the ethical standards of the institutional research committee and with the Declaration of Helsinki (1964) and its subsequent amendments. Informed consent was obtained from all participants involved in the study.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eConsent for publication\\u003c/strong\\u003e\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026nbsp;All the authors have given their consent for publication.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eAvailability of data and materials\\u003c/strong\\u003e\\u003c/p\\u003e\\n\\u003cp\\u003eThe data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eCompeting interests\\u0026nbsp;\\u003c/strong\\u003e\\u003c/p\\u003e\\n\\u003cp\\u003eThe authors declare that they have no competing interest.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eFunding\\u003c/strong\\u003e\\u003c/p\\u003e\\n\\u003cp\\u003eThis research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eAuthors\\u0026rsquo; contributions\\u0026nbsp;\\u003c/strong\\u003e\\u003c/p\\u003e\\n\\u003cp\\u003eMethodology:B.B.A, M.S.B and O.G.K ; validation: I.P, C.K and B.O ; formal analysis: \\u0026nbsp;H.D, M.C ; investigation: I.O, H.O.S ; writing\\u0026mdash;original draft: B.B.A, M.S.B ; \\u0026nbsp; writing\\u0026mdash;review and editing: B.B.A, M.S.B and O.G.K. All authors read and approved the final manuscript.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eAcknowledgements\\u003c/strong\\u003e\\u003c/p\\u003e\\n\\u003cp\\u003eNot applicable.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eAuthor details\\u003c/strong\\u003e\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003csup\\u003e1\\u003c/sup\\u003e\\u003cstrong\\u003e\\u0026nbsp;\\u003c/strong\\u003eDepartment of Cardiovascular Surgery, Izmir Faculty of Medicine\\u003cstrong\\u003e,\\u003c/strong\\u003e University of Health Sciences, Izmir City Hospital, 35530 Izmir, Turkey; banutunada@gmail.com (B.B.A.), mbademci@gmail.com (M.S.B.)\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003csup\\u003e2\\u0026nbsp;\\u003c/sup\\u003eDepartment of Cardiovascular Surgery, Izmir City Hospital, 35530 Izmir, Turkey; drguven@gmail.com (O.G.K.), ihsanpeker35@gmail.com (I.P.), cagrikandemir@hotmail.de (C.K.)\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003csup\\u003e3 \\u0026nbsp;\\u003c/sup\\u003eDepartment of Cardiovascular Surgery, Faculty of Medicine, Near East University, Nicosia 99138, Cyprus; \\u003cem\\u003ebarcin.ozcem@med.neu.edu.tr\\u003c/em\\u003e (B.O.)\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003csup\\u003e4\\u0026nbsp;\\u003c/sup\\u003eDepartment of Cardiovascular Surgery, Konya City Hospital, 42020 Konya, Turkey; durmazz1@hotmail.com (H.D.)\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003csup\\u003e5\\u0026nbsp;\\u003c/sup\\u003eDepartment of Family Medicine,\\u0026nbsp;Izmir City Hospital, 35530 Izmir, Turkey; obgndrmeryem@hotmail.com (M.C).\\u0026nbsp;\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003csup\\u003e6\\u0026nbsp;\\u003c/sup\\u003eDepartment of Cardiovascular Surgery, Izmir Katip \\u0026Ccedil;elebi University, Atat\\u0026uuml;rk Training and Research Hospital, 35360 Izmir, Turkey; iremyesiloren@gmail.com (I.O.), onur_selcuk_48@hotmail.com (H.O.S.)\\u003c/p\\u003e\"},{\"header\":\"References\",\"content\":\"\\u003col\\u003e\\n\\u003cli\\u003eOtto CM, Nishimura RA, Bonow RO, et al. 2020 ACC/AHA Guideline for the Management of Patients With Valvular Heart Disease: Executive Summary: A Report of the American College of Cardiology/American Heart Association Joint Committee on Clinical Practice Guidelines [published correction appears in Circulation. 2021 Feb 2;143(5):e228. doi: 10.1161/CIR.0000000000000960.] [published correction appears in Circulation. 2021 Mar 9;143(10):e784. Circulation. 2021;143(5):e35-e71. doi:10.1161/CIR.0000000000000932.\\u003c/li\\u003e\\n\\u003cli\\u003eVahanian A, Beyersdorf F, Praz F, et al. 2021 ESC/EACTS Guidelines for the management of valvular heart disease: Developed by the Task Force for the management of valvular heart disease of the European Society of Cardiology (ESC) and the European Association for Cardio-Thoracic Surgery (EACTS). Rev Esp Cardiol (Engl Ed). 2022;75(6):524\\u003c/li\\u003e\\n\\u003cli\\u003eLabkoff S, Oladimeji B, Kannry J, et al. Toward a responsible future: recommendations for AI-enabled clinical decision support. J Am Med Inform Assoc. 2024;31(11):2730-2739. \\u003c/li\\u003e\\n\\u003cli\\u003eMehta R, Reitz JG, Venna A, et al. Navigating the future of pediatric cardiovascular surgery: Insights and innovation powered by Chat Generative Pre-Trained Transformer (ChatGPT). J Thorac Cardiovasc Surg. Published online February 1, 2025. \\u003c/li\\u003e\\n\\u003cli\\u003eAydin S, Karabacak M, Vlachos V, Margetis K. Large language models in patient education: a scoping review of applications in medicine. Front Med (Lausanne). 2024;11:1477898. \\u003c/li\\u003e\\n\\u003cli\\u003eKoo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research [published correction appears in J Chiropr Med. 2017 Dec;16(4):346. doi: 10.1016/j.jcm.2017.10.001.]. J Chiropr Med. 2016;15(2):155-163\\u003c/li\\u003e\\n\\u003cli\\u003eZheng C, Ye H, Guo J, et al. Development and evaluation of a large language model of ophthalmology in Chinese. Br J Ophthalmol. 2024;108(10):1390-1397. \\u003c/li\\u003e\\n\\u003cli\\u003eRahimli Ocakoglu S, Coskun B. The Emerging Role of AI in Patient Education: A Comparative Analysis of LLM Accuracy for Pelvic Organ Prolapse. Med Princ Pract. Published online March 25, 2024. \\u003c/li\\u003e\\n\\u003cli\\u003eLee TJ, Campbell DJ, Rao AK, et al. Evaluating ChatGPT Responses on Atrial Fibrillation for Patient Education. Cureus. 2024;16(6):e61680.\\u003c/li\\u003e\\n\\u003cli\\u003eRao A, Pang M, Kim J, et al. Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study. J Med Internet Res. 2023;25:e48659. Published 2023 Aug 22.\\u003c/li\\u003e\\n\\u003cli\\u003eZhao, Yi-Fei, et al. \\u0026quot;Generative AI Is Not Ready for Clinical Use in Patient Education for Lower Back Pain Patients, Even With Retrieval-Augmented Generation.\\u0026quot; arXiv preprint arXiv:2409.15260 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eGe J, Sun S, Owens J, et al. Development of a liver disease-specific large language model chat interface using retrieval-augmented generation. Hepatology. 2024;80(5):1158-1168. \\u003c/li\\u003e\\n\\u003cli\\u003eWoo JJ, Yang AJ, Olsen RJ, et al. Custom Large Language Models Improve Accuracy: Comparing Retrieval Augmented Generation and Artificial Intelligence Agents to Noncustom Models for Evidence-Based Medicine. Arthroscopy. 2025;41(3):565-573.e6. \\u003c/li\\u003e\\n\\u003cli\\u003eAzamfirei R, Kudchadkar SR, Fackler J. Large language models and the perils of their hallucinations. Crit Care. 2023;27(1):120. Published 2023 Mar 21.\\u003c/li\\u003e\\n\\u003cli\\u003eHanci V, Erg\\u0026uuml;n B, G\\u0026uuml;l S, Uzun \\u0026Ouml;, Erdemir I, Hanci FB. Assessment of readability, reliability, and quality of ChatGPT\\u0026reg;, BARD\\u0026reg;, Gemini\\u0026reg;, Copilot\\u0026reg;, Perplexity\\u0026reg; responses on palliative care. Medicine (Baltimore). 2024;103(33):e39305. \\u003c/li\\u003e\\n\\u003cli\\u003eNian PP, Saleet J, Magruder M, et al. ChatGPT as a Source of Patient Information for Lumbar Spinal Fusion and Laminectomy: A Comparative Analysis Against Google Web Search. Clin Spine Surg. 2024;37(10):E394-E403. \\u003c/li\\u003e\\n\\u003cli\\u003eZhang Y, Wan XH, Kong QZ, et al. Evaluating large language models as patient education tools for inflammatory bowel disease: A comparative study. World J Gastroenterol. 2025;31(6):102090. \\u003c/li\\u003e\\n\\u003cli\\u003eJin H, Guo J, Lin Q, Wu S, Hu W, Li X. Comparative study of Claude 3.5-Sonnet and human physicians in generating discharge summaries for patients with renal insufficiency: assessment of efficiency, accuracy, and quality. Front Digit Health. 2024;6:1456911. Published 2024 Dec 5. \\u003c/li\\u003e\\n\\u003cli\\u003eZhui L, Fenghe L, Xuehu W, Qining F, Wei R. Ethical Considerations and Fundamental Principles of Large Language Models in Medical Education: Viewpoint. J Med Internet Res. 2024;26:e60083. \\u003c/li\\u003e\\n\\u003c/ol\\u003e\"}],\"fulltextSource\":\"\",\"fullText\":\"\",\"funders\":[],\"hasAdminPriorityOnWorkflow\":false,\"hasManuscriptDocX\":true,\"hasOptedInToPreprint\":true,\"hasPassedJournalQc\":\"\",\"hasAnyPriority\":false,\"hideJournal\":false,\"highlight\":\"\",\"institution\":\"\",\"isAcceptedByJournal\":false,\"isAuthorSuppliedPdf\":false,\"isDeskRejected\":\"\",\"isHiddenFromSearch\":false,\"isInQc\":false,\"isInWorkflow\":false,\"isPdf\":false,\"isPdfUpToDate\":true,\"isWithdrawnOrRetracted\":false,\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"bmc-medical-informatics-and-decision-making\",\"isNatureJournal\":false,\"hasQc\":true,\"allowDirectSubmit\":false,\"externalIdentity\":\"midm\",\"sideBox\":\"Learn more about [BMC Medical Informatics and Decision Making](http://bmcmedinformdecismak.biomedcentral.com/)\",\"snPcode\":\"\",\"submissionUrl\":\"https://www.editorialmanager.com/midm/default.aspx\",\"title\":\"BMC Medical Informatics and Decision Making\",\"twitterHandle\":\"BMC_series\",\"acdcEnabled\":true,\"dfaEnabled\":false,\"editorialSystem\":\"em\",\"reportingPortfolio\":\"BMC Series\",\"inReviewEnabled\":true,\"inReviewRevisionsEnabled\":true},\"keywords\":\"Large Language Models, Artificial Intelligence, Mitral Valve Surgery, Patient Education, Accuracy, Readability\",\"lastPublishedDoi\":\"10.21203/rs.3.rs-6965764/v1\",\"lastPublishedDoiUrl\":\"https://doi.org/10.21203/rs.3.rs-6965764/v1\",\"license\":{\"name\":\"CC BY 4.0\",\"url\":\"https://creativecommons.org/licenses/by/4.0/\"},\"manuscriptAbstract\":\"\\u003ch2\\u003eBackground\\u003c/h2\\u003e\\u003cp\\u003eCurrently, large language models (LLMs) supported by artificial intelligence (AI) are increasingly being utilized in patient education and information delivery within healthcare services. The aim of this study was to perform a comparative analysis of five different LLMs (\\u003cem\\u003ei.e.\\u003c/em\\u003e, ChatGPT-4o, Claude 3.7 Sonnet, Gemini 2.5 Pro Preview, DeepSeek-V3, and Microsoft Copilot) in terms of accuracy, completeness, and readability, based on their responses to frequently asked questions in preoperative patient education for mitral valve surgery (MVS).\\u003c/p\\u003e\\u003ch2\\u003eMethods\\u003c/h2\\u003e\\u003cp\\u003eA standardized questionnaire comprising seven frequently asked questions by patients prior to MVS was developed. These questions were presented to each LLM in an identical manner. The responses were evaluated by two academic experts in cardiac surgery using structured assessment criteria across three main dimensions: accuracy, completeness, and readability. For the readability analysis, the Simplified Measure of Gobbledygook (SMOG) Index and the Flesch-Kincaid Reading Ease (FRE) scale were utilized.\\u003c/p\\u003e\\u003ch2\\u003eResults\\u003c/h2\\u003e\\u003cp\\u003eThe ChatGPT-4o and Gemini models received statistically significantly higher scores in terms of accuracy and completeness (p\\u0026thinsp;\\u0026lt;\\u0026thinsp;0.05), while the Claude 3.7 Sonnet model achieved the highest readability scores (p\\u0026thinsp;\\u0026lt;\\u0026thinsp;0.001). This model provided reader-friendly content using simpler and more comprehensible sentence structures. The Gemini and DeepSeek models demonstrated moderate performance, whereas the Microsoft Copilot model showed limitations in semantic coherence and medical specificity. Some models were found to provide misleading or incomplete information regarding surgical risks, the postoperative course, and potential complications.\\u003c/p\\u003e\\u003ch2\\u003eConclusions\\u003c/h2\\u003e\\u003cp\\u003eThe LLMs represent valuable supplementary tools in patient education processes. However, their implementation in clinical practice must be carefully evaluated, particularly with regard to accuracy and completeness. This study highlights the potential applicability of ChatGPT-4o and Claude models for preoperative patient education in MVS, while emphasizing that all LLMs should be used under the supervision and guidance of healthcare professionals. For LLMs to be reliably utilized in the medical field, improvement in medical accuracy and standardization are essential.\\u003c/p\\u003e\",\"manuscriptTitle\":\"A Comparative Performance Analysis of AI-Assisted Language Models in Preoperative Patient Education for Mitral Valve Surgery\",\"msid\":\"\",\"msnumber\":\"\",\"nonDraftVersions\":[{\"code\":1,\"date\":\"2025-09-09 11:30:54\",\"doi\":\"10.21203/rs.3.rs-6965764/v1\",\"editorialEvents\":[{\"type\":\"communityComments\",\"content\":0},{\"type\":\"decision\",\"content\":\"Revision requested\",\"date\":\"2026-01-07T11:55:18+00:00\",\"index\":\"\",\"fulltext\":\"\"},{\"type\":\"reviewerAgreed\",\"content\":\"327343666310113847426023013310264918575\",\"date\":\"2026-01-05T00:22:42+00:00\",\"index\":\"hide\",\"fulltext\":\"\"},{\"type\":\"editorInvitedReview\",\"content\":\"\",\"date\":\"2026-01-03T13:38:40+00:00\",\"index\":\"hide\",\"fulltext\":\"\"},{\"type\":\"reviewerAgreed\",\"content\":\"315997904918063431310266763335695478574\",\"date\":\"2026-01-03T11:04:31+00:00\",\"index\":\"hide\",\"fulltext\":\"\"},{\"type\":\"editorInvitedReview\",\"content\":\"\",\"date\":\"2026-01-02T16:59:04+00:00\",\"index\":\"hide\",\"fulltext\":\"\"},{\"type\":\"reviewerAgreed\",\"content\":\"205400690756315820498856287986246459691\",\"date\":\"2026-01-01T13:11:23+00:00\",\"index\":\"hide\",\"fulltext\":\"\"},{\"type\":\"reviewerAgreed\",\"content\":\"294572433165394888407511874242065553483\",\"date\":\"2025-09-10T09:34:50+00:00\",\"index\":\"hide\",\"fulltext\":\"\"},{\"type\":\"reviewersInvited\",\"content\":\"\",\"date\":\"2025-08-29T10:54:22+00:00\",\"index\":\"\",\"fulltext\":\"\"},{\"type\":\"editorAssigned\",\"content\":\"\",\"date\":\"2025-08-28T05:43:35+00:00\",\"index\":\"\",\"fulltext\":\"\"},{\"type\":\"editorInvited\",\"content\":\"\",\"date\":\"2025-08-04T10:01:30+00:00\",\"index\":\"\",\"fulltext\":\"\"},{\"type\":\"checksComplete\",\"content\":\"\",\"date\":\"2025-08-01T12:49:17+00:00\",\"index\":\"\",\"fulltext\":\"\"},{\"type\":\"submitted\",\"content\":\"BMC Medical Informatics and Decision Making\",\"date\":\"2025-08-01T12:46:09+00:00\",\"index\":\"\",\"fulltext\":\"\"}],\"status\":\"published\",\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"bmc-medical-informatics-and-decision-making\",\"isNatureJournal\":false,\"hasQc\":true,\"allowDirectSubmit\":false,\"externalIdentity\":\"midm\",\"sideBox\":\"Learn more about [BMC Medical Informatics and Decision Making](http://bmcmedinformdecismak.biomedcentral.com/)\",\"snPcode\":\"\",\"submissionUrl\":\"https://www.editorialmanager.com/midm/default.aspx\",\"title\":\"BMC Medical Informatics and Decision Making\",\"twitterHandle\":\"BMC_series\",\"acdcEnabled\":true,\"dfaEnabled\":false,\"editorialSystem\":\"em\",\"reportingPortfolio\":\"BMC Series\",\"inReviewEnabled\":true,\"inReviewRevisionsEnabled\":true}}],\"origin\":\"\",\"ownerIdentity\":\"a540bbe2-5a6d-46af-a734-a11dfe4d75c3\",\"owner\":[],\"postedDate\":\"September 9th, 2025\",\"published\":true,\"recentEditorialEvents\":[],\"rejectedJournal\":[],\"revision\":\"\",\"amendment\":\"\",\"status\":\"in-revision\",\"subjectAreas\":[],\"tags\":[],\"updatedAt\":\"2026-05-07T05:09:49+00:00\",\"versionOfRecord\":[],\"versionCreatedAt\":\"2025-09-09 11:30:54\",\"video\":\"\",\"vorDoi\":\"\",\"vorDoiUrl\":\"\",\"workflowStages\":[]},\"version\":\"v1\",\"identity\":\"rs-6965764\",\"journalConfig\":\"researchsquare\"},\"__N_SSP\":true},\"page\":\"/article/[identity]/[[...version]]\",\"query\":{\"redirect\":\"/article/rs-6965764\",\"identity\":\"rs-6965764\",\"version\":[\"v1\"]},\"buildId\":\"XKTyCvWXoU3ODBz1xrDgd\",\"isFallback\":false,\"isExperimentalCompile\":false,\"dynamicIds\":[84888],\"gssp\":true,\"scriptLoader\":[]}","source_license":"CC-BY-4.0","license_restricted":false}