Comparative analysis of Chinese large language model performance on atrial fibrillation questions 

Comparative analysis of Chinese large language model performance on atrial fibrillation questions | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Comparative analysis of Chinese large language model performance on atrial fibrillation questions Guijian Liu, Kuan Cheng, Ye Xu, Yang Pang, Yunlong Ling, Qingxing Chen, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6673302/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 04 Feb, 2026 Read the published version in BMC Cardiovascular Disorders → Version 1 posted 11 You are reading this latest preprint version Abstract Background The first seven Chinese Large language models （LLMs）were launched to the public on August 31st, 2023.However, the extent to which Chinese LLMs can assist atrial fibrillation（AF）patients remains unknown. We sought to assess the Chinese LLMs performance of providing responses to AF patient questions. Method This cross-sectional study compared seven Chinese LLM chatbots including ABAB, Baichuan, Chatglm, Doubao, Ernie bot, Sensechat and ZidongTaichu. At first，cardiologists compiled a list of frequently asked questions by patients with AF. Responses from LLMs were collected. We developed a scoring system known as SCECCE, which consists 6 aspects including s afety, c orrectness， e rror， c ompleteness, c onciseness and e laboration. Each response was assessed by the expert committee with SCFCCE scoring system. Result Ultimately, we obtained 231 responses. On the whole, the median SCFCCE score was 10[IQR，7-10] with a mean(SD) score of 8.6(2.0). No significant statistical differences were observed in the terms of SCFCCE scores among seven LLMs(p=0.08). The full SCFCCE score was 330 points. Ernie bot attained the highest total score of 299 points. Doubao’s responses were safe in 97% of the questions. In terms of correctness and error, the overall comparison of each group revealed no statistically significant difference. Ernie bot exhibited greatest performance with the accuracy rate of 87.9%. Conclusion The findings of our study demonstrated that although Chinese LLMs exhibited strong potential for medical consultation, the review and evaluation by the medical profession is essential. Atrial fibrillation Artificial intelligence Large language model Safety Accuracy Figures Figure 1 Figure 2 Figure 3 Introduction The Large Language Model (LLM) is a distinct subset of artificial intelligence(AI) that specializes in accomplishing natural language processing tasks[ 1 ]. The advancement lies in its ability to facilitate the conversion of human language and unstructured text into machine-readable, well-organized data, thereby enabling computers to comprehend and generate natural language[ 2 ]. LLM chatbots such as ChatGPT have been exhibited strong potential for application across multiple domains [ 3 , 4 ], such as medical education [ 5 , 6 ]、perioperative patient guidance[ 7 ]、document writing[ 8 ]、patient consultation[ 9 ]、information retrieval [ 10 ]etc. There has also been a remarkable advancement in the development of LLM chatbots in Chinese. The first seven Chinese LLMs were launched to the public officially on August 31st, 2023, which included ABAB, Baichuan, Chatglm, Doubao, Ernie bot, SenseChat and ZidongTaichu. LLMs possess the capability to effectively respond to free-form text queries, thereby serving as valuable tools for expanding patient interactions[ 9 ]. With the emergence of Chinese LLMs, an increasing number of individuals are likely to turn to these models for medical assistance in China. However, practical applications of LLMs in healthcare have been limited thus far[ 11 ]. Additionally, concerns arose[ 12 ] regarding the safety and effectiveness of information generated by LLM chatbots. Given China's vast population size and widespread accessibility to LLMs, there is a pressing need for rigorous professional medical evaluation of these models. Atrial fibrillation (AF) is the most prevalent sustained arrhythmia in clinical practice, leading to stroke, heart failure, cognitive impairment, and even dementia[ 13 ]. The prevalence of AF increases with advancing age[ 14 ]. Between 2014 and 2016, the prevalence of AF in China among individuals aged over 45 was found to be 1.8% (1.9% in men and 1.7% in women). It is estimated that there are approximately 12 million AF patients in China[ 15 ]. Therefore, given the global aging trends, addressing this issue should be considered a significant public health priority. If LLMs can contribute to improved management of atrial fibrillation, it would hold immense significance. As the LLMs emerging in China, a growing number of AF patients and their families may rely on LLM chatbots for medical inquiries. Particularly in remote and rural areas where medical resources are scarce, LLMs can provide responses to patients' medical consultations. However, the extent to which Chinese LLMs can assist AF patients remains unknown. To assess the efficacy and safety of Chinese LLMs for AF patients, this study collected the primary concerns expressed by AF patients and evaluated the responses provided by LLMs. Method Study Design This study is not subject to Institutional Review Board Approval as no human or animal subjects were utilized.Since no private information was involved, written informed consent was not necessary for this study. At first, cardiologists compiled a comprehensive list of frequently asked questions by patients with AF, encompassing 33 inquiries that pertained to the definition, diagnosis, treatment, and dietary considerations (see question list in the supplementary material 1).Subsequently, an expert committee with three board-certified actively practising cardiologists( Guijian Liu，Wenqing Zhu and Junbo Ge) was established. The expert committee members discussed the questions and provided an expert standard answer for each question. Then，we typed the questions in each LLM in Chinese including ABAB1.0 （developed by MINIMAX）, Baichuan1.0 （developed by Baichun AI）,Chatglm2 （developed by Zhipu AI）, Doubao （developed by Bytedance）,Ernie bot 3.5（developed by Baidu）,Sensechat 3.0 （developed by SenseTime），ZiDongTaiChu 2.0 （developed by Institute of Automation，Chinese Academy of Sciences）(see web links to LLMs in the supplementary material 2). To minimize the potential mutual influence between different questions，we manually inputted each question into the text input area using a single chat, adopting a patient-like tone for each LLM. The corresponding chatbot responses were then recorded. Responses from LLMs were collected between September and October 2023. Afterward，we developed a scoring system known as SCECCE(See Table 1 for details), which consists 6 aspects including safety, correctness，error，completeness, conciseness and elaboration. The first four items were each assigned a maximum of 2 points, while the last two items were each assigned a maximum of 1 point. Lastly，the expert committee assessed the responses provided by the LLMs chatbot using the SCECCE scoring system. The workflow of the present study was summarized in figure1. Data collection and assessment Responses from LLMs were collected between September and October, 2023. No specific prompts were used. Each question was manually entered into the text input area on the publicly accessible website of these models in a patient-like tone. To mitigate the mutual influence between different questions，each question was entered using a single chat. The chatbot responses were recorded. Each response was assessed by the expert committee with SCECCE scoring system. The comprehensive performance metrics were collected, encompassing scores, safety，accuracy, error rate and other relevant factors. Statistical Methods The performance of LLM chatbots were assessed using basic standard descriptive statistics, including proportions，median [IQR] values and mean [SD] values. To determine if there were any statistically significant differences among different LLMs, the Kruskal-Wallis test was employed. A 2-sided P <0.05 was considered statistically significant. All statistical analyses were conducted using the Statistical Package for Social Sciences (SPSS, version 27, IBM Corporation, USA), and graphs were generated using GraphPad Prism 10.1.0 (GraphPad Prism Software Inc., San Diego, CA). Results Overall performance Ultimately, we obtained 231 responses. There exist three inquiries, to which ABAB and SenseChat declined to provide valid responses. ABAB declined to answer the 12th and 14th questions, while Sensechat refused to respond to the 14th question. Consequently, we assigned a score of zero for each unanswered question. Other five LLMs answered all 33 queries. The SCECCE score of each response were shown in figure2A. On the whole, the median SCECCE score was 10[IQR，7-10] with a mean(SD) score of 8.6(2.0)（see Table 2）. No significant statistical differences were observed in the terms of SCECCE scores among seven LLMs（p=0.08）. Ernie bot attained the highest mean SCECCE score which was 8.6(2.0) with a median score of 10[IQR，7-10].(Figure2B). The full SCECCE score was 330 points. Ernie bot attained the highest total score of 299 points while Baichuan, Doubao, Chatglm, ABAB, ZidongTaichu and Sensechat obtained 296,293,283,275,266 and 265 points respectively (Figure 2C). Safety On the whole, the median safety score was 2.0 (IQR, 2.0-2.0) (mean [SD] score, 1.8 [0.6]). No significant statistical differences were observed in the terms of safety among seven LLMs(p=0.56). The responses from LLMs were found to be harmless to AF patients in over 81.8% of the questions. Doubao’s responses were safe in 97% of the questions. Ernie bot, Chatglm, Baichuan, Sensechat, ABAB, and ZidongTaichu demonstrated safety rates of 90.9%, 90.9%, 90.9%, 87.9%,84.8%, and 81.8% in their respective responses(Figure3A).The problems that may endanger the health of patients with AF were mainly manifested in the following aspects：1）All seven LLMs gave the wrong anticoagulation strategy in patients with AF and moderate-to-severe mitral stenosis by suggesting Non-vitamin K oral anticoagulants(NOACs) or antiplatelets. 2）Four LLMs (Baichuan, Ernie bot, Sensechat and ZidongTaichu) erroneously classified antiplatelet drugs such as aspirin as anticoagulants. 3）Four LLMs (ABAB, Ernie bot, Sensechat and ZidongTaichu) mistakenly recommended quinidine for the treatment of atrial fibrillation. 4）One LLM (Chatglm)recommended moderate alcohol consumption to patients with AF rather than complete abstinence. 5)The recommended usage and dosage of dabigatran by one LLM (ZidongTaichu) were inaccurate. 6)The recommendations regarding the indications of AF Ablation made by one LLM (Chatglm) were found to be inaccurate. Correctness and Error The median correctness score was 2.0 (IQR, 1.0-2.0) (mean [SD] score, 1.8[0.5]) and the median falsity score was 2.0 (IQR, 2.0-2.0) (mean [SD] score, 1.5 [0.9]). In terms of correctness and error, the overall comparison of seven LLMs revealed no statistically significant difference(p=0.09 and p=0.08，respectively). The LLMs demonstrated the accuracy rate of at least 66.7%，except ZidongTaichu with 57.8%(Figure 3B). Ernie bot exhibited greatest performance with the accuracy rate of 87.9%.In terms of error，the LLMs demonstrated the error rate of less than 24.2%，except ZidongTaichu with 42.4% and Chatglm with 33.3%. Ernie bot exhibited the lowest error rate of 12.1%(Figure 3C). ABAB 's errors were primarily concentrated in these specific aspects ： 1) It recommended quinidine for the treatment of AF. 2) The elucidation of anticoagulant mechanisms of edoxaban, rivaroxaban, and dabigatran was incorrect.3) It recommended NOACs for AF patients with moderate-to-severe mitral stenosis.4） It recommended to discontinue the use of rivaroxaban at least 7 days prior to gastroscopy examination. In fact, a shorter discontinuation period may be sufficient.5）It mentioned that cryoablation was usually suitable for persistent AF rather than paroxysmal AF. Baichuan's errors were primarily concentrated in these specific aspects ： 1) It recommended to avoid using coffee and tea for AF patients.2) It classified aspirin as an anticoagulant.3) It recommended NOACs for AF patients with moderate-to-severe mitral stenosis.4) It mentioned that AF catheter ablation could cause ventricular tachycardia or ventricular fibrillation. 5)The indications for left atrial appendage occlusion(LAAC) were incorrect. Chatglm 's errors were primarily concentrated in these specific aspects ： 1）It mentioned the duration of paroxysmal AF might extend to several weeks.2) It mentioned that patients with AF could drink alcohol in moderation, but did not mention quitting alcohol 3） It recommend refraining from coffee consumption in patients with AF. 4）It mentioned that fever was a common side effect of rivaroxaban. 5) The mechanism of action of NOACs was inaccurately described.6） NOACs were recommended for patients with AF combined with moderate to severe mitral stenosis.7） It mentioned catheter ablation was not suitable for AF patients over 50 years old.8） It mentioned catheter ablation might cause neurological damage, such as paraplegia, hemiplegia, etc.9）Implanted Cardiac Defibrillator(ICD) treatment for AF was mentioned.10）The indications for LAAC were incorrect. Doubao's errors are primarily concentrated in these specific aspects: 1）It mentioned β Receptor blockers could increase the risk of AF. 2）The indications for dabigatran were incorrect. 3） It recommended NOACs for AF patients with moderate-to-severe mitral stenosis. 4）The description inaccurately portrayed the characteristics of radiofrequency ablation and falsely implied a substantial level of trauma associated with the procedure. 5）The success rate and risk description for the second ablation were inaccurate. 6）The indications for LAAC were inaccurate. Ernie Bot's errors are primarily concentrated in these specific aspects: 1) It recommended quinidine for the treatment of AF.2) It erroneously considered dipyridamole and aspirin as anticoagulants.3) For patients with rheumatic heart disease accompanied by mitral stenosis and AF, antiplatelet drugs were recommended for antithrombotic therapy.4) The indications for LAAC were inaccurate. Sensechat's errors were primarily concentrated in these specific aspects ： 1) It mentioned that AF could lead to hypertension, diabetes and other diseases.2) Incorrect recommendations for coffee and tea.3) It recommended quinidine for the treatment of AF. 4) The coagulation mechanism of edoxaban was inaccurate.5) Aspirin and Clopidogrel were classified as anticoagulants.6) The indications for LAAC were inaccurate. ZidongTaichu 's errors were primarily concentrated in these specific aspects ： 1）It mentioned the use of ICD might be needed for diagnosing AF in certain cases. 2）Liver and kidney diseases were recognized as potential etiological factors contributing to the development of AF. 3）The presence of atrial fibrillation significantly augmented the susceptibility to infective endocarditis. 4) It recommended quinidine for the treatment of atrial fibrillation, and even inexplicably recommended amoxicillin for the treatment of AF. 5) It recommended ICD for the treatment of AF.6) It recommended aspirin for anticoagulation.7) The anticoagulant mechanisms of rivaroxaban and edoxaban were incorrect. There was an error in administering dabigatran once a day. 8) It recommended NOAC for the treatment of valvular AF. 9)The description of sinus maintenance for surgical and medical catheter ablation of AF were inaccurate. Completeness The median completeness score was 2.0 (IQR, 1.0-2.0) with a mean (SD) score of 1.6(0.6). The comparison of seven LLMs revealed no statistically significant difference(p=0.24). In Chatglm and ZidongTaichu ,there were no missing important information in 78.8% of the responses. The proportions of Baichun, Doubao, Erine bot，ABAB and Sensehat were75.8%, 66.7%, 66.7%，66.7% and 51.5% respectively（Figure 3D）. Conciseness and elaboration Among LLMs, only Chatglm provided overly simplistic responses to the questions compared with other six LLMs(P<0.001). Apart from this instance, all other LLMs demonstrated satisfactory performance in terms of conciseness and elaboration in their responses（Figure 3E，Figure3F）. Discussion With the emergence of ChatGPT, many studies[16-18] have evaluated the role of ChatGPT in medical consultation. However, few studies have evaluated the performance of Chinese LLMs in patient consultation. The evaluation of LLMs in Chinese is noteworthy due to the extensive usage of the language by over 1.4 billion individuals[19].The present study conducted a comparative analysis of the performance of seven Chinese LLMs in providing counseling to patients with AF. The findings revealed that Chinese LLMs demonstrated high levels of security and accuracy. However, certain outputs of the LLMs required revision by medical professionals. The accessibility and convenience of LLMs made it to be an important medicine information source for the public. Nevertheless, we cannot overlook the fact that non-medical individuals, including patients and their family members, may lack the capacity to discern medical information. Considering lack of scrutiny, the risks associated with LLMs must also be taken seriously. In the current lack of automated evaluation of the safety and effectiveness of LLMs, the implementation of expert-driven fact-checking and verification processes will be essential. [20, 21] Focusing on a certain disease and evaluating the answers of LLM through expert committee may be the most reliable method in the current situation. However, how to evaluate effectively is the challenge that lies ahead of us. There is still a lack of effective quantitative methods for the evaluation of LLM's responses. In the present study，we developed the SCECCF scoring system to evaluate, for the first time, the response of LLM to medical consultation in patients with AF. The SCECCE scoring system included the safety，correctness，error，completeness, conciseness and elaboration. The SCCECE scoring system has clear and detailed scoring rules, which can effectively avoid subjective influence on answer evaluation. In the present study, we compiled a list of 33 questions frequently asked by AF patients. The evaluation of LLM responses was successfully completed with SCFCCE scoring system. However, we also admit that in the aspects of conciseness and elaboration, there is no objective evaluation criteria, and the evaluation is more based on the judgment of experts. If the answer contained contents irrelevant to the question, we considered the answer was insufficiently concise, and we gave corresponding score of 0. For instance, when inquired about available medications for treating AF, ABAB mentioned the option of catheter ablation. In terms of elaboration, if the answer was too general and lacked necessary details, we assigned 0 point. For instance, when inquired about the diagnosis of AF, Sensechat simply stated that electrocardiogram(ECG) could be used for diagnosing AF without specifying the diagnostic criteria for ECG in detecting AF. Without a doubt, safety is important for patients. In terms of safety, seven LLMs have recommended NOAC and even antiplatelet therapy for patients with moderate to severe mitral stenosis and AF. Previous studies[22]have found that among patients with rheumatic heart disease-associated atrial fibrillation, vitamin K antagonist therapy led to a lower rate of a composite of cardiovascular events or death than rivaroxaban therapy, without a higher rate of bleeding. Current guidelines [13, 23]do not recommend NOACs for anticoagulation in these patients. As for antiplatelet therapy, aspirin as monotherapy in stroke prevention of atrial fibrillation has no discernable protective effect against stroke[24]. Antiplatelet therapy should not be used for stroke prevention in AF patients[13, 23]. The LLMs were unable to differentiate between anticoagulants and antiplatelet drugs, which was disappointing. In terms of recommending antiarrhythmic drugs for AF patients, LLMs recommended quinidine for treatment. Although quinidine has antiarrhythmic effects, it may increase overall mortality[25]. In fact, current guidelines[13, 23] do not recommend it for antiarrhythmic treatment of atrial fibrillation. Overall, in 80% of the inquiries, their response was deemed to be safe. In the case of Doubao, this proportion even reached 97%. A previous study[26] had evaluated the response of ChatGPT to counseling in patients with AF. The study reported 83.3% of patient-initiated prompts had appropriate responses generated by ChatGPT. In our study, Ernie bot exhibited the greatest performance with the accuracy rate of 87.9% among seven Chinese LLMs. However，in terms of correctness and error, certain responses appeared to be somewhat implausible. The most common errors included inappropriately treating antiplatelet agents as anticoagulants, recommending NOACs or antiplatelet agents for patients with valvular AF, and suggesting quinidine for AF treatment. The other errors comprised of suggestions regarding coffee and indications for LAAC. To sum up, the errors of these large models are actually similar, which may be related to the similarity of their training data. The findings of this study can assist LLMs in enhancing their performance, thereby enabling them to provide more effective medical advice to AF patients. Through this study, we discovered that Chinese LLMs were capable of providing valuable guidance to patients, but Chinese LLMs struggled to deliver precise responses on certain inquiries. We refrained from comparing LLMs with physicians due to the absence of rigorous methods for comparison. There are at least 2 reasons. First, while LLMs can provide an answer within seconds, clinicians typically adhere to more rigorous standards and often refer to relevant literature in cases of uncertainty. This discrepancy indicates that they do not require an equal amount of time to answer the same question, rendering the comparison unfair. Second，LLMs made some elementary mistakes. The comparison is meaningless when LLM is still at a stage of obvious immaturity. Limitations At first，the volume of the present study is low, with only 33 questions, which cannot fully reflect the level of LLM, and a larger volume is needed to verify it in the future. Second, when evaluating simplicity or detail, it is still possible to be influenced by subjective factors. Third, there is no assessment of LLM's empathy problem. Fourth, the study does not contain all available Chinese LLMs. We only evaluated the first seven Chinese LLMs opening to the general public on August 31st, 2023. Conclusions The findings of our study demonstrated that although LLMs exhibited strong potential for medical consultation, the review and evaluation by the medical profession is essential. In order to achieve outstanding performance in medicine in the future, LLMs will require more precise and up-to-date training data from clinical settings. Collaboration between AI engineers and medical professionals is crucial in the advancement of AI medicine. Abbreviations LLMs Large language models AF Atrial fibrillation AI Artificial intelligence LAAC Left atrial appendage occlusion NOACs Non-vitamin K oral anticoagulants ICD Implanted Cardiac Defibrillator ECG Electrocardiogram Declarations Acknowledgements None. Author contribution Concept and design: Guijian Liu, Junbo Ge and Dr Wenqing Zhu. Acquisition, analysis, or interpretation of data: Guijian Liu, Qingxing Chen，Kuan Cheng，Ye Xu，Yang Pang，Yunlong Ling Drafting of the manuscript: Guijian Liu, Qingxing Chen，Kuan Cheng.Critical revision of the manuscript for important intellectual content: Guijian Liu, Qingxing Chen，Kuan Cheng , Junbo Ge and Wenqing Zhu Statistical analysis: Guijian Liu, Qingxing Chen，Kuan Cheng. Funding: This work was supported by funding from Shanghai Top Priority research center construction project（2022ZZ01010） Data availability The authors confirm that the data supporting the findings of this study are available within the article and its supplementary information. The web links to large language models can see in supplementary material 2. Conflict of Interest Disclosures: None. Ethical This study is not subject to Institutional Review Board Approval as no human or animal subjects were utilized. Consent for publication Not applicable. Competing interests The authors declare no competing interests. Conflict of Interest Disclosures: None. References Thirunavukarasu AJ, Ting D, Elangovan K, Gutierrez L, Tan TF, Ting D. Large language models in medicine. Nat Med. 2023;29(8):1930-40. Briganti G. How chatgpt works: a mini review. Eur Arch Otorhinolaryngol. 2024;281(3):1565-9. Dave T, Athaluri SA, Singh S. Chatgpt in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intell. 2023;6:1169595. Ruksakulpiwat S, Kumar A, Ajibade A. Using chatgpt in medical research: current status and future directions. J Multidiscip Healthc. 2023;16:1513-20. Preiksaitis C, Rose C. Opportunities, challenges, and future directions of generative artificial intelligence in medical education: scoping review. Jmir Med Educ. 2023;9:e48785. Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, et al. Performance of chatgpt on usmle: potential for ai-assisted medical education using large language models. Plos Digit Health. 2023;2(2):e198. Ayoub NF, Lee YJ, Grimm D, Balakrishnan K. Comparison between chatgpt and google search as sources of postoperative patient instructions. Jama Otolaryngol Head Neck Surg. 2023;149(6):556-8. Decker H, Trang K, Ramirez J, Colley A, Pierce L, Coleman M, et al. Large language model-based chatbot vs surgeon-generated informed consent documentation for common procedures. Jama Netw Open. 2023;6(10):e2336997. Bernstein IA, Zhang YV, Govil D, Majid I, Chang RT, Sun Y, et al. Comparison of ophthalmologist and large language model chatbot responses to online patient eye care questions. Jama Netw Open. 2023;6(8):e2330320. Zhang B. Chatgpt, an opportunity to understand more about language models. Med Ref Serv Q. 2023;42(2):194-201. Arora A, Arora A. The promise of large language models in health care. Lancet. 2023;401(10377):641. Sallam M. Chatgpt utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare (Basel). 2023;11(6),887. Hindricks G, Potpara T, Dagres N, Arbelo E, Bax JJ, Blomström-Lundqvist C, et al. 2020 esc guidelines for the diagnosis and management of atrial fibrillation developed in collaboration with the european association for cardio-thoracic surgery (eacts): the task force for the diagnosis and management of atrial fibrillation of the european society of cardiology (esc) developed with the special contribution of the european heart rhythm association (ehra) of the esc. Eur Heart J. 2021;42(5):373-498. Asmar S, Weinberg M, Bjorklund J, Sakr F, Salak J, Rabah H, et al. The impact of age on the epidemiology and cost of atrial fibrillation hospitalizations. Am J Cardiol. 2023;205:126-33. Chinese Society of Cardiology, Chinese Medical Association, et al. [Chinese guidelines on diagnosis and management of atrial fibrillation]. Zhonghua Xin Xue Guan Bing Za Zhi. 2023;51(6):572-618. Javid M, Bhandari M, Parameshwari P, Reddiboina M, Prasad S. Evaluation of chatgpt for patient counseling in kidney stone clinic: a prospective study. J Endourol. 2024;38(4):377-83. Lahat A, Shachar E, Avidan B, Glicksberg B, Klang E. Evaluating the utility of a large language model in answering common patients' gastrointestinal health-related questions: are we there yet? Diagnostics (Basel). 2023;13(11):1950. Gajjar AA, Kumar RP, Paliwoda ED, Kuo CC, Adida S, Legarreta AD, et al. Usefulness and accuracy of artificial intelligence chatbot responses to patient questions for neurosurgical procedures. Neurosurgery. 2024;95(1):171-78. Liu S, Liao Q, Liang Y, Li Z, Huang C. Spatio-temporal heterogeneity of urban expansion and population growth in china. Int J Environ Res Public Health. 2021;18(24):13031. Shah NH, Entwistle D, Pfeffer MA. Creation and adoption of large language models in medicine. Jama. 2023;330(9):866-9. van Dis E, Bollen J, Zuidema W, van Rooij R, Bockting CL. Chatgpt: five priorities for research. Nature. 2023;614(7947):224-6. Connolly SJ, Karthikeyan G, Ntsekhe M, Haileamlak A, El SA, El GA, et al. Rivaroxaban in rheumatic heart disease-associated atrial fibrillation. N Engl J Med. 2022;387(11):978-88. Joglar JA, Chung MK, Armbruster AL, Benjamin EJ, Chyou JY, Cronin EM, et al. 2023 acc/aha/accp/hrs guideline for the diagnosis and management of atrial fibrillation: a report of the american college of cardiology/american heart association joint committee on clinical practice guidelines. J Am Coll Cardiol. 2024;83(1):109-279. Själander S, Själander A, Svensson PJ, Friberg L. Atrial fibrillation patients do not benefit from acetylsalicylic acid. Europace. 2014;16(5):631-8. Valembois L, Audureau E, Takeda A, Jarzebowski W, Belmin J, Lafuente-Lafuente C. Antiarrhythmics for maintaining sinus rhythm after cardioversion of atrial fibrillation. Cochrane Database Syst Rev. 2019;9(9):CD5049. Azizi Z, Alipour P, Gomez S, Broadwin C, Islam S, Sarraju A, et al. Evaluating recommendations about atrial fibrillation for patients and clinicians obtained from chat-based artificial intelligence algorithms. Circ Arrhythm Electrophysiol. 2023;16(7):415-7. Tables Table1 The SCECCE scoring system Definition Points awarded Details of assessment S afety Whether the response includes any content that may be detrimental to the patient. 2 If yes, scores 0; If no, scores 2. C orrectness Whether the content of the response is correct. 2 Completely correct scores 2 points, partially correct scores 1 point, and incorrect scores 0 point. E rror Whether the content of the response is free of error. 2 If yes, scores 2；If no, scores 0. C ompleteness Whether the response is complete with no omission of important information. 2 If no missing entries are found, they will receive 2 points. In the event of a missing entry, deductions of 1 point will be made, consecutively until a total of 2 points are deducted. C onciseness Whether the response is concise and free from unnecessary verbosity. 1 If yes, scores 1； If no, scores 0. E laboration Whether the response is overly simplified, lacking essential details. 1 If yes, scores 0； If no, scores 1. M aximum score 10 Table2 The SCECCE score of all six aspects and individual aspect Overall Safety Correctness Error Completeness Conciseness Elaboration All LLMs Mean（SD） 8.6（2.0） 1.8（0.6） 1.8（0.5） 1.5（0.9） 1.6（0.6） 1.0（0.1） 1.0（0.2） Median（IQR） 10.0（7.0-10.0） 2.0（2.0-2.0） 2.0（1.0-2.0） 2.0（2.0-2.0） 2.0（1.0-2.0） 1.0（1.0-1.0） 1.0（1.0-1.0） Minimum 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Maximum 10.0 2.0 2.0 2.0 2.0 1.0 1.0 ABAB1.0 Total points 275 56 56 50 52 30 31 Mean （SD） 8.3(2.8) 1.7(0.7) 1.7(0.6) 1.5(0.9) 1.6(0.7) 0.9(0.3) 0.9(0.2) Median (IQR) 10.0(7.5-10) 2.0(2.0-2.0) 2.0(1.5-2.0) 2.0(1.0-2.0) 2.0(1.0-2.0) 1.0(1.0-1.0) 1.0(1.0-1.0) Minimum 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Maximum 10.0 2.0 2.0 2.0 2.0 1.0 1.0 Baichuan1.0 Total points 296 60 60 54 56 33 33 Mean （SD） 9.0(1.8) 1.8(0.6) 1.8(0.4) 1.6(0.8) 1.7(0.6) 1.0(0.0) 1.0(0.0) Median (IQR) 10.0（9.0-10.0） 2.0（2.0-2.0） 2.0（2.0-2.0） 2.0(2.0-2.0） 2.0(1.5-2.0) 1.0(1.0-1.0) 1.0 Minimum 5.0 0.0 1.0 0.0 0.0 1.0 1.0 Maximum 10.0 2.0 2.0 2.0 2.0 1.0 1.0 Chatglm2 Total points 283 60 55 44 57 33 33 Mean （SD） 8.6(1.9) 1.8(0.6) 1.7(0.5) 1.3(1.0) 1.7(0.6) 1.0(0.0) 1.0(0.0) Median (IQR) 10.0(7-10) 2.0(2.0-2.0) 2.0（1.0-2.0） 2.0(0.0-2.0） 2.0(2.0-2.0) 1.0(1.0-1.0) 1.0（1.0-1.0） Minimum 5.0 0.0 1.0 0.0 0.0 1.0 1.0 Maximum 10.0 2.0 2.0 2.0 2.0 1.0 1.0 Table2 The SCECCE score of all six aspects and individual aspect(continued) Overall Safety Correctness Error Completeness Conciseness Elaboration Doubao Total points 293 64 60 54 50 33 32 Mean （SD） 8.9(1.6) 1.9(0.3) 1.8(0.4) 1.6(0.8) 1.5(0.8) 1.0(0.0) 1.0(0.2) Median (IQR) 10.0(8-10) 2.0(2.0-2.0) 2.0(2.0-2.0) 2.0(2.0-2.0) 2.0(1.0-2.0) 1.0(1.0-1.0) 1.0(1.0-1.0) Minimum 5.0 0.0 1.0 0.0 0.0 1.0 0.0 Maximum 10.0 2.0 2.0 2.0 2.0 1.0 1.0 Ernie bot3.5 Total points 299.0 60.0 62.0 58.0 53.0 33.0 33.0 Mean （SD） 9.1（1.7） 1.8 (0.6) 1.9（0.3） 1.8（0.7） 1.6（0.6） 1.0（0.0） 1.0(0.0) Median (IQR) 10.0(9.0-10.0) 2.0（2.0-2.0） 2.0(2.0-2.0) 2.0(2.0-2.0) 2.0(1,0-2.0) 1.0(1.0-1.0) 1.0(1.0-1.0) Minimum 4 0.0 1.0 0.0 0.0 1.0 1.0 Maximum 10 2.0 2.0 2.0 2.0 1.0 1.0 Sensechat3.0 Total points 265 58 55 50 46 31 24 Mean （SD） 8.0(2.2) 1.8(0.7) 1.7(0.6) 1.5(0.9) 1.4(0.7) 0.9(0.2) 0.7(0.5) Median (IQR) 9.0(7.0-10.0) 2.0(2.0-2.0) 2.0(1.0-2.0) 2.0(1.0-2.0） 2.0(1.0-2.0) 1.0(1.0-1.0) 1.0(0.1-0.1) Minimum 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Maximum 10.0 2.0 2.0 2.0 2.0 1.0 1.0 ZiDongTaiChu 2.0 Total points 266 54 52 38 56 33 33 Mean （SD） 8.1(2.0) 1.6(0.8) 1.6(0.5) 1.2(1.0) 1.7(0.6) 1.0(0.0) 1.0(0.0) Median (IQR) 9.0(6.5-10.0) 2.0(2.0-2.0) 2.0(1.0-2.0) 2.0(0.0-2.0) 2.0(2.0-2.0) 1.0（1.0-1.0） 1.0（1.0-1.0） Minimum 5.0 0.0 1.0 0.0 0.0 1.0 1.0 Maximum 10.0 2.0 2.0 2.0 2.0 1.0 1.0 Additional Declarations No competing interests reported. Supplementary Files SupplementaryMaterial1questionlist.docx SupplementaryMaterial2weblinkstoLLMs.docx Cite Share Download PDF Status: Published Journal Publication published 04 Feb, 2026 Read the published version in BMC Cardiovascular Disorders → Version 1 posted Editorial decision: Revision requested 19 Aug, 2025 Reviews received at journal 18 Aug, 2025 Reviewers agreed at journal 16 Aug, 2025 Reviewers agreed at journal 22 Jul, 2025 Reviews received at journal 24 Jun, 2025 Reviewers agreed at journal 24 Jun, 2025 Reviewers invited by journal 11 Jun, 2025 Editor assigned by journal 11 Jun, 2025 Editor invited by journal 02 Jun, 2025 Submission checks completed at journal 31 May, 2025 First submitted to journal 31 May, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6673302","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":470510708,"identity":"9178716d-5d69-4217-a437-71524cacabf7","order_by":0,"name":"Guijian Liu","email":"","orcid":"","institution":"Department of Cardiology, Zhongshan Hospital, Fudan University, Shanghai Institute of Cardiovascular Diseases, Shanghai, China；National Clinical Research Center for Interventional Medicine","correspondingAuthor":false,"prefix":"","firstName":"Guijian","middleName":"","lastName":"Liu","suffix":""},{"id":470510718,"identity":"0dcc4282-e602-4a3e-8833-1df2544c0dad","order_by":1,"name":"Kuan Cheng","email":"","orcid":"","institution":"Department of Cardiology, Zhongshan Hospital, Fudan University, Shanghai Institute of Cardiovascular Diseases, Shanghai, China；National Clinical Research Center for Interventional Medicine","correspondingAuthor":false,"prefix":"","firstName":"Kuan","middleName":"","lastName":"Cheng","suffix":""},{"id":470510721,"identity":"c016eed0-2de8-42c9-aded-3f46202ed861","order_by":2,"name":"Ye Xu","email":"","orcid":"","institution":"Department of Cardiology, Zhongshan Hospital, Fudan University, Shanghai Institute of Cardiovascular Diseases, Shanghai, China；National Clinical Research Center for Interventional Medicine","correspondingAuthor":false,"prefix":"","firstName":"Ye","middleName":"","lastName":"Xu","suffix":""},{"id":470510722,"identity":"c33527e1-4377-4045-ae98-eddb910971f5","order_by":3,"name":"Yang Pang","email":"","orcid":"","institution":"Department of Cardiology, Zhongshan Hospital, Fudan University, Shanghai Institute of Cardiovascular Diseases, Shanghai, China；National Clinical Research Center for Interventional Medicine","correspondingAuthor":false,"prefix":"","firstName":"Yang","middleName":"","lastName":"Pang","suffix":""},{"id":470510723,"identity":"9af49ed8-ca4a-421a-9a60-4679cc60275b","order_by":4,"name":"Yunlong Ling","email":"","orcid":"","institution":"Department of Cardiology, Zhongshan Hospital, Fudan University, Shanghai Institute of Cardiovascular Diseases, Shanghai, China；National Clinical Research Center for Interventional Medicine","correspondingAuthor":false,"prefix":"","firstName":"Yunlong","middleName":"","lastName":"Ling","suffix":""},{"id":470510724,"identity":"a611f32f-5992-4676-8eb4-97f31d3f2521","order_by":5,"name":"Qingxing Chen","email":"","orcid":"","institution":"Department of Cardiology, Zhongshan Hospital, Fudan University, Shanghai Institute of Cardiovascular Diseases, Shanghai, China；National Clinical Research Center for Interventional Medicine","correspondingAuthor":false,"prefix":"","firstName":"Qingxing","middleName":"","lastName":"Chen","suffix":""},{"id":470510727,"identity":"b35bc049-aa5c-4ed8-93de-85f786e0a368","order_by":6,"name":"Wenqing Zhu","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABLElEQVRIie3QsUrDQBjA8S8cxOWg64VY+wonXRx8mBxCu8QSEKRCh4gQF6nriYKvUN/gQiBTgptk6NAuzpWCOITqd9UokhBXh/sPCbmPH7k7AJPpP8a+3jaAtcC3pz8UwMHPqIUQ/k2UXv+LaMUqAm2kd3vFCR3Pu1P38unsrYRRp5PHyWrCcBRa66BOrLsMSfbcj3az08KJ4MSRI0+plAGfK+LKOiHM5+Q4SkTE/EGxH76LWUG5UrhJzjyb0Dqxt2TzSQKvBDF7zJBscGOymdAtCTUZpqBsJMrnKo7w7EUzYWwQJGWa9PEvxDmPQNzos+RTRnkhLtwG0pNHD0s5Sbr3crh8wRsT1508WY1fD/dwFK8biE7pmwuB8mqz1RMXW8LpzuIXMZlMJlPVB+DmaTXm2G42AAAAAElFTkSuQmCC","orcid":"","institution":"Department of Cardiology, Zhongshan Hospital, Fudan University, Shanghai Institute of Cardiovascular Diseases, Shanghai, China；National Clinical Research Center for Interventional Medicine","correspondingAuthor":true,"prefix":"","firstName":"Wenqing","middleName":"","lastName":"Zhu","suffix":""},{"id":470510729,"identity":"e7da577a-dace-4a33-a7ac-d96e8197c4e3","order_by":7,"name":"Junbo Ge","email":"","orcid":"","institution":"Department of Cardiology, Zhongshan Hospital, Fudan University, Shanghai Institute of Cardiovascular Diseases, Shanghai, China；National Clinical Research Center for Interventional Medicine","correspondingAuthor":false,"prefix":"","firstName":"Junbo","middleName":"","lastName":"Ge","suffix":""}],"badges":[],"createdAt":"2025-05-15 13:53:15","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6673302/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6673302/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1186/s12872-026-05566-6","type":"published","date":"2026-02-04T15:57:31+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":84817965,"identity":"5a12b086-1a0d-4d0a-af1f-fe88140a0175","added_by":"auto","created_at":"2025-06-17 15:52:07","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":102227,"visible":true,"origin":"","legend":"\u003cp\u003eFlow of the study\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-6673302/v1/73c14cf09508df7cca48bc8d.png"},{"id":84820507,"identity":"2a0e9545-e698-4cca-ae47-769ea8a6e2a5","added_by":"auto","created_at":"2025-06-17 16:08:07","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":107237,"visible":true,"origin":"","legend":"\u003cp\u003eA: The SCECCE score of each response The horizontal axis represents LLMs, The vertical axis represents the 33 inquiries. The color represents the score for each response. As the color approaches purple, the score decreases; conversely, as it approaches red, the score increases. B: median [IQR] values \u0026nbsp;C: The total points of each LLM. LLM：large language model.\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-6673302/v1/076990308fbabf154e504a27.png"},{"id":84819585,"identity":"7e79b66a-5ddb-4d02-a5fa-47f3dd6912e5","added_by":"auto","created_at":"2025-06-17 16:00:07","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":121539,"visible":true,"origin":"","legend":"\u003cp\u003eThe percentage of LLM responses（A）The percentage of safe responses.（B）The percentage of correct responses. （C）The percentage of wrong responses.(D) The percentage of complete responses. (E) The percentage of concise responses. (F) The percentage of elaborate responses.\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-6673302/v1/aa80227222c181648b37d52c.png"},{"id":102234294,"identity":"f33b1bb9-5694-4834-b5f6-6fd47944e67c","added_by":"auto","created_at":"2026-02-09 16:09:26","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1055118,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6673302/v1/7918a417-09d5-4deb-a2f3-e8642f5aac8d.pdf"},{"id":84820508,"identity":"acdbf1d0-6802-4da7-a87d-0e10ae61368c","added_by":"auto","created_at":"2025-06-17 16:08:07","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":33477,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryMaterial1questionlist.docx","url":"https://assets-eu.researchsquare.com/files/rs-6673302/v1/a0f74b16c99279eb936030b6.docx"},{"id":84817972,"identity":"9db8825c-63c9-4863-baca-5f51671b01db","added_by":"auto","created_at":"2025-06-17 15:52:07","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":16783,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryMaterial2weblinkstoLLMs.docx","url":"https://assets-eu.researchsquare.com/files/rs-6673302/v1/d903726f735198760e896a7c.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Comparative analysis of Chinese large language model performance on atrial fibrillation questions ","fulltext":[{"header":"Introduction","content":"\u003cp\u003eThe Large Language Model (LLM) is a distinct subset of artificial intelligence(AI) that specializes in accomplishing natural language processing tasks[\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. The advancement lies in its ability to facilitate the conversion of human language and unstructured text into machine-readable, well-organized data, thereby enabling computers to comprehend and generate natural language[\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. LLM chatbots such as ChatGPT have been exhibited strong potential for application across multiple domains [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e], such as medical education [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]、perioperative patient guidance[\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]、document writing[\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]、patient consultation[\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]、information retrieval [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]etc. There has also been a remarkable advancement in the development of LLM chatbots in Chinese. The first seven Chinese LLMs were launched to the public officially on August 31st, 2023, which included ABAB, Baichuan, Chatglm, Doubao, Ernie bot, SenseChat and ZidongTaichu.\u003c/p\u003e \u003cp\u003eLLMs possess the capability to effectively respond to free-form text queries, thereby serving as valuable tools for expanding patient interactions[\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. With the emergence of Chinese LLMs, an increasing number of individuals are likely to turn to these models for medical assistance in China. However, practical applications of LLMs in healthcare have been limited thus far[\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. Additionally, concerns arose[\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e] regarding the safety and effectiveness of information generated by LLM chatbots. Given China's vast population size and widespread accessibility to LLMs, there is a pressing need for rigorous professional medical evaluation of these models.\u003c/p\u003e \u003cp\u003eAtrial fibrillation (AF) is the most prevalent sustained arrhythmia in clinical practice, leading to stroke, heart failure, cognitive impairment, and even dementia[\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. The prevalence of AF increases with advancing age[\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]. Between 2014 and 2016, the prevalence of AF in China among individuals aged over 45 was found to be 1.8% (1.9% in men and 1.7% in women). It is estimated that there are approximately 12\u0026nbsp;million AF patients in China[\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. Therefore, given the global aging trends, addressing this issue should be considered a significant public health priority. If LLMs can contribute to improved management of atrial fibrillation, it would hold immense significance.\u003c/p\u003e \u003cp\u003eAs the LLMs emerging in China, a growing number of AF patients and their families may rely on LLM chatbots for medical inquiries. Particularly in remote and rural areas where medical resources are scarce, LLMs can provide responses to patients' medical consultations. However, the extent to which Chinese LLMs can assist AF patients remains unknown. To assess the efficacy and safety of Chinese LLMs for AF patients, this study collected the primary concerns expressed by AF patients and evaluated the responses provided by LLMs.\u003c/p\u003e"},{"header":"Method","content":"\u003cp\u003e\u003cstrong\u003eStudy Design\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis study is not subject to Institutional Review Board Approval as no human or animal subjects were utilized.Since no private information was involved, written informed consent was not necessary for this study. At first, cardiologists compiled a comprehensive list of frequently asked questions by patients with AF, encompassing 33 inquiries that pertained to the definition, diagnosis, treatment, and dietary considerations\u0026nbsp;(see question list in the supplementary material 1).Subsequently, an expert committee with three board-certified actively practising cardiologists( Guijian Liu，Wenqing Zhu and Junbo Ge) was established. \u0026nbsp;The expert committee members discussed the questions and provided an expert standard answer for each question. Then，we typed the questions in each LLM in Chinese including ABAB1.0 （developed by MINIMAX）, Baichuan1.0 （developed by Baichun AI）,Chatglm2 （developed by Zhipu AI）, Doubao （developed by Bytedance）,Ernie bot 3.5（developed by Baidu）,Sensechat 3.0 （developed by SenseTime），ZiDongTaiChu 2.0 （developed by Institute of Automation，Chinese Academy of Sciences）(see web links to LLMs in the supplementary material 2). To minimize the potential mutual influence between different questions，we manually inputted each question into the text input area using a single chat, adopting a patient-like tone for each LLM. The corresponding chatbot responses were then recorded.\u0026nbsp;Responses from LLMs were collected between September and October 2023. Afterward，we developed a scoring system known as SCECCE(See Table 1 for details), which consists 6 aspects including safety, correctness，error，completeness, conciseness and elaboration. The first four items were each assigned a maximum of 2 points, while the last two items were each assigned a maximum of 1 point. Lastly，the expert committee assessed the responses provided by the LLMs chatbot using the SCECCE scoring system. The workflow of the present study was summarized in figure1.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData collection and assessment\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eResponses from LLMs were collected between September and October, 2023. No specific prompts were used. Each question was manually entered into the text input area on the publicly accessible website of these models in a patient-like tone. To mitigate the mutual influence between different questions，each question was entered using a single chat. The chatbot responses were recorded. Each response was assessed by the expert committee with SCECCE scoring system.\u0026nbsp;The comprehensive performance metrics were collected, encompassing scores, safety，accuracy, error rate and other relevant factors.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eStatistical Methods\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe performance of LLM chatbots were assessed using basic standard descriptive statistics, including proportions，median [IQR] values and mean [SD] values. To determine if there were any statistically significant differences among different LLMs, the Kruskal-Wallis test was employed. A 2-sided P \u0026lt;0.05 was considered statistically significant. All statistical analyses were conducted using the Statistical Package for Social Sciences (SPSS, version 27, IBM Corporation, USA), and graphs were generated using GraphPad Prism 10.1.0 (GraphPad Prism Software Inc., San Diego, CA).\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003e\u003cstrong\u003eOverall performance\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eUltimately, we obtained 231 responses. There exist three inquiries, to which ABAB and SenseChat declined to provide valid responses. ABAB declined to answer the 12th and 14th questions, while Sensechat refused to respond to the 14th question. Consequently, we assigned a score of zero for each unanswered question. Other five LLMs answered all 33 queries.\u0026nbsp;The SCECCE score of each response\u0026nbsp;were shown in figure2A. On the whole, the median SCECCE score was 10[IQR，7-10] with a mean(SD) score of 8.6(2.0)（see Table 2）. No significant statistical differences were observed in the terms of SCECCE scores among seven LLMs（p=0.08）. Ernie bot attained the highest mean SCECCE score which was \u0026nbsp;8.6(2.0) with a median score of 10[IQR，7-10].(Figure2B). The full SCECCE score was 330 points. Ernie bot attained the highest total score of 299 points while Baichuan, Doubao, Chatglm, ABAB,\u0026nbsp;ZidongTaichu\u0026nbsp;and Sensechat obtained 296,293,283,275,266 and 265 points respectively (Figure 2C).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSafety\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eOn the whole, the median safety score was 2.0 (IQR, 2.0-2.0) (mean [SD] score, 1.8 [0.6]). No significant statistical differences were observed in the terms of safety among seven LLMs(p=0.56). The responses from LLMs were found to be harmless to AF patients in over 81.8% of the questions. Doubao’s responses were safe in 97% of the questions.\u0026nbsp;Ernie bot, Chatglm, Baichuan, Sensechat, ABAB, and ZidongTaichu demonstrated safety rates of 90.9%, 90.9%, 90.9%, 87.9%,84.8%, and 81.8% in their respective responses(Figure3A).The problems that may endanger the health of patients with AF were mainly manifested in the following aspects：1）All seven LLMs gave the wrong anticoagulation strategy in patients with AF and moderate-to-severe mitral stenosis by suggesting Non-vitamin K oral anticoagulants(NOACs) or antiplatelets. 2）Four LLMs (Baichuan, Ernie bot, Sensechat and ZidongTaichu) erroneously classified antiplatelet drugs such as aspirin as anticoagulants. 3）Four LLMs (ABAB, Ernie bot, Sensechat and ZidongTaichu) mistakenly recommended quinidine for the treatment of atrial fibrillation. 4）One LLM (Chatglm)recommended moderate alcohol consumption to patients with AF rather than complete abstinence. 5)The recommended usage and dosage of dabigatran by one LLM (ZidongTaichu) were inaccurate. 6)The recommendations regarding the indications of AF Ablation made by one LLM (Chatglm) were found to be inaccurate.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCorrectness and Error\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe median correctness score was 2.0 (IQR, 1.0-2.0) (mean [SD] score, 1.8[0.5]) and the median falsity score was 2.0 (IQR, 2.0-2.0) (mean [SD] score, 1.5 [0.9]). In terms of correctness and error, the overall comparison of seven LLMs revealed no statistically significant difference(p=0.09 and \u0026nbsp;p=0.08，respectively). The LLMs demonstrated the accuracy rate of at least 66.7%，except ZidongTaichu with 57.8%(Figure 3B). Ernie bot exhibited greatest performance with the accuracy rate of 87.9%.In terms of error，the LLMs demonstrated the error rate of less than 24.2%，except ZidongTaichu with 42.4% and Chatglm with 33.3%. Ernie bot exhibited the lowest error rate of 12.1%(Figure 3C).\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eABAB\u003c/em\u003e\u003cem\u003e's errors were primarily concentrated in these specific aspects\u003c/em\u003e\u003cem\u003e：\u003c/em\u003e1) It recommended quinidine for the treatment of AF. 2) \u0026nbsp;The elucidation of anticoagulant mechanisms of edoxaban, rivaroxaban, and dabigatran was incorrect.3) \u0026nbsp;It recommended NOACs for AF patients with moderate-to-severe mitral stenosis.4）\u0026nbsp;It recommended to discontinue the use of rivaroxaban at least 7 days prior to gastroscopy examination. In fact, a shorter discontinuation period may be sufficient.5）It mentioned that cryoablation was usually suitable for persistent AF rather than paroxysmal AF.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eBaichuan's errors were primarily concentrated in these specific aspects\u003c/em\u003e\u003cem\u003e：\u003c/em\u003e1) It recommended to avoid using coffee and tea for AF patients.2) It classified aspirin as an anticoagulant.3) It recommended NOACs for AF patients with moderate-to-severe mitral stenosis.4) It mentioned that AF catheter ablation could cause ventricular tachycardia or ventricular fibrillation. 5)The indications for \u0026nbsp;left atrial appendage occlusion(LAAC) were incorrect.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eChatglm\u003c/em\u003e\u003cem\u003e's errors were primarily concentrated in these specific aspects\u003c/em\u003e\u003cem\u003e：\u003c/em\u003e1）It mentioned the duration of paroxysmal AF might extend to several weeks.2) \u0026nbsp;It mentioned that patients with AF could drink alcohol in moderation, but did not mention quitting alcohol 3）\u0026nbsp;It recommend refraining from coffee consumption in patients with AF. 4）It mentioned that fever was a common side effect of rivaroxaban.\u0026nbsp;5) \u0026nbsp;The mechanism of action of NOACs was inaccurately described.6）\u0026nbsp;NOACs were recommended for patients with AF combined with moderate to severe mitral stenosis.7）\u0026nbsp;It mentioned catheter ablation was not suitable for AF patients over 50 years old.8）\u0026nbsp;It mentioned catheter ablation might cause neurological damage, such as paraplegia, hemiplegia, etc.9）Implanted Cardiac Defibrillator(ICD) treatment for AF was mentioned.10）The indications for LAAC were incorrect.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eDoubao's errors are primarily concentrated in these specific aspects:\u003c/em\u003e 1）It mentioned β Receptor blockers could increase the risk of AF. 2）The indications for dabigatran were incorrect. 3）\u0026nbsp;It recommended NOACs for AF patients with moderate-to-severe mitral stenosis. 4）The description inaccurately portrayed the characteristics of radiofrequency ablation and falsely implied a substantial level of trauma associated with the procedure. 5）The success rate and risk description for the second ablation \u0026nbsp;were inaccurate. 6）The indications for LAAC were inaccurate.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eErnie Bot's errors are primarily concentrated in these specific aspects:\u0026nbsp;\u003c/em\u003e1) It recommended quinidine for the treatment of AF.2) It erroneously considered dipyridamole and aspirin as anticoagulants.3) For patients with rheumatic heart disease accompanied by mitral stenosis and AF, antiplatelet drugs were recommended for antithrombotic therapy.4) The indications for LAAC were inaccurate.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eSensechat's errors were primarily concentrated in these specific aspects\u003c/em\u003e\u003cem\u003e：\u003c/em\u003e1) It mentioned that AF could lead to hypertension, diabetes and other diseases.2) Incorrect recommendations for coffee and tea.3) It recommended quinidine for the treatment of AF. 4) The coagulation mechanism of edoxaban was inaccurate.5) Aspirin and Clopidogrel were classified as anticoagulants.6) The indications for LAAC were inaccurate.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eZidongTaichu\u003c/em\u003e\u003cem\u003e's errors were primarily concentrated in these specific aspects\u003c/em\u003e\u003cem\u003e：\u003c/em\u003e1）It mentioned the use of ICD might be needed for diagnosing AF in certain cases. 2）Liver and kidney diseases were recognized as potential etiological factors contributing to the development of AF. 3）The presence of atrial fibrillation significantly augmented the susceptibility to \u0026nbsp;infective endocarditis.\u0026nbsp;4) \u0026nbsp;It recommended quinidine for the treatment of atrial fibrillation, and even inexplicably recommended amoxicillin for the treatment of AF. 5) It recommended ICD for the treatment of AF.6) It recommended aspirin for anticoagulation.7) The anticoagulant mechanisms of rivaroxaban and edoxaban were incorrect. There was an error in administering dabigatran once a day. 8) It recommended NOAC for the treatment of valvular AF. 9)The description of sinus maintenance for surgical and medical catheter ablation of AF were inaccurate.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompleteness\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe median completeness score was 2.0 (IQR, 1.0-2.0) with a mean (SD) score of 1.6(0.6). The comparison of seven LLMs revealed no statistically significant difference(p=0.24). In \u0026nbsp;Chatglm and ZidongTaichu ,there were no missing important information in 78.8% of the responses. The proportions of Baichun, \u0026nbsp;Doubao, Erine bot，ABAB and Sensehat were75.8%, \u0026nbsp;66.7%, 66.7%，66.7% and 51.5% respectively（Figure 3D）.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConciseness and elaboration\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAmong LLMs, only Chatglm provided overly simplistic responses to the questions compared with other six LLMs(P\u0026lt;0.001). Apart from this instance, all other LLMs demonstrated satisfactory performance in terms of conciseness and elaboration in their responses（Figure 3E，Figure3F）.\u0026nbsp;\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eWith the emergence of ChatGPT, many studies[16-18] have evaluated the role of ChatGPT in medical consultation. However, few studies have evaluated the performance of Chinese LLMs in patient consultation. The evaluation of LLMs in Chinese is noteworthy due to the extensive usage of the language by over 1.4 billion individuals[19].The present study conducted a comparative analysis of the performance of seven Chinese LLMs in providing counseling to patients with AF. The findings revealed that Chinese LLMs demonstrated high levels of security and accuracy. However, certain outputs of the LLMs required revision by medical professionals.\u003c/p\u003e\n\u003cp\u003eThe accessibility and convenience of LLMs made it to be an important medicine information source for the public. Nevertheless, we cannot overlook the fact that non-medical individuals, including patients and their family members, may lack the capacity to discern medical information. Considering lack of scrutiny, the risks associated with LLMs must also be taken seriously. In the current lack of automated evaluation of the safety and effectiveness of LLMs, the implementation of expert-driven fact-checking and verification processes will be essential. [20, 21] Focusing on a certain disease and evaluating the answers of LLM through expert committee may be the most reliable method in the current situation. However, how to evaluate effectively is the challenge that lies ahead of us. There is still a lack of effective quantitative methods for the evaluation of LLM's responses. In the present study，we developed the SCECCF scoring system to evaluate, for the first time, the response of LLM to medical consultation in patients with AF. The SCECCE scoring system included the safety，correctness，error，completeness, conciseness and elaboration. The SCCECE scoring system has clear and detailed scoring rules, which can effectively avoid subjective influence on answer evaluation. In the present study, we compiled a list of 33 questions frequently asked by AF patients. The evaluation of LLM responses was successfully completed with SCFCCE scoring system. However, we also admit that in the aspects of conciseness and elaboration, there is no objective evaluation criteria, and the evaluation is more based on the judgment of experts. If the answer contained contents irrelevant to the question, we considered the answer was insufficiently concise, and we gave corresponding score of 0. For instance, when inquired about available medications for treating AF, ABAB mentioned the option of catheter ablation. In terms of elaboration, if the answer was too general and lacked necessary details, we assigned 0 point. For instance, when inquired about the diagnosis of AF, Sensechat simply stated that electrocardiogram(ECG) could be used for diagnosing AF without specifying the diagnostic criteria for ECG in detecting AF.\u003c/p\u003e\n\u003cp\u003eWithout a doubt, safety is important for patients. In terms of safety, seven LLMs have recommended NOAC and even antiplatelet therapy for patients with moderate to severe mitral stenosis and AF. Previous studies[22]have found that\u0026nbsp;among patients with rheumatic heart disease-associated atrial fibrillation, vitamin K antagonist therapy led to a lower rate of a composite of cardiovascular events or death than rivaroxaban therapy, without a higher rate of bleeding.\u0026nbsp;Current guidelines\u0026nbsp;[13, 23]do not recommend NOACs for anticoagulation in these patients. As for antiplatelet therapy, aspirin as monotherapy in stroke prevention of atrial fibrillation has no discernable protective effect against stroke[24].\u0026nbsp;Antiplatelet therapy should not be used for stroke prevention in AF patients[13, 23]. The LLMs were unable to differentiate between anticoagulants and antiplatelet drugs, which was disappointing. In terms of recommending antiarrhythmic drugs for AF patients, LLMs recommended quinidine for treatment. Although quinidine has antiarrhythmic effects, it may increase overall mortality[25]. In fact, current guidelines[13, 23]\u0026nbsp;do not recommend it for antiarrhythmic treatment of atrial fibrillation. Overall, in 80% of the inquiries, their response was deemed to be safe. In the case of Doubao, this proportion even reached 97%.\u003c/p\u003e\n\u003cp\u003eA previous study[26]\u0026nbsp;had evaluated the response of ChatGPT to counseling in patients with AF. The study reported 83.3% of patient-initiated prompts had appropriate responses generated by ChatGPT. In our study, Ernie bot exhibited the greatest performance with the accuracy rate of 87.9% among seven Chinese LLMs. However，in terms of correctness and error, certain responses appeared to be somewhat implausible. The most common errors included inappropriately treating antiplatelet agents as anticoagulants, recommending NOACs or antiplatelet agents for patients with valvular AF, and suggesting quinidine for AF treatment. The other errors comprised of suggestions regarding coffee and indications for LAAC. To sum up, the errors of these large models are actually similar, which may be related to the similarity of their training data. The findings of this study can assist LLMs in enhancing their performance, thereby enabling them to provide more effective medical advice to AF patients.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThrough this study, we discovered that Chinese LLMs were capable of providing valuable guidance to patients, but Chinese LLMs struggled to deliver precise responses on certain inquiries. We refrained from comparing LLMs with physicians due to the absence of rigorous methods for comparison. There are at least 2 reasons. First, while LLMs can provide an answer within seconds, clinicians typically adhere to more rigorous standards and often refer to relevant literature in cases of uncertainty. This discrepancy indicates that they do not require an equal amount of time to answer the same question, rendering the comparison unfair. Second，LLMs made some elementary mistakes. The comparison is meaningless when LLM is still at a stage of obvious immaturity.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eLimitations\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAt first，the volume of the present study is low, with only 33 questions, which cannot fully reflect the level of LLM, and a larger volume is needed to verify it in the future. Second, when evaluating simplicity or detail, it is still possible to be influenced by subjective factors. Third, there is no assessment of LLM's empathy problem. Fourth, the study does not contain all available Chinese LLMs. We only evaluated the first seven Chinese LLMs opening to the general public on August 31st, 2023.\u003c/p\u003e"},{"header":"Conclusions","content":"\u003cp\u003eThe findings of our study demonstrated that although LLMs exhibited strong potential for medical consultation, the review and evaluation by the medical profession is essential. In order to achieve outstanding performance in medicine in the future, LLMs will require more precise and up-to-date training data from clinical settings. Collaboration between AI engineers and medical professionals is crucial in the advancement of AI medicine.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cp\u003eLLMs \u0026nbsp; \u0026nbsp; \u0026nbsp;Large language models\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAF \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;Atrial\u0026nbsp;fibrillation\u003c/p\u003e\n\u003cp\u003eAI \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;Artificial\u0026nbsp;intelligence\u003c/p\u003e\n\u003cp\u003eLAAC \u0026nbsp; \u0026nbsp; Left\u0026nbsp;atrial appendage occlusion\u003c/p\u003e\n\u003cp\u003eNOACs \u0026nbsp; \u0026nbsp;Non-vitamin K oral anticoagulants\u003c/p\u003e\n\u003cp\u003eICD \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;Implanted Cardiac Defibrillator\u003c/p\u003e\n\u003cp\u003eECG \u0026nbsp; \u0026nbsp; \u0026nbsp; Electrocardiogram\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eAcknowledgements\u0026nbsp;\u003c/strong\u003eNone.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor contribution\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eConcept and design: Guijian Liu, Junbo Ge and Dr Wenqing Zhu. Acquisition, analysis, or interpretation of data: Guijian Liu, Qingxing Chen，Kuan Cheng，Ye Xu，Yang Pang，Yunlong Ling Drafting of the manuscript: Guijian Liu, Qingxing Chen，Kuan Cheng.Critical revision of the manuscript for important intellectual content: Guijian Liu, Qingxing Chen，Kuan Cheng , Junbo Ge and Wenqing Zhu Statistical analysis: Guijian Liu, Qingxing Chen，Kuan Cheng.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding:\u003c/strong\u003e This work was supported by funding from Shanghai Top Priority research center construction project（2022ZZ01010）\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData availability\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors confirm that the data supporting the findings of this study are available within the article and its supplementary information. The web \u0026nbsp;links to large language models can see in\u003c/p\u003e\n\u003cp\u003esupplementary material 2.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConflict of Interest Disclosures:\u003c/strong\u003e None.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEthical\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis study is not subject to Institutional Review Board Approval as no human or animal subjects were utilized.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent for publication\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors declare no competing interests.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConflict of Interest Disclosures:\u003c/strong\u003e None.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eThirunavukarasu AJ, Ting D, Elangovan K, Gutierrez L, Tan TF, Ting D. Large language models in medicine. Nat Med. 2023;29(8):1930-40.\u003c/li\u003e\n\u003cli\u003eBriganti G. How chatgpt works: a mini review. Eur Arch Otorhinolaryngol. 2024;281(3):1565-9.\u003c/li\u003e\n\u003cli\u003eDave T, Athaluri SA, Singh S. Chatgpt in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intell. 2023;6:1169595.\u003c/li\u003e\n\u003cli\u003eRuksakulpiwat S, Kumar A, Ajibade A. Using chatgpt in medical research: current status and future directions. J Multidiscip Healthc. 2023;16:1513-20.\u003c/li\u003e\n\u003cli\u003ePreiksaitis C, Rose C. Opportunities, challenges, and future directions of generative artificial intelligence in medical education: scoping review. Jmir Med Educ. 2023;9:e48785.\u003c/li\u003e\n\u003cli\u003eKung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepa\u0026ntilde;o C, et al. Performance of chatgpt on usmle: potential for ai-assisted medical education using large language models. Plos Digit Health. 2023;2(2):e198.\u003c/li\u003e\n\u003cli\u003eAyoub NF, Lee YJ, Grimm D, Balakrishnan K. Comparison between chatgpt and google search as sources of postoperative patient instructions. Jama Otolaryngol Head Neck Surg. 2023;149(6):556-8.\u003c/li\u003e\n\u003cli\u003eDecker H, Trang K, Ramirez J, Colley A, Pierce L, Coleman M, et al. Large language model-based chatbot vs surgeon-generated informed consent documentation for common procedures. Jama Netw Open. 2023;6(10):e2336997.\u003c/li\u003e\n\u003cli\u003eBernstein IA, Zhang YV, Govil D, Majid I, Chang RT, Sun Y, et al. Comparison of ophthalmologist and large language model chatbot responses to online patient eye care questions. Jama Netw Open. 2023;6(8):e2330320.\u003c/li\u003e\n\u003cli\u003eZhang B. Chatgpt, an opportunity to understand more about language models. Med Ref Serv Q. 2023;42(2):194-201.\u003c/li\u003e\n\u003cli\u003eArora A, Arora A. The promise of large language models in health care. Lancet. 2023;401(10377):641.\u003c/li\u003e\n\u003cli\u003eSallam M. Chatgpt utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare (Basel). 2023;11(6),887.\u003c/li\u003e\n\u003cli\u003eHindricks G, Potpara T, Dagres N, Arbelo E, Bax JJ, Blomstr\u0026ouml;m-Lundqvist C, et al. 2020 esc guidelines for the diagnosis and management of atrial fibrillation developed in collaboration with the european association for cardio-thoracic surgery (eacts): the task force for the diagnosis and management of atrial fibrillation of the european society of cardiology (esc) developed with the special contribution of the european heart rhythm association (ehra) of the esc. Eur Heart J. 2021;42(5):373-498.\u003c/li\u003e\n\u003cli\u003eAsmar S, Weinberg M, Bjorklund J, Sakr F, Salak J, Rabah H, et al. The impact of age on the epidemiology and cost of atrial fibrillation hospitalizations. Am J Cardiol. 2023;205:126-33.\u003c/li\u003e\n\u003cli\u003eChinese Society of Cardiology, Chinese Medical Association, et al. [Chinese guidelines on diagnosis and management of atrial fibrillation]. Zhonghua Xin Xue Guan Bing Za Zhi. 2023;51(6):572-618.\u003c/li\u003e\n\u003cli\u003eJavid M, Bhandari M, Parameshwari P, Reddiboina M, Prasad S. Evaluation of chatgpt for patient counseling in kidney stone clinic: a prospective study. J Endourol. 2024;38(4):377-83.\u003c/li\u003e\n\u003cli\u003eLahat A, Shachar E, Avidan B, Glicksberg B, Klang E. Evaluating the utility of a large language model in answering common patients\u0026apos; gastrointestinal health-related questions: are we there yet? Diagnostics (Basel). 2023;13(11):1950.\u003c/li\u003e\n\u003cli\u003eGajjar AA, Kumar RP, Paliwoda ED, Kuo CC, Adida S, Legarreta AD, et al. Usefulness and accuracy of artificial intelligence chatbot responses to patient questions for neurosurgical procedures. Neurosurgery. 2024;95(1):171-78.\u003c/li\u003e\n\u003cli\u003eLiu S, Liao Q, Liang Y, Li Z, Huang C. Spatio-temporal heterogeneity of urban expansion and population growth in china. Int J Environ Res Public Health. 2021;18(24):13031.\u003c/li\u003e\n\u003cli\u003eShah NH, Entwistle D, Pfeffer MA. Creation and adoption of large language models in medicine. Jama. 2023;330(9):866-9.\u003c/li\u003e\n\u003cli\u003evan Dis E, Bollen J, Zuidema W, van Rooij R, Bockting CL. Chatgpt: five priorities for research. Nature. 2023;614(7947):224-6.\u003c/li\u003e\n\u003cli\u003eConnolly SJ, Karthikeyan G, Ntsekhe M, Haileamlak A, El SA, El GA, et al. Rivaroxaban in rheumatic heart disease-associated atrial fibrillation. N Engl J Med. 2022;387(11):978-88.\u003c/li\u003e\n\u003cli\u003eJoglar JA, Chung MK, Armbruster AL, Benjamin EJ, Chyou JY, Cronin EM, et al. 2023 acc/aha/accp/hrs guideline for the diagnosis and management of atrial fibrillation: a report of the american college of cardiology/american heart association joint committee on clinical practice guidelines. J Am Coll Cardiol. 2024;83(1):109-279.\u003c/li\u003e\n\u003cli\u003eSj\u0026auml;lander S, Sj\u0026auml;lander A, Svensson PJ, Friberg L. Atrial fibrillation patients do not benefit from acetylsalicylic acid. Europace. 2014;16(5):631-8.\u003c/li\u003e\n\u003cli\u003eValembois L, Audureau E, Takeda A, Jarzebowski W, Belmin J, Lafuente-Lafuente C. Antiarrhythmics for maintaining sinus rhythm after cardioversion of atrial fibrillation. Cochrane Database Syst Rev. 2019;9(9):CD5049.\u003c/li\u003e\n\u003cli\u003eAzizi Z, Alipour P, Gomez S, Broadwin C, Islam S, Sarraju A, et al. Evaluating recommendations about atrial fibrillation for patients and clinicians obtained from chat-based artificial intelligence algorithms. Circ Arrhythm Electrophysiol. 2023;16(7):415-7.\u003c/li\u003e\n\u003c/ol\u003e"},{"header":"Tables","content":"\u003cp\u003eTable1 The SCECCE scoring system\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"100%\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 21px;\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 35px;\"\u003e\n \u003cp\u003eDefinition\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 13px;\"\u003e\n \u003cp\u003ePoints awarded\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 30px;\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003eDetails of assessment\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 21px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eS\u003c/strong\u003eafety\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 35px;\"\u003e\n \u003cp\u003eWhether the response includes any content that may be detrimental to the patient.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 13px;\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 30px;\"\u003e\n \u003cp\u003eIf yes, scores 0; If no, scores 2.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 21px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eC\u003c/strong\u003eorrectness\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 35px;\"\u003e\n \u003cp\u003eWhether the content of the response is correct.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 13px;\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 30px;\"\u003e\n \u003cp\u003eCompletely correct scores 2 points, partially correct scores 1 point, and incorrect scores 0 point.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 21px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eE\u003c/strong\u003error\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 35px;\"\u003e\n \u003cp\u003eWhether the content of the response is free of error.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 13px;\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 30px;\"\u003e\n \u003cp\u003eIf yes, scores 2；If no, scores 0.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 21px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eC\u003c/strong\u003eompleteness\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 35px;\"\u003e\n \u003cp\u003eWhether the response is complete with no omission of important information.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 13px;\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 30px;\"\u003e\n \u003cp\u003eIf no missing entries are found, they will receive 2 points. In the event of a missing entry, deductions of 1 point will be made, consecutively until a total of 2 points are deducted.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 21px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eC\u003c/strong\u003eonciseness\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 35px;\"\u003e\n \u003cp\u003eWhether the response is concise and free from unnecessary verbosity.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 13px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 30px;\"\u003e\n \u003cp\u003eIf yes, scores 1；\u0026nbsp;If no, scores 0.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 21px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eE\u003c/strong\u003elaboration\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 35px;\"\u003e\n \u003cp\u003eWhether the response is overly simplified, lacking essential details.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 13px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 30px;\"\u003e\n \u003cp\u003eIf yes, scores 0；\u0026nbsp;If no, scores 1.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 21px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eM\u003c/strong\u003eaximum score\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 35px;\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 13px;\"\u003e\n \u003cp\u003e10\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 30px;\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eTable2 The SCECCE score of all six aspects and individual aspect\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"915\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eOverall\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eSafety\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eCorrectness\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eError\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eCompleteness\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eConciseness\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eElaboration\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eAll LLMs\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMean（SD）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e8.6（2.0）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.8（0.6）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.8（0.5）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.5（0.9）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.6（0.6）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0（0.1）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0（0.2）\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMedian（IQR）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e10.0（7.0-10.0）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0（2.0-2.0）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0（1.0-2.0）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0（2.0-2.0）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0（1.0-2.0）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0（1.0-1.0）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0（1.0-1.0）\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMinimum\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMaximum\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e10.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eABAB1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eTotal points\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e275\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e56\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e56\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e50\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e52\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e30\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e31\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMean\u003c/p\u003e\n \u003cp\u003e（SD）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e8.3(2.8)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.7(0.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.7(0.6)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.5(0.9)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.6(0.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.9(0.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.9(0.2)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMedian\u003c/p\u003e\n \u003cp\u003e(IQR)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e10.0(7.5-10)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(2.0-2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(1.5-2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(1.0-2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(1.0-2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0(1.0-1.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0(1.0-1.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMinimum\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMaximum\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e10.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eBaichuan1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eTotal points\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e296\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e60\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e60\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e54\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e56\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e33\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e33\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMean\u003c/p\u003e\n \u003cp\u003e（SD）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e9.0(1.8)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.8(0.6)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.8(0.4)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.6(0.8)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.7(0.6)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0(0.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0(0.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMedian\u003c/p\u003e\n \u003cp\u003e(IQR)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e10.0（9.0-10.0）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0（2.0-2.0）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0（2.0-2.0）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(2.0-2.0）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(1.5-2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0(1.0-1.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMinimum\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e5.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMaximum\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e10.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eChatglm2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eTotal points\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e283\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e60\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e55\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e44\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e57\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e33\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e33\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMean\u003c/p\u003e\n \u003cp\u003e（SD）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e8.6(1.9)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.8(0.6)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.7(0.5)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.3(1.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.7(0.6)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0(0.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0(0.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMedian\u003c/p\u003e\n \u003cp\u003e(IQR)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e10.0(7-10)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(2.0-2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0（1.0-2.0）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(0.0-2.0）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(2.0-2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0(1.0-1.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0（1.0-1.0）\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMinimum\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e5.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMaximum\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e10.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eTable2 The SCECCE score of all six aspects and individual aspect(continued)\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"961\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eOverall\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eSafety\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eCorrectness\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eError\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eCompleteness\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eConciseness\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eElaboration\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eDoubao\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eTotal points\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e293\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e64\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e60\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e54\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e50\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e33\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e32\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMean\u003c/p\u003e\n \u003cp\u003e（SD）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e8.9(1.6)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.9(0.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.8(0.4)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.6(0.8)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.5(0.8)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0(0.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0(0.2)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMedian\u003c/p\u003e\n \u003cp\u003e(IQR)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e10.0(8-10)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(2.0-2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(2.0-2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(2.0-2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(1.0-2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0(1.0-1.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0(1.0-1.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMinimum\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e5.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMaximum\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e10.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eErnie bot3.5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eTotal points\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e299.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e60.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e62.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e58.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e53.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e33.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e33.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMean\u003c/p\u003e\n \u003cp\u003e（SD）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e9.1（1.7）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.8 (0.6)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.9（0.3）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.8（0.7）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.6（0.6）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0（0.0）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0(0.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMedian\u003c/p\u003e\n \u003cp\u003e(IQR)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e10.0(9.0-10.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0（2.0-2.0）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(2.0-2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(2.0-2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(1,0-2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0(1.0-1.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0(1.0-1.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMinimum\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMaximum\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e10\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eSensechat3.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eTotal points\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e265\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e58\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e55\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e50\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e46\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e31\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e24\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMean\u003c/p\u003e\n \u003cp\u003e（SD）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e8.0(2.2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.8(0.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.7(0.6)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.5(0.9)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.4(0.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.9(0.2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.7(0.5)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMedian\u003c/p\u003e\n \u003cp\u003e(IQR)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e9.0(7.0-10.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(2.0-2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(1.0-2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(1.0-2.0）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(1.0-2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0(1.0-1.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0(0.1-0.1)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMinimum\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMaximum\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e10.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003eZiDongTaiChu 2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eTotal points\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e266\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e54\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e52\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e38\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e56\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e33\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e33\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMean\u003c/p\u003e\n \u003cp\u003e（SD）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e8.1(2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.6(0.8)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.6(0.5)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.2(1.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.7(0.6)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0(0.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0(0.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMedian\u003c/p\u003e\n \u003cp\u003e(IQR)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e9.0(6.5-10.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(2.0-2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(1.0-2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(0.0-2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0(2.0-2.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0（1.0-1.0）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0（1.0-1.0）\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMinimum\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e5.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e0.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003eMaximum\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e10.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e2.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-cardiovascular-disorders","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"bcar","sideBox":"Learn more about [BMC Cardiovascular Disorders](http://bmccardiovascdisord.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/bcar/default.aspx","title":"BMC Cardiovascular Disorders","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Atrial fibrillation, Artificial intelligence, Large language model, Safety, Accuracy","lastPublishedDoi":"10.21203/rs.3.rs-6673302/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6673302/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground \u003c/strong\u003eThe first seven Chinese Large language models （LLMs）were launched to the public on August 31st, 2023.However, the extent to which Chinese LLMs can assist atrial fibrillation（AF）patients remains unknown. We sought to assess the Chinese LLMs performance of providing responses to AF patient questions.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethod\u003c/strong\u003e This cross-sectional study compared seven Chinese LLM chatbots including ABAB, Baichuan, Chatglm, Doubao, Ernie bot, Sensechat and ZidongTaichu. At first，cardiologists compiled a list of frequently asked questions by patients with AF. Responses from LLMs were collected. We developed a scoring system known as SCECCE, which consists 6 aspects including \u003cstrong\u003es\u003c/strong\u003eafety, \u003cstrong\u003ec\u003c/strong\u003eorrectness，\u003cstrong\u003ee\u003c/strong\u003error，\u003cstrong\u003ec\u003c/strong\u003eompleteness, \u003cstrong\u003ec\u003c/strong\u003eonciseness and \u003cstrong\u003ee\u003c/strong\u003elaboration. Each response was assessed by the expert committee with SCFCCE scoring system.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResult\u003c/strong\u003e \u0026nbsp;Ultimately, we obtained 231 responses. On the whole, the median SCFCCE score was 10[IQR，7-10] with a mean(SD) score of 8.6(2.0). No significant statistical differences were observed in the terms of SCFCCE scores among seven LLMs(p=0.08). The full SCFCCE score was 330 points. Ernie bot attained the highest total score of 299 points. Doubao’s responses were safe in 97% of the questions. In terms of correctness and error, the overall comparison of each group revealed no statistically significant difference. Ernie bot exhibited greatest performance with the accuracy rate of 87.9%.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusion \u003c/strong\u003eThe findings of our study demonstrated that although Chinese LLMs exhibited strong potential for medical consultation, the review and evaluation by the medical profession is essential.\u003c/p\u003e","manuscriptTitle":"Comparative analysis of Chinese large language model performance on atrial fibrillation questions ","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-06-17 15:52:02","doi":"10.21203/rs.3.rs-6673302/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-08-19T11:18:23+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-08-18T15:08:57+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"164241715243361035970722588323214054901","date":"2025-08-16T04:41:30+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"339537420454425954671117667404582616482","date":"2025-07-22T06:41:37+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-06-24T17:30:40+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"76892699042882188917964442581646046443","date":"2025-06-24T17:19:44+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-06-12T03:50:34+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-06-12T03:49:53+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2025-06-02T05:04:54+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-05-31T08:16:16+00:00","index":"","fulltext":""},{"type":"submitted","content":"BMC Cardiovascular Disorders","date":"2025-05-31T08:13:29+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-cardiovascular-disorders","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"bcar","sideBox":"Learn more about [BMC Cardiovascular Disorders](http://bmccardiovascdisord.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/bcar/default.aspx","title":"BMC Cardiovascular Disorders","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"f85d4620-6812-495b-abe7-11a52b6dd8d9","owner":[],"postedDate":"June 17th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2026-02-09T16:05:18+00:00","versionOfRecord":{"articleIdentity":"rs-6673302","link":"https://doi.org/10.1186/s12872-026-05566-6","journal":{"identity":"bmc-cardiovascular-disorders","isVorOnly":false,"title":"BMC Cardiovascular Disorders"},"publishedOn":"2026-02-04 15:57:31","publishedOnDateReadable":"February 4th, 2026"},"versionCreatedAt":"2025-06-17 15:52:02","video":"","vorDoi":"10.1186/s12872-026-05566-6","vorDoiUrl":"https://doi.org/10.1186/s12872-026-05566-6","workflowStages":[]},"version":"v1","identity":"rs-6673302","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6673302","identity":"rs-6673302","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}