Constructing a Smart-Assistant for Improving the Outpatient Service Quality in Real-time: a Prospective Single-center Cohort Study

Constructing a Smart-Assistant for Improving the Outpatient Service Quality in Real-time: a Prospective Single-center Cohort Study | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Constructing a Smart-Assistant for Improving the Outpatient Service Quality in Real-time: a Prospective Single-center Cohort Study Hongliu Du, Jialing Li, Bing Xiao, Xueying Wang, Wenxin Xue, Mei Deng, and 9 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7066667/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background The quality of consultation and outpatient electronic medical records (EMRs) varies among physicians. We aimed to construct an intelligent system (SMART-ASSISTANT) to assist physicians in history taking and the composing of EMRs for patients presenting with the chief complaint of abdominal pain. Methods Anonymized EMRs of 1249 cases, free-text-structured EMRs pairs of 119 cases, and a hot words dictionary were used to train the SMART-ASSISTANT. The SMART-ASSISTANT is constructed with four components: audio transcription, structured EMRs generation, EMRs quality control, and assisted diagnosis. The functions were validated through the simulated set, the retrospective set, and a multi-reader multi-case (MRMC) study. A prospective cohort study including 62 participants was conducted to evaluate the utility of SMART-ASSISTANT to transcribe the consultation audio into standardized EMRs text. Results SMART-ASSISTANT outperformed GPT-4 in identifying symptoms, characteristics, onset characteristics, and disease progression significantly (100.00 vs 19.20%, P <0.001; 92.00 vs 85.80%, P <0.001; 85.60 vs 45.60%, P <0.001; 89.60 vs 45.40%, P <0.001). Physicians’ rating of the completeness and diagnostic correlation of EMRs in the AI-assisted set were significantly superior to those in the human-generated set (4.27 vs 3.92, P <0.001; 2.53 vs 2.33, P <0.001). In prospective cohort study, the mean semantic textual similarity (STS) of audio transcription reached 0.9253. Physicians rated no significant difference between AI and human-generated EMRs in normativity (3.23 vs 3.25, P = 0.806), readability (3.22 vs 3.38, P = 0.068), and logicality (3.19 vs 3.32, P = 0.122). AI-generated EMRs demonstrated significantly superior performance in terms of integrity (3.50 vs 3.21, P = 0.008) than human-generated EMRs. Conclusions The quality of the outpatient EMRs have been significantly improved by the utilization of SMART-ASSISTANT. The system has the potential to confer benefits on both patients and healthcare organizations through assisted consultation and automated EMRs generation. Biological sciences/Computational biology and bioinformatics Health sciences/Health care Physical sciences/Mathematics and computing Health sciences/Medical research large language model electronic medical record fine-tuning quality control Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Introduction Electronic medical records (EMRs) constitute indispensable repositories of integrated patient information, capturing presenting symptoms, therapeutic interventions, thus take a pivotal role during patient care and referral 1 . EMRs are being widely used in the excitement over health information technology, and are associated with medical malpractice liability and may exert effectiveness as well 2 – 4 . High-quality EMRs can reduce adverse events and claims by reducing discontinuities and errors in care and by improving clinical decisions 5 . Incomplete medical records and difficulties in accessing a patient’s medical history compromise the overall quality of care that clinicians can provide 6 . Therefore, both the patients' welfare and the interests of medical institutions can be protected as the quality of EMRs improves. Consultation is one of the essential steps in medical history collection, which could lead to appropriate diagnosis thinking and prompt treatment choice. Standardized consultation is a powerful guarantee for the improvement of EMRs 7 – 9 . In the case of patients with acute abdominal pain (AAP), accurate and rapid consultation can help physicians diagnose or eliminate fatal diseases promptly. However, the quality of consultation and EMRs varies considerably attributed to the large volumes of consultations, the high level of fatigue, and the wide variations in clinical experience and expertise of the physicians 10 – 12 . Misdiagnosis and delayed diagnosis may occur if physicians fail to conduct a complete and comprehensive interview based on the patients’ complaints, which directly harm the patients’ benefits and may lead to compensation for medical malpractice liability 5 . There is a dearth of efficacious assistance to regulate the process of medical history collection. Large language models (LLM) and generative artificial intelligence (AI) may have vast potential in patient care, especially in medical document management compared with image recognition 13 – 15 . The accelerated evolution of LLM has prompted experts to believe that we are likely 'on the cusp' of a monumental shift in healthcare delivery and evaluation 16 . Recent studies have demonstrated that the large language model is capable of generate EMRs of lung cancer cases, can accurately extracting the headache frequency, and is comparable with that of radiologists in detecting five types of errors in radiology reports 17 – 19 . However, the majority of existing studies have concentrated on 'prompt engineering', which is likely to cause unreliable advice due to hallucination, and there is no precedent for the application of LLM in outpatient EMRs quality improvement 20 . This study aims to develop a fine-tuning LLM-based system (SMART-ASSISTANT) to automatically generate standardized EMRs based on consultation audio and to improve the quality of EMRs with abdominal pain as the chief complaint. This study is the first to use AI to assist the entire process of outpatient work, including consultation, audio transcription, structured EMRs generation, EMRs quality improvement, and assisted diagnosis. We conducted a prospective cohort study to evaluate the performance of the system in a real clinical setting. The outcomes demonstrate that the system is an effective means of improving the quality of outpatient EMRs and assisting physicians in daily outpatient service (Fig. 1 ). Results Outcome The SMART-ASSISTANT consists of four parts: audio transcription, structured EMRs generation, quality control, and assisted diagnosis. We validated the above mentioned functions through the simulated set, the retrospective set, a multi-reader multi-case (MRMC) study, and a prospective cohort study. The semantic textual similarity (STS) and character error rate (CER) were used to evaluate the ability of audio-to-text transcription and structured EMRs transcription. The accuracy was used to evaluate the capabilities of AI in recognizing quality control points (QCP). Using physician-labelled results as the gold standard, accuracy = concordant predictions/total number of cases. The QCP completion rate (QCR) was used to evaluate the quality of human generated EMRs. QCR = number of completed mandatory points/total number of mandatory points. The omission detection rate (ODR) and the standardization rate (SR) were used to evaluate the capabilities of AI in recognizing omission points. ODR = number of omission points correctly detected by AI/total number of omission points. SR = number of omission points correctly detected by AI/total number of mandatory points. The theoretical completion rate (TCR) was used to evaluate the quality of EMRs recorded by human with the assistance of AI. TCR = QCR + SR. The TCR indicates the completeness of EMRs in case whenever the AI correctly identifies an omission point in the human consultation records, it prompts the physician to conduct further inquiry and fill in the omission point. A Likert scale was used to assess AI's ability to improve EMRs quality. Performance of audio transcription and structured EMRs transcription The STS and CER of audio-to-text transcription using the original engine powered by OpenAI-whisper were 0.5654 ± 0.0908 and 0.4559 ± 0.1900. After the entity correction, the WER improved significantly meanwhile the CER decreased significantly (WER, 0.5912 ± 0.0853, P <0.01; CER, 0.3665 ± 0.1407, P <0.01). FunASR with entity correction achieved the best performance thus was used for prospective validation as the audio transcribe module in SMART-ASSISTANT (WER, 0.9080 ± 0.1348, P <0.001; CER, 0.2643 ± 0.3534, P <0.01) (Table 1 ). In the performance of structured EMRs transcription, the model was found to have a greater proportion of cases with high reliability for allergy history, auxiliary examination, past medical history and other history (91.30%, 78.26%, 73.91%). The STS and CER of structured EMRs transcription were 0.8114 ± 0.1259 and 0.7139 ± 0.4372 (Table 1 ). Performance in detecting quality control key points The performance of the three LLMs was evaluated both before and after fine-tuning, and a comparison was made with the performance of GPT-4 and DeepSeek. Prior to fine-tuning, the accuracy of HuatuoGPT-II, Llama-3, and ChatGLM-3 for QCP extraction was 58.23% [95%CI, 55.32–61.08%], 68.74% [95%CI, 65.98–71.39%], and 60.31% [95%CI, 57.42–63.13%], respectively. These values were all inferior to that of GPT-4 significantly (71.61% [95%CI, 68.92–74.18%], P <0.001). It appears that there is a great challenge for LLMs in interpreting the connotative content of EMRs due to the lack of medical domain knowledge without the support of supervised learning. After fine-tuning, the accuracy of the LLMs for QCP extraction was significantly improved ( P <0.001). All three LLMs outperformed the GPT-4 and DeepSeek significantly ( P <0.001), indicating that fine-tuning can facilitate greater progress in specific tasks for LLMs without medical domain knowledge. HuatuoGPT-II-FT demonstrated the most optimal performance (91.53% [95%CI, 89.76–93.02%]). Among them, HuatuoGPT-II-FT achieved the best performance for identifying onset characteristics, location, duration, and accompanying symptoms (85.60% [95%CI, 78.38–90.69%], 91.20% [95%CI, 84.93–95.02%], 94.40% [95%CI, 88.89–97.26%], and 90.13% [95%CI, 86.69–92.75%]), while Llama-3-FT exhibited the greatest efficacy in identifying predisposing conditions and disease progression (97.60%, 91.00%) (Table 2 ). The QCR and the TCR in the original test set were 73.60% [95%CI, 70.33–76.63%] and 97.87% [95%CI, 96.57–98.69%], indicating that there will be a significant improvement in EMRs completeness as long as physicians are able to follow the prompts of SMART-ASSISTANT without compromise ( P <0.01) (Fig. 3 ). Performance in disease diagnosis We omitted diagnoses from the original EMRs and models were trained to predict the most likely 3–5 diagnoses. If the output contains the diagnosis that was originally documented in the EMRs, it will be deemed accurate. Before fine-tuning, the diagnostic accuracy of HuatuoGPT-II, Llama-3, ChatGLM-3, and GPT-4 were 43.20% [95%CI, 34.85–51.96%], 45.60% [95%CI, 37.13–54.33%], 62.40% [95%CI, 53.66–70.40%], 47.20% [95%CI, 38.66–55.90%]. After fine-tuning, the accuracy of diagnosis was significantly improved in three LLMs (78.40% vs. 43.20%, P <0.001; 75.20% vs. 45.60%, P <0.001; 72.00% vs. 62.40%, P <0.05) (Table S1 ). Table 1 Performance in audio-free text transcription and structured EMRs transcription. Subject STS (mean ± SD) CER (mean ± SD) Percentage of Different Reliability high medium low Simulated dataset Audio transcription Whisper 0.5654 ± 0.0908 0.4559 ± 0.1900 0 26.67% 73.33% Whisper + Entity Correction 0.5912**±0.0853 0.3665**±0.1407 0 53.33% 46.67% FunASR + Entity Correction 0.9080***±0.1348 0.2643**±0.3534 100.00% 0 0 SMART-ASSISTANT in prospective cohort 0.9253 ± 0.1179 0.2058 ± 0.2995 100.00% 0 0 Structured EMRs transcription Chief Complaint 0.7388 ± 0.1732 0.7385 ± 0.4610 39.13% 34.78% 26.09% History of Present Illness 0.7390 ± 0.1225 1.6016 ± 1.2129 21.74% 65.22% 13.04% Past Medical History and Other History 0.8427 ± 0.2428 0.5571 ± 0.7940 73.91% 8.70% 17.39% Allergy History 0.9384 ± 0.1979 0.0408 ± 0.1210 91.30% 4.35% 4.35% Auxiliary Examination 0.7981 ± 0.3576 0.4016 ± 0.3358 78.26% 4.35% 17.39% Total 0.8114 ± 0.1259 0.7139 ± 0.4372 56.52% 39.13% 4.35% STS , semantic textual similarity. CER , character error rate. We consider STS above 80% as a high reliability, 60–80% as a medium reliability, and below 60% as a low reliability. *Compared with OpenAI. *Significant at 5% level. **Significant at 1% level. ***Significant at 0.1% level Table 2 Performance between LLMs in quality-control points extraction. Quality Control Key Points Model performance in retrospective EMRs Model performance in prospective cohort HuatuoGPT-II Llama-3 ChatGLM3 ChatGPT-4 DeepSeek HuatuoGPT-II-FT Llama-3-FT ChatGLM3-FT Symptom (95%CI), % 32.80 (25.19–41.44) 27.20 (20.17–35.59) 22.40 (15.98–30.47) 19.20 (13.26–26.98) ### 23.20 (16.67–31.33) ### 100 (97.12–100.00)*** 100 (97.12–100.00)*** 100 (97.12–100.00)*** 98.39 (91.42–99.72) Characteristics (95%CI), % 65.60 (61.33–69.63) 74.60 (70.61–78.22) 62.20 (57.87–66.34) 85.80 (82.47–88.59) ### 59.20 (54.84–63.42) ### 92.00 (89.29–94.07)*** 92.00 (89.29–94.07)*** 89.80 (86.84–92.16)*** 95.97 (92.74–97.80) Predisposing conditions (95%CI), % 72.00 (63.56–79.12) 83.20 (75.65–88.74) 80.00 (72.14–86.07) 96.00 (90.98–98.28) 60.00 (51.24–68.17) ### 96.00 (90.98–98.28)*** 97.60 (93.18–99.18)*** 92.80 (86.88–96.17)** 83.87 (72.79-91.00) Influencing factors (95%CI), % 56.80 (48.04–65.15) 76.80 (68.67–83.33) 72.80 (64.41–79.83) 85.60 (78.38–90.96) 8.00 (4.40–14.10) ### 84.80 (77.48–90.05)*** 80.80 (73.02–86.74) 80.80 (73.02–86.74) 93.55 (84.55–97.46) Onset characteristics (95%CI), % 24.00 (17.36–32.19) 34.40 (26.65–43.08) 14.40 (9.31–21.62) 45.60 (37.13–54.33) ### 3.20 (1.25–7.94) ### 85.60 (78.38–90.69)*** 84.80 (77.48–90.05)*** 84.80 (77.48–90.05)*** 79.03 (67.36–87.31) Location (95%CI), % 68.00 (59.39–75.54) 72.00 (63.56–79.12) 71.20 (62.72–78.41) 85.60 (78.38–90.96) 45.60 (37.13–54.33) ### 91.20 (84.93–95.02)*** 88.80 (82.08–93.21)*** 88.00 (81.14–92.59)*** 87.10 (76.55–93.32) Duration (95%CI), % 75.20 (66.95–81.94) 83.20 (75.65–88.74) 91.20 (84.93–95.02) 91.20 (84.93–95.02) 50.40 (41.75–59.02) ### 94.40 (88.89–97.26)*** 91.20 (84.93–95.02) 92.00 (85.90–95.60) 98.39 (91.42–99.72) Accompanied symptoms (95%CI), % 75.46 (70.87–79.55) 86.93 (83.14–89.97) 80.80 (76.51–84.47) 90.13 (86.69–92.75) 47.47 (42.47–52.52) ### 90.13 (86.69–92.75)*** 84.40 (80.81–88.08) 83.73 (79.66–87.12) 94.62 (90.38–97.05) Disease progression (95%CI), % 54.20 (49.82–58.52) 80.40 (76.69–83.64) 47.80 (43.46–52.18) 45.40 (41.09–49.78) ### 5.00 (3.41–7.28) ### 89.60 (86.62–91.98)*** 91.00 (88.17–93.21)*** 86.80 (83.55–89.49)*** 99.19 (97.10-99.78) Total (95%CI), % 58.23 (55.32–61.08) 68.74 (65.98–71.39) 60.31 (57.42–63.13) 71.61 (68.92–74.18) ### 33.56 (30.90-36.41) ### 91.53 (89.76–93.02)*** 90.11 (88.23–91.72)*** 88.75 (86.77–90.47)*** 94.40 (92.84–95.63) LLM , large language model *Compared with large language models before fine-tuning. Since fine-tuning requires an open-source model, we cannot choose GPT-4 or DeeSseek for fine-tuning. # Compared with HuatuoGPT-II-FT. */ # Significant at 5% level. **/ ## Significant at 1% level. ***/ ### Significant at 0.1% level Multi-reader multi-case study Since HuatuoGPT-II-FT performs better than the other models in both quality control and diagnosis, it was regarded as the SMART-ASSISTANT and the results of HuatuoGPT-II-FT are provided to physicians to assist EMRs recording and diagnosis. The average ensemble completeness and diagnostic correlation rating in AI-first group was mean 4.45 ± SD 0.68 and 2.64 ± 0.59 in the first round. After a 3-week washout period, physicians in AI-first group rated the original test set with a significantly lower mean completeness and diagnostic correlation score than in the first round (4.04 ± 0.87, P <0.01, 2.45 ± 0.71, P <0.01). Physicians in the control group had a mean completeness rating of 3.80 ± 0.93 and a mean diagnostic correlation rating of 2.21 ± 0.77 in the first round. After a three-week washout period, they rated the AI-assisted test set significantly better than the first round on both mean completeness rating and diagnostic correlation rating (4.09 ± 0.90, P <0.01, 2.43 ± 0.63, P <0.01). Physicians in the AI-first and control groups rated significantly higher on the AI-assisted test set than on the original test set when assessing the completeness and diagnostic correlation of the EMRs. This suggests that the majority of physicians in both the AI-first and control groups perceived the AI-assisted EMRs to be more complete and were able to infer a higher degree of diagnostic correlation (Fig. 2 ). Prospective observational cohort study The STS and CER of audio transcription were 0.9253 ± 0.1179 and 0.2058 ± 0.2995 (Table 1 ). In QCP recognition, SMART-ASSISTANT achieved the accuracy of 94.40% [95%CI, 92.84–95.63%], which was better than the retrospective test. In preliminary diagnosis, SMART-ASSISTANT achieved the accuracy of 50.00% [95%CI, 37.92–62.08%], which was lower than the retrospective test (Table S1 ). This discrepancy may be attributed to the fact that the original test set encompasses EMRs from residents, attending physicians, and consultants, with a relatively higher occurrence of common diseases. In contrast, the prospective cohort consists of data collected by four consultants, which includes a higher proportion of cases of rare disease and difficult and complicated disease (Table S3). The QCR and the TCR were 59.14% [95%CI, 54.08–64.02%] and 97.85% [95%CI, 95.82–98.91%] ( P <0.05), indicating a significant improvement in EMRs completeness if physicians are able to follow the prompt from SMART-ASSISTANT. Physicians rated no significant difference between AI and human-generated EMRs in normativity (3.23 ± 0.96 vs 3.25 ± 0.91, P = 0.806), readability (3.22 ± 0.98 vs 3.38 ± 0.84, P = 0.068), and logicality (3.19 ± 0.94 vs 3.32 ± 0.83, P = 0.122). The average ensemble factuality score was 4.01 ± 0.78 for human-generated EMRs and 3.34 ± 1.22 for AI-generated EMRs ( P <0.001). AI-generated EMRs achieved a significantly higher integrity score than human-generated EMRs (3.50 ± 0.97 vs 3.21 ± 0.96, P = 0.008). This indicates that at this stage, with only the free text of the doctor-patient dialogue available, the model may still generate implausible information and illusory outputs. Nevertheless, in terms of extracting key information from these consultation dialogues, the model performs significantly better than that of physicians. Furthermore, it may even capture certain details that physicians might overlook (Fig. 3 ). Discussion Abdominal pain can be the presenting symptom of a life-threatening abdominal catastrophe ('acute abdomen'). When approaching a patient with acute abdominal pain, it is the responsibility of the physician to assess the patient's overall physiologic state and make a definitive diagnosis rapidly. Despite the advances made in various investigations including clinical imaging, history taking remains the most important component of the initial evaluation of the patient with acute abdominal pain 21 . LLM is increasingly being integrated into medical decision-making. Peter et al. suggested three scenarios of potential medical use of GPT-4: medical note taking, innate medical knowledge, and medical consultation 22 . Carrie et al. conducted a cross-sectional study to evaluate LLM responses to rheumatology patient questions and compared with the doctors’ responses 23 . Dhavalkumar et al. evaluated the performance of GPT-3.5 and GPT-4 in USMLE-style medical calculation 24 . The majority of existing research has concentrated on the field of prompt engineering, which is less labour-intensive but can cause hallucinations. In this approach, LLM does not appear to be an optimal tool for making modifications to enhance the content of clinical documents. We use a logical and behavioral anthropomorphic way to simulate the comprehensive workflow of human physicians in outpatient settings, encompassing consultation, EMRs generation, and clinical decision-making, in order to train the SMART-ASSISTANT. We chose LoRA to enable the general LLM lacking medical domain knowledge to accurately identify QCP (91.53%, 90.11%, 88.75%) in the outpatient EMRs of abdominal pain patients. The propensity of generative models to generate hallucinatory in outputs has the potential to be highly detrimental of medical decision-making 22 . The evaluation approach adopted in this study considers the potential impact of hallucinations of the LLMs. In the gold standard, a physician is permitted to mark nulls based on the actual medical record text. Should the model generate a value for the nulls section that is not mentioned in the EMRs, or a value that is mentioned in the EMRs but is not the appropriate part of this metric, it will be deemed inconsistent. To illustrate, if the gold standard is assigned a null value for Onset Characteristics and the Characteristics is labelled as Sharp, and the model erroneously populates Sharp for Onset Characteristics, it will be deemed inconsistent. By meticulously categorizing the inconsistencies, this study illustrates that through the fine-tuning of the LLM, it is feasible to optimize it to a clinically utilitarian level by decreasing the inconsistencies including omission, error, and extra. The results of this study indicate that, during the QCP recognition phase, the general model demonstrated better performance in the extraction of categories particularly within the more expansive domain. For example, GPT-4 achieved high accuracy in identifying predisposing conditions, location, duration, and accompanied symptoms. These items also demonstrated satisfied performance for the other LLMs before fine-tuning. In specialized areas of medicine, such as distinguishing between the Characteristics of abdominal pain, Onset Characteristics, and Disease Progression, the fine-tuned model can achieve significant progress. This suggests that through fine-tuning, physicians can adapt a general model that lacks medical domain knowledge to fit a specific task such as identifying QCP. There were several limitations in this study. First, this study only included the outpatient EMRs, whereas the inpatient EMRs are more complicated and regarded as more challenging. We will continue to collect impatient records and develop a more widely applicable system for inpatient EMRs quality control. Second, this study only included patients with the main complaint of 'abdominal pain', we will further expand the data set to include patients with a variety of complaints. Third, despite the satisfied STS that audio transcription has achieved, its application is subject to a number of demanding conditions. It requires meeting several criteria simultaneously, including not speaking in dialects, having a clean background without noise, limiting the number of speakers, and ensuring that speakers do not talk over each other but instead take turns speaking. Additionally, our preliminary study indicated that the application of fine-tuning in LLMs can improve the model performance in EMRs quality control in both Chinese and English. However, due to the fact that the majority of the medical records collected by RHWU are from Chinese-speaking individuals, the medical record included in this study is exclusively written in Chinese. We will further conduct an international multi-center study to validate the applicability of SMART-ASSISTANT in English language settings. In conclusion, this study is the first to use AI to assist the entire process of outpatient work. SMART-ASSISTANT can accurately extract features from outpatient EMRs and significantly improve the completeness and diagnostic correlation of EMRs. The system has the potential to improve the quality of EMRs in real clinic and to automatically transcribe the consultation audio to structured EMRs, meanwhile protecting the interests of both physicians and patients and reducing the incidence of medical malpractice. Methods Study design and ethics The construction of the SMART-ASSISTANT can be divided into four parts: audio transcription, structured EMRs generation, quality control, and assisted diagnosis. We validated the above mentioned functions through the simulated set, the retrospective set, a multi-reader multi-case (MRMC) study, and a prospective cohort study. The study was approved by the Ethics Committee of Renmin Hospital of Wuhan University (RHWU) and was registered on https://www.chictr.org.cn/ with registration number ChiCTR2400086606. Patient sensitive information was removed during the initial extraction of EMRs data. The EMRs data was anonymised by replacing the identifiers used for the encounter notes with universal unique identifiers. The translation table was kept securely on a second server and could be unlocked only with the approval of the institutional review board or for a public health authority. The EMRs system was developed by a Chinese vendor named Donghua Electronic Medical Records (Beijing, China). Audio transcription model 30 cases of simulated consultation dialogues were conducted by 2 researchers to simulate clinical consultation scenarios. The audio files of simulated doctor-patient clinic dialogues were recorded. Three different audio transcription strategies (OpenAI-Whisper, OpenAI-Whisper with entity correction, and FunASR with entity correction) were used to convert the audio files into text and the best performing one was used for prospective validation. One researcher transcribed the text on the basis of the recording file as gold standard. OpenAI-Whisper is a product of Open-AI company and the version is 20231117. FunASR is a product of Alibaba and the version number is 1.1.9. The details of audio-to-text transcription methods were presented in the supplementary materials 25 – 30 . Structured EMRs generation model A total of 119 cases of simulated free text of consultation dialogue and paired structured EMRs were written by five researchers based on different chief complaints, forming 119 free text-EMRs pairs. These pairs were randomly split 4:1 into a fine-tuning set and a test set. The fine-tuning set was used to fine-tune the foundation model (HuatuoGPT-II), enabling it to summarize free text into structured outpatient medical records. The test set was used to evaluate the capability of the model. The models are expected to automatically generate a structured medical record with five sections: chief complaint, history of present illness, past history and other history, allergy history, and auxiliary examination, based on the free text of the dialogue. Quality improving model Datasets and preprocessing The EMRs of patients who presented to RHWU with the chief complaint of 'abdominal pain' were retrospectively collected from December 12th 2020 to May 31st 2023. Anonymized EMRs of 3456 cases with the chief complaint of abdominal pain were retrospectively collected. A total of 1249 outpatient EMRs met the inclusion criteria (Fig. 4 ). The EMRs were then labelled with symptoms related information according to a quality control template by 10 residents and doctor of medicine (MD) students. Two consultants reviewed the labels. The patient-based EMRs were randomly divided into a fine-tuning set and an original test set in a ratio of 9:1, ensuring that the medical records of the same patient would not appear in both sets. 1124 cases of EMRs were used for the fine-tuning set and 125 cases of EMRs consisted the original test set. Each medical record was annotated with the corresponding QCP and preliminary diagnosis as the input data. Finally, the models were expected to identify the QCP and to predict the most relevant top 3 to 5 diagnoses according to the EMRs. Inclusion criteria: patients presented with a chief complaint of abdominal pain.Exclusion criteria: (1) patients with a history of substance abuse or mental disorder in the past 5 years; (2) chief complaint containing multiple symptoms; (3) EMRs of obstetrics and gynaecology; (4) defective EMRs; (5) subsequent visit patient with previously confirmed diagnoses (Fig. 4 ). The details of datasets and preprocessing were presented in the supplementary materials. Quality-control template establishment The quality-control template was established in accordance with clinical guidelines and textbooks pertaining to acute abdominal pain-related disorders and key points of consultation 21,31–33 . Two consultants conducted template establishment. They selected characteristics, predisposing conditions, influencing factors, onset characteristics, location, duration, accompanied symptoms, and disease progression as quality-control points (QCP). Among them, characteristics, predisposing conditions, influencing factors, onset characteristics, location, and duration were regarded as mandatory points, which could be filled with 'null' only if a negative description of the relevant information is documented in the EMRs. The accompanied symptoms and disease progression were regarded as optional, which could be empty. The templates were employed to train and fine-tune the LLM, enabling it to extract the aforementioned characteristics from outpatient medical records and to indicate the omission of mandatory items (Table S2). Training process We input the annotated fine-tuning set into the LoRA (Low-Rank Adaptation) framework to fine-tune three selected base large language models: ChatGLM3-6B, LLAMA-3-8B, and HuatuoGPT-II. During training, we monitor the models' performance on the fine-tuning set, adjust hyperparameters accordingly, and optimize model performance. Subsequently, an independent test set is employed to evaluate each model's accuracy, and other critical metrics in predicting medical record quality control points. The model demonstrating the best performance will be selected as EMRs-ANGEL for further experimentation. The details of model construction were presented in the supplementary materials. Inconsistency Inconsistency can be classified into three categories. Features that are labelled by the physician and not extracted by the model are defined as 'missed’. Features that are labelled by the physician and also labelled by the model but inconsistent with the physician's results are defined as 'error’. Features that are labelled by the physician as null but labelled by the model are defined as 'extra’. For each case, there were 4 items of characteristics, each with a weight of 0.25; 3 items of accompanied symptoms, each with a weight of 0.33; and 4 items of disease progression, each with a weight of 0.25. Symptom, Predisposing conditions, Influencing factors, Seizure characteristics, Location, and Duration each with a weight of 1. Therefore, in a given case, if one characteristic is incorrectly filled, it is counted as 0.25 errors; if an extra accompanied symptom is recorded, it is counted as 0.33 extra-recording; if a predisposing condition is omitted, it is counted as one missed-item (Table S1). AI-assisted EMRs recording 125 cases of EMRs from the original test set and the prompts of the SMART-ASSISTANT were presented to five attending physicians with at least 5 years of clinical experience. They were asked to revise and improve the original medical records based on the prompted omission items from the SMART-ASSISTANT. This is to replicate the EMRs recording process in a real-world clinical application scenario, where the EMRs is incomplete and the physician makes further inquiries according to SMART-ASSISTANT cues and finally completes the EMRs. Eventually, an assisted test set consisting of 125 medical records was obtained. Multi-reader multi-case study Physicians were asked to evaluate each EMRs for completeness on a Likert scale from 1 to 5 (1 being poor, and 5 being excellent) and diagnostic correlation on a Likert scale from 1 to 3 (1 being poor, and 3 being excellent) and to pick their preferred answer. For completeness, a higher score indicates a higher degree of completeness of the EMRs, corresponding to better history taking and more comprehensive EMRs recording. For diagnostic correlation, a higher score indicates a higher correlation of inferring the diagnosis based on the content of the medical record. Ten physicians participated in this study including 2 consultants, 4 attending physicians and 6 residents. They were randomized into two groups: an AI-first group and a control group, with five in each group. Physicians in the AI-first group first assessed 125 EMRs from the assisted test set and then, after a 3-week washout period, 125 EMRs from the original test set. In contrast, physicians in the control group assessed 125 EMRs from the original test set first, and after a 3-week washout period, they assessed 125 EMRs from the assisted test set. The physicians were blinded to their subgroups and to which test set they were reading throughout the study. The original medical record is displayed on the left. This area can be amended by the physician in accordance with the details of the consultation. The right side displays a real-time assistance area which can automatically extract the content of the quality control points, indicate omission items and provide the most likely three to five preliminary diagnoses based on the EMRs entered on the left side. Prospective cohort study Four consultants were invited to participate in the recruitment process. Consecutive patients attending the outpatient service of these four consultants at RHWU from 22nd July 2024 to 15th September 2024 were recruited to the study. Inclusion criteria: (1) patients aged 18 years or older; (2) patients presented with a chief complaint of abdominal pain; (3) first visit outpatients without previous definitive diagnosis; (4) patients able to give informed consent; (5) researchers believe that the participant is able to understand the process of this clinical trial and is willing to follow all study procedures. Exclusion criteria: (1) patients who have participated in other clinical trials, signed informed consent, and are in the follow-up phase of the other clinical trials; (2) patients with a history of substance abuse or mental disorder in the past 5 years; (3) researchers believe that the patient is not suitable to participate in the trial. Sample size: based on our preliminary study, we expected the QCP completeness rate (QCR) of human generated EMRs to be 70% and that of the AI-assisted recording to be 85%. Using the test for one proportion, with a power of 80% and a two-sided significance level of 0.05, 64 participants were required. We collected the following records on the enrolled participants: (1) de-identified doctor-patient consultation dialogue audio, with all sensitive information such as names, identification numbers, and other personal identifiers removed; (2) de-identified outpatient EMRs; (3) informed consent form signed by the participant. The gold standard consists of three components: audio transcription, QCP, and preliminary diagnosis. The performance of structured EMRs transcription was evaluated by 5 physicians (including 2 consultants and 3 attending physicians) on a 5 dimension Likert scale including factuality, integrity, normativity, readability, and logicality. The gold standard of audio transcription was labeled by 3 MD students based on the consultation dialogue audio. The gold standard of QCP was labeled by 2 residents and reviewed by 1 consultant. The preliminary diagnosis written by 4 consultants were regarded as the gold standard. EMRs-ANGEL was expected to perform audio transcription, structured EMRs transcription, QCP identification, and preliminary diagnosis. Statistical analysis To evaluate the capabilities of EMRs-ANGEL, accuracy for QCP identification and diagnosis were calculated for all the tests mentioned. The McNemar test was used to compare the accuracy of QCP recognition of the models. To evaluate physicians ratings for completeness and diagnostic correlation in retrospective validation and physicians ratings for factuality, integrity, normativity, readability, and logicality in prospective cohort, we used an ensemble scoring strategy 34 . In this method, the average score is calculated across raters for each case, and the aggregate score for each rating of human-generated EMRs is compared to the aggregate score of the respective rating of AI-assited EMRs or AI-generated EMRs using paired t-tests. Mann-Whitney U test was used to compare the QCR of physicians and the TCR. STS and CER were used to evaluate the ability of audio-to-text transcription and structured medical records transcription. We consider STS above 80% as a high reliability, 60–80% as a medium reliability, and below 60% as a low reliability. Declarations Role of the funding source The funder had no role in this study. Source of support : This work was partly supported by the Key Research and Development Program of Hubei Province (grant no. 2023BCB153, to Honggang Yu) and the College-enterprise Deepening Reform Project of Wuhan University (to Honggang Yu). Description : This study constructed and validated an intelligent system which can help physicians in consultation and can automatically generate high-quality electronic health records based on the consultation audio. Author Contribution H.Y. and J.Y. conceived and supervised the overall study. H.Y. , H.D. and J.Y. contributed to the study design and methodology. J.L., H.D., B.X., X.W. nd W.X. contributed to the data curation. D.H., J.L., B.X., X.W., W.X., M.D., S.T., J.W., C.L., B.C., T.Y., Z.C. and Q.Z were involved in data validation. H.D., J.L., X.W. and W.X. wrote the manuscript. H.Y., J.Y. and J.L. revised and edited the manuscript. All authors reviewed the manuscript. All authors approved the final version of the manuscript. References Häyrinen K, Saranto K, Nykänen P. Definition, structure, content, use and impacts of electronic health records: a review of the research literature. Int J Med Inform 2008;77(5):291-304. DOI: 10.1016/j.ijmedinf.2007.09.001. Lyles CR, Nelson EC, Frampton S, Dykes PC, Cemballi AG, Sarkar U. Using Electronic Health Record Portals to Improve Patient Engagement: Research Priorities and Best Practices. Ann Intern Med 2020;172(11 Suppl):S123-s129. DOI: 10.7326/m19-0876. Chawla NV, Davis DA. Bringing big data to personalized healthcare: a patient-centered framework. J Gen Intern Med 2013;28 Suppl 3(Suppl 3):S660-5. DOI: 10.1007/s11606-013-2455-8. Swinckels L, Bennis FC, Ziesemer KA, et al. The Use of Deep Learning and Machine Learning on Longitudinal Electronic Health Records for the Early Detection and Prevention of Diseases: Scoping Review. J Med Internet Res 2024;26:e48320. DOI: 10.2196/48320. Mangalmurti SS, Murtagh L, Mello MM. Medical malpractice liability in the age of electronic health records. N Engl J Med 2010;363(21):2060-7. DOI: 10.1056/NEJMhle1005210. Raab R, Küderle A, Zakreuskaya A, et al. Federated electronic health records for the European Health Data Space. Lancet Digit Health 2023;5(11):e840-e847. DOI: 10.1016/s2589-7500(23)00156-5. Ebbers T, Kool RB, Smeele LE, et al. The Impact of Structured and Standardized Documentation on Documentation Quality; a Multicenter, Retrospective Study. J Med Syst 2022;46(7):46. DOI: 10.1007/s10916-022-01837-9. Pavitt S, McHugh A, Chi K, et al. Improving Inpatient Consult Communication Through a Standardized Tool. Pediatrics 2021;147(5). DOI: 10.1542/peds.2020-0681. Smith GR, Jr., Rost K, Kashner TM. A trial of the effect of a standardized psychiatric consultation on health outcomes and costs in somatizing patients. Arch Gen Psychiatry 1995;52(3):238-43. DOI: 10.1001/archpsyc.1995.03950150070012. Ancker JS, Kern LM, Edwards A, et al. How is the electronic health record being used? Use of EHR data to assess physician-level variability in technology use. J Am Med Inform Assoc 2014;21(6):1001-8. DOI: 10.1136/amiajnl-2013-002627. Melnick ER, Dyrbye LN, Sinsky CA, et al. The Association Between Perceived Electronic Health Record Usability and Professional Burnout Among US Physicians. Mayo Clin Proc 2020;95(3):476-487. DOI: 10.1016/j.mayocp.2019.09.024. Toscano F, O'Donnell E, Broderick JE, et al. How Physicians Spend Their Work Time: an Ecological Momentary Assessment. J Gen Intern Med 2020;35(11):3166-3172. DOI: 10.1007/s11606-020-06087-4. Han J, Park J, Huh J, Oh U, Do J, Kim D. AscleAI: A LLM-based Clinical Note Management System for Enhancing Clinician Productivity. Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems: Association for Computing Machinery; 2024:Article 50. Wiest IC, Ferber D, Zhu J, et al. From Text to Tables: A Local Privacy Preserving Large Language Model for Structured Information Retrieval from Medical Documents. medRxiv2023. Zhang P, Kamel Boulos MN. Generative AI in Medicine and Healthcare: Promises, Opportunities and Challenges. 2023;15(9):286. (https://www.mdpi.com/1999-5903/15/9/286). Goodman RS, Patrinely JR, Jr., Osterman T, Wheless L, Johnson DB. On the cusp: Considering the impact of artificial intelligence language models in healthcare. Med 2023;4(3):139-140. DOI: 10.1016/j.medj.2023.02.008. Chiang CC, Luo M, Dumkrieger G, et al. A large language model-based generative natural language processing framework fine-tuned on clinical notes accurately extracts headache frequency from electronic health records. Headache 2024;64(4):400-409. DOI: 10.1111/head.14702. Gertz RJ, Dratsch T, Bunck AC, et al. Potential of GPT-4 for Detecting Errors in Radiology Reports: Implications for Reporting Accuracy. Radiology 2024;311(1):e232714. DOI: 10.1148/radiol.232714. Ding H, Xia W, Zhou Y, et al. Evaluation and practical application of prompt-driven ChatGPTs for EMR generation. NPJ digital medicine 2025;8(1):77. DOI: 10.1038/s41746-025-01472-x. Ge J, Sun S, Owens J, et al. Development of a liver disease-specific large language model chat interface using retrieval-augmented generation. Hepatology 2024;80(5):1158-1168. DOI: 10.1097/hep.0000000000000834. Mark Feldman, Lawrence S. Friedman, Lawrence J. Brandt, editors. Sleisenger and Fordtran's Gastrointestinal and Liver Disease. 11 ed: Elsevier, 2020. Lee P, Bubeck S, Petro J. Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine. N Engl J Med 2023;388(13):1233-1239. DOI: 10.1056/NEJMsr2214184. Ye C, Zweck E, Ma Z, Smith J, Katz S. Doctor Versus Artificial Intelligence: Patient and Physician Evaluation of Large Language Model Responses to Rheumatology Patient Questions in a Cross-Sectional Study. Arthritis Rheumatol 2024;76(3):479-484. DOI: 10.1002/art.42737. Patel D, Raut G, Zimlichman E, et al. Evaluating prompt engineering on GPT-3.5's performance in USMLE-style medical calculations and clinical scenarios generated by GPT-4. Sci Rep 2024;14(1):17341. DOI: 10.1038/s41598-024-66933-x. Gao Z, Li Z, Wang J, et al. FunASR: A Fundamental End-to-End Speech Recognition Toolkit. (https://ui.adsabs.harvard.edu/abs/2023arXiv230511013G). He H, Choi JDJae-p. The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders. (https://ui.adsabs.harvard.edu/abs/2021arXiv210906939H). Plaquet A, Bredin HJae-p. Powerset multi-class cross entropy loss for neural speaker diarization. (https://ui.adsabs.harvard.edu/abs/2023arXiv231013025P). Radford A, Kim JW, Xu T, Brockman G, McLeavey C, Sutskever IJae-p. Robust Speech Recognition via Large-Scale Weak Supervision. (https://ui.adsabs.harvard.edu/abs/2022arXiv221204356R). Wang H, Zheng S, Chen Y, Cheng L, Chen QJae-p. CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking. (https://ui.adsabs.harvard.edu/abs/2023arXiv230300332W). Zhang M, Qiao X, Zhao Y, et al. Knowledge Prompt for Whisper: An ASR Entity Correction Approach with Knowledge Base. 2023:2975-2979. Gans SL, Pols MA, Stoker J, Boermeester MA. Guideline for the diagnostic pathway in patients with acute abdominal pain. Dig Surg 2015;32(1):23-31. DOI: 10.1159/000371583. Lukic S, Mijac D, Filipovic B, et al. Chronic Abdominal Pain: Gastroenterologist Approach. Dig Dis 2022;40(2):181-186. DOI: 10.1159/000516977. Zeller JL, Burke AE, Glass RM. JAMA patient page. Acute abdominal pain. JAMA 2006;296(14):1800. DOI: 10.1001/jama.296.14.1800. Ayers JW, Poliak A, Dredze M, et al. Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA internal medicine 2023;183(6):589-596. DOI: 10.1001/jamainternmed.2023.1838. Video 1 Video 1 is not available with this version. Additional Declarations No competing interests reported. Supplementary Files npjSupplements.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7066667","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":491418266,"identity":"65a1d055-e16f-4294-b18e-73c571aa0d9f","order_by":0,"name":"Hongliu Du","email":"","orcid":"","institution":"Renmin Hospital of Wuhan University","correspondingAuthor":false,"prefix":"","firstName":"Hongliu","middleName":"","lastName":"Du","suffix":""},{"id":491418267,"identity":"a60ac34e-cc9e-435f-bb17-a53573d663e8","order_by":1,"name":"Jialing Li","email":"","orcid":"","institution":"Renmin Hospital of Wuhan University","correspondingAuthor":false,"prefix":"","firstName":"Jialing","middleName":"","lastName":"Li","suffix":""},{"id":491418268,"identity":"8b89c6eb-1f18-4b6e-a6b2-42bc17da0604","order_by":2,"name":"Bing Xiao","email":"","orcid":"","institution":"Renmin Hospital of Wuhan University","correspondingAuthor":false,"prefix":"","firstName":"Bing","middleName":"","lastName":"Xiao","suffix":""},{"id":491418269,"identity":"60b346f1-47ae-432a-90fe-cb255a68b5c0","order_by":3,"name":"Xueying Wang","email":"","orcid":"","institution":"Renmin Hospital of Wuhan University","correspondingAuthor":false,"prefix":"","firstName":"Xueying","middleName":"","lastName":"Wang","suffix":""},{"id":491418270,"identity":"907032f8-b88e-4a89-a88b-5b3e73519b36","order_by":4,"name":"Wenxin Xue","email":"","orcid":"","institution":"Renmin Hospital of Wuhan University","correspondingAuthor":false,"prefix":"","firstName":"Wenxin","middleName":"","lastName":"Xue","suffix":""},{"id":491418271,"identity":"754c9949-7f56-4fcd-9404-788e198be92b","order_by":5,"name":"Mei Deng","email":"","orcid":"","institution":"Renmin Hospital of Wuhan University","correspondingAuthor":false,"prefix":"","firstName":"Mei","middleName":"","lastName":"Deng","suffix":""},{"id":491418272,"identity":"6196712d-adda-4864-b5a2-bb7c04220a1e","order_by":6,"name":"Shuzhe Tan","email":"","orcid":"","institution":"Renmin Hospital of Wuhan University","correspondingAuthor":false,"prefix":"","firstName":"Shuzhe","middleName":"","lastName":"Tan","suffix":""},{"id":491418273,"identity":"b53e18b4-bf93-4833-9cee-ae418b9462d1","order_by":7,"name":"Jiamin Wang","email":"","orcid":"","institution":"Renmin Hospital of Wuhan University","correspondingAuthor":false,"prefix":"","firstName":"Jiamin","middleName":"","lastName":"Wang","suffix":""},{"id":491418274,"identity":"b9716860-aec8-475d-ac2e-48888add6c92","order_by":8,"name":"Chaijie Luo","email":"","orcid":"","institution":"Renmin Hospital of Wuhan University","correspondingAuthor":false,"prefix":"","firstName":"Chaijie","middleName":"","lastName":"Luo","suffix":""},{"id":491418275,"identity":"6719609c-7293-4ef7-858e-4dd99444f91b","order_by":9,"name":"Boru Chen","email":"","orcid":"","institution":"Renmin Hospital of Wuhan University","correspondingAuthor":false,"prefix":"","firstName":"Boru","middleName":"","lastName":"Chen","suffix":""},{"id":491418276,"identity":"7d0f6263-bab9-4511-b43f-f6b388cd1109","order_by":10,"name":"Ting Yang","email":"","orcid":"","institution":"Renmin Hospital of Wuhan University","correspondingAuthor":false,"prefix":"","firstName":"Ting","middleName":"","lastName":"Yang","suffix":""},{"id":491418277,"identity":"daa4f8db-4005-449c-82c0-9c99d3fe0f52","order_by":11,"name":"Zhan Chen","email":"","orcid":"","institution":"Renmin Hospital of Wuhan University","correspondingAuthor":false,"prefix":"","firstName":"Zhan","middleName":"","lastName":"Chen","suffix":""},{"id":491418278,"identity":"bb45dbd7-0360-4b32-873e-088ef517b572","order_by":12,"name":"Qinxuan Zu","email":"","orcid":"","institution":"Renmin Hospital of Wuhan University","correspondingAuthor":false,"prefix":"","firstName":"Qinxuan","middleName":"","lastName":"Zu","suffix":""},{"id":491418279,"identity":"8f801aeb-4b97-4fad-8b39-609a5b120072","order_by":13,"name":"Joseph J. Y. Sung","email":"","orcid":"","institution":"Nanyang Technological University","correspondingAuthor":false,"prefix":"","firstName":"Joseph","middleName":"J. Y.","lastName":"Sung","suffix":""},{"id":491418280,"identity":"df62ece0-8f45-476b-ac67-ba0e97f2d81c","order_by":14,"name":"Honggang Yu","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA6ElEQVRIiWNgGAWjYFACHhiD+QAzmD5AvBa2BJK18BgQp8XgRu7BxwW/auXN+dd8ky5sY5Dju5HA+LkAr5a8ZOOZfccNd854u016ZhuDseSNBGbpGXi15JhJ8/YcY9xw4+y227xtDIkbbiSwMfMQocV+w40zz0Ba6onTwvOjJnHD+R42kJYEA0JaJM+8MTbmbTiQvOEGm/lvnnMShjPPPGyWxqeF73iO4WOeP3W2G84ffmzMU2Yjz3c8+eBnfFoUDgAJxrbDDAwSCSC+BIjbgEcDA4M8WPpPHQMD/wG8CkfBKBgFo2AEAwCAPVLupJ33vgAAAABJRU5ErkJggg==","orcid":"","institution":"Renmin Hospital of Wuhan University","correspondingAuthor":true,"prefix":"","firstName":"Honggang","middleName":"","lastName":"Yu","suffix":""}],"badges":[],"createdAt":"2025-07-07 14:53:07","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7066667/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7066667/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":87827649,"identity":"66d6bdbf-7bdd-42c4-8545-097cb7e3ffc3","added_by":"auto","created_at":"2025-07-29 11:57:58","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":318980,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eApplication Scenario Diagram.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003ea. Consultation dialogue between a physician and a patient suffering from abdominal pain. b. SMART-ASSISTANT converts the audio of the consultation dialogue into free text. c. SMART-ASSISTANT automatically generates structured electronic medical records (EMRs) based on free text. d. SMART-ASSISTANT performs quality control of structured EMRs, indicates omission items and provides preliminary diagnosis.\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7066667/v1/640feed2798f3e18024a7257.png"},{"id":87828539,"identity":"19d6954b-a7a9-45c1-882f-cc3e04fec27c","added_by":"auto","created_at":"2025-07-29 12:05:58","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":496004,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003ePhysician evaluation of EMRs completeness on a Likert scale from 1 to 5 and diagnostic correlation on a Likert scale from 1 to 3 between AI-assisted set (AI, indicated by red color) and original test set (Control, indicated by blue color).\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003ea. The frequency distribution of the 10 physicians’ ratings of the EMRs completeness. b. The frequency distribution of the 10 physicians’ ratings of the diagnostic correlation. c. Each physician’s rating of the EMRs completeness. d. Each physician’s rating of the diagnostic correlation.\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-7066667/v1/0563a73259008514907ea129.png"},{"id":87827651,"identity":"0d07357a-f961-404d-ba9e-828194656d8c","added_by":"auto","created_at":"2025-07-29 11:57:58","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":647832,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003ePhysician evaluation of EMRs quality and objective evaluation of EMRs completeness.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003ea. Physician evaluation on a five dimensions scale for human-generated EMRs (blue) and AI-generated EMRs (red) based on consultation dialogue, with error bars indicating the standard deviation. b. Each physician’s rating of the diagnostic correlation. c. Radar map showing TCR and QCR in original test set. d. Radar map showing TCR and QCR in prospective cohort.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eTCR\u003c/em\u003e, theoretical completion rate. \u003cem\u003eQCR\u003c/em\u003e, quality control points completion rate.\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-7066667/v1/38dce390f085e047d7d46ca9.png"},{"id":87828540,"identity":"d2775a4e-2adc-4350-b931-f46e82feb5a9","added_by":"auto","created_at":"2025-07-29 12:05:58","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":441963,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eStudy flowchart.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003ea. model construction and retrospective validation. b. prospective cohort study.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eEMRs\u003c/em\u003e, electronic medical records; \u003cem\u003eLLM\u003c/em\u003e, large language model.\u003c/p\u003e","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-7066667/v1/3646fe6b0800ac4c4eafb621.png"},{"id":87828538,"identity":"ce8c4826-f787-4ed7-a342-195745eec878","added_by":"auto","created_at":"2025-07-29 12:05:58","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":710328,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eInterface of SMART-ASSISTANT.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe original medical record is displayed on the left. This area can be amended by the physician in accordance with the details of the consultation. The right side displays a real-time assistance area which can automatically extract the content of the quality control points, indicate omission items and provide the most likely three to five preliminary diagnoses based on the EMRs entered on the left side.\u003c/p\u003e","description":"","filename":"floatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-7066667/v1/6547738279b5813834b784bc.png"},{"id":88839889,"identity":"aa322a4c-4890-42b5-92c7-0c1a5688b989","added_by":"auto","created_at":"2025-08-12 02:16:37","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":3900534,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7066667/v1/393eb2aa-02d6-4f08-97a1-0d590d5df1ca.pdf"},{"id":87828537,"identity":"4aa99959-2b91-4d42-86ea-212b97035bcc","added_by":"auto","created_at":"2025-07-29 12:05:58","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":40729,"visible":true,"origin":"","legend":"","description":"","filename":"npjSupplements.docx","url":"https://assets-eu.researchsquare.com/files/rs-7066667/v1/cfd041b00d38a6f0e876c2df.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Constructing a Smart-Assistant for Improving the Outpatient Service Quality in Real-time: a Prospective Single-center Cohort Study","fulltext":[{"header":"Introduction","content":"\u003cp\u003eElectronic medical records (EMRs) constitute indispensable repositories of integrated patient information, capturing presenting symptoms, therapeutic interventions, thus take a pivotal role during patient care and referral\u003csup\u003e\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u003c/sup\u003e. EMRs are being widely used in the excitement over health information technology, and are associated with medical malpractice liability and may exert effectiveness as well\u003csup\u003e\u003cspan additionalcitationids=\"CR3\" citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u003c/sup\u003e. High-quality EMRs can reduce adverse events and claims by reducing discontinuities and errors in care and by improving clinical decisions\u003csup\u003e\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u003c/sup\u003e. Incomplete medical records and difficulties in accessing a patient\u0026rsquo;s medical history compromise the overall quality of care that clinicians can provide\u003csup\u003e\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u003c/sup\u003e. Therefore, both the patients' welfare and the interests of medical institutions can be protected as the quality of EMRs improves.\u003c/p\u003e\u003cp\u003eConsultation is one of the essential steps in medical history collection, which could lead to appropriate diagnosis thinking and prompt treatment choice. Standardized consultation is a powerful guarantee for the improvement of EMRs\u003csup\u003e\u003cspan additionalcitationids=\"CR8\" citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e\u003c/sup\u003e. In the case of patients with acute abdominal pain (AAP), accurate and rapid consultation can help physicians diagnose or eliminate fatal diseases promptly. However, the quality of consultation and EMRs varies considerably attributed to the large volumes of consultations, the high level of fatigue, and the wide variations in clinical experience and expertise of the physicians\u003csup\u003e\u003cspan additionalcitationids=\"CR11\" citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u003c/sup\u003e. Misdiagnosis and delayed diagnosis may occur if physicians fail to conduct a complete and comprehensive interview based on the patients\u0026rsquo; complaints, which directly harm the patients\u0026rsquo; benefits and may lead to compensation for medical malpractice liability\u003csup\u003e\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u003c/sup\u003e. There is a dearth of efficacious assistance to regulate the process of medical history collection.\u003c/p\u003e\u003cp\u003eLarge language models (LLM) and generative artificial intelligence (AI) may have vast potential in patient care, especially in medical document management compared with image recognition\u003csup\u003e\u003cspan additionalcitationids=\"CR14\" citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u003c/sup\u003e. The accelerated evolution of LLM has prompted experts to believe that we are likely 'on the cusp' of a monumental shift in healthcare delivery and evaluation\u003csup\u003e\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e\u003c/sup\u003e. Recent studies have demonstrated that the large language model is capable of generate EMRs of lung cancer cases, can accurately extracting the headache frequency, and is comparable with that of radiologists in detecting five types of errors in radiology reports\u003csup\u003e\u003cspan additionalcitationids=\"CR18\" citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e. However, the majority of existing studies have concentrated on 'prompt engineering', which is likely to cause unreliable advice due to hallucination, and there is no precedent for the application of LLM in outpatient EMRs quality improvement\u003csup\u003e\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\u003cp\u003eThis study aims to develop a fine-tuning LLM-based system (SMART-ASSISTANT) to automatically generate standardized EMRs based on consultation audio and to improve the quality of EMRs with abdominal pain as the chief complaint. This study is the first to use AI to assist the entire process of outpatient work, including consultation, audio transcription, structured EMRs generation, EMRs quality improvement, and assisted diagnosis. We conducted a prospective cohort study to evaluate the performance of the system in a real clinical setting. The outcomes demonstrate that the system is an effective means of improving the quality of outpatient EMRs and assisting physicians in daily outpatient service (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003e\u003cb\u003eOutcome\u003c/b\u003e\u003c/p\u003e\u003cp\u003eThe SMART-ASSISTANT consists of four parts: audio transcription, structured EMRs generation, quality control, and assisted diagnosis. We validated the above mentioned functions through the simulated set, the retrospective set, a multi-reader multi-case (MRMC) study, and a prospective cohort study.\u003c/p\u003e\u003cp\u003eThe semantic textual similarity (STS) and character error rate (CER) were used to evaluate the ability of audio-to-text transcription and structured EMRs transcription. The accuracy was used to evaluate the capabilities of AI in recognizing quality control points (QCP). Using physician-labelled results as the gold standard, accuracy\u0026thinsp;=\u0026thinsp;concordant predictions/total number of cases. The QCP completion rate (QCR) was used to evaluate the quality of human generated EMRs. QCR\u0026thinsp;=\u0026thinsp;number of completed mandatory points/total number of mandatory points. The omission detection rate (ODR) and the standardization rate (SR) were used to evaluate the capabilities of AI in recognizing omission points. ODR\u0026thinsp;=\u0026thinsp;number of omission points correctly detected by AI/total number of omission points. SR\u0026thinsp;=\u0026thinsp;number of omission points correctly detected by AI/total number of mandatory points. The theoretical completion rate (TCR) was used to evaluate the quality of EMRs recorded by human with the assistance of AI. TCR\u0026thinsp;=\u0026thinsp;QCR\u0026thinsp;+\u0026thinsp;SR. The TCR indicates the completeness of EMRs in case whenever the AI correctly identifies an omission point in the human consultation records, it prompts the physician to conduct further inquiry and fill in the omission point. A Likert scale was used to assess AI's ability to improve EMRs quality.\u003c/p\u003e\u003cp\u003e\u003cb\u003ePerformance of audio transcription and structured EMRs transcription\u003c/b\u003e\u003c/p\u003e\u003cp\u003eThe STS and CER of audio-to-text transcription using the original engine powered by OpenAI-whisper were 0.5654\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0908 and 0.4559\u0026thinsp;\u0026plusmn;\u0026thinsp;0.1900. After the entity correction, the WER improved significantly meanwhile the CER decreased significantly (WER, 0.5912\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0853, \u003cem\u003eP\u003c/em\u003e\u0026lt;0.01; CER, 0.3665\u0026thinsp;\u0026plusmn;\u0026thinsp;0.1407, \u003cem\u003eP\u003c/em\u003e\u0026lt;0.01). FunASR with entity correction achieved the best performance thus was used for prospective validation as the audio transcribe module in SMART-ASSISTANT (WER, 0.9080\u0026thinsp;\u0026plusmn;\u0026thinsp;0.1348, \u003cem\u003eP\u003c/em\u003e\u0026lt;0.001; CER, 0.2643\u0026thinsp;\u0026plusmn;\u0026thinsp;0.3534, \u003cem\u003eP\u003c/em\u003e\u0026lt;0.01) (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e\u003cp\u003eIn the performance of structured EMRs transcription, the model was found to have a greater proportion of cases with high reliability for allergy history, auxiliary examination, past medical history and other history (91.30%, 78.26%, 73.91%). The STS and CER of structured EMRs transcription were 0.8114\u0026thinsp;\u0026plusmn;\u0026thinsp;0.1259 and 0.7139\u0026thinsp;\u0026plusmn;\u0026thinsp;0.4372 (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e\u003cp\u003e\u003cb\u003ePerformance in detecting quality control key points\u003c/b\u003e\u003c/p\u003e\u003cp\u003eThe performance of the three LLMs was evaluated both before and after fine-tuning, and a comparison was made with the performance of GPT-4 and DeepSeek.\u003c/p\u003e\u003cp\u003ePrior to fine-tuning, the accuracy of HuatuoGPT-II, Llama-3, and ChatGLM-3 for QCP extraction was 58.23% [95%CI, 55.32\u0026ndash;61.08%], 68.74% [95%CI, 65.98\u0026ndash;71.39%], and 60.31% [95%CI, 57.42\u0026ndash;63.13%], respectively. These values were all inferior to that of GPT-4 significantly (71.61% [95%CI, 68.92\u0026ndash;74.18%], \u003cem\u003eP\u003c/em\u003e\u0026lt;0.001). It appears that there is a great challenge for LLMs in interpreting the connotative content of EMRs due to the lack of medical domain knowledge without the support of supervised learning.\u003c/p\u003e\u003cp\u003eAfter fine-tuning, the accuracy of the LLMs for QCP extraction was significantly improved (\u003cem\u003eP\u003c/em\u003e\u0026lt;0.001). All three LLMs outperformed the GPT-4 and DeepSeek significantly (\u003cem\u003eP\u003c/em\u003e\u0026lt;0.001), indicating that fine-tuning can facilitate greater progress in specific tasks for LLMs without medical domain knowledge. HuatuoGPT-II-FT demonstrated the most optimal performance (91.53% [95%CI, 89.76\u0026ndash;93.02%]). Among them, HuatuoGPT-II-FT achieved the best performance for identifying onset characteristics, location, duration, and accompanying symptoms (85.60% [95%CI, 78.38\u0026ndash;90.69%], 91.20% [95%CI, 84.93\u0026ndash;95.02%], 94.40% [95%CI, 88.89\u0026ndash;97.26%], and 90.13% [95%CI, 86.69\u0026ndash;92.75%]), while Llama-3-FT exhibited the greatest efficacy in identifying predisposing conditions and disease progression (97.60%, 91.00%) (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e).\u003c/p\u003e\u003cp\u003eThe QCR and the TCR in the original test set were 73.60% [95%CI, 70.33\u0026ndash;76.63%] and 97.87% [95%CI, 96.57\u0026ndash;98.69%], indicating that there will be a significant improvement in EMRs completeness as long as physicians are able to follow the prompts of SMART-ASSISTANT without compromise (\u003cem\u003eP\u003c/em\u003e\u0026lt;0.01) (Fig.\u0026nbsp;\u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e3\u003c/span\u003e).\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003e\u003cb\u003ePerformance in disease diagnosis\u003c/b\u003e\u003c/p\u003e\u003cp\u003eWe omitted diagnoses from the original EMRs and models were trained to predict the most likely 3\u0026ndash;5 diagnoses. If the output contains the diagnosis that was originally documented in the EMRs, it will be deemed accurate.\u003c/p\u003e\u003cp\u003eBefore fine-tuning, the diagnostic accuracy of HuatuoGPT-II, Llama-3, ChatGLM-3, and GPT-4 were 43.20% [95%CI, 34.85\u0026ndash;51.96%], 45.60% [95%CI, 37.13\u0026ndash;54.33%], 62.40% [95%CI, 53.66\u0026ndash;70.40%], 47.20% [95%CI, 38.66\u0026ndash;55.90%]. After fine-tuning, the accuracy of diagnosis was significantly improved in three LLMs (78.40% vs. 43.20%, \u003cem\u003eP\u003c/em\u003e\u0026lt;0.001; 75.20% vs. 45.60%, \u003cem\u003eP\u003c/em\u003e\u0026lt;0.001; 72.00% vs. 62.40%, \u003cem\u003eP\u003c/em\u003e\u0026lt;0.05) (Table \u003cspan refid=\"MOESM1\" class=\"InternalRef\"\u003eS1\u003c/span\u003e).\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003ePerformance in audio-free text transcription and structured EMRs transcription.\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"6\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\"\u0026plusmn;\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\"\u0026plusmn;\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e\u003cp\u003eSubject\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\" morerows=\"1\" rowspan=\"2\"\u003e\u003cp\u003eSTS\u003c/p\u003e\u003cp\u003e(mean\u0026thinsp;\u0026plusmn;\u0026thinsp;SD)\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\" morerows=\"1\" rowspan=\"2\"\u003e\u003cp\u003eCER\u003c/p\u003e\u003cp\u003e(mean\u0026thinsp;\u0026plusmn;\u0026thinsp;SD)\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colspan=\"3\" nameend=\"c6\" namest=\"c4\"\u003e\u003cp\u003ePercentage of Different Reliability\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c4\"\u003e\u003cp\u003ehigh\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c5\"\u003e\u003cp\u003emedium\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c6\"\u003e\u003cp\u003elow\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e\u003cb\u003eSimulated dataset\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e\u003cb\u003eAudio transcription\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eWhisper\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e\u003cp\u003e0.5654\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0908\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e\u003cp\u003e0.4559\u0026thinsp;\u0026plusmn;\u0026thinsp;0.1900\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e26.67%\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e73.33%\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eWhisper\u0026thinsp;+\u0026thinsp;Entity Correction\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e\u003cp\u003e0.5912**\u0026plusmn;0.0853\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e\u003cp\u003e0.3665**\u0026plusmn;0.1407\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e53.33%\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e46.67%\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eFunASR\u0026thinsp;+\u0026thinsp;Entity Correction\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e\u003cp\u003e0.9080***\u0026plusmn;0.1348\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e\u003cp\u003e0.2643**\u0026plusmn;0.3534\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e100.00%\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e0\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eSMART-ASSISTANT in prospective cohort\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e\u003cp\u003e0.9253\u0026thinsp;\u0026plusmn;\u0026thinsp;0.1179\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e\u003cp\u003e0.2058\u0026thinsp;\u0026plusmn;\u0026thinsp;0.2995\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e100.00%\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e0\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e\u003cb\u003eStructured EMRs transcription\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eChief Complaint\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e\u003cp\u003e0.7388\u0026thinsp;\u0026plusmn;\u0026thinsp;0.1732\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e\u003cp\u003e0.7385\u0026thinsp;\u0026plusmn;\u0026thinsp;0.4610\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e39.13%\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e34.78%\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e26.09%\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eHistory of Present Illness\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e\u003cp\u003e0.7390\u0026thinsp;\u0026plusmn;\u0026thinsp;0.1225\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e\u003cp\u003e1.6016\u0026thinsp;\u0026plusmn;\u0026thinsp;1.2129\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e21.74%\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e65.22%\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e13.04%\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003ePast Medical History and Other History\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e\u003cp\u003e0.8427\u0026thinsp;\u0026plusmn;\u0026thinsp;0.2428\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e\u003cp\u003e0.5571\u0026thinsp;\u0026plusmn;\u0026thinsp;0.7940\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e73.91%\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e8.70%\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e17.39%\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eAllergy History\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e\u003cp\u003e0.9384\u0026thinsp;\u0026plusmn;\u0026thinsp;0.1979\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e\u003cp\u003e0.0408\u0026thinsp;\u0026plusmn;\u0026thinsp;0.1210\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e91.30%\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e4.35%\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e4.35%\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eAuxiliary Examination\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e\u003cp\u003e0.7981\u0026thinsp;\u0026plusmn;\u0026thinsp;0.3576\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e\u003cp\u003e0.4016\u0026thinsp;\u0026plusmn;\u0026thinsp;0.3358\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e78.26%\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e4.35%\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e17.39%\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eTotal\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e\u003cp\u003e0.8114\u0026thinsp;\u0026plusmn;\u0026thinsp;0.1259\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e\u003cp\u003e0.7139\u0026thinsp;\u0026plusmn;\u0026thinsp;0.4372\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e56.52%\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e39.13%\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e4.35%\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003e\u003cem\u003eSTS\u003c/em\u003e, semantic textual similarity. \u003cem\u003eCER\u003c/em\u003e, character error rate.\u003c/p\u003e\u003cp\u003eWe consider STS above 80% as a high reliability, 60\u0026ndash;80% as a medium reliability, and below 60% as a low reliability.\u003c/p\u003e\u003cp\u003e*Compared with OpenAI.\u003c/p\u003e\u003cp\u003e*Significant at 5% level. **Significant at 1% level. ***Significant at 0.1% level\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003ePerformance between LLMs in quality-control points extraction.\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"10\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c10\" colnum=\"10\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e\u003cp\u003eQuality Control \u003c/p\u003e\u003cp\u003eKey Points\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colspan=\"8\" nameend=\"c9\" namest=\"c2\"\u003e\u003cp\u003eModel performance in retrospective EMRs\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c10\" morerows=\"1\" rowspan=\"2\"\u003e\u003cp\u003eModel performance in \u003c/p\u003e\u003cp\u003eprospective cohort\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eHuatuoGPT-II\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003eLlama-3\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c4\"\u003e\u003cp\u003eChatGLM3\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c5\"\u003e\u003cp\u003eChatGPT-4\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c6\"\u003e\u003cp\u003eDeepSeek\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c7\"\u003e\u003cp\u003eHuatuoGPT-II-FT\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c8\"\u003e\u003cp\u003eLlama-3-FT\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c9\"\u003e\u003cp\u003eChatGLM3-FT\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eSymptom (95%CI), %\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e32.80 (25.19\u0026ndash;41.44)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e27.20 (20.17\u0026ndash;35.59)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e22.40 (15.98\u0026ndash;30.47)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e19.20 (13.26\u0026ndash;26.98)\u003csup\u003e###\u003c/sup\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e\u003cp\u003e23.20 (16.67\u0026ndash;31.33)\u003csup\u003e###\u003c/sup\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e\u003cp\u003e100 (97.12\u0026ndash;100.00)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e\u003cp\u003e100 (97.12\u0026ndash;100.00)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e\u003cp\u003e100 (97.12\u0026ndash;100.00)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e\u003cp\u003e98.39 (91.42\u0026ndash;99.72)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eCharacteristics (95%CI), %\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e65.60 (61.33\u0026ndash;69.63)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e74.60 (70.61\u0026ndash;78.22)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e62.20 (57.87\u0026ndash;66.34)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e85.80 (82.47\u0026ndash;88.59)\u003csup\u003e###\u003c/sup\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e\u003cp\u003e59.20 (54.84\u0026ndash;63.42)\u003csup\u003e###\u003c/sup\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e\u003cp\u003e92.00 (89.29\u0026ndash;94.07)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e\u003cp\u003e92.00 (89.29\u0026ndash;94.07)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e\u003cp\u003e89.80 (86.84\u0026ndash;92.16)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e\u003cp\u003e95.97 (92.74\u0026ndash;97.80)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003ePredisposing conditions (95%CI), %\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e72.00 (63.56\u0026ndash;79.12)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e83.20 (75.65\u0026ndash;88.74)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e80.00 (72.14\u0026ndash;86.07)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e96.00 (90.98\u0026ndash;98.28)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e\u003cp\u003e60.00 (51.24\u0026ndash;68.17)\u003csup\u003e###\u003c/sup\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e\u003cp\u003e96.00 (90.98\u0026ndash;98.28)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e\u003cp\u003e97.60 (93.18\u0026ndash;99.18)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e\u003cp\u003e92.80 (86.88\u0026ndash;96.17)**\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e\u003cp\u003e83.87 (72.79-91.00)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eInfluencing factors (95%CI), %\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e56.80 (48.04\u0026ndash;65.15)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e76.80 (68.67\u0026ndash;83.33)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e72.80 (64.41\u0026ndash;79.83)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e85.60 (78.38\u0026ndash;90.96)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e\u003cp\u003e8.00 (4.40\u0026ndash;14.10)\u003csup\u003e###\u003c/sup\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e\u003cp\u003e84.80 (77.48\u0026ndash;90.05)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e\u003cp\u003e80.80 (73.02\u0026ndash;86.74)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e\u003cp\u003e80.80 (73.02\u0026ndash;86.74)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e\u003cp\u003e93.55 (84.55\u0026ndash;97.46)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eOnset characteristics (95%CI), %\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e24.00 (17.36\u0026ndash;32.19)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e34.40 (26.65\u0026ndash;43.08)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e14.40 (9.31\u0026ndash;21.62)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e45.60 (37.13\u0026ndash;54.33)\u003csup\u003e###\u003c/sup\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e\u003cp\u003e3.20 (1.25\u0026ndash;7.94)\u003csup\u003e###\u003c/sup\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e\u003cp\u003e85.60 (78.38\u0026ndash;90.69)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e\u003cp\u003e84.80 (77.48\u0026ndash;90.05)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e\u003cp\u003e84.80 (77.48\u0026ndash;90.05)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e\u003cp\u003e79.03 (67.36\u0026ndash;87.31)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eLocation (95%CI), %\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e68.00 (59.39\u0026ndash;75.54)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e72.00 (63.56\u0026ndash;79.12)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e71.20 (62.72\u0026ndash;78.41)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e85.60 (78.38\u0026ndash;90.96)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e\u003cp\u003e45.60 (37.13\u0026ndash;54.33)\u003csup\u003e###\u003c/sup\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e\u003cp\u003e91.20 (84.93\u0026ndash;95.02)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e\u003cp\u003e88.80 (82.08\u0026ndash;93.21)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e\u003cp\u003e88.00 (81.14\u0026ndash;92.59)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e\u003cp\u003e87.10 (76.55\u0026ndash;93.32)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eDuration (95%CI), %\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e75.20 (66.95\u0026ndash;81.94)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e83.20 (75.65\u0026ndash;88.74)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e91.20 (84.93\u0026ndash;95.02)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e91.20 (84.93\u0026ndash;95.02)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e\u003cp\u003e50.40 (41.75\u0026ndash;59.02)\u003csup\u003e###\u003c/sup\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e\u003cp\u003e94.40 (88.89\u0026ndash;97.26)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e\u003cp\u003e91.20 (84.93\u0026ndash;95.02)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e\u003cp\u003e92.00 (85.90\u0026ndash;95.60)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e\u003cp\u003e98.39 (91.42\u0026ndash;99.72)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eAccompanied symptoms (95%CI), %\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e75.46 (70.87\u0026ndash;79.55)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e86.93 (83.14\u0026ndash;89.97)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e80.80 (76.51\u0026ndash;84.47)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e90.13 (86.69\u0026ndash;92.75)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e\u003cp\u003e47.47 (42.47\u0026ndash;52.52)\u003csup\u003e###\u003c/sup\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e\u003cp\u003e90.13 (86.69\u0026ndash;92.75)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e\u003cp\u003e84.40 (80.81\u0026ndash;88.08)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e\u003cp\u003e83.73 (79.66\u0026ndash;87.12)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e\u003cp\u003e94.62 (90.38\u0026ndash;97.05)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eDisease progression (95%CI), %\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e54.20 (49.82\u0026ndash;58.52)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e80.40 (76.69\u0026ndash;83.64)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e47.80 (43.46\u0026ndash;52.18)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e45.40 (41.09\u0026ndash;49.78)\u003csup\u003e###\u003c/sup\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e\u003cp\u003e5.00 (3.41\u0026ndash;7.28)\u003csup\u003e###\u003c/sup\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e\u003cp\u003e89.60 (86.62\u0026ndash;91.98)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e\u003cp\u003e91.00 (88.17\u0026ndash;93.21)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e\u003cp\u003e86.80 (83.55\u0026ndash;89.49)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e\u003cp\u003e99.19 (97.10-99.78)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eTotal (95%CI), %\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e58.23 (55.32\u0026ndash;61.08)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e68.74 (65.98\u0026ndash;71.39)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e60.31 (57.42\u0026ndash;63.13)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e71.61 (68.92\u0026ndash;74.18)\u003csup\u003e###\u003c/sup\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e\u003cp\u003e33.56 (30.90-36.41)\u003csup\u003e###\u003c/sup\u003e\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e\u003cp\u003e91.53 (89.76\u0026ndash;93.02)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e\u003cp\u003e90.11 (88.23\u0026ndash;91.72)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e\u003cp\u003e88.75 (86.77\u0026ndash;90.47)***\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e\u003cp\u003e94.40 (92.84\u0026ndash;95.63)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003ctfoot\u003e\u003ctr\u003e\u003ctd colspan=\"10\"\u003e\u003cem\u003eLLM\u003c/em\u003e, large language model\u003c/td\u003e\u003c/tr\u003e\u003c/tfoot\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003e*Compared with large language models before fine-tuning. Since fine-tuning requires an open-source model, we cannot choose GPT-4 or DeeSseek for fine-tuning.\u003c/p\u003e\u003cp\u003e\u003csup\u003e#\u003c/sup\u003eCompared with HuatuoGPT-II-FT.\u003c/p\u003e\u003cp\u003e*/\u003csup\u003e#\u003c/sup\u003eSignificant at 5% level. **/\u003csup\u003e##\u003c/sup\u003eSignificant at 1% level. ***/\u003csup\u003e###\u003c/sup\u003eSignificant at 0.1% level\u003c/p\u003e\u003cp\u003e\u003cb\u003eMulti-reader multi-case study\u003c/b\u003e\u003c/p\u003e\u003cp\u003e Since HuatuoGPT-II-FT performs better than the other models in both quality control and diagnosis, it was regarded as the SMART-ASSISTANT and the results of HuatuoGPT-II-FT are provided to physicians to assist EMRs recording and diagnosis. The average ensemble completeness and diagnostic correlation rating in AI-first group was mean 4.45\u0026thinsp;\u0026plusmn;\u0026thinsp;SD 0.68 and 2.64\u0026thinsp;\u0026plusmn;\u0026thinsp;0.59 in the first round. After a 3-week washout period, physicians in AI-first group rated the original test set with a significantly lower mean completeness and diagnostic correlation score than in the first round (4.04\u0026thinsp;\u0026plusmn;\u0026thinsp;0.87, \u003cem\u003eP\u003c/em\u003e\u0026lt;0.01, 2.45\u0026thinsp;\u0026plusmn;\u0026thinsp;0.71, \u003cem\u003eP\u003c/em\u003e\u0026lt;0.01). Physicians in the control group had a mean completeness rating of 3.80\u0026thinsp;\u0026plusmn;\u0026thinsp;0.93 and a mean diagnostic correlation rating of 2.21\u0026thinsp;\u0026plusmn;\u0026thinsp;0.77 in the first round. After a three-week washout period, they rated the AI-assisted test set significantly better than the first round on both mean completeness rating and diagnostic correlation rating (4.09\u0026thinsp;\u0026plusmn;\u0026thinsp;0.90, \u003cem\u003eP\u003c/em\u003e\u0026lt;0.01, 2.43\u0026thinsp;\u0026plusmn;\u0026thinsp;0.63, \u003cem\u003eP\u003c/em\u003e\u0026lt;0.01). Physicians in the AI-first and control groups rated significantly higher on the AI-assisted test set than on the original test set when assessing the completeness and diagnostic correlation of the EMRs. This suggests that the majority of physicians in both the AI-first and control groups perceived the AI-assisted EMRs to be more complete and were able to infer a higher degree of diagnostic correlation (Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e2\u003c/span\u003e).\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003e\u003cb\u003eProspective observational cohort study\u003c/b\u003e\u003c/p\u003e\u003cp\u003eThe STS and CER of audio transcription were 0.9253\u0026thinsp;\u0026plusmn;\u0026thinsp;0.1179 and 0.2058\u0026thinsp;\u0026plusmn;\u0026thinsp;0.2995 (Table \u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e). In QCP recognition, SMART-ASSISTANT achieved the accuracy of 94.40% [95%CI, 92.84\u0026ndash;95.63%], which was better than the retrospective test. In preliminary diagnosis, SMART-ASSISTANT achieved the accuracy of 50.00% [95%CI, 37.92\u0026ndash;62.08%], which was lower than the retrospective test (Table \u003cspan class=\"InternalRef\"\u003eS1\u003c/span\u003e). This discrepancy may be attributed to the fact that the original test set encompasses EMRs from residents, attending physicians, and consultants, with a relatively higher occurrence of common diseases. In contrast, the prospective cohort consists of data collected by four consultants, which includes a higher proportion of cases of rare disease and difficult and complicated disease (Table S3).\u003c/p\u003e\n\u003cp\u003eThe QCR and the TCR were 59.14% [95%CI, 54.08\u0026ndash;64.02%] and 97.85% [95%CI, 95.82\u0026ndash;98.91%] (\u003cem\u003eP\u003c/em\u003e\u0026lt;0.05), indicating a significant improvement in EMRs completeness if physicians are able to follow the prompt from SMART-ASSISTANT.\u003c/p\u003e\n\u003cp\u003ePhysicians rated no significant difference between AI and human-generated EMRs in normativity (3.23\u0026thinsp;\u0026plusmn;\u0026thinsp;0.96 vs 3.25\u0026thinsp;\u0026plusmn;\u0026thinsp;0.91, \u003cem\u003eP\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.806), readability (3.22\u0026thinsp;\u0026plusmn;\u0026thinsp;0.98 vs 3.38\u0026thinsp;\u0026plusmn;\u0026thinsp;0.84, \u003cem\u003eP\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.068), and logicality (3.19\u0026thinsp;\u0026plusmn;\u0026thinsp;0.94 vs 3.32\u0026thinsp;\u0026plusmn;\u0026thinsp;0.83, \u003cem\u003eP\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.122). The average ensemble factuality score was 4.01\u0026thinsp;\u0026plusmn;\u0026thinsp;0.78 for human-generated EMRs and 3.34\u0026thinsp;\u0026plusmn;\u0026thinsp;1.22 for AI-generated EMRs (\u003cem\u003eP\u003c/em\u003e\u0026lt;0.001). AI-generated EMRs achieved a significantly higher integrity score than human-generated EMRs (3.50\u0026thinsp;\u0026plusmn;\u0026thinsp;0.97 vs 3.21\u0026thinsp;\u0026plusmn;\u0026thinsp;0.96, \u003cem\u003eP\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.008). This indicates that at this stage, with only the free text of the doctor-patient dialogue available, the model may still generate implausible information and illusory outputs. Nevertheless, in terms of extracting key information from these consultation dialogues, the model performs significantly better than that of physicians. Furthermore, it may even capture certain details that physicians might overlook (Fig. \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e).\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eAbdominal pain can be the presenting symptom of a life-threatening abdominal catastrophe ('acute abdomen'). When approaching a patient with acute abdominal pain, it is the responsibility of the physician to assess the patient's overall physiologic state and make a definitive diagnosis rapidly. Despite the advances made in various investigations including clinical imaging, history taking remains the most important component of the initial evaluation of the patient with acute abdominal pain\u003csup\u003e\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\u003cp\u003eLLM is increasingly being integrated into medical decision-making. Peter et al. suggested three scenarios of potential medical use of GPT-4: medical note taking, innate medical knowledge, and medical consultation\u003csup\u003e\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e\u003c/sup\u003e. Carrie et al. conducted a cross-sectional study to evaluate LLM responses to rheumatology patient questions and compared with the doctors’ responses\u003csup\u003e\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e. Dhavalkumar et al. evaluated the performance of GPT-3.5 and GPT-4 in USMLE-style medical calculation\u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e. The majority of existing research has concentrated on the field of prompt engineering, which is less labour-intensive but can cause hallucinations. In this approach, LLM does not appear to be an optimal tool for making modifications to enhance the content of clinical documents. We use a logical and behavioral anthropomorphic way to simulate the comprehensive workflow of human physicians in outpatient settings, encompassing consultation, EMRs generation, and clinical decision-making, in order to train the SMART-ASSISTANT. We chose LoRA to enable the general LLM lacking medical domain knowledge to accurately identify QCP (91.53%, 90.11%, 88.75%) in the outpatient EMRs of abdominal pain patients.\u003c/p\u003e\u003cp\u003eThe propensity of generative models to generate hallucinatory in outputs has the potential to be highly detrimental of medical decision-making\u003csup\u003e\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e\u003c/sup\u003e. The evaluation approach adopted in this study considers the potential impact of hallucinations of the LLMs. In the gold standard, a physician is permitted to mark nulls based on the actual medical record text. Should the model generate a value for the nulls section that is not mentioned in the EMRs, or a value that is mentioned in the EMRs but is not the appropriate part of this metric, it will be deemed inconsistent. To illustrate, if the gold standard is assigned a null value for Onset Characteristics and the Characteristics is labelled as Sharp, and the model erroneously populates Sharp for Onset Characteristics, it will be deemed inconsistent. By meticulously categorizing the inconsistencies, this study illustrates that through the fine-tuning of the LLM, it is feasible to optimize it to a clinically utilitarian level by decreasing the inconsistencies including omission, error, and extra.\u003c/p\u003e\u003cp\u003eThe results of this study indicate that, during the QCP recognition phase, the general model demonstrated better performance in the extraction of categories particularly within the more expansive domain. For example, GPT-4 achieved high accuracy in identifying predisposing conditions, location, duration, and accompanied symptoms. These items also demonstrated satisfied performance for the other LLMs before fine-tuning. In specialized areas of medicine, such as distinguishing between the Characteristics of abdominal pain, Onset Characteristics, and Disease Progression, the fine-tuned model can achieve significant progress. This suggests that through fine-tuning, physicians can adapt a general model that lacks medical domain knowledge to fit a specific task such as identifying QCP.\u003c/p\u003e\u003cp\u003eThere were several limitations in this study. First, this study only included the outpatient EMRs, whereas the inpatient EMRs are more complicated and regarded as more challenging. We will continue to collect impatient records and develop a more widely applicable system for inpatient EMRs quality control. Second, this study only included patients with the main complaint of 'abdominal pain', we will further expand the data set to include patients with a variety of complaints. Third, despite the satisfied STS that audio transcription has achieved, its application is subject to a number of demanding conditions. It requires meeting several criteria simultaneously, including not speaking in dialects, having a clean background without noise, limiting the number of speakers, and ensuring that speakers do not talk over each other but instead take turns speaking. Additionally, our preliminary study indicated that the application of fine-tuning in LLMs can improve the model performance in EMRs quality control in both Chinese and English. However, due to the fact that the majority of the medical records collected by RHWU are from Chinese-speaking individuals, the medical record included in this study is exclusively written in Chinese. We will further conduct an international multi-center study to validate the applicability of SMART-ASSISTANT in English language settings.\u003c/p\u003e\u003cp\u003eIn conclusion, this study is the first to use AI to assist the entire process of outpatient work. SMART-ASSISTANT can accurately extract features from outpatient EMRs and significantly improve the completeness and diagnostic correlation of EMRs. The system has the potential to improve the quality of EMRs in real clinic and to automatically transcribe the consultation audio to structured EMRs, meanwhile protecting the interests of both physicians and patients and reducing the incidence of medical malpractice.\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003e\u003cb\u003eStudy design and ethics\u003c/b\u003e\u003c/p\u003e\u003cp\u003eThe construction of the SMART-ASSISTANT can be divided into four parts: audio transcription, structured EMRs generation, quality control, and assisted diagnosis. We validated the above mentioned functions through the simulated set, the retrospective set, a multi-reader multi-case (MRMC) study, and a prospective cohort study.\u003c/p\u003e\u003cp\u003eThe study was approved by the Ethics Committee of Renmin Hospital of Wuhan University (RHWU) and was registered on \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.chictr.org.cn/\u003c/span\u003e\u003cspan address=\"https://www.chictr.org.cn/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e with registration number ChiCTR2400086606. Patient sensitive information was removed during the initial extraction of EMRs data. The EMRs data was anonymised by replacing the identifiers used for the encounter notes with universal unique identifiers. The translation table was kept securely on a second server and could be unlocked only with the approval of the institutional review board or for a public health authority. The EMRs system was developed by a Chinese vendor named Donghua Electronic Medical Records (Beijing, China).\u003c/p\u003e\u003cp\u003e\u003cb\u003eAudio transcription model\u003c/b\u003e\u003c/p\u003e\u003cp\u003e30 cases of simulated consultation dialogues were conducted by 2 researchers to simulate clinical consultation scenarios. The audio files of simulated doctor-patient clinic dialogues were recorded. Three different audio transcription strategies (OpenAI-Whisper, OpenAI-Whisper with entity correction, and FunASR with entity correction) were used to convert the audio files into text and the best performing one was used for prospective validation. One researcher transcribed the text on the basis of the recording file as gold standard. OpenAI-Whisper is a product of Open-AI company and the version is 20231117. FunASR is a product of Alibaba and the version number is 1.1.9. The details of audio-to-text transcription methods were presented in the supplementary materials\u003csup\u003e\u003cspan additionalcitationids=\"CR26 CR27 CR28 CR29\" citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e–\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\u003cp\u003e\u003cb\u003eStructured EMRs generation model\u003c/b\u003e\u003c/p\u003e\u003cp\u003eA total of 119 cases of simulated free text of consultation dialogue and paired structured EMRs were written by five researchers based on different chief complaints, forming 119 free text-EMRs pairs. These pairs were randomly split 4:1 into a fine-tuning set and a test set. The fine-tuning set was used to fine-tune the foundation model (HuatuoGPT-II), enabling it to summarize free text into structured outpatient medical records. The test set was used to evaluate the capability of the model. The models are expected to automatically generate a structured medical record with five sections: chief complaint, history of present illness, past history and other history, allergy history, and auxiliary examination, based on the free text of the dialogue.\u003c/p\u003e\u003cp\u003e\u003cb\u003eQuality improving model\u003c/b\u003e\u003c/p\u003e\u003cp\u003e\u003cb\u003eDatasets and preprocessing\u003c/b\u003e\u003c/p\u003e\u003cp\u003eThe EMRs of patients who presented to RHWU with the chief complaint of 'abdominal pain' were retrospectively collected from December 12th 2020 to May 31st 2023. Anonymized EMRs of 3456 cases with the chief complaint of abdominal pain were retrospectively collected. A total of 1249 outpatient EMRs met the inclusion criteria (Fig.\u0026nbsp;\u003cspan refid=\"Fig8\" class=\"InternalRef\"\u003e4\u003c/span\u003e). The EMRs were then labelled with symptoms related information according to a quality control template by 10 residents and doctor of medicine (MD) students. Two consultants reviewed the labels. The patient-based EMRs were randomly divided into a fine-tuning set and an original test set in a ratio of 9:1, ensuring that the medical records of the same patient would not appear in both sets. 1124 cases of EMRs were used for the fine-tuning set and 125 cases of EMRs consisted the original test set. Each medical record was annotated with the corresponding QCP and preliminary diagnosis as the input data. Finally, the models were expected to identify the QCP and to predict the most relevant top 3 to 5 diagnoses according to the EMRs.\u003c/p\u003e\u003cp\u003eInclusion criteria: patients presented with a chief complaint of abdominal pain.Exclusion criteria: (1) patients with a history of substance abuse or mental disorder in the past 5 years; (2) chief complaint containing multiple symptoms; (3) EMRs of obstetrics and gynaecology; (4) defective EMRs; (5) subsequent visit patient with previously confirmed diagnoses (Fig.\u0026nbsp;\u003cspan refid=\"Fig8\" class=\"InternalRef\"\u003e4\u003c/span\u003e). The details of datasets and preprocessing were presented in the supplementary materials.\u003c/p\u003e\u003cp\u003e\u003cstrong\u003eQuality-control template establishment\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe quality-control template was established in accordance with clinical guidelines and textbooks pertaining to acute abdominal pain-related disorders and key points of consultation\u003csup\u003e21,31–33\u003c/sup\u003e. Two consultants conducted template establishment. They selected characteristics, predisposing conditions, influencing factors, onset characteristics, location, duration, accompanied symptoms, and disease progression as quality-control points (QCP). Among them, characteristics, predisposing conditions, influencing factors, onset characteristics, location, and duration were regarded as mandatory points, which could be filled with 'null' only if a negative description of the relevant information is documented in the EMRs. The accompanied symptoms and disease progression were regarded as optional, which could be empty. The templates were employed to train and fine-tune the LLM, enabling it to extract the aforementioned characteristics from outpatient medical records and to indicate the omission of mandatory items (Table S2).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTraining process\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe input the annotated fine-tuning set into the LoRA (Low-Rank Adaptation) framework to fine-tune three selected base large language models: ChatGLM3-6B, LLAMA-3-8B, and HuatuoGPT-II. During training, we monitor the models' performance on the fine-tuning set, adjust hyperparameters accordingly, and optimize model performance. Subsequently, an independent test set is employed to evaluate each model's accuracy, and other critical metrics in predicting medical record quality control points. The model demonstrating the best performance will be selected as EMRs-ANGEL for further experimentation. The details of model construction were presented in the supplementary materials.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eInconsistency\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eInconsistency can be classified into three categories. Features that are labelled by the physician and not extracted by the model are defined as 'missed’. Features that are labelled by the physician and also labelled by the model but inconsistent with the physician's results are defined as 'error’. Features that are labelled by the physician as null but labelled by the model are defined as 'extra’. For each case, there were 4 items of characteristics, each with a weight of 0.25; 3 items of accompanied symptoms, each with a weight of 0.33; and 4 items of disease progression, each with a weight of 0.25. Symptom, Predisposing conditions, Influencing factors, Seizure characteristics, Location, and Duration each with a weight of 1. Therefore, in a given case, if one characteristic is incorrectly filled, it is counted as 0.25 errors; if an extra accompanied symptom is recorded, it is counted as 0.33 extra-recording; if a predisposing condition is omitted, it is counted as one missed-item (Table S1).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAI-assisted EMRs recording\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e125 cases of EMRs from the original test set and the prompts of the SMART-ASSISTANT were presented to five attending physicians with at least 5 years of clinical experience. They were asked to revise and improve the original medical records based on the prompted omission items from the SMART-ASSISTANT. This is to replicate the EMRs recording process in a real-world clinical application scenario, where the EMRs is incomplete and the physician makes further inquiries according to SMART-ASSISTANT cues and finally completes the EMRs. Eventually, an assisted test set consisting of 125 medical records was obtained.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMulti-reader multi-case study\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003ePhysicians were asked to evaluate each EMRs for completeness on a Likert scale from 1 to 5 (1 being poor, and 5 being excellent) and diagnostic correlation on a Likert scale from 1 to 3 (1 being poor, and 3 being excellent) and to pick their preferred answer. For completeness, a higher score indicates a higher degree of completeness of the EMRs, corresponding to better history taking and more comprehensive EMRs recording. For diagnostic correlation, a higher score indicates a higher correlation of inferring the diagnosis based on the content of the medical record.\u003c/p\u003e\n\u003cp\u003eTen physicians participated in this study including 2 consultants, 4 attending physicians and 6 residents. They were randomized into two groups: an AI-first group and a control group, with five in each group. Physicians in the AI-first group first assessed 125 EMRs from the assisted test set and then, after a 3-week washout period, 125 EMRs from the original test set. In contrast, physicians in the control group assessed 125 EMRs from the original test set first, and after a 3-week washout period, they assessed 125 EMRs from the assisted test set. The physicians were blinded to their subgroups and to which test set they were reading throughout the study.\u003c/p\u003e\n\u003cp\u003eThe original medical record is displayed on the left. This area can be amended by the physician in accordance with the details of the consultation. The right side displays a real-time assistance area which can automatically extract the content of the quality control points, indicate omission items and provide the most likely three to five preliminary diagnoses based on the EMRs entered on the left side.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eProspective cohort study\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eFour consultants were invited to participate in the recruitment process. Consecutive patients attending the outpatient service of these four consultants at RHWU from 22nd July 2024 to 15th September 2024 were recruited to the study. Inclusion criteria: (1) patients aged 18 years or older; (2) patients presented with a chief complaint of abdominal pain; (3) first visit outpatients without previous definitive diagnosis; (4) patients able to give informed consent; (5) researchers believe that the participant is able to understand the process of this clinical trial and is willing to follow all study procedures. Exclusion criteria: (1) patients who have participated in other clinical trials, signed informed consent, and are in the follow-up phase of the other clinical trials; (2) patients with a history of substance abuse or mental disorder in the past 5 years; (3) researchers believe that the patient is not suitable to participate in the trial.\u003c/p\u003e\n\u003cp\u003eSample size: based on our preliminary study, we expected the QCP completeness rate (QCR) of human generated EMRs to be 70% and that of the AI-assisted recording to be 85%. Using the test for one proportion, with a power of 80% and a two-sided significance level of 0.05, 64 participants were required.\u003c/p\u003e\n\u003cp\u003eWe collected the following records on the enrolled participants: (1) de-identified doctor-patient consultation dialogue audio, with all sensitive information such as names, identification numbers, and other personal identifiers removed; (2) de-identified outpatient EMRs; (3) informed consent form signed by the participant.\u003c/p\u003e\n\u003cp\u003eThe gold standard consists of three components: audio transcription, QCP, and preliminary diagnosis. The performance of structured EMRs transcription was evaluated by 5 physicians (including 2 consultants and 3 attending physicians) on a 5 dimension Likert scale including factuality, integrity, normativity, readability, and logicality. The gold standard of audio transcription was labeled by 3 MD students based on the consultation dialogue audio. The gold standard of QCP was labeled by 2 residents and reviewed by 1 consultant. The preliminary diagnosis written by 4 consultants were regarded as the gold standard. EMRs-ANGEL was expected to perform audio transcription, structured EMRs transcription, QCP identification, and preliminary diagnosis.\u003c/p\u003e\n\u003ch2\u003eStatistical analysis\u003c/h2\u003e\n\u003cp\u003eTo evaluate the capabilities of EMRs-ANGEL, accuracy for QCP identification and diagnosis were calculated for all the tests mentioned. The McNemar test was used to compare the accuracy of QCP recognition of the models. To evaluate physicians ratings for completeness and diagnostic correlation in retrospective validation and physicians ratings for factuality, integrity, normativity, readability, and logicality in prospective cohort, we used an ensemble scoring strategy\u003csup\u003e34\u003c/sup\u003e. In this method, the average score is calculated across raters for each case, and the aggregate score for each rating of human-generated EMRs is compared to the aggregate score of the respective rating of AI-assited EMRs or AI-generated EMRs using paired t-tests. Mann-Whitney \u003cem\u003eU\u003c/em\u003e test was used to compare the QCR of physicians and the TCR. STS and CER were used to evaluate the ability of audio-to-text transcription and structured medical records transcription. We consider STS above 80% as a high reliability, 60–80% as a medium reliability, and below 60% as a low reliability.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eRole of the funding source\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe funder had no role in this study.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSource of support\u003c/strong\u003e: This work was partly supported by the Key Research and Development Program of Hubei Province (grant no. 2023BCB153, to Honggang Yu) and the College-enterprise Deepening Reform Project of Wuhan University (to Honggang Yu).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDescription\u003c/strong\u003e: This study constructed and validated an intelligent system which can help physicians in consultation and can automatically generate high-quality electronic health records based on the consultation audio.\u0026nbsp;\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eH.Y. and J.Y. conceived and supervised the overall study. H.Y. , H.D. and J.Y. contributed to the study design and methodology. J.L., H.D., B.X., X.W. nd W.X. contributed to the data curation. D.H., J.L., B.X., X.W., W.X., M.D., S.T., J.W., C.L., B.C., T.Y., Z.C. and Q.Z were involved in data validation. H.D., J.L., X.W. and W.X. wrote the manuscript. H.Y., J.Y. and J.L. revised and edited the manuscript. All authors reviewed the manuscript. All authors approved the final version of the manuscript.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eH\u0026auml;yrinen K, Saranto K, Nyk\u0026auml;nen P. Definition, structure, content, use and impacts of electronic health records: a review of the research literature. Int J Med Inform 2008;77(5):291-304. DOI: 10.1016/j.ijmedinf.2007.09.001.\u003c/li\u003e\n\u003cli\u003eLyles CR, Nelson EC, Frampton S, Dykes PC, Cemballi AG, Sarkar U. Using Electronic Health Record Portals to Improve Patient Engagement: Research Priorities and Best Practices. Ann Intern Med 2020;172(11 Suppl):S123-s129. DOI: 10.7326/m19-0876.\u003c/li\u003e\n\u003cli\u003eChawla NV, Davis DA. Bringing big data to personalized healthcare: a patient-centered framework. J Gen Intern Med 2013;28 Suppl 3(Suppl 3):S660-5. DOI: 10.1007/s11606-013-2455-8.\u003c/li\u003e\n\u003cli\u003eSwinckels L, Bennis FC, Ziesemer KA, et al. The Use of Deep Learning and Machine Learning on Longitudinal Electronic Health Records for the Early Detection and Prevention of Diseases: Scoping Review. J Med Internet Res 2024;26:e48320. DOI: 10.2196/48320.\u003c/li\u003e\n\u003cli\u003eMangalmurti SS, Murtagh L, Mello MM. Medical malpractice liability in the age of electronic health records. N Engl J Med 2010;363(21):2060-7. DOI: 10.1056/NEJMhle1005210.\u003c/li\u003e\n\u003cli\u003eRaab R, K\u0026uuml;derle A, Zakreuskaya A, et al. Federated electronic health records for the European Health Data Space. Lancet Digit Health 2023;5(11):e840-e847. DOI: 10.1016/s2589-7500(23)00156-5.\u003c/li\u003e\n\u003cli\u003eEbbers T, Kool RB, Smeele LE, et al. The Impact of Structured and Standardized Documentation on Documentation Quality; a Multicenter, Retrospective Study. J Med Syst 2022;46(7):46. DOI: 10.1007/s10916-022-01837-9.\u003c/li\u003e\n\u003cli\u003ePavitt S, McHugh A, Chi K, et al. Improving Inpatient Consult Communication Through a Standardized Tool. Pediatrics 2021;147(5). DOI: 10.1542/peds.2020-0681.\u003c/li\u003e\n\u003cli\u003eSmith GR, Jr., Rost K, Kashner TM. A trial of the effect of a standardized psychiatric consultation on health outcomes and costs in somatizing patients. Arch Gen Psychiatry 1995;52(3):238-43. DOI: 10.1001/archpsyc.1995.03950150070012.\u003c/li\u003e\n\u003cli\u003eAncker JS, Kern LM, Edwards A, et al. How is the electronic health record being used? Use of EHR data to assess physician-level variability in technology use. J Am Med Inform Assoc 2014;21(6):1001-8. DOI: 10.1136/amiajnl-2013-002627.\u003c/li\u003e\n\u003cli\u003eMelnick ER, Dyrbye LN, Sinsky CA, et al. The Association Between Perceived Electronic Health Record Usability and Professional Burnout Among US Physicians. Mayo Clin Proc 2020;95(3):476-487. DOI: 10.1016/j.mayocp.2019.09.024.\u003c/li\u003e\n\u003cli\u003eToscano F, O\u0026apos;Donnell E, Broderick JE, et al. How Physicians Spend Their Work Time: an Ecological Momentary Assessment. J Gen Intern Med 2020;35(11):3166-3172. DOI: 10.1007/s11606-020-06087-4.\u003c/li\u003e\n\u003cli\u003eHan J, Park J, Huh J, Oh U, Do J, Kim D. AscleAI: A LLM-based Clinical Note Management System for Enhancing Clinician Productivity. Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems: Association for Computing Machinery; 2024:Article 50.\u003c/li\u003e\n\u003cli\u003eWiest IC, Ferber D, Zhu J, et al. From Text to Tables: A Local Privacy Preserving Large Language Model for Structured Information Retrieval from Medical Documents. medRxiv2023.\u003c/li\u003e\n\u003cli\u003eZhang P, Kamel Boulos MN. Generative AI in Medicine and Healthcare: Promises, Opportunities and Challenges. 2023;15(9):286. (https://www.mdpi.com/1999-5903/15/9/286).\u003c/li\u003e\n\u003cli\u003eGoodman RS, Patrinely JR, Jr., Osterman T, Wheless L, Johnson DB. On the cusp: Considering the impact of artificial intelligence language models in healthcare. Med 2023;4(3):139-140. DOI: 10.1016/j.medj.2023.02.008.\u003c/li\u003e\n\u003cli\u003eChiang CC, Luo M, Dumkrieger G, et al. A large language model-based generative natural language processing framework fine-tuned on clinical notes accurately extracts headache frequency from electronic health records. Headache 2024;64(4):400-409. DOI: 10.1111/head.14702.\u003c/li\u003e\n\u003cli\u003eGertz RJ, Dratsch T, Bunck AC, et al. Potential of GPT-4 for Detecting Errors in Radiology Reports: Implications for Reporting Accuracy. Radiology 2024;311(1):e232714. DOI: 10.1148/radiol.232714.\u003c/li\u003e\n\u003cli\u003eDing H, Xia W, Zhou Y, et al. Evaluation and practical application of prompt-driven ChatGPTs for EMR generation. NPJ digital medicine 2025;8(1):77. DOI: 10.1038/s41746-025-01472-x.\u003c/li\u003e\n\u003cli\u003eGe J, Sun S, Owens J, et al. Development of a liver disease-specific large language model chat interface using retrieval-augmented generation. Hepatology 2024;80(5):1158-1168. DOI: 10.1097/hep.0000000000000834.\u003c/li\u003e\n\u003cli\u003eMark Feldman, Lawrence S. Friedman, Lawrence J. Brandt, editors. Sleisenger and Fordtran\u0026apos;s Gastrointestinal and Liver Disease. 11 ed: Elsevier, 2020.\u003c/li\u003e\n\u003cli\u003eLee P, Bubeck S, Petro J. Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine. N Engl J Med 2023;388(13):1233-1239. DOI: 10.1056/NEJMsr2214184.\u003c/li\u003e\n\u003cli\u003eYe C, Zweck E, Ma Z, Smith J, Katz S. Doctor Versus Artificial Intelligence: Patient and Physician Evaluation of Large Language Model Responses to Rheumatology Patient Questions in a Cross-Sectional Study. Arthritis Rheumatol 2024;76(3):479-484. DOI: 10.1002/art.42737.\u003c/li\u003e\n\u003cli\u003ePatel D, Raut G, Zimlichman E, et al. Evaluating prompt engineering on GPT-3.5\u0026apos;s performance in USMLE-style medical calculations and clinical scenarios generated by GPT-4. Sci Rep 2024;14(1):17341. DOI: 10.1038/s41598-024-66933-x.\u003c/li\u003e\n\u003cli\u003eGao Z, Li Z, Wang J, et al. FunASR: A Fundamental End-to-End Speech Recognition Toolkit. (https://ui.adsabs.harvard.edu/abs/2023arXiv230511013G).\u003c/li\u003e\n\u003cli\u003eHe H, Choi JDJae-p. The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders. (https://ui.adsabs.harvard.edu/abs/2021arXiv210906939H).\u003c/li\u003e\n\u003cli\u003ePlaquet A, Bredin HJae-p. Powerset multi-class cross entropy loss for neural speaker diarization. (https://ui.adsabs.harvard.edu/abs/2023arXiv231013025P).\u003c/li\u003e\n\u003cli\u003eRadford A, Kim JW, Xu T, Brockman G, McLeavey C, Sutskever IJae-p. Robust Speech Recognition via Large-Scale Weak Supervision. (https://ui.adsabs.harvard.edu/abs/2022arXiv221204356R).\u003c/li\u003e\n\u003cli\u003eWang H, Zheng S, Chen Y, Cheng L, Chen QJae-p. CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking. (https://ui.adsabs.harvard.edu/abs/2023arXiv230300332W).\u003c/li\u003e\n\u003cli\u003eZhang M, Qiao X, Zhao Y, et al. Knowledge Prompt for Whisper: An ASR Entity Correction Approach with Knowledge Base. 2023:2975-2979.\u003c/li\u003e\n\u003cli\u003eGans SL, Pols MA, Stoker J, Boermeester MA. Guideline for the diagnostic pathway in patients with acute abdominal pain. Dig Surg 2015;32(1):23-31. DOI: 10.1159/000371583.\u003c/li\u003e\n\u003cli\u003eLukic S, Mijac D, Filipovic B, et al. Chronic Abdominal Pain: Gastroenterologist Approach. Dig Dis 2022;40(2):181-186. DOI: 10.1159/000516977.\u003c/li\u003e\n\u003cli\u003eZeller JL, Burke AE, Glass RM. JAMA patient page. Acute abdominal pain. JAMA 2006;296(14):1800. DOI: 10.1001/jama.296.14.1800.\u003c/li\u003e\n\u003cli\u003eAyers JW, Poliak A, Dredze M, et al. Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA internal medicine 2023;183(6):589-596. DOI: 10.1001/jamainternmed.2023.1838.\u003c/li\u003e\n\u003c/ol\u003e"},{"header":"Video 1","content":"Video 1 is not available with this version."}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"large language model, electronic medical record, fine-tuning, quality control","lastPublishedDoi":"10.21203/rs.3.rs-7066667/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7066667/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e\u003cp\u003eThe quality of consultation and outpatient electronic medical records (EMRs) varies among physicians. We aimed to construct an intelligent system (SMART-ASSISTANT) to assist physicians in history taking and the composing of EMRs for patients presenting with the chief complaint of abdominal pain.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e\u003cp\u003eAnonymized EMRs of 1249 cases, free-text-structured EMRs pairs of 119 cases, and a hot words dictionary were used to train the SMART-ASSISTANT. The SMART-ASSISTANT is constructed with four components: audio transcription, structured EMRs generation, EMRs quality control, and assisted diagnosis. The functions were validated through the simulated set, the retrospective set, and a multi-reader multi-case (MRMC) study. A prospective cohort study including 62 participants was conducted to evaluate the utility of SMART-ASSISTANT to transcribe the consultation audio into standardized EMRs text.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e\u003cp\u003eSMART-ASSISTANT outperformed GPT-4 in identifying symptoms, characteristics, onset characteristics, and disease progression significantly (100.00 vs 19.20%, \u003cem\u003eP\u003c/em\u003e\u0026lt;0.001; 92.00 vs 85.80%, \u003cem\u003eP\u003c/em\u003e\u0026lt;0.001; 85.60 vs 45.60%, \u003cem\u003eP\u003c/em\u003e\u0026lt;0.001; 89.60 vs 45.40%, \u003cem\u003eP\u003c/em\u003e\u0026lt;0.001). Physicians\u0026rsquo; rating of the completeness and diagnostic correlation of EMRs in the AI-assisted set were significantly superior to those in the human-generated set (4.27 vs 3.92, \u003cem\u003eP\u003c/em\u003e\u0026lt;0.001; 2.53 vs 2.33, \u003cem\u003eP\u003c/em\u003e\u0026lt;0.001). In prospective cohort study, the mean semantic textual similarity (STS) of audio transcription reached 0.9253. Physicians rated no significant difference between AI and human-generated EMRs in normativity (3.23 vs 3.25, \u003cem\u003eP\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.806), readability (3.22 vs 3.38, \u003cem\u003eP\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.068), and logicality (3.19 vs 3.32, \u003cem\u003eP\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.122). AI-generated EMRs demonstrated significantly superior performance in terms of integrity (3.50 vs 3.21, \u003cem\u003eP\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.008) than human-generated EMRs.\u003c/p\u003e\u003ch2\u003eConclusions\u003c/h2\u003e\u003cp\u003eThe quality of the outpatient EMRs have been significantly improved by the utilization of SMART-ASSISTANT. The system has the potential to confer benefits on both patients and healthcare organizations through assisted consultation and automated EMRs generation.\u003c/p\u003e","manuscriptTitle":"Constructing a Smart-Assistant for Improving the Outpatient Service Quality in Real-time: a Prospective Single-center Cohort Study","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-07-29 11:57:53","doi":"10.21203/rs.3.rs-7066667/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"54a676c6-ef1b-4936-aaa2-009ee44e7451","owner":[],"postedDate":"July 29th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":52176575,"name":"Biological sciences/Computational biology and bioinformatics"},{"id":52176576,"name":"Health sciences/Health care"},{"id":52176577,"name":"Physical sciences/Mathematics and computing"},{"id":52176578,"name":"Health sciences/Medical research"}],"tags":[],"updatedAt":"2025-08-12T02:08:20+00:00","versionOfRecord":[],"versionCreatedAt":"2025-07-29 11:57:53","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7066667","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7066667","identity":"rs-7066667","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}