MRQC-LLM: A Novel Large Language Model Framework for Enhancing Medical Record Quality Control

MRQC-LLM: A Novel Large Language Model Framework for Enhancing Medical Record Quality Control | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article MRQC-LLM: A Novel Large Language Model Framework for Enhancing Medical Record Quality Control Zhenqi Zhang, Xuchen Yang, Xun Yao, Hao Yang, Shutong Zhang, Sikai Liu, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6765575/v1 This work is licensed under a CC BY 4.0 License Status: Under Revision Version 1 posted 15 You are reading this latest preprint version Abstract Objective: This study introduces MRQC-LLM, a novel large language model (LLM)-based framework designed to enhance quality control in electronic medical records (EMRs). By leveraging fine-tuned LLMs, we compare the performance of several models in clinical documentation review to identify the most effective approach. Methods: We selected xunfei Spark LLM as the base model, enhanced through LoRA-based fine-tuning for resource optimization. The study employed multiple AI agents tailored to specific quality control tasks, including surgical records, test results, and antibiotic treatment evaluations. Additionally, Retrieval-Augmented Generation technology was integrated with a quality control knowledge base to further improve the precision of context-aware analysis and decision-making processes. Results: Our evaluation, using 2,600 medical records, demonstrated that the xunfei Spark Model with LoRA fine-tuning outperformed other models, including GPT-4o, ChatGLM3, Qwen2, and LLama3, achieving accuracy rates between 91%-100% and consistency consistently above 90%. The model excelled particularly in tasks such as surgical record consistency and antibiotic treatment effectiveness, showcasing its adaptability to complex clinical scenarios. Conclusion: The xunfei Spark model with LoRA fine-tuning demonstrates strong potential for improving EMR quality control, offering high accuracy, efficiency, and consistency across clinical scenarios. Future research will focus on expanding MRQC-LLM’s capabilities to encompass a wider array of quality control tasks in increasingly complex clinical environments. LLMs EMR quality control LoRA RAG Figures Figure 1 Figure 2 Figure 3 1. Introduction In the healthcare sector, medical record quality control is a crucial component for ensuring the overall quality of medical services. It not only relates to the development of hospitals' overall medical standards but also directly impacts patient treatment processes and medical safety. According to incomplete statistics, in 2019, medical disputes arising from improper medical record documentation accounted for 45% of cases, reflecting a 16% increase compared to the same period the previous year [ 1 ]. To further standardize electronic medical record documentation and protect the legitimate rights and interests of both patients and healthcare providers, the government has introduced regulations such as the "Basic Norms for Electronic Medical Records (Trial)" and the "Medical Quality Management Measures." These regulations aim to ensure the objectivity, authenticity, accuracy, timeliness, completeness, and standardization of medical records [ 2 ]. In recent years, particularly with the implementation of policies like DRG payment systems and performance assessments for public hospitals, there has been a growing emphasis on the quality of medical records. The recent guidelines on enhancing performance assessments for tier-three public hospitals have included medical record quality as a metric, mandating hospitals to strengthen their management of record quality and improve data accuracy [ 3 ]. Traditional methods of medical record quality control remain primarily rule-based and retrospective, which not only consumes significant time and effort from medical record departments but also leads to inefficient feedback processes. This imposes a heavy burden on frontline staff and often results in incomplete coverage of quality control points, leading to overlooked issues and suboptimal quality control outcomes. Therefore, establishing high-quality, digitized quality control and management of medical records has become a focal point of research and practice in hospitals[ 4 ]. With the rapid advancement of artificial intelligence technology, large models like ChatGPT are continuously evolving, driving healthcare AI toward greater intelligence[ 5 ]. In April 2023, Microsoft introduced OpenAI’s GPT-4 AI language model into the healthcare sector to assist healthcare personnel in responding to patient inquiries and analyzing medical records [ 6 ]. In May 2023, Google unveiled its latest healthcare model, Med-PaLM2, at the Google I/O conference, capable of retrieving medical knowledge based on user inputs and reasoning to answer medical questions. This model can assist physicians in completing daily reports and even support diagnostic processes [ 7 ]. In May 2024, Google released the Med-Gemini model, achieving an accuracy rate of 91.1% in medical diagnostics [ 8 ]. Furthermore, some studies have utilized LLaMa2-based fine-tuned models to generate electronic medical records based on doctor-patient dialogues, showcasing the remarkable capabilities of large models in understanding medical records and extracting key medical terms [ 9 ]. The evolution of healthcare large models—from reasoning and question-answering to assisting in diagnosis and analyzing medical texts—has brought revolutionary changes to the medical field. However, there has yet to be similar research focused on leveraging AI large models for medical record quality control. Our study will focus on the intrinsic quality control of electronic medical records using large models, guided by national electronic medical record system application evaluation standards, DRG payment policies, and common documentation norms in clinical practice. The goal is to establish a standardized electronic medical record quality control system. By employing large models, we aim to provide new avenues for enhancing the accuracy and comprehensiveness of quality control, thereby improving overall healthcare quality. 2. Data Processing 2.1 Data collection This study sources its data from the Electronic Medical Records (EMR) system of a tertiary hospital. Collaborating with key departments such as Orthopedics, Respiratory Medicine, Hepatobiliary Surgery, and Cardiology, we conducted data selection and annotation. To ensure the scientific rigor and representativeness of the research, the data collection timeframe spans the past three years, focusing exclusively on inpatient records. Ultimately, 4,500 patient electronic medical records were included, covering a wide range of disease types and treatment plans. Additionally, the study incorporated critical diagnostic data related to patients, including laboratory test records, physician orders, and examination reports, ensuring the completeness of medical record information and the feasibility of conducting multidimensional analyses with the model. 2.2 Design of Quality Control points Based on the national electronic medical record system application evaluation standards, DRG payment policies, and common documentation norms encountered in clinical practice, we have constructed a novel quality control system for medical records focused on their intrinsic characteristics. This system identifies 11 key points for medical record quality control, encompassing multiple dimensions such as patient auxiliary examinations, surgical records, treatment plans, and medication usage. Different types of data are matched as inputs for the large model according to these quality control points, as detailed in Table 1 below. Table 1 The design of Quality Control Points and Corresponding Input Data for Medical Record Evaluation. # Quality Control Point Input 1 Determine whether consultation opinions and their execution status are recorded in the progress notes Progress notes + Consultation records 2 Check if the surgical steps in the surgical record match the procedure name and if the surgical site is consistent with the progress notes Progress notes + Procedure name + Surgical procedure 3 Assess whether the treatment effects of antibiotics are recorded in the progress notes Progress notes + Medication orders 4 Confirm if examination results are documented in the progress notes Progress notes + Imaging reports 5 Verify if laboratory test results are recorded in the progress notes Progress notes + Laboratory test reports 6 Check if the reasons for ordering imaging examinations are documented in the progress notes Progress notes + Imaging reports 7 Assess whether the progress notes include an analysis of positive results from examinations Progress notes + Laboratory test reports 8 Determine if the reasons for ordering laboratory tests are included in the progress notes Progress notes + Laboratory test reports 9 Verify if the progress notes include an analysis of positive results from laboratory tests Progress notes + Laboratory test reports 10 Check for the presence of antibiotic prescriptions without indications Progress notes + Medication orders 11 Determine whether records related to issued treatment orders are documented in the progress notes Progress notes + Treatment orders 2.3 Data Clean and Data Annotation We strictly adhere to the Medical Quality Management Regulations [ 10 ], ensuring that all medical record data is objective, authentic, and accurate. To protect patient privacy, all data was anonymized during collection. Multiple rounds of data cleaning were conducted to remove redundant and invalid entries, standardizing data formats and units to maintain consistency and completeness. High-quality annotated data is essential for training effective quality control models, and our annotation process followed rigorous steps. Team Formation: A team of 45 medical professionals and trained annotators was assembled to ensure high annotation quality. Each team member has at least two years of clinical experience, including 27 junior-level, 16 mid-level, and 2 senior-level members. The team spans various clinical departments—internal medicine, surgery, pediatrics, obstetrics, and both traditional Chinese and Western medicine—allowing for specialized insights relevant to the quality control tasks. The selection of annotators from diverse departments was intentional to align their expertise with the specific quality control areas in our study. Annotation Guideline Development: To ensure consistency and accuracy in the annotation process, we developed comprehensive annotation guidelines. These guidelines clearly define and provide examples for various types of annotations, focusing on identifying key information such as diagnoses, treatments, and medications. Use of Annotation Tools (shown in supplementary): We utilized specialized annotation tools to assist annotators in efficiently and accurately highlighting and marking relevant content within the medical records. Quality Assurance Measures: To maintain high annotation quality, we employed a dual-annotation method, where two annotators independently labeled each record. Any discrepancies were resolved through discussion or arbitration by a third-party expert annotator. Additionally, we conducted regular quality checks and feedback meetings to ensure the ongoing high quality of the annotations. Based on 4,500 original electronic medical records and 11 quality control points, we utilized a front-end engine to semi-structure the decomposition of the medical documents. These documents were sampled in chronological order according to the sequence of writing and the logic of events. Specific instructions, prompts, inputs, and outputs were developed for different quality control items to meet the standards for instruction fine-tuning of large models. This process ultimately generated a training set of 21,000 pairs and a test set of 2,600 pairs. Relevant data samples are shown in Fig. 1. 3. LLM-based EMR quality control This study selects the xunfei Spark Medical LLM [ 11 ] as the foundational model. According to third-party testing by the National Center for Comprehensive Utilization of Science and Technology Information Resources (STI), the xunfei Spark Medical LLM surpasses GPT-4 in six core capabilities within the healthcare field: knowledge Q&A, complex language understanding, professional document generation, diagnostic and treatment recommendations, as well as multi-turn and multimodal interactions. This research introduces an intelligent EMR quality control framework based on LLMs, integrating parameter-efficient fine-tuning, multi-agent systems, and retrieval-augmented generation (RAG) technology to achieve resource optimization, intelligent decision-making, and flexible task allocation for quality control. Additionally, the study considers open-source models such as ChatGLM3 [ 12 ], Qwen2 [ 13 ], and Llama3 [ 14 ] for comparison, ensuring that the deployed model size does not exceed 13 billion parameters, considering computational resources and internal deployment within healthcare institutions. 3.1 Fine-tune In the context of the widespread adoption of large language models, optimizing computational efficiency and resource utilization has become particularly important. The goal of resource optimization during fine-tuning is to reduce resource consumption. Against this backdrop, a parameter-efficient fine-tuning method has been proposed, leading to the promotion and development of Parameter-Efficient Fine-Tuning (PEFT) [ 15 ]. In our study, we will employ the Low-Rank Adaptation (LoRA) [ 16 ] algorithm to fine-tune the foundational model. LoRA approximates updates to the original model parameters by introducing low-rank matrices, which reduces the number of parameters that need to be adjusted, thereby lowering computational costs and resource consumption. The core idea of LoRA is to represent the weight matrix in the model as the product of two low-rank matrices, i.e., w = A×B , where adjusting A and B facilitates model fine-tuning. This approach not only enhances the efficiency of the fine-tuning process but also preserves the original performance of the model. We will use CrossEntropyLoss[ 17 ] as the training loss function. 3.2 Multi AI-agent In our study, we have developed a comprehensive process for the intrinsic quality control of EMR based on LLMs. This process primarily consists of three components: data input, rule triggering with quality control (QC) items, and QC output. The system preprocesses multi-source medical record data to extract patients' admission information, serving as the initial point for quality control. It collects daily patient progress notes, detailed records of surgical procedures, examination results, laboratory tests, and medical orders, conducting intrinsic quality control throughout the patient's hospitalization. To address various quality control points, we have established multi-AI agent [ 18 ] approach to perform quality control on the intrinsic aspects of electronic medical records, including surgical records, examination results, and antibiotic usage records. The system extracts data and segments content based on predefined rules, intelligently selecting the appropriate agent for quality control assessment. For instance, the system verifies the existence of surgical records, retrieves relevant surgical procedures and names from the electronic medical record, inputs this data into the model, and ultimately produces quality control results. These results consist of three components: the interpretability of the quality control conclusion, traceability of the quality control reasons, and the quality control conclusion itself. The interpretability of the quality control conclusion might indicate that the surgical site described in the patient's surgical record is the left knee joint, while the progress notes refer to it as the right knee medial side. The traceability of quality control reasons allows the system to identify specific entry points for the traced issues. For example, the quality control conclusion may reveal inconsistencies between the surgical site noted in the surgical record and the information on the medical record front page. By employing this method, our system can efficiently and accurately conduct quality control of electronic medical records, ensuring the completeness and accuracy of medical data, thereby improving healthcare quality and patient safety. Furthermore, the system integrates a user feedback mechanism, continuously optimizing and enhancing the intelligent quality control model through collaborative reasoning based on user input, forming a closed-loop process for ongoing improvement. The specific architecture of the system is illustrated in Fig. 2 . 3.3 RAG The establishment of an authoritative quality control knowledge base aims to integrate and manage a wide range of quality control information, ensuring the authority and currency of the information. Regular updates and maintenance are critical to aligning the content of the knowledge base with the latest research findings and practical applications. When the model conducts quality control of medical records, it can retrieve relevant information from the knowledge base (shown in Fig. 3 ). For instance, creating a knowledge base linking antibiotics to symptoms can assist the antibiotic usage agent in making more informed decisions within the medication quality control system. By leveraging RAG technology [ 19 ], the knowledge base not only provides precise quality control background information and domain knowledge but also combines dynamic retrieval with generation to offer specific contexts for the large model's prompts. This integration supports a more accurate and efficient analysis and decision-making process, significantly enhancing the intelligence level of quality control efforts. 4. Results In this quality control assessment, we focused on evaluating the performance of different models across multiple quality control points, encompassing intelligent quality control tasks involving 2,600 medical data entries. The quality control points included surgical records, examination and test result documentation, and antibiotic treatment effect evaluations, with each point involving 200 data entries, maintaining a positive-to-negative sample ratio of 1:1. Among these tasks, the Xunfei Spark model stood out, particularly after optimization with the LoRA technique, which significantly enhanced its overall performance shown in Table 2 . Additionally, a radar chart is provided to visually compare the model performances, as illustrated in Fig. 4 . Table 2 Evaluation of Accuracy and Consistency: A Comparative Analysis of Different LLMs in Quality Control (w/o means without; w/ means with) QC Point Xunfei Spark w/o lora Xunfei Spark w/ lora ChatGLM3 Llama3 Claude GPT-4o 1 91.8%±0.07% / 92.2%±0.15% 91.7%±0.01% / 93.3%±0.11% 52.82%±0.61% / 58.65%±0.39% 78.78%±1.93% / 82.14%±0.96% 88.4%±0.46% / 91.1%±0.12% 97.6%±0.13% / 90.1%±0.23% 2 100%±0.00% / 88.7%±0.19% 100%±0.00% / 91.4%±0.16% 73.36%±3.23% / 70.40%±2.83% 88.13%±0.77% / 83.63%±1.16% 95.6%±0.01% / 89.5%±0.22% 98.2%±0.05% / 96.3%±0.09% 3 90.8%±0.29% / 94.3%±0.05% 93.6%±0.12% / 93.7%±0.07% 69.11%±1.21% / 52.00%±1.79% 85.98%±0.99% / 76.35%±1.30% 91.6%±0.54% / 85.4%±0.73% 90.9%±0.50% / 86.9%±0.47% 4 97.5%±0.11% / 91.5%±0.21% 98.1%±0.08% / 94.5%±0.03% 38.79%±2.98% / 78.12%±3.23% 72.53%±1.42% / 91.56%±1.08% 79.3%±0.20% / 91.6%±0.10% 97.7%±0.10% / 93.9%±0.13% 5 88.9%±0.19% / 72.2%±0.24% 91.3%±0.13% / 88.8%±0.13% 42.58%±4.04% / 34.41%±3.48% 53.64%±0.20% / 60.09%±1.28% 57.9%±0.39% / 71.3%±0.53% 42.9%±3.86% / 66.3%±1.44% 6 95.6%±0.03% / 96.1%±0.05% 95.1%±0.09% / 96.8%±0.04% 64.64%±1.35% / 83.95%±1.65% 89.57%±0.95% / 88.93%±1.41% 97.2%±0.71% / 95.9%±1.06% 94.6%±0.43% / 96.8%±0.31% 7 99.7%±0.24% / 90.3%±0.14% 97.7%±0.07% / 92.6%±0.03% 40.72%±2.69% / 37.54%±2.06% 95.41%±0.71% / 73.40%±0.59% 96.2%±0.16% / 77.9%±0.24% 95.2%±0.02% / 64.0%±0.91% 8 96.3%±0.11% / 85.4%±0.15% 96.7%±0.03% / 90.6%±0.17% 56.85%±0.68% / 66.93%±1.22% 90.62%±0.06% / 84.85%±1.26% 93.3%±0.05% / 92.1%±0.08% 97.8%±0.07% / 90.3%±0.15% 9 76.9%±0.13% / 70.1%±0.27% 83.4%±0.17% / 87.4%±0.22% 45.69%±2.96% / 42.62%±2.10% 62.67%±0.58% / 80.14%±0.86% 67.4%±1.34% / 81.8%±0.83% 63.2%±2.28% / 66.7%±1.78% 10 91.7%±0.04% / 82.8%±0.10% 92.5%±0.05% / 83.5%±0.12% 53.45%±2.55% / 81.51%±3.20% 79.36%±1.05% / 83.97%±0.59% 79.9%±0.60% / 86.4%±0.35% 96.4%±0.39% / 78.2%±0.90% 11 68.7%±0.10% / 90.4%±0.04% 79.6%±0.26% / 89.3%±0.15% 29.55%±1.65% / 30.28%±1.59% 52.16%±1.28% / 66.17%±1.63% 56.5%±0.97% / 77.3%±1.01% 63.3%±1.39% / 86.5%±0.25% Merge 93.5%±0.14% / 87.8%±0.12% 96.1%±0.10% / 89.7%±0.16% 55.40%±2.34% / 57.34%±2.47% 84.56%±0.87% / 72.17%±1.39% 86.1%±0.56% / 86.5%±0.69% 93.8%±0.79% / 84.4%±0.68% Overall, the Xunfei Spark w/ LoRA model demonstrated exceptionally high accuracy and consistency in most quality control tasks. In critical tasks such as assessing the consistency of surgical records, evaluating antibiotic treatment effects, and analyzing test results, the performance of Xunfei Spark w/ LoRA surpassed that of other comparative models, achieving accuracy rates consistently between 91% and 100%, with consistency also remaining above 90%. This outstanding performance not only reflects the model's strong adaptability in handling complex medical record scenarios but also validates the effectiveness of LoRA optimization in LLMs. In contrast, while GPT-4o and ChatGLM3 performed exceptionally well on certain quality control points, their overall consistency and noise robustness did not match that of Xunfei Spark. The Xunfei Spark w/ LoRA model excelled particularly in tasks related to the consistency judgment of surgical steps and progress notes, as well as the analysis of positive examination results, achieving accuracy rates as high as 97.7–100%. This indicates that Xunfei Spark w/ LoRA possesses efficient and reliable application potential in medical quality control tasks, especially in terms of accuracy and comprehensiveness in analyzing surgical and examination-related records. Moreover, the LoRA optimization enhances the model's robustness when dealing with intricate medical record details, showcasing superior intelligent analytical capabilities across key tasks such as antibiotic usage judgment and positive result analysis in imaging examinations. In summary, the performance of Xunfei Spark demonstrates its significant application potential in the field of medical quality control, particularly after LoRA optimization, which has markedly improved the model's accuracy and consistency. Compared to other models, Xunfei Spark w/ LoRA exhibits clear advantages in managing the complexity of medical records and addressing various quality control tasks, especially in terms of noise resilience and multi-tasking capabilities. In the future, this model is expected to be further utilized in practical medical quality control work, enhancing the efficiency and accuracy of medical data processing. Table 2 should be placed here. 5. Discussion In this study, we developed the MRQC-LLM framework, leveraging a large language model to improve the quality control of EMRs. This framework aligns with national EMR grading standards, DRG payment policies, and clinical practices, focusing on 11 key quality control points across multiple dimensions, such as patient examinations, surgical records, treatment plans, and medication usage. Through these dimensions, we established a comprehensive quality control dataset with 21,000 training pairs and 2,600 test pairs, enhancing quality control in terms of efficiency, breadth, and accuracy. Our approach employed the Xunfei Spark medical large model, fine-tuned with LoRA technology, and incorporated RAG to dynamically access domain-specific knowledge from an authoritative quality control database. This use of RAG technology enabled the model to contextualize its responses based on the latest and most relevant background information, thereby improving the precision and relevance of the quality control assessments.The MRQC-LLM framework introduces a multi-agent model architecture, a novel approach in EMR quality control, which comprises dedicated agent modules for each key quality control point. This multi-agent setup allows the model to target specific tasks, such as surgical and examination records or antibiotic usage, by performing tailored data extraction and analysis according to established quality control rules. This structure enhances both the accuracy and flexibility of the model, as each agent can specialize in particular aspects of quality control. Moreover, we compared the performance of several large language models in EMR quality control tasks, including the Xunfei Spark medical large model, ChatGLM3, Qwen2, and LLama3. Experimental results indicate that the Xunfei Spark medical large model exhibits significant advantages in consistency and accuracy for quality control tasks, achieving an accuracy rate of approximately 90%. Despite the Xunfei Spark model having a parameter size of only 13B, its performance after LoRA fine-tuning is comparable to that of GPT-4o. Its outstanding performance and lower computational overhead provide it with greater deployment flexibility and practical application potential. This suggests that the Xunfei Spark model can operate efficiently in resource-constrained environments and has the potential to further enhance healthcare quality and patient safety in actual medical applications [ 20 ]. Considering the costs associated with data labeling, our current research focuses on 11 quality control points and does not yet cover all scenarios in the medical field. Moving forward, we also plan to add new quality control points to expand the coverage, such as process quality control in nursing and diagnostic behaviors, to further improve the quality and accuracy of medical data. Additionally, we will explore adaptive learning [ 21 ] and active learning [ 22 ] techniques to reduce the labor costs associated with labeling new quality control points. These technologies will enable the model to enhance its understanding and judgment capabilities regarding new quality control points through continuous learning and optimization after being exposed to a limited amount of labeled data. This approach will facilitate the rapid iteration and expansion of the quality control model. To improve the model's applicability and precision across various medical scenarios, we plan to continuously enrich the quality control knowledge base and regularly update industry standards and the latest clinical practice guidelines, ensuring the timeliness and authority of the quality control system. In practical deployment, we will further strengthen the validation of the model in real healthcare environments, including multi-center experiments and cross-regional testing, to ensure consistent performance across different hospitals and departments. Furthermore, to achieve efficient integration and information interoperability within the intelligent quality control system, we will advance the deep integration with Hospital Information Systems (HIS), Laboratory Information Systems (LIS), and other electronic medical record systems, allowing the quality control process to permeate the entire medical data flow. 5. Conclusion Overall, the quality control system based on LLMs demonstrates significant advantages in enhancing the quality of medical record quality control, but it also reveals some areas for improvement. Future research can focus on further optimizing the model's performance in complex domains, experimenting with larger models, and integrating additional practical quality control needs to cover a broader range of quality control scenarios. Through continuous refinement and optimization, we aim to achieve a higher level of quality control management in the healthcare field, ultimately enhancing healthcare quality and patient safety. Declarations Ethics approval and consent to participate This retrospective study was approved by the Ethics Committee of West China Hospital, Sichuan University, and was conducted in accordance with the Declaration of Helsinki. No interventions or additional risks were involved. All patient data were de-identified to ensure confidentiality. The requirement for informed consent was waived by the Ethics Committee of West China Hospital, in accordance with relevant national regulations and institutional guidelines. Consent for publication Not applicable. Availability of data and material The datasets generated and analyzed during the current study are not publicly available due to data security concerns but are available from the corresponding author on reasonable request. The code used in this study is available from the corresponding author upon reasonable request. Competing interests The authors declare no competing of interest. Figure Legends All figures are original and created by the authors. Funding Not applicable. Authors and Affiliations Anesthesiology Department, The Third People’s Hospital of Chengdu, Chengdu, China Zhenqi Zhang Information Center, West China Hospital, Sichuan University, Chengdu, China Xuchen Yang Information Center, West China Hospital, Sichuan University, Chengdu, China Xun Yao Information Center, West China Hospital, Sichuan University, Chengdu, China Hao Yang Xunfei Healthcare Technology Co., Ltd, Hefei, China Shutong Zhang Xunfei Healthcare Technology Co., Ltd, Hefei, China Sikai Liu Xunfei Healthcare Technology Co., Ltd, Hefei, China Jing Wang Information Center, West China Hospital, Sichuan University, Chengdu, China Rui Shi Authors’ contributions XCY and ZZ contributed equally to the study; they led the design and conceptualization of the framework and collaborated on the development of the methodology. HY directed the study, performed model analysis, conducted data processing, and drafted the manuscript. XY and SZ supported data collection, clinical analysis, and model validation. SL contributed to data analysis and result interpretation, ensuring clinical relevance in findings. JW supervised the project, provided essential feedback on data analysis, and assisted with manuscript revision. RS conceptualized the study, provided strategic guidance on research direction, and reviewed the manuscript. All authors contributed to editorial adjustments, read, and approved the final version of the manuscript. Each author has participated substantially in this work and agrees to be accountable for all aspects of this research. Acknowledgments We are grateful to the staff in our research groups involved in the study for their valuable contributions and discussions. References Zhao X. Practice status of electronic medical records and improvement measures in EMRs. Chinese Medical Record English Edition. 2013;1(8):343–6.Zhao X: Practice Status of Electronic Medical Records and Improvement Measures in EMRs. Chinese Medical Record English Edition 2013, 1(8):343–346. Tian J, Wang YC, Bao SX, Chen HL, Lu JF. Construction and research on the mode of medical record quality control. Chin Res Hosp. 2020;7(5):63–6. General Office of the State Council. Opinions on strengthening the performance assessment of tertiary public hospitals [EB/OL]. [April 15, 2020]. https://www.gov.cn/gongbao/content/2019/content_5366476.htm Wen Y, Li X, Zeng FF, Lei JY, Chen SY. Application of medical record quality control system based on artificial intelligence. J Sichuan Univ (Medical Sciences). 2023;54(6):1263–8. Biswas SS. Role of chat gpt in public health. Ann Biomed Eng. 2023;51(5):868–9. Abu-Haifa M, Etawi B, Alkhatatbeh H, Ababneh A. Comparative analysis of ChatGPT, GPT-4, and Microsoft Copilot Chatbots for GRE test. Int J Learn Teach Educational Res. 2024;23(6):327–47. Argasiński K, Grabska-Gradzińska J, Przystalski I, Ochab KK, Walkowiak J. T. Stylometric analysis of large language model-generated commentaries in the context of medical neuroscience. In: International Conference on Computational Science: 2024: Springer; 2024. pp. 281 – 95. Saab K, Tu T, Weng WH, Tanno R, Stutz D, Wulczyn E et al. Capabilities of gemini models in medicine. 2024. arXiv preprint, arXiv:2404.18416. Teja NS, Kumar K, Malarvel M. Multilingual Text Enhancer (MLTE)–A LLaMA2 based Model for Prompt Generation. In: 2024 3rd International Conference on Applied Artificial Intelligence and Computing (ICAAIC): 2024: IEEE; 2024. pp. 895–900. Huang XT, Jing C, Wang CY. The practice of medical quality control system construction based on the Medical Quality Management Procedures. Chin Health Qual Manage. 2021;28(2):20–2. Liu Z, Chen Z. Cross-Cultural Perspectives on Artificial Intelligence Generated Content (AIGC): A Comparative Study of Attitudes and Acceptance Among Global Products. In: International Conference on Human-Computer Interaction: 2024: Springer; 2024. pp. 287 – 98. Ye C, Shi X. Optimizing News Topic Classification with Instructional Fine-Tuning of Chatglm3. In: Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering: 2024; 2024. pp. 573-7. Yang A, Yang B, Hui B, Zheng B, Yu B, Zhou C et al. Qwen2 technical report. 2024. arXiv preprint, arXiv:2407.10671. Grattafiori A, Dubey A, Jauhri A, Pandey A, Kadian A, Al-Dahle A et al. The llama 3 herd of models. 2024. arXiv preprint, arXiv:2407.21783. Liu H, Tam D, Muqeeth M, Mohta J, Huang T, Bansal M, et al. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Adv Neural Inf Process Syst. 2022;35:1950–65. Hu EJ, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S et al. Lora: Low-rank adaptation of large language models. 2021. arXiv preprint, arXiv:2106.09685. Mao A, Mohri M, Zhong Y. Cross-entropy loss functions: Theoretical analysis and applications. In: International conference on Machine learning: 2023: PMLR; 2023. pp. 23803-28. Masterman T, Besen S, Sawtell M, Chao A. The landscape of emerging ai agent architectures for reasoning, planning, and tool calling: A survey. 2024. arXiv preprint, arXiv:2404.11584. Lewis P, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Adv Neural Inf Process Syst. 2020;33:9459–74. Cheng X, Jiao F, Ji G, Tian Y. The Artificial Intelligence Revolution Led by ChatGPT. In: 2023 International Seminar on Computer Science and Engineering Technology (SCSET). 2023: IEEE; 2023. pp. 360 – 63. Kabir MR, Lin F, An LLM-P. Adaptive Practicing System. In: LLM@ AIED: 2023; 2023. pp. 43–52. Lu Y, Yao B, Zhang S, Wang Y, Zhang P, Lu T et al. Human still wins over llm: An empirical study of active learning on domain-specific annotation tasks. 2023. arXiv preprint, arXiv:2311.09825. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Revision Version 1 posted Editorial decision: Revision requested 13 Nov, 2025 Reviews received at journal 11 Jul, 2025 Reviews received at journal 11 Jul, 2025 Reviews received at journal 03 Jul, 2025 Reviewers agreed at journal 01 Jul, 2025 Reviewers agreed at journal 29 Jun, 2025 Reviewers agreed at journal 28 Jun, 2025 Reviews received at journal 28 Jun, 2025 Reviewers agreed at journal 27 Jun, 2025 Reviewers agreed at journal 21 Jun, 2025 Reviewers invited by journal 18 Jun, 2025 Editor assigned by journal 18 Jun, 2025 Editor invited by journal 18 Jun, 2025 Submission checks completed at journal 17 Jun, 2025 First submitted to journal 17 Jun, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6765575","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":473092579,"identity":"9b795ba8-8a3b-4b4d-8f79-ba9e847fe109","order_by":0,"name":"Zhenqi Zhang","email":"","orcid":"","institution":"The Third People’s Hospital of Chengdu","correspondingAuthor":false,"prefix":"","firstName":"Zhenqi","middleName":"","lastName":"Zhang","suffix":""},{"id":473092580,"identity":"2dcc3111-9a87-402e-ae2e-9f98a18700d6","order_by":1,"name":"Xuchen Yang","email":"","orcid":"","institution":"Sichuan University","correspondingAuthor":false,"prefix":"","firstName":"Xuchen","middleName":"","lastName":"Yang","suffix":""},{"id":473092581,"identity":"90c560c8-12c0-4fd6-9a12-d26c75b2af18","order_by":2,"name":"Xun Yao","email":"","orcid":"","institution":"Sichuan University","correspondingAuthor":false,"prefix":"","firstName":"Xun","middleName":"","lastName":"Yao","suffix":""},{"id":473092582,"identity":"41bf4624-acb1-4c2f-a005-da9bd90e7a3e","order_by":3,"name":"Hao Yang","email":"","orcid":"","institution":"Sichuan University","correspondingAuthor":false,"prefix":"","firstName":"Hao","middleName":"","lastName":"Yang","suffix":""},{"id":473092583,"identity":"0569bad6-dc38-4c6a-84b6-29e4428aea1c","order_by":4,"name":"Shutong Zhang","email":"","orcid":"","institution":"Xunfei Healthcare Technology Co., Ltd","correspondingAuthor":false,"prefix":"","firstName":"Shutong","middleName":"","lastName":"Zhang","suffix":""},{"id":473092584,"identity":"2e0e5a0a-8c38-4b74-9103-94f52888511c","order_by":5,"name":"Sikai Liu","email":"","orcid":"","institution":"Xunfei Healthcare Technology Co., Ltd","correspondingAuthor":false,"prefix":"","firstName":"Sikai","middleName":"","lastName":"Liu","suffix":""},{"id":473092585,"identity":"37093f13-4ada-4b98-a142-78e09aeb08a4","order_by":6,"name":"Jing Wang","email":"","orcid":"","institution":"Xunfei Healthcare Technology Co., Ltd","correspondingAuthor":false,"prefix":"","firstName":"Jing","middleName":"","lastName":"Wang","suffix":""},{"id":473092586,"identity":"78bb075e-347f-4e85-a98b-1a9553a2dfed","order_by":7,"name":"Rui Shi","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA1ElEQVRIie3PMQuCQBjG8VdeOJcXWxXrO1wIjfVVFNcjhKZoKAhsiea2voWz4tByNQdBZM0N4dqQ3tigjg33357jfnAHoNP9YykAAgdg5lZtsroTkmoQ60bqbKEGtBLrmmIZRbdg57yy51uM+wyweFwaiHP2mbvnsyB2pyHPkrB6GPM80UC4BIbE/YqIkZ0lWBFibgvBUhFH1mTZiYCriE01yduJI424Jl5MwuOn5EgMW/5iScxL+viDw0YO7/NkMemZ6+LZRACM1c8BNl7X6XQ6XZe+u8ZADTpEkycAAAAASUVORK5CYII=","orcid":"","institution":"Sichuan University","correspondingAuthor":true,"prefix":"","firstName":"Rui","middleName":"","lastName":"Shi","suffix":""}],"badges":[],"createdAt":"2025-05-28 07:53:33","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6765575/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6765575/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":85386795,"identity":"bfbdaeac-20d0-40c1-868d-51bb6da32183","added_by":"auto","created_at":"2025-06-25 09:54:25","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":199812,"visible":true,"origin":"","legend":"\u003cp\u003eStructured Prompt Design for Medical Document Quality Control\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-6765575/v1/3bfe04f5bd0eae4c3e249505.png"},{"id":85387563,"identity":"e819eba6-94a8-4e5c-8cd0-344a438f530d","added_by":"auto","created_at":"2025-06-25 10:02:25","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":60270,"visible":true,"origin":"","legend":"\u003cp\u003eArchitecture of a Multi-Agent Approach to Quality Control in Healthcare Records\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-6765575/v1/4d64069692ee0858bb33c0cb.png"},{"id":85386798,"identity":"8fe24868-b323-4b75-a126-ff3714365501","added_by":"auto","created_at":"2025-06-25 09:54:25","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":324232,"visible":true,"origin":"","legend":"\u003cp\u003eIntegrating RAG Technology in Quality Control LLMs\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-6765575/v1/41ad1acf88aa88c82f695130.png"},{"id":85389064,"identity":"770ea0f4-5277-4f3d-8f30-c670da3e9714","added_by":"auto","created_at":"2025-06-25 10:18:25","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1257903,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6765575/v1/807f94f1-1889-44c7-b930-2b0628e7f089.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"MRQC-LLM: A Novel Large Language Model Framework for Enhancing Medical Record Quality Control","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eIn the healthcare sector, medical record quality control is a crucial component for ensuring the overall quality of medical services. It not only relates to the development of hospitals' overall medical standards but also directly impacts patient treatment processes and medical safety. According to incomplete statistics, in 2019, medical disputes arising from improper medical record documentation accounted for 45% of cases, reflecting a 16% increase compared to the same period the previous year [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. To further standardize electronic medical record documentation and protect the legitimate rights and interests of both patients and healthcare providers, the government has introduced regulations such as the \"Basic Norms for Electronic Medical Records (Trial)\" and the \"Medical Quality Management Measures.\" These regulations aim to ensure the objectivity, authenticity, accuracy, timeliness, completeness, and standardization of medical records [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eIn recent years, particularly with the implementation of policies like DRG payment systems and performance assessments for public hospitals, there has been a growing emphasis on the quality of medical records. The recent guidelines on enhancing performance assessments for tier-three public hospitals have included medical record quality as a metric, mandating hospitals to strengthen their management of record quality and improve data accuracy [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. Traditional methods of medical record quality control remain primarily rule-based and retrospective, which not only consumes significant time and effort from medical record departments but also leads to inefficient feedback processes. This imposes a heavy burden on frontline staff and often results in incomplete coverage of quality control points, leading to overlooked issues and suboptimal quality control outcomes. Therefore, establishing high-quality, digitized quality control and management of medical records has become a focal point of research and practice in hospitals[\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eWith the rapid advancement of artificial intelligence technology, large models like ChatGPT are continuously evolving, driving healthcare AI toward greater intelligence[\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. In April 2023, Microsoft introduced OpenAI\u0026rsquo;s GPT-4 AI language model into the healthcare sector to assist healthcare personnel in responding to patient inquiries and analyzing medical records [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. In May 2023, Google unveiled its latest healthcare model, Med-PaLM2, at the Google I/O conference, capable of retrieving medical knowledge based on user inputs and reasoning to answer medical questions. This model can assist physicians in completing daily reports and even support diagnostic processes [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. In May 2024, Google released the Med-Gemini model, achieving an accuracy rate of 91.1% in medical diagnostics [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. Furthermore, some studies have utilized LLaMa2-based fine-tuned models to generate electronic medical records based on doctor-patient dialogues, showcasing the remarkable capabilities of large models in understanding medical records and extracting key medical terms [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. The evolution of healthcare large models\u0026mdash;from reasoning and question-answering to assisting in diagnosis and analyzing medical texts\u0026mdash;has brought revolutionary changes to the medical field. However, there has yet to be similar research focused on leveraging AI large models for medical record quality control.\u003c/p\u003e \u003cp\u003e Our study will focus on the intrinsic quality control of electronic medical records using large models, guided by national electronic medical record system application evaluation standards, DRG payment policies, and common documentation norms in clinical practice. The goal is to establish a standardized electronic medical record quality control system. By employing large models, we aim to provide new avenues for enhancing the accuracy and comprehensiveness of quality control, thereby improving overall healthcare quality.\u003c/p\u003e"},{"header":"2. Data Processing","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Data collection\u003c/h2\u003e \u003cp\u003eThis study sources its data from the Electronic Medical Records (EMR) system of a tertiary hospital. Collaborating with key departments such as Orthopedics, Respiratory Medicine, Hepatobiliary Surgery, and Cardiology, we conducted data selection and annotation. To ensure the scientific rigor and representativeness of the research, the data collection timeframe spans the past three years, focusing exclusively on inpatient records. Ultimately, 4,500 patient electronic medical records were included, covering a wide range of disease types and treatment plans. Additionally, the study incorporated critical diagnostic data related to patients, including laboratory test records, physician orders, and examination reports, ensuring the completeness of medical record information and the feasibility of conducting multidimensional analyses with the model.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Design of Quality Control points\u003c/h2\u003e \u003cp\u003eBased on the national electronic medical record system application evaluation standards, DRG payment policies, and common documentation norms encountered in clinical practice, we have constructed a novel quality control system for medical records focused on their intrinsic characteristics. This system identifies 11 key points for medical record quality control, encompassing multiple dimensions such as patient auxiliary examinations, surgical records, treatment plans, and medication usage. Different types of data are matched as inputs for the large model according to these quality control points, as detailed in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e below.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eThe design of Quality Control Points and Corresponding Input Data for Medical Record Evaluation.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003e#\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eQuality Control Point\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eInput\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDetermine whether consultation opinions and their execution status are recorded in the progress notes\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eProgress notes\u0026thinsp;+\u0026thinsp;Consultation records\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCheck if the surgical steps in the surgical record match the procedure name and if the surgical site is consistent with the progress notes\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eProgress notes\u0026thinsp;+\u0026thinsp;Procedure name\u0026thinsp;+\u0026thinsp;Surgical procedure\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAssess whether the treatment effects of antibiotics are recorded in the progress notes\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eProgress notes\u0026thinsp;+\u0026thinsp;Medication orders\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eConfirm if examination results are documented in the progress notes\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eProgress notes\u0026thinsp;+\u0026thinsp;Imaging reports\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eVerify if laboratory test results are recorded in the progress notes\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eProgress notes\u0026thinsp;+\u0026thinsp;Laboratory test reports\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCheck if the reasons for ordering imaging examinations are documented in the progress notes\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eProgress notes\u0026thinsp;+\u0026thinsp;Imaging reports\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAssess whether the progress notes include an analysis of positive results from examinations\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eProgress notes\u0026thinsp;+\u0026thinsp;Laboratory test reports\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDetermine if the reasons for ordering laboratory tests are included in the progress notes\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eProgress notes\u0026thinsp;+\u0026thinsp;Laboratory test reports\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eVerify if the progress notes include an analysis of positive results from laboratory tests\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eProgress notes\u0026thinsp;+\u0026thinsp;Laboratory test reports\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e10\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCheck for the presence of antibiotic prescriptions without indications\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eProgress notes\u0026thinsp;+\u0026thinsp;Medication orders\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDetermine whether records related to issued treatment orders are documented in the progress notes\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eProgress notes\u0026thinsp;+\u0026thinsp;Treatment orders\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.3 Data Clean and Data Annotation\u003c/h2\u003e \u003cp\u003eWe strictly adhere to the Medical Quality Management Regulations [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e], ensuring that all medical record data is objective, authentic, and accurate. To protect patient privacy, all data was anonymized during collection. Multiple rounds of data cleaning were conducted to remove redundant and invalid entries, standardizing data formats and units to maintain consistency and completeness. High-quality annotated data is essential for training effective quality control models, and our annotation process followed rigorous steps.\u003c/p\u003e \u003cp\u003eTeam Formation: A team of 45 medical professionals and trained annotators was assembled to ensure high annotation quality. Each team member has at least two years of clinical experience, including 27 junior-level, 16 mid-level, and 2 senior-level members. The team spans various clinical departments\u0026mdash;internal medicine, surgery, pediatrics, obstetrics, and both traditional Chinese and Western medicine\u0026mdash;allowing for specialized insights relevant to the quality control tasks. The selection of annotators from diverse departments was intentional to align their expertise with the specific quality control areas in our study.\u003c/p\u003e \u003cp\u003e Annotation Guideline Development: To ensure consistency and accuracy in the annotation process, we developed comprehensive annotation guidelines. These guidelines clearly define and provide examples for various types of annotations, focusing on identifying key information such as diagnoses, treatments, and medications.\u003c/p\u003e \u003cp\u003eUse of Annotation Tools (shown in supplementary): We utilized specialized annotation tools to assist annotators in efficiently and accurately highlighting and marking relevant content within the medical records.\u003c/p\u003e \u003cp\u003eQuality Assurance Measures: To maintain high annotation quality, we employed a dual-annotation method, where two annotators independently labeled each record. Any discrepancies were resolved through discussion or arbitration by a third-party expert annotator. Additionally, we conducted regular quality checks and feedback meetings to ensure the ongoing high quality of the annotations.\u003c/p\u003e \u003cp\u003eBased on 4,500 original electronic medical records and 11 quality control points, we utilized a front-end engine to semi-structure the decomposition of the medical documents. These documents were sampled in chronological order according to the sequence of writing and the logic of events. Specific instructions, prompts, inputs, and outputs were developed for different quality control items to meet the standards for instruction fine-tuning of large models. This process ultimately generated a training set of 21,000 pairs and a test set of 2,600 pairs. Relevant data samples are shown in Fig.\u0026nbsp;1.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"3. LLM-based EMR quality control","content":"\u003cp\u003eThis study selects the xunfei Spark Medical LLM [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e] as the foundational model. According to third-party testing by the National Center for Comprehensive Utilization of Science and Technology Information Resources (STI), the xunfei Spark Medical LLM surpasses GPT-4 in six core capabilities within the healthcare field: knowledge Q\u0026amp;A, complex language understanding, professional document generation, diagnostic and treatment recommendations, as well as multi-turn and multimodal interactions. This research introduces an intelligent EMR quality control framework based on LLMs, integrating parameter-efficient fine-tuning, multi-agent systems, and retrieval-augmented generation (RAG) technology to achieve resource optimization, intelligent decision-making, and flexible task allocation for quality control. Additionally, the study considers open-source models such as ChatGLM3 [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e], Qwen2 [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e], and Llama3 [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e] for comparison, ensuring that the deployed model size does not exceed 13\u0026nbsp;billion parameters, considering computational resources and internal deployment within healthcare institutions.\u003c/p\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Fine-tune\u003c/h2\u003e \u003cp\u003eIn the context of the widespread adoption of large language models, optimizing computational efficiency and resource utilization has become particularly important. The goal of resource optimization during fine-tuning is to reduce resource consumption. Against this backdrop, a parameter-efficient fine-tuning method has been proposed, leading to the promotion and development of Parameter-Efficient Fine-Tuning (PEFT) [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. In our study, we will employ the Low-Rank Adaptation (LoRA) [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e] algorithm to fine-tune the foundational model. LoRA approximates updates to the original model parameters by introducing low-rank matrices, which reduces the number of parameters that need to be adjusted, thereby lowering computational costs and resource consumption. The core idea of LoRA is to represent the weight matrix in the model as the product of two low-rank matrices, i.e.,\u003cem\u003ew\u0026thinsp;=\u0026thinsp;A\u0026times;B\u003c/em\u003e, where adjusting A and B facilitates model fine-tuning. This approach not only enhances the efficiency of the fine-tuning process but also preserves the original performance of the model. We will use CrossEntropyLoss[\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e] as the training loss function.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003e3.2 Multi AI-agent\u003c/h2\u003e \u003cp\u003eIn our study, we have developed a comprehensive process for the intrinsic quality control of EMR based on LLMs. This process primarily consists of three components: data input, rule triggering with quality control (QC) items, and QC output. The system preprocesses multi-source medical record data to extract patients' admission information, serving as the initial point for quality control. It collects daily patient progress notes, detailed records of surgical procedures, examination results, laboratory tests, and medical orders, conducting intrinsic quality control throughout the patient's hospitalization.\u003c/p\u003e \u003cp\u003eTo address various quality control points, we have established multi-AI agent [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e] approach to perform quality control on the intrinsic aspects of electronic medical records, including surgical records, examination results, and antibiotic usage records. The system extracts data and segments content based on predefined rules, intelligently selecting the appropriate agent for quality control assessment. For instance, the system verifies the existence of surgical records, retrieves relevant surgical procedures and names from the electronic medical record, inputs this data into the model, and ultimately produces quality control results. These results consist of three components: the interpretability of the quality control conclusion, traceability of the quality control reasons, and the quality control conclusion itself.\u003c/p\u003e \u003cp\u003eThe interpretability of the quality control conclusion might indicate that the surgical site described in the patient's surgical record is the left knee joint, while the progress notes refer to it as the right knee medial side. The traceability of quality control reasons allows the system to identify specific entry points for the traced issues. For example, the quality control conclusion may reveal inconsistencies between the surgical site noted in the surgical record and the information on the medical record front page.\u003c/p\u003e \u003cp\u003eBy employing this method, our system can efficiently and accurately conduct quality control of electronic medical records, ensuring the completeness and accuracy of medical data, thereby improving healthcare quality and patient safety. Furthermore, the system integrates a user feedback mechanism, continuously optimizing and enhancing the intelligent quality control model through collaborative reasoning based on user input, forming a closed-loop process for ongoing improvement. The specific architecture of the system is illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e2\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003e3.3 RAG\u003c/h2\u003e \u003cp\u003eThe establishment of an authoritative quality control knowledge base aims to integrate and manage a wide range of quality control information, ensuring the authority and currency of the information. Regular updates and maintenance are critical to aligning the content of the knowledge base with the latest research findings and practical applications. When the model conducts quality control of medical records, it can retrieve relevant information from the knowledge base (shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e3\u003c/span\u003e). For instance, creating a knowledge base linking antibiotics to symptoms can assist the antibiotic usage agent in making more informed decisions within the medication quality control system.\u003c/p\u003e \u003cp\u003eBy leveraging RAG technology [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e], the knowledge base not only provides precise quality control background information and domain knowledge but also combines dynamic retrieval with generation to offer specific contexts for the large model's prompts. This integration supports a more accurate and efficient analysis and decision-making process, significantly enhancing the intelligence level of quality control efforts.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"4. Results","content":"\u003cp\u003eIn this quality control assessment, we focused on evaluating the performance of different models across multiple quality control points, encompassing intelligent quality control tasks involving 2,600 medical data entries. The quality control points included surgical records, examination and test result documentation, and antibiotic treatment effect evaluations, with each point involving 200 data entries, maintaining a positive-to-negative sample ratio of 1:1. Among these tasks, the Xunfei Spark model stood out, particularly after optimization with the LoRA technique, which significantly enhanced its overall performance shown in Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e. Additionally, a radar chart is provided to visually compare the model performances, as illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e4\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eEvaluation of Accuracy and Consistency: A Comparative Analysis of Different LLMs in Quality Control (w/o means without; w/ means with)\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"7\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\"\u0026plusmn;\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\"\u0026plusmn;\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\"\u0026plusmn;\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\"\u0026plusmn;\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\"\u0026plusmn;\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\"\u0026plusmn;\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eQC Point\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eXunfei Spark w/o lora\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eXunfei Spark w/ lora\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eChatGLM3\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eLlama3\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eClaude\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eGPT-4o\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e91.8%\u0026plusmn;0.07% / 92.2%\u0026plusmn;0.15%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e91.7%\u0026plusmn;0.01% / 93.3%\u0026plusmn;0.11%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c4\"\u003e \u003cp\u003e52.82%\u0026plusmn;0.61% / 58.65%\u0026plusmn;0.39%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e78.78%\u0026plusmn;1.93% / 82.14%\u0026plusmn;0.96%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e88.4%\u0026plusmn;0.46% / 91.1%\u0026plusmn;0.12%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c7\"\u003e \u003cp\u003e97.6%\u0026plusmn;0.13% / 90.1%\u0026plusmn;0.23%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e100%\u0026plusmn;0.00% / 88.7%\u0026plusmn;0.19%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e100%\u0026plusmn;0.00% / 91.4%\u0026plusmn;0.16%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c4\"\u003e \u003cp\u003e73.36%\u0026plusmn;3.23% / 70.40%\u0026plusmn;2.83%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e88.13%\u0026plusmn;0.77% / 83.63%\u0026plusmn;1.16%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e95.6%\u0026plusmn;0.01% / 89.5%\u0026plusmn;0.22%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c7\"\u003e \u003cp\u003e98.2%\u0026plusmn;0.05% / 96.3%\u0026plusmn;0.09%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e90.8%\u0026plusmn;0.29% / 94.3%\u0026plusmn;0.05%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e93.6%\u0026plusmn;0.12% / 93.7%\u0026plusmn;0.07%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c4\"\u003e \u003cp\u003e69.11%\u0026plusmn;1.21% / 52.00%\u0026plusmn;1.79%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e85.98%\u0026plusmn;0.99% / 76.35%\u0026plusmn;1.30%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e91.6%\u0026plusmn;0.54% / 85.4%\u0026plusmn;0.73%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c7\"\u003e \u003cp\u003e90.9%\u0026plusmn;0.50% / 86.9%\u0026plusmn;0.47%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e97.5%\u0026plusmn;0.11% / 91.5%\u0026plusmn;0.21%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e98.1%\u0026plusmn;0.08% / 94.5%\u0026plusmn;0.03%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c4\"\u003e \u003cp\u003e38.79%\u0026plusmn;2.98% / 78.12%\u0026plusmn;3.23%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e72.53%\u0026plusmn;1.42% / 91.56%\u0026plusmn;1.08%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e79.3%\u0026plusmn;0.20% / 91.6%\u0026plusmn;0.10%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c7\"\u003e \u003cp\u003e97.7%\u0026plusmn;0.10% / 93.9%\u0026plusmn;0.13%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e88.9%\u0026plusmn;0.19% / 72.2%\u0026plusmn;0.24%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e91.3%\u0026plusmn;0.13% / 88.8%\u0026plusmn;0.13%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c4\"\u003e \u003cp\u003e42.58%\u0026plusmn;4.04% / 34.41%\u0026plusmn;3.48%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e53.64%\u0026plusmn;0.20% / 60.09%\u0026plusmn;1.28%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e57.9%\u0026plusmn;0.39% / 71.3%\u0026plusmn;0.53%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c7\"\u003e \u003cp\u003e42.9%\u0026plusmn;3.86% / 66.3%\u0026plusmn;1.44%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e95.6%\u0026plusmn;0.03% / 96.1%\u0026plusmn;0.05%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e95.1%\u0026plusmn;0.09% / 96.8%\u0026plusmn;0.04%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c4\"\u003e \u003cp\u003e64.64%\u0026plusmn;1.35% / 83.95%\u0026plusmn;1.65%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e89.57%\u0026plusmn;0.95% / 88.93%\u0026plusmn;1.41%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e97.2%\u0026plusmn;0.71% / 95.9%\u0026plusmn;1.06%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c7\"\u003e \u003cp\u003e94.6%\u0026plusmn;0.43% / 96.8%\u0026plusmn;0.31%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e99.7%\u0026plusmn;0.24% / 90.3%\u0026plusmn;0.14%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e97.7%\u0026plusmn;0.07% / 92.6%\u0026plusmn;0.03%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c4\"\u003e \u003cp\u003e40.72%\u0026plusmn;2.69% / 37.54%\u0026plusmn;2.06%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e95.41%\u0026plusmn;0.71% / 73.40%\u0026plusmn;0.59%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e96.2%\u0026plusmn;0.16% / 77.9%\u0026plusmn;0.24%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c7\"\u003e \u003cp\u003e95.2%\u0026plusmn;0.02% / 64.0%\u0026plusmn;0.91%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e96.3%\u0026plusmn;0.11% / 85.4%\u0026plusmn;0.15%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e96.7%\u0026plusmn;0.03% / 90.6%\u0026plusmn;0.17%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c4\"\u003e \u003cp\u003e56.85%\u0026plusmn;0.68% / 66.93%\u0026plusmn;1.22%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e90.62%\u0026plusmn;0.06% / 84.85%\u0026plusmn;1.26%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e93.3%\u0026plusmn;0.05% / 92.1%\u0026plusmn;0.08%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c7\"\u003e \u003cp\u003e97.8%\u0026plusmn;0.07% / 90.3%\u0026plusmn;0.15%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e76.9%\u0026plusmn;0.13% / 70.1%\u0026plusmn;0.27%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e83.4%\u0026plusmn;0.17% / 87.4%\u0026plusmn;0.22%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c4\"\u003e \u003cp\u003e45.69%\u0026plusmn;2.96% / 42.62%\u0026plusmn;2.10%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e62.67%\u0026plusmn;0.58% / 80.14%\u0026plusmn;0.86%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e67.4%\u0026plusmn;1.34% / 81.8%\u0026plusmn;0.83%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c7\"\u003e \u003cp\u003e63.2%\u0026plusmn;2.28% / 66.7%\u0026plusmn;1.78%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e10\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e91.7%\u0026plusmn;0.04% / 82.8%\u0026plusmn;0.10%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e92.5%\u0026plusmn;0.05% / 83.5%\u0026plusmn;0.12%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c4\"\u003e \u003cp\u003e53.45%\u0026plusmn;2.55% / 81.51%\u0026plusmn;3.20%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e79.36%\u0026plusmn;1.05% / 83.97%\u0026plusmn;0.59%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e79.9%\u0026plusmn;0.60% / 86.4%\u0026plusmn;0.35%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c7\"\u003e \u003cp\u003e96.4%\u0026plusmn;0.39% / 78.2%\u0026plusmn;0.90%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e68.7%\u0026plusmn;0.10% / 90.4%\u0026plusmn;0.04%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e79.6%\u0026plusmn;0.26% / 89.3%\u0026plusmn;0.15%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c4\"\u003e \u003cp\u003e29.55%\u0026plusmn;1.65% / 30.28%\u0026plusmn;1.59%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e52.16%\u0026plusmn;1.28% / 66.17%\u0026plusmn;1.63%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e56.5%\u0026plusmn;0.97% / 77.3%\u0026plusmn;1.01%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c7\"\u003e \u003cp\u003e63.3%\u0026plusmn;1.39% / 86.5%\u0026plusmn;0.25%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMerge\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e93.5%\u0026plusmn;0.14% / 87.8%\u0026plusmn;0.12%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e96.1%\u0026plusmn;0.10% / 89.7%\u0026plusmn;0.16%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c4\"\u003e \u003cp\u003e55.40%\u0026plusmn;2.34% / 57.34%\u0026plusmn;2.47%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c5\"\u003e \u003cp\u003e84.56%\u0026plusmn;0.87% / 72.17%\u0026plusmn;1.39%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c6\"\u003e \u003cp\u003e86.1%\u0026plusmn;0.56% / 86.5%\u0026plusmn;0.69%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c7\"\u003e \u003cp\u003e93.8%\u0026plusmn;0.79% / 84.4%\u0026plusmn;0.68%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eOverall, the Xunfei Spark w/ LoRA model demonstrated exceptionally high accuracy and consistency in most quality control tasks. In critical tasks such as assessing the consistency of surgical records, evaluating antibiotic treatment effects, and analyzing test results, the performance of Xunfei Spark w/ LoRA surpassed that of other comparative models, achieving accuracy rates consistently between 91% and 100%, with consistency also remaining above 90%. This outstanding performance not only reflects the model's strong adaptability in handling complex medical record scenarios but also validates the effectiveness of LoRA optimization in LLMs.\u003c/p\u003e \u003cp\u003eIn contrast, while GPT-4o and ChatGLM3 performed exceptionally well on certain quality control points, their overall consistency and noise robustness did not match that of Xunfei Spark. The Xunfei Spark w/ LoRA model excelled particularly in tasks related to the consistency judgment of surgical steps and progress notes, as well as the analysis of positive examination results, achieving accuracy rates as high as 97.7\u0026ndash;100%. This indicates that Xunfei Spark w/ LoRA possesses efficient and reliable application potential in medical quality control tasks, especially in terms of accuracy and comprehensiveness in analyzing surgical and examination-related records. Moreover, the LoRA optimization enhances the model's robustness when dealing with intricate medical record details, showcasing superior intelligent analytical capabilities across key tasks such as antibiotic usage judgment and positive result analysis in imaging examinations.\u003c/p\u003e \u003cp\u003eIn summary, the performance of Xunfei Spark demonstrates its significant application potential in the field of medical quality control, particularly after LoRA optimization, which has markedly improved the model's accuracy and consistency. Compared to other models, Xunfei Spark w/ LoRA exhibits clear advantages in managing the complexity of medical records and addressing various quality control tasks, especially in terms of noise resilience and multi-tasking capabilities. In the future, this model is expected to be further utilized in practical medical quality control work, enhancing the efficiency and accuracy of medical data processing.\u003c/p\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e should be placed here.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e"},{"header":"5. Discussion","content":"\u003cp\u003eIn this study, we developed the MRQC-LLM framework, leveraging a large language model to improve the quality control of EMRs. This framework aligns with national EMR grading standards, DRG payment policies, and clinical practices, focusing on 11 key quality control points across multiple dimensions, such as patient examinations, surgical records, treatment plans, and medication usage. Through these dimensions, we established a comprehensive quality control dataset with 21,000 training pairs and 2,600 test pairs, enhancing quality control in terms of efficiency, breadth, and accuracy.\u003c/p\u003e \u003cp\u003eOur approach employed the Xunfei Spark medical large model, fine-tuned with LoRA technology, and incorporated RAG to dynamically access domain-specific knowledge from an authoritative quality control database. This use of RAG technology enabled the model to contextualize its responses based on the latest and most relevant background information, thereby improving the precision and relevance of the quality control assessments.The MRQC-LLM framework introduces a multi-agent model architecture, a novel approach in EMR quality control, which comprises dedicated agent modules for each key quality control point. This multi-agent setup allows the model to target specific tasks, such as surgical and examination records or antibiotic usage, by performing tailored data extraction and analysis according to established quality control rules. This structure enhances both the accuracy and flexibility of the model, as each agent can specialize in particular aspects of quality control.\u003c/p\u003e \u003cp\u003eMoreover, we compared the performance of several large language models in EMR quality control tasks, including the Xunfei Spark medical large model, ChatGLM3, Qwen2, and LLama3. Experimental results indicate that the Xunfei Spark medical large model exhibits significant advantages in consistency and accuracy for quality control tasks, achieving an accuracy rate of approximately 90%. Despite the Xunfei Spark model having a parameter size of only 13B, its performance after LoRA fine-tuning is comparable to that of GPT-4o. Its outstanding performance and lower computational overhead provide it with greater deployment flexibility and practical application potential. This suggests that the Xunfei Spark model can operate efficiently in resource-constrained environments and has the potential to further enhance healthcare quality and patient safety in actual medical applications [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eConsidering the costs associated with data labeling, our current research focuses on 11 quality control points and does not yet cover all scenarios in the medical field. Moving forward, we also plan to add new quality control points to expand the coverage, such as process quality control in nursing and diagnostic behaviors, to further improve the quality and accuracy of medical data. Additionally, we will explore adaptive learning [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e] and active learning [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e] techniques to reduce the labor costs associated with labeling new quality control points. These technologies will enable the model to enhance its understanding and judgment capabilities regarding new quality control points through continuous learning and optimization after being exposed to a limited amount of labeled data. This approach will facilitate the rapid iteration and expansion of the quality control model. To improve the model's applicability and precision across various medical scenarios, we plan to continuously enrich the quality control knowledge base and regularly update industry standards and the latest clinical practice guidelines, ensuring the timeliness and authority of the quality control system. In practical deployment, we will further strengthen the validation of the model in real healthcare environments, including multi-center experiments and cross-regional testing, to ensure consistent performance across different hospitals and departments. Furthermore, to achieve efficient integration and information interoperability within the intelligent quality control system, we will advance the deep integration with Hospital Information Systems (HIS), Laboratory Information Systems (LIS), and other electronic medical record systems, allowing the quality control process to permeate the entire medical data flow.\u003c/p\u003e"},{"header":"5. Conclusion","content":"\u003cp\u003eOverall, the quality control system based on LLMs demonstrates significant advantages in enhancing the quality of medical record quality control, but it also reveals some areas for improvement. Future research can focus on further optimizing the model's performance in complex domains, experimenting with larger models, and integrating additional practical quality control needs to cover a broader range of quality control scenarios. Through continuous refinement and optimization, we aim to achieve a higher level of quality control management in the healthcare field, ultimately enhancing healthcare quality and patient safety.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eEthics approval and consent to participate\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis retrospective study was approved by the Ethics Committee of West China Hospital, Sichuan University, and was conducted in accordance with the Declaration of Helsinki. No interventions or additional risks were involved. All patient data were de-identified to ensure confidentiality. The requirement for informed consent was waived by the Ethics Committee of West China Hospital, in accordance with relevant national regulations and institutional guidelines.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent for publication\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAvailability of data and material\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe datasets generated and analyzed during the current study are not publicly available due to data security concerns but are available from the corresponding author on reasonable request. The code used in this study is available from the corresponding author upon reasonable request.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors declare no competing of interest.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFigure Legends\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAll figures are original and created by the authors.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003ch3\u003e\u003cstrong\u003eAuthors and Affiliations\u003c/strong\u003e\u003c/h3\u003e\n\u003cp\u003eAnesthesiology Department, The Third People\u0026rsquo;s Hospital of Chengdu, Chengdu, China\u003c/p\u003e\n\u003cp\u003eZhenqi Zhang\u003c/p\u003e\n\u003cp\u003eInformation Center, West China Hospital, Sichuan University, Chengdu, China\u003c/p\u003e\n\u003cp\u003eXuchen Yang\u003c/p\u003e\n\u003cp\u003eInformation Center, West China Hospital, Sichuan University, Chengdu, China\u003c/p\u003e\n\u003cp\u003eXun Yao\u003c/p\u003e\n\u003cp\u003eInformation Center, West China Hospital, Sichuan University, Chengdu, China\u003c/p\u003e\n\u003cp\u003eHao Yang\u003c/p\u003e\n\u003cp\u003eXunfei Healthcare Technology Co., Ltd, Hefei, China\u003c/p\u003e\n\u003cp\u003eShutong Zhang\u003c/p\u003e\n\u003cp\u003eXunfei Healthcare Technology Co., Ltd, Hefei, China\u003c/p\u003e\n\u003cp\u003eSikai Liu\u003c/p\u003e\n\u003cp\u003eXunfei Healthcare Technology Co., Ltd, Hefei, China\u003c/p\u003e\n\u003cp\u003eJing Wang\u003c/p\u003e\n\u003cp\u003eInformation Center, West China Hospital, Sichuan University, Chengdu, China\u003c/p\u003e\n\u003cp\u003eRui Shi\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthors\u0026rsquo; contributions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eXCY and ZZ contributed equally to the study; they led the design and conceptualization of the framework and collaborated on the development of the methodology. HY directed the study, performed model analysis, conducted data processing, and drafted the manuscript. XY and SZ supported data collection, clinical analysis, and model validation. SL contributed to data analysis and result interpretation, ensuring clinical relevance in findings. JW supervised the project, provided essential feedback on data analysis, and assisted with manuscript revision. RS conceptualized the study, provided strategic guidance on research direction, and reviewed the manuscript. All authors contributed to editorial adjustments, read, and approved the final version of the manuscript. Each author has participated substantially in this work and agrees to be accountable for all aspects of this research.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgments\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe are grateful to the staff in our research groups involved in the study for their valuable contributions and discussions.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eZhao X. Practice status of electronic medical records and improvement measures in EMRs. Chinese Medical Record English Edition. 2013;1(8):343\u0026ndash;6.Zhao X: Practice Status of Electronic Medical Records and Improvement Measures in EMRs. Chinese Medical Record English Edition 2013, 1(8):343\u0026ndash;346.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTian J, Wang YC, Bao SX, Chen HL, Lu JF. Construction and research on the mode of medical record quality control. Chin Res Hosp. 2020;7(5):63\u0026ndash;6.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGeneral Office of the State Council. Opinions on strengthening the performance assessment of tertiary public hospitals [EB/OL]. [April 15, 2020]. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.gov.cn/gongbao/content/2019/content_5366476.htm\u003c/span\u003e\u003cspan address=\"https://www.gov.cn/gongbao/content/2019/content_5366476.htm\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWen Y, Li X, Zeng FF, Lei JY, Chen SY. Application of medical record quality control system based on artificial intelligence. J Sichuan Univ (Medical Sciences). 2023;54(6):1263\u0026ndash;8.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBiswas SS. Role of chat gpt in public health. Ann Biomed Eng. 2023;51(5):868\u0026ndash;9.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAbu-Haifa M, Etawi B, Alkhatatbeh H, Ababneh A. Comparative analysis of ChatGPT, GPT-4, and Microsoft Copilot Chatbots for GRE test. Int J Learn Teach Educational Res. 2024;23(6):327\u0026ndash;47.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eArgasiński K, Grabska-Gradzińska J, Przystalski I, Ochab KK, Walkowiak J. T. Stylometric analysis of large language model-generated commentaries in the context of medical neuroscience. In: International Conference on Computational Science: 2024: Springer; 2024. pp. 281\u0026thinsp;\u0026ndash;\u0026thinsp;95.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSaab K, Tu T, Weng WH, Tanno R, Stutz D, Wulczyn E et al. Capabilities of gemini models in medicine. 2024. arXiv preprint, arXiv:2404.18416.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTeja NS, Kumar K, Malarvel M. Multilingual Text Enhancer (MLTE)\u0026ndash;A LLaMA2 based Model for Prompt Generation. In: 2024 3rd International Conference on Applied Artificial Intelligence and Computing (ICAAIC): 2024: IEEE; 2024. pp. 895\u0026ndash;900.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHuang XT, Jing C, Wang CY. The practice of medical quality control system construction based on the Medical Quality Management Procedures. Chin Health Qual Manage. 2021;28(2):20\u0026ndash;2.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu Z, Chen Z. Cross-Cultural Perspectives on Artificial Intelligence Generated Content (AIGC): A Comparative Study of Attitudes and Acceptance Among Global Products. In: International Conference on Human-Computer Interaction: 2024: Springer; 2024. pp. 287\u0026thinsp;\u0026ndash;\u0026thinsp;98.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYe C, Shi X. Optimizing News Topic Classification with Instructional Fine-Tuning of Chatglm3. In: Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering: 2024; 2024. pp. 573-7.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYang A, Yang B, Hui B, Zheng B, Yu B, Zhou C et al. Qwen2 technical report. 2024. arXiv preprint, arXiv:2407.10671.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGrattafiori A, Dubey A, Jauhri A, Pandey A, Kadian A, Al-Dahle A et al. The llama 3 herd of models. 2024. arXiv preprint, arXiv:2407.21783.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu H, Tam D, Muqeeth M, Mohta J, Huang T, Bansal M, et al. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Adv Neural Inf Process Syst. 2022;35:1950\u0026ndash;65.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHu EJ, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S et al. Lora: Low-rank adaptation of large language models. 2021. arXiv preprint, arXiv:2106.09685.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMao A, Mohri M, Zhong Y. Cross-entropy loss functions: Theoretical analysis and applications. In: International conference on Machine learning: 2023: PMLR; 2023. pp. 23803-28.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMasterman T, Besen S, Sawtell M, Chao A. The landscape of emerging ai agent architectures for reasoning, planning, and tool calling: A survey. 2024. arXiv preprint, arXiv:2404.11584.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLewis P, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Adv Neural Inf Process Syst. 2020;33:9459\u0026ndash;74.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCheng X, Jiao F, Ji G, Tian Y. The Artificial Intelligence Revolution Led by ChatGPT. In: 2023 International Seminar on Computer Science and Engineering Technology (SCSET). 2023: IEEE; 2023. pp. 360\u0026thinsp;\u0026ndash;\u0026thinsp;63.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKabir MR, Lin F, An LLM-P. Adaptive Practicing System. In: LLM@ AIED: 2023; 2023. pp. 43\u0026ndash;52.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLu Y, Yao B, Zhang S, Wang Y, Zhang P, Lu T et al. Human still wins over llm: An empirical study of active learning on domain-specific annotation tasks. 2023. arXiv preprint, arXiv:2311.09825.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-informatics-and-decision-making","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"midm","sideBox":"Learn more about [BMC Medical Informatics and Decision Making](http://bmcmedinformdecismak.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/midm/default.aspx","title":"BMC Medical Informatics and Decision Making","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"LLMs, EMR, quality control, LoRA, RAG","lastPublishedDoi":"10.21203/rs.3.rs-6765575/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6765575/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eObjective:\u003c/strong\u003e This study introduces MRQC-LLM, a novel large language model (LLM)-based framework designed to enhance quality control in electronic medical records (EMRs). By leveraging fine-tuned LLMs, we compare the performance of several models in clinical documentation review to identify the most effective approach.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods: \u003c/strong\u003eWe selected xunfei Spark LLM as the base model, enhanced through LoRA-based fine-tuning for resource optimization. The study employed multiple AI agents tailored to specific quality control tasks, including surgical records, test results, and antibiotic treatment evaluations. Additionally, Retrieval-Augmented Generation technology was integrated with a quality control knowledge base to further improve the precision of context-aware analysis and decision-making processes.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults:\u003c/strong\u003e Our evaluation, using 2,600 medical records, demonstrated that the xunfei Spark Model with LoRA fine-tuning outperformed other models, including GPT-4o, ChatGLM3, Qwen2, and LLama3, achieving accuracy rates between 91%-100% and consistency consistently above 90%. The model excelled particularly in tasks such as surgical record consistency and antibiotic treatment effectiveness, showcasing its adaptability to complex clinical scenarios.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusion:\u003c/strong\u003e The xunfei Spark model with LoRA fine-tuning demonstrates strong potential for improving EMR quality control, offering high accuracy, efficiency, and consistency across clinical scenarios. Future research will focus on expanding MRQC-LLM’s capabilities to encompass a wider array of quality control tasks in increasingly complex clinical environments.\u003c/p\u003e","manuscriptTitle":"MRQC-LLM: A Novel Large Language Model Framework for Enhancing Medical Record Quality Control","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-06-25 09:54:21","doi":"10.21203/rs.3.rs-6765575/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-11-13T12:55:11+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-07-11T21:41:20+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-07-11T10:16:44+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-07-03T13:48:15+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"322070253091203652401720333085288941708","date":"2025-07-01T12:39:49+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"43179720924568610505334522146820030039","date":"2025-06-29T15:32:02+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"171047089748050473739710316276246542865","date":"2025-06-28T14:31:03+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-06-28T08:19:01+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"334631806226681411095838193762727434903","date":"2025-06-27T16:07:32+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"266439385728354854767069535692495943169","date":"2025-06-21T17:41:03+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-06-18T12:07:16+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-06-18T09:31:56+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2025-06-18T06:31:36+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-06-17T09:45:14+00:00","index":"","fulltext":""},{"type":"submitted","content":"BMC Medical Informatics and Decision Making","date":"2025-06-17T09:42:34+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-informatics-and-decision-making","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"midm","sideBox":"Learn more about [BMC Medical Informatics and Decision Making](http://bmcmedinformdecismak.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/midm/default.aspx","title":"BMC Medical Informatics and Decision Making","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"228a4584-9e02-4ac2-a2d2-8d635acc1061","owner":[],"postedDate":"June 25th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"in-revision","subjectAreas":[],"tags":[],"updatedAt":"2025-11-13T13:08:17+00:00","versionOfRecord":[],"versionCreatedAt":"2025-06-25 09:54:21","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6765575","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6765575","identity":"rs-6765575","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}