Shifts in Emergency Physicians’ Attitudes Toward Large Language Model-based Documentation: A Pre- and Post-Implementation Study

Shifts in Emergency Physicians’ Attitudes Toward Large Language Model-based Documentation: A Pre- and Post-Implementation Study | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Shifts in Emergency Physicians’ Attitudes Toward Large Language Model-based Documentation: A Pre- and Post-Implementation Study Seongwon Lee, Ji Woo Song, Seng Chan You, Ji Hoon Kim This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6912295/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 24 Nov, 2025 Read the published version in Scientific Reports → Version 1 posted 12 You are reading this latest preprint version Abstract Large language models (LLMs) can help physicians write medical notes more efficiently. We studied whether using an LLM assistant for writing emergency department discharge notes would reduce doctors' workload and address their concerns about using AI in real medical practice. Eight emergency doctors with an average of 12 years of experience participated in our study. We surveyed them before using the LLM assistant, after 3 days, and after 5 weeks of use. The results showed that doctors' concerns about using LLMs decreased significantly and stayed low throughout the study period. Their perceived workload also dropped considerably. Additionally, the time needed to write each discharge note was reduced to one-third of the original time. These findings demonstrate that doctors readily accepted and benefited from the LLM assistant in their daily practice. Our study provides the first real-world evidence of how doctors' attitudes toward AI assistants change over time in clinical settings, offering valuable insights for future implementation of LLM-based documentation tools in healthcare. Health sciences/Health care Health sciences/Medical research Surveys and Questionnaires Natural Language Processing Workload Emergency Service Hospital Longitudinal Studies Figures Figure 1 Figure 2 Figure 3 INTRODUCTION Large language models (LLMs) have drawn considerable attention as a promising technology for clinical documentation due to their ability to rapidly assimilate, summarize, and rephrase information 1 – 5 . LLM-based clinical documentation system is expected to enhance the efficiency of medical note writing and reduce healthcare professionals’ administrative burden 6 . Numerous studies have elaborated on the development and evaluation of LLM-based clinical documentation technologies 6 – 9 , but there has been little studies empirically investigating the effects of LLMs system on healthcare professionals' work and their attitude toward the systems in real-world clinical practice. According to a survey of American Medical Association in 2024, 40% of 1,183 physicians equally expressed excitements and concerns regarding the increase usage of AI in the healthcare 10 . This ambivalence highlights the critical role of physician acceptance and attitudes in the successful integration of AI-based systems. Prior studies have underscored that trust and acceptance among healthcare professionals are indispensable prerequisites for the widespread implementation of AI in clinical practice 11 . We conducted a longitudinal survey study on implementation of ‘Your-Knowledgeable Navigator of Treatment-Emergency department Discharge Note assistant’ (Y-KNOT-EDN), an on-premise LLM-based clinical documentation system at Severance Hospital (Seoul, South Korea) in November 2024 that generates discharge note drafts from EHR data for physician review and finalization 12 . We aimed to investigate physicians’ perceptions and acceptance of LLM-based clinical documentation system in clinical settings, focusing on emergency department discharge notes. Specifically, we examine changes in perceived workload and concerns related to LLM use, as well as key technology acceptance factors including perceived usefulness, attitude, intention to use, and intention to delegate drafting tasks to the LLM assistant system. Three surveys were administered to emergency physicians responsible for documenting discharge notes, at three time points: prior to Y-KNOT-DEN implementation (T1), three days after implementation (T2), and five weeks after implementation (T3) (see Fig. 1 ). RESULT Eight emergency medicine attending physicians (4 males, 4 females; mean age 39.9 years) participated in the surveys, with an average of 12.1 years of ED experience. On average, participants handled 10.9 discharge notes daily. Figure 2 shows responses on concerns about using LLMs in discharge note writing over time. Overall concern, defined as the mean across dimensions, reduced from 3.4 at T1 to 2.7 at T2 and further to 2.5 at T3 (p = 0.008), showing 26% reduction from T1 to T3. Pairwise comparisons indicated reductions from T1 to T2 (p = 0.023) and from T1 to T3 (p = 0.008), while T2 and T3 were comparable (p = 0.202). Four dimensions of concern on LLMs—worsening patient care, loss of control, generating impersonal draft, and legal and ethical issues—significantly dropped overtime (p = 0.004, 0.002, 0.010, and 0.028, respectively). The other four dimensions of concern—false information, data bias, privacy, and worsening physician reasoning—declined without statistical significance (p = 0.089, 0.268, 0.143, and 0.156, respectively). Detailed statistics are provided in Supplementary Table 1 and Supplementary Table 2. Figure 3 shows perceived workload scores across dimensions. Overall workload dropped from 11.0 at T1 to 8.0 at T2, and further to 6.9 at T3 (p = 0.040), showing 37% reduction from T1 to T3. Paired comparisons indicated reductions between T1 and T2 (p = 0.023), and T1 and T3 (p = 0.035); T2 and T3 (p = 0.402) were comparable. Two dimensions of perceived workload—temporal demand, and effort required for accomplishment—significantly decreased over time (p = 0.002, and 0.021, respectively). Three dimensions of perceived workload—mental demand, physical demand, and frustration—consistently declined without statistical significance (p = 0.381, 0.409, and 0.122, respectively), while unsatisfaction with performance remained consistently low. Detailed statistics are provided in Supplementary Table 3 and Supplementary Table 4. Participants reported that completing a single discharge note manually took 127.5 seconds at T1, whereas it took 42.8 seconds using the LLM assistant at T3—indicating a time reduction of approximately two-thirds (p = 0.002). Participants’ perceptions toward the LLM assistant at T2 and T3 are presented in Supplementary Fig. 1. The mean perceived usefulness of the assistant was 3.8 at T2 and increased to 4.2 at T3 (p = 0.038). Attitude toward the assistant (T2 mean = 3.7; T3 mean = 3.9), the intention to use the assistant (T2 mean = 3.9; T3 mean = 4.0), and the intention to delegate discharge drafting to the assistant (T2 mean = 4.0; T3 mean = 4.0) remained consistently high. It is worth mentioning that the LLM assistant was unexpectedly shut down once between T2 and T3, and it was quickly restored within a week. Among the six participants who were affected, three reported that writing discharge notes manually without the aid of the LLM assistant was “Difficult” (score = 4), two found it “Not difficult” (score = 2), and one responded “Not at all difficult” (score = 1), indicating varying levels of perceived disruption. In response to the open-ended question regarding the generated drafts, participants emphasized the need for improved accuracy and more patient-tailored content. They also expressed a high willingness to delegate the task of drafting discharge notes to the assistant. DISCUSSION The key findings of this longitudinal field study can be summarized as follows. First, concerns about LLMs were significantly alleviated over time (26% reduction from T1 to T3), particularly in relation to worsening patient care, loss of control, impersonal drafts, and legal/ethical issues. Second, the perceived workload significantly declined over time (37% reduction from T1 to T3), especially in terms of temporal demand and effort. Recall-based documentation time reduced approximately by two-thirds. Moreover, participants reported high levels of perceived usefulness, attitude, intention to use, and intention to delegate discharge note drafting throughout the study period, indicating that the positive results were not merely driven by a novelty effect but reflected genuine acceptance of the system. Despite these positive outcomes, open-ended feedback indicated a need for further system refinement to improve accuracy and provide more physician-tailored content. The success of AI implementation in healthcare depends not only on technical performance but also on how healthcare professionals respond to and interact with these systems 11 . Lambert et al. identified key barriers to healthcare AI adoption, including concerns about loss of professional autonomy, difficulties with clinical workflow integration, and alert fatigue from oversensitive systems 13 . To our knowledge, this study is the first to examine physicians' perceptions as potential enablers and barriers to the adoption of an LLM-based clinical documentation system. By documenting how these metrics evolved from pre-implementation to five weeks post-implementation, we provide valuable insights for future AI integration efforts in healthcare. Our findings underscore the necessity of conducting similar pre-post implementation studies when deploying AI systems to understand user experience, address potential barriers to adoption, and optimize implementation strategies. The primary goal of implementing an LLM assistant is to reduce physicians' workload 14 , 15 . However, a previous study reported AI-powered clinical documentation did not enhance clinician efficiency in primary care settings 16 . While this study did not measure times using EHRs, our findings demonstrated significant reductions in perceived temporal demand and required effort from pre-implementation to five weeks post-implementation. Interestingly, aspects less directly related to discharge note writing, such as physical demand and dissatisfaction with performance, showed minimal changes, as did mental demand, suggesting the cognitive aspects of documentation remained relatively consistent. These results are particularly meaningful for emergency physicians who must manage multiple patients under time constraints, as the LLM assistant provided substantial benefits in perceived burden while preserving physicians’ role in the process. Understanding these nuanced workload changes is critical for guiding the development and acceptance of future AI documentation technologies in healthcare settings. LIMITATIONS Our study has several limitations. First, the sample size was small, with only eight emergency physicians, due to the current shortage of physicians in South Korea following a temporal mass resignation of residents 17 , 18 . Future studies with larger samples are needed. Second, the study was conducted at a single institution, which may limit generalizability. Third, the reliance on self-reported measures introduces subjectivity. Finally, documentation time was measured using recall-based methods 19 . CONCLUSION In summary, this study is the first to implement an on-premise LLM system for ED discharge note drafting in a clinical setting and to evaluate the physician acceptance by comparing perceptions before and after the deployment of the system. The survey results revealed strong physician acceptance: as both the initial concerns on using LLM in clinical documentation and the perceived workload of writing a discharge note subsided within days, highlighting the system’s practical value. Future research will actively incorporate user feedback and track satisfaction changes to further refine and optimize the system. METHODS Participants and Data Sources We recruited 15 attending ED physicians at Severance Hospital, Seoul, South Korea, and eight of them voluntarily participated in this study. We obtained online informed consent and conducted a pre-implementation survey prior to the implementation of the LLM system. They received instruction on the Y-KNOT-EDN system and were required to use it for discharge note drafting throughout the study period. Two follow-up surveys were administered thereafter. The survey was developed by two researchers (S.L. and J.W.S.) using the SurveyMonkey online survey platform (San Mateo, California, USA; www.surveymonkey.com ). The study was approved by the Institutional Review Board of Severance Hospital, Yonsei University Health System (No. 4-2024-1622), and was conducted in accordance with the Declaration of Helsinki and all relevant institutional and national guidelines and regulations. Survey Phases To conduct a longitudinal evaluation of the clinical application of the Y-KNOT-EDN, we carried out a survey-based study in three phases (Fig. 1 ). T1 (Pre-implementation) : During this phase, participants provided baseline data that included their concerns about using LLMs in clinical documentation and their perceived workload associated with manually drafting ED discharge notes. Measures of concerns related to the use of LLMs in clinical documentation were adapted from Spotnitz et al. 4 Spotnitz et al. surveyed 30 physicians using open-ended questions about their concerns regarding LLMs in healthcare and identified eight key dimensions: false information, worsen patient care, data bias, loss of human control, impersonal draft, legal/ethical issues, privacy, and worsen clinicians’ reasoning. We used these dimensions as measurement items, assessed on a 5-point Likert scale. The perceived workload of ED discharge documentation was assessed using the NASA-TLX 20 , a widely adopted instrument that evaluates workload across six dimensions: mental demand, physical demand, temporal demand, dissatisfaction with one’s own performance, effort required for accomplishment, and frustration level. Each dimension was rated on a scale ranging from 0 to 20. In addition, demographic information was collected, including sex, age, work experience in the ED, and the average time required for manual documentation per note. T2 (Three days post-implementation) : At this point, concerns and workload were reassessed, along with system acceptance measures—perceived usefulness 21 , attitude 22 , intention to use 23 , and intention to delegate drafting 24 by adapting measurement items from prior studies. T3 (Five weeks post-implementation) : This phase repeated the survey items from T2, with the addition of questions related to the documentation time per note when using the system. Initially, the third survey was planned to be administered four weeks after the implementation of Y-KNOT-EDN into the clinical workflow. However, an unexpected one-week system shutdown occurred one week after implementation, compelling emergency department physicians to revert to manual documentation. Consequently, the survey was postponed to five weeks post-launch, and additional questions were included to capture the impact of this shutdown. Specifically, participants were asked whether they had experienced a temporary shutdown of the system during their work period. Those who had were then requested to rate the difficulty of manually drafting discharge notes during the downtime on a 5-point scale, with 1 indicating “not at all difficult” and 5 indicating “very difficult.” Open-ended feedback about the system was also collected. The full survey questions and the scoring scales are summarized in Table 1 . Table 1 Measurements Variables (Phase) Dimensions Items Concern on Using LLM in Clinical Settings* (T1, T2, T3) False Information I worry that using the LLM system for discharge note documentation may generate inaccurate or false information. Worsen Patient Care I worry that using the LLM system for discharge note documentation may adversely affect patient care. Data Bias I worry that using the LLM system for discharge note documentation may distort content due to low-quality and biased training data. Loss of Control I worry that using the LLM system for discharge note documentation may make it difficult for physicians to control and supervise the process. Impersonal Draft I worry that using the LLM system for discharge note documentation may produce impersonal records with low levels of empathy. Legal/Ethical Issue I worry that using the LLM system for discharge note documentation may lead to ethical and legal issues. Privacy I worry that using the LLM system for discharge note documentation may raise privacy concerns. Worsen Physicians’ Reasoning I worry that using the LLM system for discharge note documentation may reduce physicians’ opportunities to interpret and synthesize data. Perceived Workload † (T1, T2, T3) Mental Demand How mentally demanding was the discharge note documentation task? Physical Demand How physically demanding was the discharge note documentation task? Temporal Demand How hurried or rushed was the pace of the discharge note documentation task? Dissatisfaction with Performance ‡ How successful were you in accomplishing the discharge note documentation? Effort for Accomplishment How hard did you have to work to accomplish your level of performance in the discharge note documentation? Frustration How insecure, discouraged, irritated, stressed, and annoyed did you feel during the discharge note documentation task? Perceived Usefulness* (T2, T3) - The LLM system increases productivity in discharge note documentation. The LLM system is effective for discharge note documentation. I believe that the LLM system is useful for discharge note documentation. Attitude* (T2, T3) - Using the LLM system for discharge note documentation is a good idea. Using the LLM system for discharge note documentation is a wise idea. I like using the LLM system for discharge note documentation. Using the LLM system for discharge note documentation makes me feel good. Intention to Use* (T2, T3) - I intend to use the LLM system for discharge note documentation. I have an intention to use the LLM system for discharge note documentation. I will try to use the LLM system for discharge note documentation. Intention to Delegate* (T2, T3) - I plan to delegate discharge note drafting to the LLM system. I intend to delegate discharge note drafting to the LLM system. I have chosen to use the LLM system for discharge note drafting. Documentation Time (T1, T3) - How much time does it take to write a single discharge note? Experience of Shutdown (T3) - During your work period, did you experience any downtime or errors that rendered Y-KNOT-EDN unusable? (Yes/No) (If “Yes”) How challenging was the task during the shutdown? * These items are measured using a 1-to-5 Likert scale. † Workload is measured using a 0-to-20 scale. ‡The actual question is phrased positively; its score is calculated by subtracting the reported value from the maximum score of 20. Statistical Analysis The statistical analysis employed a Friedman test to assess changes across three time points (T1, T2, and T3) for both the individual LLM concern items (as well as the overall concern score) and the six workload dimensions (and their aggregated overall score). When the Friedman test indicated a significant effect, post-hoc Wilcoxon signed-rank tests with Holm-Bonferroni correction were conducted to perform pairwise comparisons between time points. In addition, a paired t-test was used to compare both times required to complete a note between T1 and T3 and perceived usefulness, attitude, intention to use, and intention to delegate drafting between T2 and T3. Declarations ACKNOWLEDGEMENT This research was supported by a grant of the MD-Phd/Medical Scientist Training Program through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea. This research was also supported by a grant of the Korea Health Technology R&D Project through the KHIDI (grant number: RS-2023-KH135326). FUNDING This research was supported by a grant of the MD-Phd/Medical Scientist Training Program through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea. This research was also supported by a grant of the Korea Health Technology R&D Project through the KHIDI (grant number: RS-2023-KH135326). AUTHOR CONTRIBUTIONS S.L. developed the study protocol, analysed data and edited manuscript. J.W.S. analysed data, wrote and edited manuscript. S.C.Y. developed the study protocol and edited manuscript. J.H.K. gathered the interviewees and edited manuscript S.L. and J.W.S. are co-first authors and have contributed equally to this work. S.C.Y. and J.H.K. are corresponding authors and have contributed equally to this work. The corresponding authors attest that all listed authors meet the authorship criteria and that others who met the criteria have not been omitted. COMPETING INTERESTS S.C.Y reports grants from Daiichi Sankyo. He is a coinventor of granted Korea Patent DP-2023-1223 and DP-2023-0920, and pending Patent Applications DP-2024-0909, DP-2024-0908, DP-2022-1658, DP-2022-1478, and DP-2022-1365 unrelated to current work. S.C.Y. is a chief executive officer of PHI Digital Healthcare. Other authors have no potential conflicts of interest to disclose. DATA AVAILABILITY The raw data are disclosed in Supplementary Table S5 References Boussina, A. et al. Large language models for more efficient reporting of hospital quality measures. NEJM AI . 1 , AIcs2400420 (2024). Ong, J. C. L. et al. Medical Ethics of Large Language Models in Medicine. NEJM AI , AIra2400038 (2024). Gallifant, J. et al. The TRIPOD-LLM reporting guideline for studies using large language models. Nature Medicine , 1–10 (2025). Spotnitz, M. et al. A Survey of Clinicians' Views of the Utility of Large Language Models. Appl. Clin. Inf. 15 , 306–312. https://doi.org/10.1055/a-2281-7092 (2024). Tripathi, S., Sukumaran, R. & Cook, T. S. Efficient healthcare with large language models: optimizing clinical workflow and enhancing patient care. J. Am. Med. Inform. Assoc. 31 , 1436–1440 (2024). Hartman, V. et al. Developing and evaluating large language model–generated emergency medicine handoff notes. JAMA Netw. Open. 7 , e2448723–e2448723 (2024). Hartman, V. C. et al. A method to automate the discharge summary hospital course for neurology patients. J. Am. Med. Inform. Assoc. 30 , 1995–2003 (2023). Chua, C. E. et al. Integration of customised LLM for discharge summary generation in real-world clinical settings: a pilot study on RUSSELL GPT. The Lancet Reg. Health–Western Pacific 51 (2024). Heilmeyer, F. et al. Viability of Open Large Language Models for Clinical Documentation in German Health Care: Real-World Model Evaluation Study. JMIR Med. Inf. 12 , e59617 (2024). Association, A. M. AMA Augmented Intelligence Research: Physician Sentiments Around the Use of AI in Health Care: Motivations, Opportunities, Risks, and Use Cases – Shifts from 2023 to 2024. (2025). Park, S. H. & Langlotz, C. P. Crucial Role of Understanding in Human-Artificial Intelligence Interaction for Successful Clinical Adoption. Korean J. Radiol. 26 , 287–290 (2025). Kim, H. et al. A Bilingual On-premise AI agent for Clinical Drafting: Seamless EHR integration in the Y-KNOT Project. medRxiv , 2025.2004. 25325003 (2025). (2003). Lambert, S. I. et al. An integrative review on the acceptance of artificial intelligence among healthcare professionals in hospitals. NPJ Digit. Med. 6 , 111 (2023). Jindal, J. A., Lungren, M. P. & Shah, N. H. Ensuring useful adoption of generative artificial intelligence in healthcare. J. Am. Med. Inform. Assoc. 31 , 1441–1444. https://doi.org/10.1093/jamia/ocae043 (2024). Gandhi, T. K. et al. How can artificial intelligence decrease cognitive and work burden for front line practitioners? JAMIA open. 6 , ooad079 (2023). Tierney, A. A. et al. Ambient artificial intelligence scribes to alleviate the burden of clinical documentation. NEJM Catalyst Innovations in Care Delivery 5, CAT. 23.0404 (2024). McCurry, J. South Korean doctors threaten mass resignation. Lancet 403 , 1124 (2024). Moon, J. & Lee, J. Y. Why I decide to leave South Korea healthcare system. The Lancet Reg. Health–Western Pacific 52 (2024). Murad, M. H. et al. Measuring Documentation Burden in Healthcare. Journal Gen. Intern. medicine , 1–12 (2024). Hart, S. G. in Proceedings of the human factors and ergonomics society annual meeting. 904–908 (Sage publications Sage CA: Los Angeles, CA). Venkatesh, V. & Davis, F. D. A theoretical extension of the technology acceptance model: Four longitudinal field studies. Manage. Sci. 46 , 186–204 (2000). Davis, F. D. & Perceived Usefulness Perceived Ease of Use, and User Acceptance of Information Technology. MIS Q. 13 , 319–340. https://doi.org/10.2307/249008 (1989). Ajzen, I., Fishbein, M., Lohmann, S. & Albarracín, D. The influence of attitudes on behavior. The handbook of attitudes, volume 1: Basic principles , 197–255 (2018). Stout, N., Dennis, A. R. & Wells, T. M. The buck stops there: The impact of perceived accountability and control on the intention to delegate to software agents. AIS Trans. Hum Comput Interact. 6 , 1–15 (2014). Additional Declarations Competing interest reported. S.C.Y reports grants from Daiichi Sankyo. He is a coinventor of granted Korea Patent DP-2023-1223 and DP-2023-0920, and pending Patent Applications DP-2024-0909, DP-2024-0908, DP-2022-1658, DP-2022-1478, and DP-2022-1365 unrelated to current work. S.C.Y. is a chief executive officer of PHI Digital Healthcare. Other authors have no potential conflicts of interest to disclose. Supplementary Files SupplementaryMaterialver.4.0.pdf Cite Share Download PDF Status: Published Journal Publication published 24 Nov, 2025 Read the published version in Scientific Reports → Version 1 posted Editorial decision: Revision requested 11 Aug, 2025 Reviews received at journal 28 Jul, 2025 Reviews received at journal 17 Jul, 2025 Reviews received at journal 13 Jul, 2025 Reviewers agreed at journal 08 Jul, 2025 Reviewers agreed at journal 08 Jul, 2025 Reviewers agreed at journal 03 Jul, 2025 Reviewers invited by journal 03 Jul, 2025 Editor assigned by journal 03 Jul, 2025 Editor invited by journal 27 Jun, 2025 Submission checks completed at journal 22 Jun, 2025 First submitted to journal 22 Jun, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6912295","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":480234454,"identity":"b9a56c8a-0af1-4fd4-ac87-95a175b8d076","order_by":0,"name":"Seongwon Lee","email":"","orcid":"","institution":"Institute for Innovation in Digital Healthcare, Yonsei University Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Seongwon","middleName":"","lastName":"Lee","suffix":""},{"id":480234455,"identity":"fb014a5e-a118-4ea4-ba62-bf383bd8bc52","order_by":1,"name":"Ji Woo Song","email":"","orcid":"","institution":"Yonsei University College of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Ji","middleName":"Woo","lastName":"Song","suffix":""},{"id":480234457,"identity":"f4beb962-c074-47c8-8be5-5eb30af4c5d0","order_by":2,"name":"Seng Chan You","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAyUlEQVRIiWNgGAWjYDACCSDmATHYGxiYSdTCc4BkLRIJRGrhn9387MHbPYflDG6+MZMu+MMgz99AyJI7x8wN5zw7bGxwO8dMemYbg+GMAwS0GEgkmEnzHDicuAGkhbeBgXEDIYcZSKR/g2i5eQao9w+DPRFacqC23OABMtgYEglqkbiRUyY550C6seSZtGJr3jaJZIJ+4Z+Rvk3izQFrOb7jhzfe5vljY9vfQMgaCGgGYg4DBkg0EQfqgJj9AdHKR8EoGAWjYGQBAKOaPe76PrPCAAAAAElFTkSuQmCC","orcid":"","institution":"Yonsei University College of Medicine","correspondingAuthor":true,"prefix":"","firstName":"Seng","middleName":"Chan","lastName":"You","suffix":""},{"id":480234460,"identity":"fc5ddd67-79c8-45b5-b877-657bb661cb90","order_by":3,"name":"Ji Hoon Kim","email":"","orcid":"","institution":"Yonsei University College of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Ji","middleName":"Hoon","lastName":"Kim","suffix":""}],"badges":[],"createdAt":"2025-06-17 08:53:12","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6912295/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6912295/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41598-025-24659-4","type":"published","date":"2025-11-24T15:58:37+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":86148097,"identity":"96bc7869-4404-456f-8d4e-2b70877eea68","added_by":"auto","created_at":"2025-07-07 09:33:29","extension":"jpeg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":328872,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eTimeline of the Study and Items Asked at Each Phase\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe study timeline illustrates three survey phases: T1 (before Y-KNOT-EDN implementation), T2 (3 days after implementation), and T3 (5 weeks after implementation). Between T2 and T3, Y-KNOT-EDN was shut down unexpectedly and then restored within a week. Survey measures conducted at each phase are indicated below the timeline.\u003c/p\u003e","description":"","filename":"floatimage1.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-6912295/v1/ae5e00eff1c8d14802d2f76a.jpeg"},{"id":86148099,"identity":"6a48391d-2219-462a-8cb5-ae9f7743849d","added_by":"auto","created_at":"2025-07-07 09:33:29","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":119784,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eConcerns on Using LLM in Clinical Documentation Over Time (T1, T2, T3)\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eNote.\u003c/em\u003e ***: \u003cem\u003ep\u003c/em\u003e\u0026lt; .001; **: \u003cem\u003ep\u003c/em\u003e \u0026lt; .01; *: \u003cem\u003ep\u003c/em\u003e \u0026lt; .05\u003c/p\u003e\n\u003cp\u003eMean scores for eight major concerns regarding the use of the LLM-based documentation system, as well as overall concerns, are shown across T1, T2, and T3. Gray lines connect each physician’s individual trajectory; large purple circles and solid lines denote the group mean at T1, T2, and T3. Significant reductions were observed in overall concerns (p = 0.008), particularly for worsening patient care (p = 0.004), loss of control (p = 0.002), impersonal drafts (p = 0.010), and legal/ethical issues (p = 0.028), reflecting increased trust and acceptance of the LLM-based documentation system over time.\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-6912295/v1/455ae9c1f510d90333375b5b.png"},{"id":86149114,"identity":"a9e3012e-ac9e-496b-80ef-fbb87cd4f945","added_by":"auto","created_at":"2025-07-07 09:41:29","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":152734,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003ePerceived Workload Over Time (T1, T2, T3)\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eNote.\u003c/em\u003e ***: \u003cem\u003ep\u003c/em\u003e\u0026lt; .001; **: \u003cem\u003ep\u003c/em\u003e \u0026lt; .01; *: \u003cem\u003ep\u003c/em\u003e \u0026lt; .05\u003c/p\u003e\n\u003cp\u003eMean of NASA-TLX workload scores (0–20 scale) across six sub-dimensions—mental demand, physical demand, temporal demand, dissatisfaction with performance, effort required for accomplishment, and frustration—as well as overall workload are shown across T1, T2, and T3. Gray lines show individual physicians’ workload trajectories; large pink circles and connecting lines indicate the group mean at each time point. Notable decreases in temporal demand (p = 0.002), effort (p = 0.021), and overall workload (p = 0.040) suggest a reduced perceived burden over the three survey phases.\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-6912295/v1/24ea0a7e1593f3296a8b9959.png"},{"id":97178693,"identity":"887d0045-0b9c-4333-a346-090a0c081a2c","added_by":"auto","created_at":"2025-12-01 16:12:49","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1148090,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6912295/v1/d2277afb-c100-4abd-bd73-49ed83caf468.pdf"},{"id":86149111,"identity":"088cf77c-38cc-415d-9120-873f80f227f8","added_by":"auto","created_at":"2025-07-07 09:41:29","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":300397,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryMaterialver.4.0.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6912295/v1/75273de2013b6d46ec772b03.pdf"}],"financialInterests":"Competing interest reported. S.C.Y reports grants from Daiichi Sankyo. He is a coinventor of granted Korea Patent DP-2023-1223 and DP-2023-0920, and pending Patent Applications DP-2024-0909, DP-2024-0908, DP-2022-1658, DP-2022-1478, and DP-2022-1365 unrelated to current work. S.C.Y. is a chief executive officer of PHI Digital Healthcare. Other authors have no potential conflicts of interest to disclose.","formattedTitle":"Shifts in Emergency Physicians’ Attitudes Toward Large Language Model-based Documentation: A Pre- and Post-Implementation Study","fulltext":[{"header":"INTRODUCTION","content":"\u003cp\u003eLarge language models (LLMs) have drawn considerable attention as a promising technology for clinical documentation due to their ability to rapidly assimilate, summarize, and rephrase information\u003csup\u003e\u003cspan additionalcitationids=\"CR2 CR3 CR4\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u003c/sup\u003e. LLM-based clinical documentation system is expected to enhance the efficiency of medical note writing and reduce healthcare professionals\u0026rsquo; administrative burden\u003csup\u003e\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u003c/sup\u003e. Numerous studies have elaborated on the development and evaluation of LLM-based clinical documentation technologies\u003csup\u003e\u003cspan additionalcitationids=\"CR7 CR8\" citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e\u003c/sup\u003e, but there has been little studies empirically investigating the effects of LLMs system on healthcare professionals' work and their attitude toward the systems in real-world clinical practice.\u003c/p\u003e \u003cp\u003eAccording to a survey of American Medical Association in 2024, 40% of 1,183 physicians equally expressed excitements and concerns regarding the increase usage of AI in the healthcare\u003csup\u003e\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u003c/sup\u003e. This ambivalence highlights the critical role of physician acceptance and attitudes in the successful integration of AI-based systems. Prior studies have underscored that trust and acceptance among healthcare professionals are indispensable prerequisites for the widespread implementation of AI in clinical practice\u003csup\u003e\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eWe conducted a longitudinal survey study on implementation of \u0026lsquo;Your-Knowledgeable Navigator of Treatment-Emergency department Discharge Note assistant\u0026rsquo; (Y-KNOT-EDN), an on-premise LLM-based clinical documentation system at Severance Hospital (Seoul, South Korea) in November 2024 that generates discharge note drafts from EHR data for physician review and finalization\u003csup\u003e\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eWe aimed to investigate physicians\u0026rsquo; perceptions and acceptance of LLM-based clinical documentation system in clinical settings, focusing on emergency department discharge notes. Specifically, we examine changes in perceived workload and concerns related to LLM use, as well as key technology acceptance factors including perceived usefulness, attitude, intention to use, and intention to delegate drafting tasks to the LLM assistant system.\u003c/p\u003e \u003cp\u003eThree surveys were administered to emergency physicians responsible for documenting discharge notes, at three time points: prior to Y-KNOT-DEN implementation (T1), three days after implementation (T2), and five weeks after implementation (T3) (see Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e"},{"header":"RESULT","content":"\u003cp\u003eEight emergency medicine attending physicians (4 males, 4 females; mean age 39.9 years) participated in the surveys, with an average of 12.1 years of ED experience. On average, participants handled 10.9 discharge notes daily.\u003c/p\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e2\u003c/span\u003e shows responses on concerns about using LLMs in discharge note writing over time. Overall concern, defined as the mean across dimensions, reduced from 3.4 at T1 to 2.7 at T2 and further to 2.5 at T3 (p\u0026thinsp;=\u0026thinsp;0.008), showing 26% reduction from T1 to T3. Pairwise comparisons indicated reductions from T1 to T2 (p\u0026thinsp;=\u0026thinsp;0.023) and from T1 to T3 (p\u0026thinsp;=\u0026thinsp;0.008), while T2 and T3 were comparable (p\u0026thinsp;=\u0026thinsp;0.202).\u003c/p\u003e \u003cp\u003eFour dimensions of concern on LLMs\u0026mdash;worsening patient care, loss of control, generating impersonal draft, and legal and ethical issues\u0026mdash;significantly dropped overtime (p\u0026thinsp;=\u0026thinsp;0.004, 0.002, 0.010, and 0.028, respectively). The other four dimensions of concern\u0026mdash;false information, data bias, privacy, and worsening physician reasoning\u0026mdash;declined without statistical significance (p\u0026thinsp;=\u0026thinsp;0.089, 0.268, 0.143, and 0.156, respectively). Detailed statistics are provided in Supplementary Table\u0026nbsp;1 and Supplementary Table\u0026nbsp;2.\u003c/p\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e3\u003c/span\u003e shows perceived workload scores across dimensions. Overall workload dropped from 11.0 at T1 to 8.0 at T2, and further to 6.9 at T3 (p\u0026thinsp;=\u0026thinsp;0.040), showing 37% reduction from T1 to T3. Paired comparisons indicated reductions between T1 and T2 (p\u0026thinsp;=\u0026thinsp;0.023), and T1 and T3 (p\u0026thinsp;=\u0026thinsp;0.035); T2 and T3 (p\u0026thinsp;=\u0026thinsp;0.402) were comparable.\u003c/p\u003e \u003cp\u003eTwo dimensions of perceived workload\u0026mdash;temporal demand, and effort required for accomplishment\u0026mdash;significantly decreased over time (p\u0026thinsp;=\u0026thinsp;0.002, and 0.021, respectively). Three dimensions of perceived workload\u0026mdash;mental demand, physical demand, and frustration\u0026mdash;consistently declined without statistical significance (p\u0026thinsp;=\u0026thinsp;0.381, 0.409, and 0.122, respectively), while unsatisfaction with performance remained consistently low. Detailed statistics are provided in Supplementary Table\u0026nbsp;3 and Supplementary Table\u0026nbsp;4.\u003c/p\u003e \u003cp\u003e Participants reported that completing a single discharge note manually took 127.5 seconds at T1, whereas it took 42.8 seconds using the LLM assistant at T3\u0026mdash;indicating a time reduction of approximately two-thirds (p\u0026thinsp;=\u0026thinsp;0.002).\u003c/p\u003e \u003cp\u003eParticipants\u0026rsquo; perceptions toward the LLM assistant at T2 and T3 are presented in Supplementary Fig.\u0026nbsp;1. The mean perceived usefulness of the assistant was 3.8 at T2 and increased to 4.2 at T3 (p\u0026thinsp;=\u0026thinsp;0.038). Attitude toward the assistant (T2 mean\u0026thinsp;=\u0026thinsp;3.7; T3 mean\u0026thinsp;=\u0026thinsp;3.9), the intention to use the assistant (T2 mean\u0026thinsp;=\u0026thinsp;3.9; T3 mean\u0026thinsp;=\u0026thinsp;4.0), and the intention to delegate discharge drafting to the assistant (T2 mean\u0026thinsp;=\u0026thinsp;4.0; T3 mean\u0026thinsp;=\u0026thinsp;4.0) remained consistently high.\u003c/p\u003e \u003cp\u003eIt is worth mentioning that the LLM assistant was unexpectedly shut down once between T2 and T3, and it was quickly restored within a week. Among the six participants who were affected, three reported that writing discharge notes manually without the aid of the LLM assistant was \u0026ldquo;Difficult\u0026rdquo; (score\u0026thinsp;=\u0026thinsp;4), two found it \u0026ldquo;Not difficult\u0026rdquo; (score\u0026thinsp;=\u0026thinsp;2), and one responded \u0026ldquo;Not at all difficult\u0026rdquo; (score\u0026thinsp;=\u0026thinsp;1), indicating varying levels of perceived disruption.\u003c/p\u003e \u003cp\u003eIn response to the open-ended question regarding the generated drafts, participants emphasized the need for improved accuracy and more patient-tailored content. They also expressed a high willingness to delegate the task of drafting discharge notes to the assistant.\u003c/p\u003e"},{"header":"DISCUSSION","content":"\u003cp\u003eThe key findings of this longitudinal field study can be summarized as follows. First, concerns about LLMs were significantly alleviated over time (26% reduction from T1 to T3), particularly in relation to worsening patient care, loss of control, impersonal drafts, and legal/ethical issues. Second, the perceived workload significantly declined over time (37% reduction from T1 to T3), especially in terms of temporal demand and effort. Recall-based documentation time reduced approximately by two-thirds. Moreover, participants reported high levels of perceived usefulness, attitude, intention to use, and intention to delegate discharge note drafting throughout the study period, indicating that the positive results were not merely driven by a novelty effect but reflected genuine acceptance of the system. Despite these positive outcomes, open-ended feedback indicated a need for further system refinement to improve accuracy and provide more physician-tailored content.\u003c/p\u003e \u003cp\u003eThe success of AI implementation in healthcare depends not only on technical performance but also on how healthcare professionals respond to and interact with these systems\u003csup\u003e\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e. Lambert et al. identified key barriers to healthcare AI adoption, including concerns about loss of professional autonomy, difficulties with clinical workflow integration, and alert fatigue from oversensitive systems\u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e. To our knowledge, this study is the first to examine physicians' perceptions as potential enablers and barriers to the adoption of an LLM-based clinical documentation system. By documenting how these metrics evolved from pre-implementation to five weeks post-implementation, we provide valuable insights for future AI integration efforts in healthcare. Our findings underscore the necessity of conducting similar pre-post implementation studies when deploying AI systems to understand user experience, address potential barriers to adoption, and optimize implementation strategies.\u003c/p\u003e \u003cp\u003eThe primary goal of implementing an LLM assistant is to reduce physicians' workload\u003csup\u003e\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e,\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u003c/sup\u003e. However, a previous study reported AI-powered clinical documentation did not enhance clinician efficiency in primary care settings\u003csup\u003e\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e\u003c/sup\u003e. While this study did not measure times using EHRs, our findings demonstrated significant reductions in perceived temporal demand and required effort from pre-implementation to five weeks post-implementation. Interestingly, aspects less directly related to discharge note writing, such as physical demand and dissatisfaction with performance, showed minimal changes, as did mental demand, suggesting the cognitive aspects of documentation remained relatively consistent. These results are particularly meaningful for emergency physicians who must manage multiple patients under time constraints, as the LLM assistant provided substantial benefits in perceived burden while preserving physicians\u0026rsquo; role in the process. Understanding these nuanced workload changes is critical for guiding the development and acceptance of future AI documentation technologies in healthcare settings.\u003c/p\u003e"},{"header":"LIMITATIONS","content":"\u003cp\u003eOur study has several limitations. First, the sample size was small, with only eight emergency physicians, due to the current shortage of physicians in South Korea following a temporal mass resignation of residents\u003csup\u003e\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e,\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e\u003c/sup\u003e. Future studies with larger samples are needed. Second, the study was conducted at a single institution, which may limit generalizability. Third, the reliance on self-reported measures introduces subjectivity. Finally, documentation time was measured using recall-based methods\u003csup\u003e\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e"},{"header":"CONCLUSION","content":"\u003cp\u003eIn summary, this study is the first to implement an on-premise LLM system for ED discharge note drafting in a clinical setting and to evaluate the physician acceptance by comparing perceptions before and after the deployment of the system. The survey results revealed strong physician acceptance: as both the initial concerns on using LLM in clinical documentation and the perceived workload of writing a discharge note subsided within days, highlighting the system\u0026rsquo;s practical value. Future research will actively incorporate user feedback and track satisfaction changes to further refine and optimize the system.\u003c/p\u003e"},{"header":"METHODS","content":"\u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003eParticipants and Data Sources\u003c/h2\u003e \u003cp\u003e We recruited 15 attending ED physicians at Severance Hospital, Seoul, South Korea, and eight of them voluntarily participated in this study. We obtained online informed consent and conducted a pre-implementation survey prior to the implementation of the LLM system. They received instruction on the Y-KNOT-EDN system and were required to use it for discharge note drafting throughout the study period. Two follow-up surveys were administered thereafter.\u003c/p\u003e \u003cp\u003eThe survey was developed by two researchers (S.L. and J.W.S.) using the SurveyMonkey online survey platform (San Mateo, California, USA; www.surveymonkey.com\u003c/a\u003e\u003c/span\u003e\u003cspan address=\"http://www.surveymonkey.com\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e). The study was approved by the Institutional Review Board of Severance Hospital, Yonsei University Health System (No. 4-2024-1622), and was conducted in accordance with the Declaration of Helsinki and all relevant institutional and national guidelines and regulations.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eSurvey Phases\u003c/h2\u003e \u003cp\u003eTo conduct a longitudinal evaluation of the clinical application of the Y-KNOT-EDN, we carried out a survey-based study in three phases (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003e \u003cem\u003eT1 (Pre-implementation)\u003c/em\u003e: During this phase, participants provided baseline data that included their concerns about using LLMs in clinical documentation and their perceived workload associated with manually drafting ED discharge notes. Measures of concerns related to the use of LLMs in clinical documentation were adapted from Spotnitz et al.\u003csup\u003e\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u003c/sup\u003e Spotnitz et al. surveyed 30 physicians using open-ended questions about their concerns regarding LLMs in healthcare and identified eight key dimensions: false information, worsen patient care, data bias, loss of human control, impersonal draft, legal/ethical issues, privacy, and worsen clinicians\u0026rsquo; reasoning. We used these dimensions as measurement items, assessed on a 5-point Likert scale. The perceived workload of ED discharge documentation was assessed using the NASA-TLX\u003csup\u003e\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e\u003c/sup\u003e, a widely adopted instrument that evaluates workload across six dimensions: mental demand, physical demand, temporal demand, dissatisfaction with one\u0026rsquo;s own performance, effort required for accomplishment, and frustration level. Each dimension was rated on a scale ranging from 0 to 20. In addition, demographic information was collected, including sex, age, work experience in the ED, and the average time required for manual documentation per note.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cem\u003eT2 (Three days post-implementation)\u003c/em\u003e: At this point, concerns and workload were reassessed, along with system acceptance measures\u0026mdash;perceived usefulness\u003csup\u003e\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e\u003c/sup\u003e, attitude\u003csup\u003e\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e\u003c/sup\u003e, intention to use\u003csup\u003e\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e, and intention to delegate drafting\u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e by adapting measurement items from prior studies.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cem\u003eT3 (Five weeks post-implementation)\u003c/em\u003e: This phase repeated the survey items from T2, with the addition of questions related to the documentation time per note when using the system. Initially, the third survey was planned to be administered four weeks after the implementation of Y-KNOT-EDN into the clinical workflow. However, an unexpected one-week system shutdown occurred one week after implementation, compelling emergency department physicians to revert to manual documentation. Consequently, the survey was postponed to five weeks post-launch, and additional questions were included to capture the impact of this shutdown. Specifically, participants were asked whether they had experienced a temporary shutdown of the system during their work period. Those who had were then requested to rate the difficulty of manually drafting discharge notes during the downtime on a 5-point scale, with 1 indicating \u0026ldquo;not at all difficult\u0026rdquo; and 5 indicating \u0026ldquo;very difficult.\u0026rdquo; Open-ended feedback about the system was also collected.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003eThe full survey questions and the scoring scales are summarized in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eMeasurements\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eVariables (Phase)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDimensions\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eItems\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"7\" rowspan=\"8\"\u003e \u003cp\u003eConcern on Using LLM in Clinical Settings*\u003c/p\u003e \u003cp\u003e(T1, T2, T3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFalse Information\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eI worry that using the LLM system for discharge note documentation may generate inaccurate or false information.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eWorsen Patient Care\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eI worry that using the LLM system for discharge note documentation may adversely affect patient care.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eData Bias\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eI worry that using the LLM system for discharge note documentation may distort content due to low-quality and biased training data.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eLoss of Control\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eI worry that using the LLM system for discharge note documentation may make it difficult for physicians to control and supervise the process.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eImpersonal Draft\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eI worry that using the LLM system for discharge note documentation may produce impersonal records with low levels of empathy.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eLegal/Ethical Issue\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eI worry that using the LLM system for discharge note documentation may lead to ethical and legal issues.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePrivacy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eI worry that using the LLM system for discharge note documentation may raise privacy concerns.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eWorsen Physicians\u0026rsquo; Reasoning\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eI worry that using the LLM system for discharge note documentation may reduce physicians\u0026rsquo; opportunities to interpret and synthesize data.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"5\" rowspan=\"6\"\u003e \u003cp\u003ePerceived Workload\u003cb\u003e\u0026dagger;\u003c/b\u003e\u003c/p\u003e \u003cp\u003e(T1, T2, T3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMental Demand\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHow mentally demanding was the discharge note documentation task?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePhysical Demand\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHow physically demanding was the discharge note documentation task?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTemporal Demand\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHow hurried or rushed was the pace of the discharge note documentation task?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDissatisfaction with Performance\u003cb\u003e\u0026Dagger;\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHow successful were you in accomplishing the discharge note documentation?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eEffort for Accomplishment\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHow hard did you have to work to accomplish your level of performance in the discharge note documentation?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFrustration\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHow insecure, discouraged, irritated, stressed, and annoyed did you feel during the discharge note documentation task?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003ePerceived Usefulness*\u003c/p\u003e \u003cp\u003e(T2, T3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eThe LLM system increases productivity in discharge note documentation.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eThe LLM system is effective for discharge note documentation.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eI believe that the LLM system is useful for discharge note documentation.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"3\" rowspan=\"4\"\u003e \u003cp\u003eAttitude*\u003c/p\u003e \u003cp\u003e(T2, T3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\" morerows=\"3\" rowspan=\"4\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eUsing the LLM system for discharge note documentation is a good idea.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eUsing the LLM system for discharge note documentation is a wise idea.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eI like using the LLM system for discharge note documentation.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eUsing the LLM system for discharge note documentation makes me feel good.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003eIntention to Use*\u003c/p\u003e \u003cp\u003e(T2, T3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eI intend to use the LLM system for discharge note documentation.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eI have an intention to use the LLM system for discharge note documentation.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eI will try to use the LLM system for discharge note documentation.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003eIntention to Delegate*\u003c/p\u003e \u003cp\u003e(T2, T3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eI plan to delegate discharge note drafting to the LLM system.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eI intend to delegate discharge note drafting to the LLM system.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eI have chosen to use the LLM system for discharge note drafting.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDocumentation Time\u003c/p\u003e \u003cp\u003e(T1, T3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHow much time does it take to write a single discharge note?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eExperience of Shutdown\u003c/p\u003e \u003cp\u003e(T3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eDuring your work period, did you experience any downtime or errors that rendered Y-KNOT-EDN unusable? (Yes/No)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(If \u0026ldquo;Yes\u0026rdquo;) How challenging was the task during the shutdown?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"3\"\u003e\u003cb\u003e*\u003c/b\u003eThese items are measured using a 1-to-5 Likert scale.\u003c/td\u003e\u003c/tr\u003e \u003ctr\u003e\u003ctd colspan=\"3\"\u003e\u003cb\u003e\u0026dagger;\u003c/b\u003eWorkload is measured using a 0-to-20 scale.\u003c/td\u003e\u003c/tr\u003e \u003ctr\u003e\u003ctd colspan=\"3\"\u003e\u0026Dagger;The actual question is phrased positively; its score is calculated by subtracting the reported value from the maximum score of 20.\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003eStatistical Analysis\u003c/h2\u003e \u003cp\u003eThe statistical analysis employed a Friedman test to assess changes across three time points (T1, T2, and T3) for both the individual LLM concern items (as well as the overall concern score) and the six workload dimensions (and their aggregated overall score). When the Friedman test indicated a significant effect, post-hoc Wilcoxon signed-rank tests with Holm-Bonferroni correction were conducted to perform pairwise comparisons between time points. In addition, a paired t-test was used to compare both times required to complete a note between T1 and T3 and perceived usefulness, attitude, intention to use, and intention to delegate drafting between T2 and T3.\u003c/p\u003e \u003c/div\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eACKNOWLEDGEMENT\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis research was supported by a grant of the MD-Phd/Medical Scientist Training Program through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health \u0026amp; Welfare, Republic of Korea. This research was also supported by a grant of the Korea Health Technology R\u0026amp;D Project through the KHIDI (grant number: RS-2023-KH135326).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFUNDING\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis research was supported by a grant of the MD-Phd/Medical Scientist Training Program through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health \u0026amp; Welfare, Republic of Korea. This research was also supported by a grant of the Korea Health Technology R\u0026amp;D Project through the KHIDI (grant number: RS-2023-KH135326).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAUTHOR CONTRIBUTIONS\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eS.L. developed the study protocol, analysed data and edited manuscript.\u003c/p\u003e\n\u003cp\u003eJ.W.S. analysed data, wrote and edited manuscript.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eS.C.Y. developed the study protocol and edited manuscript.\u003c/p\u003e\n\u003cp\u003eJ.H.K. gathered the interviewees and edited manuscript\u003c/p\u003e\n\u003cp\u003eS.L. and J.W.S. are co-first authors and have contributed equally to this work. S.C.Y. and J.H.K. are corresponding authors and have contributed equally to this work. The corresponding authors attest that all listed authors meet the authorship criteria and that others who met the criteria have not been omitted.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCOMPETING INTERESTS\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eS.C.Y reports grants from Daiichi Sankyo. He is a coinventor of granted Korea Patent DP-2023-1223 and DP-2023-0920, and pending Patent Applications DP-2024-0909, DP-2024-0908, DP-2022-1658, DP-2022-1478, and DP-2022-1365 unrelated to current work. S.C.Y. is a chief executive officer of PHI Digital Healthcare. Other authors have no potential conflicts of interest to disclose.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDATA AVAILABILITY\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe raw data are disclosed in Supplementary Table S5\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eBoussina, A. et al. Large language models for more efficient reporting of hospital quality measures. \u003cem\u003eNEJM AI\u003c/em\u003e. \u003cb\u003e1\u003c/b\u003e, AIcs2400420 (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOng, J. C. L. et al. Medical Ethics of Large Language Models in Medicine. \u003cem\u003eNEJM AI\u003c/em\u003e, AIra2400038 (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGallifant, J. et al. The TRIPOD-LLM reporting guideline for studies using large language models. \u003cem\u003eNature Medicine\u003c/em\u003e, 1\u0026ndash;10 (2025).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSpotnitz, M. et al. A Survey of Clinicians' Views of the Utility of Large Language Models. \u003cem\u003eAppl. Clin. Inf.\u003c/em\u003e \u003cb\u003e15\u003c/b\u003e, 306\u0026ndash;312. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1055/a-2281-7092\u003c/span\u003e\u003cspan address=\"10.1055/a-2281-7092\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTripathi, S., Sukumaran, R. \u0026amp; Cook, T. S. Efficient healthcare with large language models: optimizing clinical workflow and enhancing patient care. \u003cem\u003eJ. Am. Med. Inform. Assoc.\u003c/em\u003e \u003cb\u003e31\u003c/b\u003e, 1436\u0026ndash;1440 (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHartman, V. et al. Developing and evaluating large language model\u0026ndash;generated emergency medicine handoff notes. \u003cem\u003eJAMA Netw. Open.\u003c/em\u003e \u003cb\u003e7\u003c/b\u003e, e2448723\u0026ndash;e2448723 (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHartman, V. C. et al. A method to automate the discharge summary hospital course for neurology patients. \u003cem\u003eJ. Am. Med. Inform. Assoc.\u003c/em\u003e \u003cb\u003e30\u003c/b\u003e, 1995\u0026ndash;2003 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChua, C. E. et al. Integration of customised LLM for discharge summary generation in real-world clinical settings: a pilot study on RUSSELL GPT. \u003cem\u003eThe Lancet Reg. Health\u0026ndash;Western Pacific\u003c/em\u003e \u003cb\u003e51\u003c/b\u003e (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHeilmeyer, F. et al. Viability of Open Large Language Models for Clinical Documentation in German Health Care: Real-World Model Evaluation Study. \u003cem\u003eJMIR Med. Inf.\u003c/em\u003e \u003cb\u003e12\u003c/b\u003e, e59617 (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAssociation, A. M. AMA Augmented Intelligence Research: Physician Sentiments Around the Use of AI in Health Care: Motivations, Opportunities, Risks, and Use Cases \u0026ndash; Shifts from 2023 to 2024. (2025).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePark, S. H. \u0026amp; Langlotz, C. P. Crucial Role of Understanding in Human-Artificial Intelligence Interaction for Successful Clinical Adoption. \u003cem\u003eKorean J. Radiol.\u003c/em\u003e \u003cb\u003e26\u003c/b\u003e, 287\u0026ndash;290 (2025).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKim, H. et al. A Bilingual On-premise AI agent for Clinical Drafting: Seamless EHR integration in the Y-KNOT Project. \u003cem\u003emedRxiv\u003c/em\u003e, 2025.2004. 25325003 (2025). (2003).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLambert, S. I. et al. An integrative review on the acceptance of artificial intelligence among healthcare professionals in hospitals. \u003cem\u003eNPJ Digit. Med.\u003c/em\u003e \u003cb\u003e6\u003c/b\u003e, 111 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJindal, J. A., Lungren, M. P. \u0026amp; Shah, N. H. Ensuring useful adoption of generative artificial intelligence in healthcare. \u003cem\u003eJ. Am. Med. Inform. Assoc.\u003c/em\u003e \u003cb\u003e31\u003c/b\u003e, 1441\u0026ndash;1444. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1093/jamia/ocae043\u003c/span\u003e\u003cspan address=\"10.1093/jamia/ocae043\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGandhi, T. K. et al. How can artificial intelligence decrease cognitive and work burden for front line practitioners? \u003cem\u003eJAMIA open.\u003c/em\u003e \u003cb\u003e6\u003c/b\u003e, ooad079 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTierney, A. A. et al. Ambient artificial intelligence scribes to alleviate the burden of clinical documentation. \u003cem\u003eNEJM Catalyst Innovations in Care Delivery\u003c/em\u003e 5, CAT. 23.0404 (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMcCurry, J. South Korean doctors threaten mass resignation. \u003cem\u003eLancet\u003c/em\u003e \u003cb\u003e403\u003c/b\u003e, 1124 (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMoon, J. \u0026amp; Lee, J. Y. Why I decide to leave South Korea healthcare system. \u003cem\u003eThe Lancet Reg. Health\u0026ndash;Western Pacific\u003c/em\u003e \u003cb\u003e52\u003c/b\u003e (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMurad, M. H. et al. Measuring Documentation Burden in Healthcare. \u003cem\u003eJournal Gen. Intern. medicine\u003c/em\u003e, 1\u0026ndash;12 (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHart, S. G. in \u003cem\u003eProceedings of the human factors and ergonomics society annual meeting.\u003c/em\u003e 904\u0026ndash;908 (Sage publications Sage CA: Los Angeles, CA).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVenkatesh, V. \u0026amp; Davis, F. D. A theoretical extension of the technology acceptance model: Four longitudinal field studies. \u003cem\u003eManage. Sci.\u003c/em\u003e \u003cb\u003e46\u003c/b\u003e, 186\u0026ndash;204 (2000).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDavis, F. D. \u0026amp; Perceived Usefulness Perceived Ease of Use, and User Acceptance of Information Technology. \u003cem\u003eMIS Q.\u003c/em\u003e \u003cb\u003e13\u003c/b\u003e, 319\u0026ndash;340. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.2307/249008\u003c/span\u003e\u003cspan address=\"10.2307/249008\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e (1989).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAjzen, I., Fishbein, M., Lohmann, S. \u0026amp; Albarrac\u0026iacute;n, D. The influence of attitudes on behavior. \u003cem\u003eThe handbook of attitudes, volume 1: Basic principles\u003c/em\u003e, 197\u0026ndash;255 (2018).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eStout, N., Dennis, A. R. \u0026amp; Wells, T. M. The buck stops there: The impact of perceived accountability and control on the intention to delegate to software agents. \u003cem\u003eAIS Trans. Hum Comput Interact.\u003c/em\u003e \u003cb\u003e6\u003c/b\u003e, 1\u0026ndash;15 (2014).\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Surveys and Questionnaires, Natural Language Processing, Workload, Emergency Service, Hospital, Longitudinal Studies","lastPublishedDoi":"10.21203/rs.3.rs-6912295/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6912295/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eLarge language models (LLMs) can help physicians write medical notes more efficiently. We studied whether using an LLM assistant for writing emergency department discharge notes would reduce doctors' workload and address their concerns about using AI in real medical practice. Eight emergency doctors with an average of 12 years of experience participated in our study. We surveyed them before using the LLM assistant, after 3 days, and after 5 weeks of use. The results showed that doctors' concerns about using LLMs decreased significantly and stayed low throughout the study period. Their perceived workload also dropped considerably. Additionally, the time needed to write each discharge note was reduced to one-third of the original time. These findings demonstrate that doctors readily accepted and benefited from the LLM assistant in their daily practice. Our study provides the first real-world evidence of how doctors' attitudes toward AI assistants change over time in clinical settings, offering valuable insights for future implementation of LLM-based documentation tools in healthcare.\u003c/p\u003e","manuscriptTitle":"Shifts in Emergency Physicians’ Attitudes Toward Large Language Model-based Documentation: A Pre- and Post-Implementation Study","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-07-07 09:33:24","doi":"10.21203/rs.3.rs-6912295/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-08-11T23:12:52+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-07-28T15:53:43+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-07-17T15:36:34+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-07-13T12:04:56+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"323972193811989908793311294697264021667","date":"2025-07-08T09:40:28+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"137630412549782966241761786861783718614","date":"2025-07-08T09:31:56+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"113712398598055668923159308514278078201","date":"2025-07-03T13:50:15+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-07-03T08:49:55+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-07-03T07:53:32+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2025-06-27T17:24:25+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-06-22T13:23:29+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2025-06-22T13:21:01+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"20969745-5196-41cf-b8a9-5e424b310108","owner":[],"postedDate":"July 7th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":50999831,"name":"Health sciences/Health care"},{"id":50999833,"name":"Health sciences/Medical research"}],"tags":[],"updatedAt":"2025-12-01T16:05:16+00:00","versionOfRecord":{"articleIdentity":"rs-6912295","link":"https://doi.org/10.1038/s41598-025-24659-4","journal":{"identity":"scientific-reports","isVorOnly":false,"title":"Scientific Reports"},"publishedOn":"2025-11-24 15:58:37","publishedOnDateReadable":"November 24th, 2025"},"versionCreatedAt":"2025-07-07 09:33:24","video":"","vorDoi":"10.1038/s41598-025-24659-4","vorDoiUrl":"https://doi.org/10.1038/s41598-025-24659-4","workflowStages":[]},"version":"v1","identity":"rs-6912295","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6912295","identity":"rs-6912295","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}