{"paper_id":"0c28537e-b32f-4ffd-bec5-eacd4279dfac","body_text":"A Systematic Review of Early Evidence on Generative AI for Drafting Responses to Patient Messages | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article A Systematic Review of Early Evidence on Generative AI for Drafting Responses to Patient Messages Di Hu, Yawen Guo, Yiliang Zhou, Lidia Flores, Kai Zheng This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6713507/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 9 You are reading this latest preprint version Abstract This systematic review synthesizes currently available empirical evidence on generative artificial intelligence (GenAI) tools for drafting responses to patient messages. Across a total of 23 studies identified, GenAI was found to produce empathetic replies with quality comparable to that of responses drafted by human experts, demonstrating its potential to facilitate patient–provider communication and alleviate clinician burnout. Challenges include inconsistent performance, risks to patient safety, and ethical concerns around transparency and oversight. Additionally, utilization of the technology remains limited in real-world settings, and existing evaluation efforts vary greatly in study design and methodological rigor. As this field evolves, there is a critical need to establish robust and standardized evaluation frameworks, develop practical guidelines for disclosure and accountability, and meaningfully engage clinicians, patients, and other stakeholders. This review may provide timely insights into informing future research of GenAI and guiding the responsible integration of this technology into day-to-day clinical work. Business and commerce/Information systems and information technology Health sciences/Health care Figures Figure 1 Introduction Patient portals, as an integral part of electronic health records (EHRs), are now available in nearly 90% of health systems in the United States, enhancing patient engagement and transforming patient–provider communication 1 , 2 . Through portal messaging, patients can contact their care teams outside of scheduled visits to ask questions, request medication refills, and follow up on lab-test results 3 , 4 . Incentivized by the Health Information Technology for Economic and Clinical Health Act 5 , messaging has become one of the most frequently used patient portal features 6 . Over the past decade, message volume has grown substantially 7 , 8 , with the COVID-19 pandemic further accelerating this surge, driving a 157% increase compared to pre-pandemic levels. This elevated rate of use has persisted since 9 . While improved communication is associated with improved patient care, it has also increased the burden on clinicians. The influx of messages has overwhelmed clinicians’ “in-baskets”—the EHR-based inboxes—resulting in a workload that often extends beyond regular work hours 8 , 10 – 12 . This sustained burden has been linked to clinician burnout, job dissatisfaction, and challenges in maintaining work–life balance 10 , 13 – 16 . Human, technological, and policy-level strategies have been adopted to address this growing burden. These efforts have included forming designated administrative teams 17 , refining management workflows 2 , 18 , developing artificial intelligence (AI) applications for message triage 19 , 20 , and introducing new billing codes for e-visits 21 , 22 . Most recently, generative AI (GenAI), particularly generative large language models (LLMs), has emerged as a potential solution for alleviating clinician in-basket overload 23 . Capable of interpreting complex texts and generating human-like responses, these models have demonstrated the ability to answer medical questions with expert-level knowledge 24 , 25 and to respond to patient forum posts in a more empathetic tone than physicians 26 , 27 . With such capability, GenAI offers a novel approach to assist clinicians by creating draft replies to patient messages. Several large health systems in the United States have begun implementing Health Insurance Portability and Accountability Act (HIPAA)-compliant GenAI tools for this purpose within their EHR systems 28 – 30 . These early adopters have highlighted GenAI’s potential to generate useful first drafts, reduce clinician exhaustion, and ultimately enable and improve the efficiency of asynchronous care 31 , 32 . Despite encouraging progress, use of GenAI in facilitating clinical communication is still at an early stage, and the existing literature is highly fragmented. Prior reviews on GenAI in medical question-answering have mainly focused on its performance and efficacy for medical exams, clinical decision support, and patient education, rather than its role in supporting clinicians with in-basket management 33 – 35 . Further, there has been no consensus to date on how to evaluate AI-drafted responses or what lessons can be learned from initial efforts. Current studies vary widely in design, outcome measures, and evaluation methods. Clinical contexts for these studies also differ, spanning various specialties and diverse patient populations, each of which may present nuanced differences in patient–provider communication practices. The lack of standardization poses challenges for synthesizing findings. It also emphasizes the need to understand the body of early evidence in order to inform the development and implementation of GenAI tools as they continue to evolve and become better integrated into care delivery. This review addresses these gaps by systematically identifying and synthesizing empirical studies that evaluate GenAI for drafting replies to patient messages through EHR-embedded patient portals. By examining study settings, objectives, designs, outcomes, and key findings, we aimed to provide a timely overview of the current evidence and outline directions for future research. Specifically, this review seeks to answer the following questions: How have GenAI tools been studied for drafting responses to patient portal messages and what are the study settings, objectives, and designs? In what clinical contexts have these tools been evaluated? Who are the users participating in these evaluations, and what approaches and outcome measures have been used? What early consensus has emerged from the findings? In answering these questions, we also discuss the challenges and opportunities surfaced by early studies and highlight ethical considerations that should be prioritized to ensure the responsible use of GenAI in replying to patient messages. Our synthesis offers guidance for future evaluations and implementations of GenAI designed to aid in patient–provider communication particularly through patient portals. Results This section presents our synthesis of current research on the use of GenAI for drafting responses to patient messages. Table 1 provides an overview of key information extracted from the included studies. Results of Literature Search and Screening Figure 1 illustrates the literature search and screening process following the PRISMA flow diagram format. Our search across five databases (ACM Digital Library, IEEE Xplore, PubMed, Scopus, and Web of Science) resulted in 3,980 potentially relevant papers. After removing 2,003 duplicates, 1,977 articles remained. Screening based on title and abstract further excluded 1,284 additional papers, leaving 693 for full-text review. After reviewing the full-text of these 693 papers, 23 were deemed to meet the inclusion and exclusion criteria (detailed in the Methods section), and were included in the final review. Study Characteristics All 23 studies were conducted in the United States and published between 2023 and 2025. The majority of them (n = 16) appeared in medical and informatics journals, including JAMA Network Open , Journal of the American Medical Informatics Association , JAMIA Open , and Mayo Clinic Proceedings:Digital Health . The remainder (n = 7) were published in clinical specialty-focused venues, such as Urology Practice , Ophthalmology Science , and Annals of Plastic Surgery . In terms of type of publication: 17 were full-length research articles, while the others consisted of three research letters, one brief communication, one perspective, and one commentary. Study Setting, Objective, and Design The included studies evaluated GenAI for drafting replies to patient messages in two settings: live EHR systems (n = 7) and simulated environments (n = 16). Across these settings, studies primarily aimed to evaluate: (1) the content of AI-generated drafts (n = 19); (2) the implementation of GenAI and its impact on clinical efficiency (n = 8); (3) user perceptions, preferences, or experiences (n = 12); and (4) the effects of prompt engineering (n = 5). Many studies addressed more than one objective. The live EHR system setting refers to real-world clinical environments where GenAI tools are embedded into existing in-basket workflows to generate draft replies for patient messages. All seven studies in this setting evaluated GenAI tools integrated into the Epic EHR (Epic Systems Corporation, Verona, Wisconsin, USA), using OpenAI’s Generative Pre-Trained Transformer (GPT)-4 for response generation 31,32,36–40 . Several studies explored GenAI implementation and its impact on clinical efficiency and user experience, using measures such as draft utilization 31,36,37 , extent of edits 36 , time spent 31,32 , and clinician burden 31,39 . Some evaluated user perceptions of AI-drafted content, aiming to understand how clinicians in different roles judged its usefulness 37 or how linguistic features were associated with perceived empathy 38 . Three studies primarily examined prompt engineering. Two iteratively modified prompts in live settings and assessed their effect on draft usability and clinician feedback 36,37 . The third adopted a structured, human-in-the-loop process to refine prompts in a test environment before deploying them in production, evaluating their impact on provider acceptance and patient satisfaction 39 . A distinct effort sought to bridge qualitative and quantitative evaluation metrics by proposing a unified framework for assessing LLM performance in healthcare, which was then applied to Epic’s in-basket GenAI feature 40 . All implementations were conducted as pilot-scale efforts, with most framed as quality improvement projects 31,32,37–39 . The majority employed prospective 31,37,39 or quasi-experimental 36 designs to evaluate GenAI deployment or prompt modifications in real time. One study used a modified waitlist randomized design to compare outcomes between clinicians with and without access to GenAI drafts 32 , while another used a cross-sectional survey to capture provider perceptions of draft quality 38 . In contrast, studies conducted in the simulated setting evaluated GenAI tools outside of live clinical workflows. These controlled experiments involved generating replies to either de-identified patient messages extracted from the EHR 41–51 or hypothetical inquiries modeled on real patient portal communications 52–56 . The 16 studies conducted in this setting primarily focused on assessing the content quality 42–55 and user perceptions 41,45,46,49,51,53,56 of AI-drafted replies, using a range of outcome measures detailed later. Many compared GenAI drafts across different models 43,45,47,51,52,54 or against human-authored responses 41–43,45–49,51,53,54,56 . Although not embedded in actual workflows, a few studies examined GenAI’s impact on efficiency through subjective ratings and self-reported time for responding or editing drafts 47,48,53,54 . While GPT-3.5 and GPT-4 were the most frequently used models, some studies evaluated alternative or customized variants. These included GPT-4-based, specialty-specific retrieval-augmented models 45,48 , fine-tuned versions of LLaMA adapted for clinical use 43,52 , and institution-developed GPT-powered tools 45,51 . One study also compared multiple commercial models, including Bard, Claude, and Bing, alongside GPT variants 54 . To approximate the EHR context, one study included simulated medical records alongside messages when prompting GPT 53 , while another connected a customized model to the local EHR system to access clinical notes and patient details in support of drafting 48 . A few studies also incorporated clinician editing processes to examine the human–AI synergistic effect in message drafting 53,54 as well as to explore patient preferences regarding the disclosure of AI involvement 56 . Cross-sectional or multi-stage surveys were commonly used to gather user perceptions and assessments, often through randomized and blinded review designs comparing responses by different GenAI models or humans 41,45–47,49,51,53,54,56 . Two studies had distinct objectives and approaches. One developed and fine-tuned a large language model on portal message data and evaluated its performance against baseline GPT and clinician drafts 43 . The other conducted a retrospective qualitative study focused specifically on GenAI responses to negative patient messages, using thematic analysis to compare content differences between GenAI and clinical care team replies 42 . Clinical Context, Message Topic, and Participant (Evaluator) GenAI for responding to patient messages was evaluated across various clinical contexts, with primary care , including internal medicine, family medicine, and pediatrics, being the most common setting (n = 10) 31,32,36–39,43,49–51 . Four studies conducted evaluations across primary care and specialties 31,36,37,51 , while the remaining studies focused on specific specialty domains, including dermatology 46 , urology 44,45,48 , ophthalmology 54,55 , oncology 53 , and surgery 47,52 . Across the included studies, GenAI was used to generate draft replies for patient messages covering a variety of topics, ranging from administrative inquiries , such as appointment scheduling and medication refills 38,43,49,52,56 , to medical advice requests related to symptoms, postoperative concerns, and test results 44,47,48,55,56 . Several studies deliberately varied message seriousness 56 , complexity 52 , or level of detail 55 to include comprehensive test cases, while others focused solely on clinical questions involving decision-making implications or condition-specific issues 39,41,44–48,51,54,55 . Some studies curated representative messages to ensure that evaluations covered the most commonly asked topics in patient portals 41,43,45,54 , while others randomly selected samples from the in-baskets of participating clinicians or from messages sent by a particular patient group 38,39,46,50 . To ensure fair GenAI evaluation under simulated conditions without integrated data sources, two studies excluded messages that required access to external information 38,43 . In addition, some studies collected unique message samples to meet specific research objectives. For example, one study evaluated GenAI using a mix of adolescent patient and proxy messages 50 , while another tested GenAI in emotionally sensitive scenarios using negative patient messages 42 . Participants in the studies can be grouped into two high-level categories: clinical experts and non-experts . Over 90% of the studies (n = 21) involved clinical experts across diverse contexts and roles, including physicians 31,36–39,42–47,49,50,52–55 , advanced practice providers 31,37,39,47,49 , nurses 31,36,37,48 , medical trainees (medical students, residents, and fellows) 47,48 , medical assistants 37 , and clinical pharmacists 31 . Clinician participants reviewed GenAI draft replies, shared pilot experiences, and either edited AI drafts or created human-authored counterparts for comparative analysis. In contrast, only a few studies engaged non-expert participants, including patient advisors or laypeople 39,41,45,51,56 . These participants were typically asked to rate tone, identify the authorship, or share their personal preferences for the given responses, rather than evaluate their clinical quality. While considered laypersons in the medical context, many were active stakeholders, such as long-term patient advisors with extensive experience using patient portals 39 , participants recruited from institutional research registries 51 , or pre-screened volunteers with a clinical condition of interest 45 . Only two studies focused exclusively on non-expert perspectives. One surveyed over 1,400 members of a patient advisory committee to evaluate preferences for AI-generated responses under varying disclosure conditions 56 . Another recruited a nationally representative sample of laypersons through a crowdsourcing platform to assess their ability to distinguish between human and AI responses as well as their trust in AI’s advice 41 . Evaluation Method and Outcome Measure All 23 studies incorporated human ratings using Likert scales to assess GenAI responses and their impact. Ten evaluations also calculated basic computational metrics , including text length, utilization rate, and time changes 31,32,36–38,42,47,51,53,54 , while only six studies employed more advanced computational metrics , such as BERTScore and Flesch-Kincaid grade level 36,38,40,43,47,48 . Among these six, one evaluation framework study intentionally mapped outcome measures to both human ratings and computational metrics to assess their alignment 40 . Building on the grouping approach used in prior work 38 , we categorized the common outcome measures examined in these studies into five groups: (1) information quality, (2) communication quality, (3) user perception, experience, and preference, (4) utilization and efficiency, and (5) composite measures. Information Quality (n = 14). Information quality was consistently evaluated across the studies, encompassing measures such as accuracy 38,43–47,49 , completeness 38,40,44,45,48,52 , relevance 38,40,48–50 , and factuality 40,50 . These measures assess the integrity of the information presented in AI drafts, with accuracy (n = 7) being the most frequently evaluated. While similar or identical terms were often used, they may reflect subtle differences. For instance, some studies evaluated completeness by checking whether drafts lacked essential information needed to answer patients’ questions 38,40,44,48,52 , while others examined whether responses were comprehensive beyond the minimum required content 45,46 . Relevance or responsiveness was typically rated based on whether AI responses addressed patients’ concerns 38,39,43,49,50 . In contrast, studies using computational methods defined relevance by how well AI drafts inferred patient inquiries 48 or matched clinician-authored replies 40 . One study treated information quality as a single dimension, without specifying the subcomponents it included 51,54 . Communication Quality (n = 14) .Communication quality was another important focus, capturing the patient-centered aspects of responses, with empathy (n = 9) being the most commonly measured aspect 37,43,45–49,51,54 . AI drafts were also evaluated for their tone or style, determining whether their wording and expressions were appropriate for the context of the conversation 37–39,50 . Readability 46,49 and related subdimensions, such as clarity 40,48 , understandability 38,44 , and brevity (or verbosity) 38,40,50 , were assessed to ensure that responses could be easily comprehended by patients without semantic confusions, literacy challenges, or distractions. Several studies used computational metrics to assess these aspects, including DiscoScore, lexical diversity, and the Flesch reading ease score 38,40,47 . Two studies analyzed the overall sentiment of the replies 38,48 , while another calculated BERT Toxicity to detect any pejorative terms or non-inclusive language 40 . User Perception, Experience, and Preference (n = 12) . A variety of measures were used to evaluate user perceptions, experiences, and preferences regarding the use of GenAI to draft replies to patient messages. A recurring focus was the perception of authorship, specifically whether participants could distinguish between AI-generated and human-written replies 39,41,45,49 . Some studies also asked participants to indicate their preference or rate their satisfaction with each response 36,45,46,51,56 . Additionally, patients’ trust and perceived level of care from AI-generated replies were assessed 41,56 . GenAI’s impact on clinician burnout was examined in one study by comparing physician task load and work exhaustion scores 31 . In parallel, a few other studies captured clinicians’ perceptions of reduced cognitive load, time savings, and improved efficiency to evaluate the potential benefits of GenAI assistance 37,39,53 . The net promoter score was also used as a measure of clinicians’ overall support for the GenAI tool 31,32,37,39 . Utilization and Efficiency (n = 11) . Measures related to utilization and efficiency often focused on basic characteristics or objective metrics of GenAI drafts and their implementation. Length (n = 7) was the most frequently evaluated characteristic 32,38,42,47,51,53,54 , as it may affect how efficiently clinicians review AI-generated drafts. Time-related measures included the time spent reading messages, as well as writing or editing replies. Some studies measured time changes before and after GenAI implementations 31,32 , while others compared the completion time of AI-generated, clinician-written, and clinician-edited replies 47,48,54 . Three implementation studies reported real-world utilization rates of AI drafts in clinical practices, typically measured by the how often clinicians selected “Start with Draft” instead of “Start Blank Reply.” 31,36,37 One of these studies also calculated the Damerau-Levenshtein distance between AI-generated drafts and final replies as a metric for estimating the editing effort required before clinical use 36 . Composite Measures (n = 10). In several studies, some measures were framed to assess multiple dimensions of quality through subjective judgment. Appropriateness, treated as a composite concept, involves evaluating whether the tone of the message was suitable and whether the information provided was adequate in a reply 52,55 . Potential harm associated with AI-generated responses was assessed by considering not only the risk of incorrect content compromising patient safety but also the possibility of communication that appeared unfriendly or perpetuated bias 37,44,53,54 . Usefulness or acceptability was often rated based on whether clinicians believed that AI-generated responses could be directly sent to patients, used as a starting point, or help improve the quality of a final response 31,37–39,43,44 . Consensus from Early Findings GenAI responses were generally found to match, and in some cases exceed, the quality of human-authored replies across several dimensions, though risks and limitations remain. GenAI drafts were frequently rated as comparable in information qualityand more favorable in communication style compared to clinician-authored responses 38,43,45,47–49,51,54 . Empathy consistently emerged as a notable strength of GenAI drafts 37,38,43,45,47–49,51,54 , with GPT-4 most often recognized as the top-performing model 43,45,47,52 . However, potential harms associated with these responses, although minimal, were documented 37,40,42,44,48,50,52–55 . Common challenges included hallucinations, incoherent language, and limited contextual understanding 31,40,48,52,55 . One study found that 7% of AI-generated replies posed a risk of severe harm or death 53 , while another reported two instances in which AI disclosed unsolicited confidential information in messaging involving proxies 50 . Additionally, studies noted inconsistent AI performance across message types, with reduced reliability in clinically complex inquiries and an increased risk of escalating emotionally charged conversations 40–42,44,52,55 . Laypersons—and even clinicians—often could not accurately distinguish between AI- and human-authored responses in blinded reviews. Relatedly, several studies asked participants to identify the authorship of responses and reported only low to modest accuracy, with averages ranging from 24% (among patients) to 73% (among clinicians), suggesting that AI-generated replies closely resemble human communication 39,41,45,49 . While several blinded evaluations showed that participants, particularly patients, tended to favor AI responses 45,51,56 , one study found that more empathetic and preferable responses were often attributed to human authorship, even when they were actually generated by AI 45 . Similarly, another study noted that disclosing AI involvement in reply drafting led to a slight decrease in patient satisfaction, although participants still valued transparency over nondisclosure 56 . Despite positive attitudes, adoption of GenAI for drafting patient message replies in real-world settings remained limited. While recognizing GenAI’s limitations and the need for edits, clinicians still viewed these tools as helpful aids for managing inbox burden. Several studies highlighted that clinicians found the drafts acceptable or useful as starting points and appreciated features such as templates or pleasantries 32,38–40,42–44,46 . Many clinicians also expressed willingness to recommend GenAI tools to colleagues and to retain the tools in future workflows 31,32,37,39 . However, this enthusiasm did not translate into consistent use. Across pilot implementations, actual utilization of AI-generated drafts was low, with average usage rates no higher than 20% 31,36,37 . Few studies have explored the disconnect between perceived benefits and limited uptake. While no strong evidence suggesting time savings, current GenAI implementations were associated with perceived efficiency gains and burnout relief. Despite expectations for streamlining workflows, early implementations reported no statistically significant changes in message reply time 31,32 , and one study even observed an increase in read time following GenAI integration 32 . Moreover, all studies comparing length found that AI-generated responses were considerably longer than human-written ones, raising concerns about increased burden for draft review and editing 32,38,42,47,51,53,54 . Nevertheless, survey data revealed that clinicians perceived a meaningful reduction in task and cognitive load, along with decreased work exhaustion 31,39 . Some also reported a subjective sense of time saved and improved efficiency 31,37,39,53 , even in the absence of objective evidence for reduced time or workload. Prompt engineering consistently emerged as an effective strategy for enhancing the quality and usability of GenAI drafts. Across studies, prompt optimization was associated with measurable improvements in response quality, tone, and user acceptance. One study reported a significant increase in clinician acceptance of AI-generated replies after three rounds of prompt refinement, alongside improvements in patient-rated tone and overall message quality 39 . Another found that a revised prompt led to a reduction in negative clinician feedback on drafts 36 . Incorporating the most recent assessment and plan into prompts was shown to improve perceived usefulness among clinicians 37 , while purposely designed prompts helped mitigate inconsistencies between AI- and clinician-authored responses, particularly in relational tone, content relevance, and clinical recommendations 42 . Table 1. Summary of studies included in this review. Study Setting & Objective Design Clinical context Message and/or Response Evaluated Responses by? Participant Outcome Measure Key Finding Expert Patient/ Laypeople Automated Athavale et al. 52 , 2023 Simulated, O1 Experimental evaluation study Vascular surgery Devised 20 administrative non-complex and 20 complex medical questions on CVD based on actual messages via patient portal GPT-4, GPT-3.5, Clinical Camel (a healthcare chatbot based on LLaMA) 1 internist and 1 vascular medicine specialist Appropriateness, Completeness N/A N/A ChatGPT-4 performed the best across both non-complex and complex question sets (100% and 75% appropriate and complete responses respectively). ChatGPT3.5 ranked second for both sets. Reported one hallucination case from ChatGPT-3.5 Nov et al. 41 , 2023 Simulated, O3 Cross-sectional survey study Not specified 10 representative, nonadministrative patient–provider interactions were extracted from EHRs GPT-3.5, Human 392 layerson respondents from a US representative sample N/A Perception of authorship, Trust N/A On average, respondents correctly classified both AI and human responses around 65% of the time, with trust in chatbots being weakly positive but decreasing as task complexity increased. Afshar et al. 36 , 2024 Live EHR, O2, O3, O4 Pre-post quasi-experimental study Primary care, dermatology, oncology, psychiatry AI drafts generated: 3882 (Pre); 3723 (Post); 2573 (Follow-up) GPT-4 Human + GPT-4 Pre & post: 27 physicians Follow up: 44 with 17 nurses added Thumbs up/down feedback N/A Utilization Edit distance Total usage: 17.5%. Usage increased in the follow-up: 35.8%. Post prompt engineering, no change for utilization but decreased in “thumb down”. Only 2.6% AI drafts were used without or with minimal provider edits. Table 1. Continued. Study Setting & Objective Design Clinical Context Message and/or Response Evaluated Responses by? Participant Outcome Measure Key Findings Expert Patient/ Laypeople Automated Baxter et al. 42 , 2024 Simulated, O1, O4, to evaluate whether LLMs can help address negative patient messages. Retrospective qualitative evaluation study Not specified A random sample of 50 negative sentiment messages and responses extracted from EHRs GPT-3.5, Human Two researcher coders. One is an ophthalmologist and another has a master of public health and a doctoral degree Themes in response N/A Length AI responses were about triple the length of clinicians’. Differences were noted in relational connection, content, and next-step recommendations. Prompting mitigated some issues but not all. AI drafts could be helpful starting points but could escalate emotional conversations. Chen et al. 53 , 2024 Simulated, O1, O2, O3 Two-stage observational study Oncology 100 hypothetical patient messages with simulated EHRs GPT-4, Human, GPT-4 + Human Stage 1 & 2 surveys: 6 attending radiation oncologists 2 additional physicians for content analysis Helpfulness, Risk/Harm, Subjective efficiency, Content categories in response N/A Length AI/AI-assisted responses were longer than human ones. About 7% of AI drafts posed a risk of severe harm or death. AI responses contained less direct action but provided more extensive education. Physicians reported improved efficiency, and responses became more consistent with AI assistance. English et al. 37 , 2024 Live EHR, O1, O2, O3, O4 Prospective quality improvement study Primary care and specialty (non-specified) AI drafts generated: 21323 GPT-4 12 nurses,14 MAs, 93 physicians and APPs Empathy, Tone, Perceived efficiency, NPS, Minimal risk N/A Utilization Overall, 12% utilization rate. Nurses were more likely to recommend the AI tool to others than MAs and clinicians, with more than 90% believing that it improved efficiency, empathy, and tone. Including the last A/P in prompts made some replies useful. Table 1. Continued. Study Setting & Objective Design Clinical Context Message and/or Response Evaluated Responses by? Participant Outcome Measure Key Findings Expert Patient/ Laypeople Automated Garcia et al. 31 , 2024 Live EHR, O1, O2, O3 Prospective quality improvement study Primary care, Gastroenterology and hepatology AI drafts generated: 9621 GPT-4 Total: 162 Primary care: 83 physicians and APPs, 4 nurses, 8 clinical pharmacists Gastroenterology and hepatology: 58 physicians and APPs, 10 nurses. Surveyed: 73 NASA TLX with a 4-item physician task load score derivative, PFI-WE score, NPS, Usability (utility, quality, perceived time saved), Free-text comment N/A Utilization, Change in reply action time, write time, or read time Mean AI- draft utilization rate was 20%. No changes in reply action time, write time, or read time between pre-pilot and pilot periods. Task load and work exhaustion scores significantly decreased. Comments identified facilitators such as readiness, utility, and time-saving. Barriers included tone, content relevance, and accuracy. Kim et al. 51 , 2024 Simulated, O1, O3, O4 Cross-sectional survey study Primary care, Endocrinology, Cardiology 59 messages selected from PMARs in EHRs GPT-4, Stanford Health Care and Stanford School of Medicine GPT, Human 6 clinicians 30 survey participants (layperson) recruited through the Stanford Research Registry Empathy, Information quality Satisfaction Length Promoted Stanford GPT was rated best for information quality and empathy. Satisfaction was higher with AI responses than with clinicians’ across specialties. Clinician responses were shorter. Satisfaction was not necessarily concordant with clinician-rated information quality and empathy. Clinician response length was associated with satisfaction while AI response length was not. Table 1. Continued. Study Setting & Objective Design Clinical Context Message and/or Response Evaluated Responses by? Participant Outcome Measure Key Findings Expert Patient/ Laypeople Automated Liu et al. 43 , 2024 Simulated, O1, aimed to fine-tune a LLM using patient portal interactions as well as evaluate its responses Model development and evaluation Primary care Fine-tune: CLARE-Short (499286 portal message -response pairs) CLAIR-Long (the pairs + 5000 open-source patient questions with GPT-4 responses) Evaluate set: 10 representative, de-identified, and rephrased patient messages and responses GPT-4, GPT-3.5, CLARE-Short, CLAIR-Long (based on LLaMA-65B), Human 4 primary care physicians Accuracy, Empathy, Responsiveness, Usefulness, Free-text comment N/A BERTScore GPT-4 responses were rated the best. ChatGPT models outperformed CLAIR-Long across accuracy, empathy, responsiveness, and usefulness. They all outperformed CLAIR-Short and the provider’ responses significantly. ChatGPT 3.5 achieved the highest BERTScore compared to actual provider responses. Reynolds et al. 46 , 2024 Simulated, O1, O3 Cross-sectional study Dermatology 31 patient messages with questions related to dermatological conditions or management and their responses extracted from EHRs GPT-3.5, Human 7 dermatology physicians, 3 nonphysicians Overall quality‚ Readability‚ Accuracy‚ Thoroughness, Empathy, Preference N/A N/A Both physicians and non-physicians preferred physician-generated responses over ChatGPT’s in most cases. Physician responses were rated significantly better in readability, empathy, accuracy, and overall quality. No hallucinations observed. Robinson et al. 45 , 2024 Simulated, O1, O3 Cross-sectional study Urology 20 common BPH-related patient questions from phone or EHR-messaging were pooled, anonymized, and compiled GPT-4, KPGPT (GPT-4-0613) SurgiChat (GPT-4-0613 with RAG on BPH literature), Human 2 urologists, 5 pre-screened non-medical volunteers that relevant to BPH Accuracy, Empathy, Comprehensiveness Perception of authorship, Preference, Empathy N/A Chatbot and urologist responses had similar accuracy, but chatbots rated significantly higher in completeness and empathy. Volunteers identified the correct author 59% and preferred chatbot responses. However, responses labeled as human scored higher in empathy than those labeled as chatbot. Table 1. Continued. Study Setting & Objective Design Clinical Context Message and/or Response Evaluated Responses by? Participant Outcome Measure Key Findings Expert Patient/ Laypeople Automated Scott et al. 44 , 2024 Simulated, O1 Cross-sectional study Urology 100 patient messages requesting medical advice were collected from the in-basket of a urologist specializing in andrology. GPT-3.5 5 urologists Accuracy, Helpfulness, Completeness, Harmfulness, Intelligibleness, Acceptability N/A N/A Overall, ChatGPT was rated to give accurate and intelligible answers, while completeness and helpfulness were rated lower. Harm was minimal. Performance was better on easier questions than harder ones. 47% of responses were considered acceptable. Small et al. 38 , 2024 Live EHR O1, O2, O3 Cross-sectional quality improvement study Primary care 117 unique HCP and 126 unique AI message-response pairs from pilot users’ in-basket AI drafts from silent validation, not being seen before) GPT-4, Human A convenience sample of 16 primary care physicians Usability, Information content quality (completeness, accuracy, relevance) Communication quality (understandable, appropriate, tone; verbosity) N/A Length, Complexity (lexical diversity, Flesch-Kincaid grade level), Sentiment Both AI and HCP responses were rated favorably. AI scored higher in communication style and matched HCPs in information content quality and usable draft proportion. Usable AI responses were seen as more empathetic, possibly due to their subjective and positive tone. They were also longer and more linguistically complex. Soroudi et al. 47 , 2024 Simulated, O1, O2 Cross-sectional survey study Plastic surgery 10 queries from patients undergoing breast reconstruction with highest level of complexity and decision-making implications were extracted from EHRs GPT-3, GPT-4, Human 2 APPs and 2 plastic surgeons for generated responses 2 medical students and 1 plastic surgeon, and 1 microsurgery fellow for review Accuracy (surgeon and fellow) Empathy (medical students) N/A Length, FRE score, Time to compilation (self-reported) Combined provider responses were more readable compared to combined chatbot responses. Empathy scores were higher in chatbot response. No significant differences in accuracy between providers and chatbot responses. Prompts increased readability. Table 1. Continued. Study Setting & Objective Design Clinical Context Message and/or Response Evaluated Responses by? Participant Outcome Measure Key Findings Expert Patient/ Laypeople Automated Tai-Seale et al. 32 , 2024a Live EHR, O1, O2, O3 Modified waiting list randomized quality improvement study Primary care 10 679 replies to patient messages were examined GPT-4 Immediate activation group: 25 physicians Delayed activation group: 27 Contemporary control group: 70 NPS N/A Length, Time spent reading and replying to messages Access to AI drafts was associated with a significant increase in read time, no change in reply time, and significantly longer replies. Physicians’ views of AI replies ranged from helpful as starting drafts and for adding empathy, to ineffective, overly focused on visits, and having an overly nice tone. Examples showed physicians kept pleasantries from AI drafts but made substantive edits. Tailor et al. 55 , 2024a Simulated, O1 Cross-sectional study Ophthalmology (across 9 subspecialties) For each ophthalmic subspecialty, about 20 clinical questions were generated on the basis of common patient questions received via the clinic or patient portal GPT-4 25 subspecialists participated. 22 of them both wrote and graded questions, and 3 only wrote questions. Appropriateness N/A N/A Reported robust aggregate appropriateness of an LLM across ophthalmic subspecialties both in the context of a patient information site (56%-100%) and as responses to EHR patient messages (54%-90%). Generally, inappropriate responses were inappropriate recommendations and incorrect or missing information. Tailor et al. 54 , 2024b Simulated, O1, Q2 Randomized cross-sectional study Ophthalmology Created 21 retina questions similar to common patient inquiries received in clinic or via patient portals GPT-3.5, GPT-4, Bard, Claude 2, Bing, Human, Human + GPT-4, 13 retinal specialists Information quality Empathy, Safety (inappropriate/incorrect/missing content, likelihood of possible harm) N/A Length, Time spent (self-reported) For quality, Expert+AI performed the best overall while GPT-3.5 was the top performing AI. For empathy, GPT-3.5 got the best score followed by Expert+AI. There were time savings for an Expert+AI response versus expert-created response. ChatGPT-4 performed similarly to Expert for safety metrics. Table 1. Continued. Study Setting & Objective Design Clinical Context Message and/or Response Evaluated Responses by? Participant Outcome Measure Key Findings Expert Patient/ Laypeople Automated Yan et al. 39 , 2024 Live EHR, O1, O2, O3, O4 Prospective quality improvement study Primary care Train: 116 random PMARs and responses from 5 pilot physicians’ in-basket Validation: 200 additional PMARs and responses Test: AI drafts to 761 PMARs in production Patient validation 1&2: 250 PMARs and responses including the 116 in train set Patient validation 3: 250 PMARs and AI responses from production including the 200 in validation GPT-4, Human Test: 5 primary care physicians, 5 patient advisors Production: 69 primary care clinicians (physicians and APPs) including the 5 in test, the same patient advisors Post production survey: 40 out of 69 Acceptance (Send, Edit or Reject), Helpfulness, Want to retain the tool? Recommend to colleagues? Perception of cognitive load reduced, Perception of time saved Perception of authorship, Tone, Overall quality, Responsiveness N/A After prompt iterations, physician acceptance (send/edit) of AI drafts rose significantly, with 74% rated as helpful. Patients also reported improved tone and overall quality, noting most responses addressed patient questions. Patients were unable to distinguish between humans and AI for 76% of messages. Majority clinicians would like to keep the tool and would recommend it, 72% believe it can reduce cognitive load, and 41% believe it has potential to reduce in-basket time. Cavalier et al. 56 , 2025 Simulated, O3 Survey-based randomized factorial experiment Not specified Created 3 hypothetical patient messages representing low, medium, or high clinical seriousness 3 physicians wrote responses AI drafts were reviewed and minimally edited by two physician authors GPT-3.5 + Human, Human 1455 respondents from patient advisory committee N/A Satisfaction, Perceived level of care, Usefulness, Preference for disclosure N/A Participants preferred AI responses over human responses regardless of the disclosure or seriousness of the topic. However, there was a slight decrease in satisfaction when told AI was involved. Participants preferred the shortest disclosure statement. Table 1. Continued. Study Setting & Objective Design Clinical Context Message and/or Response Evaluated Responses by? Participant Outcome Measure Key Findings Expert Patient/ Laypeople Automated Hao et al. 48 , 2025 Simulated, O1, O2 Retrospective observational study Urology, Radiation oncology 58 in-basket message interactions, selected from 90 patients with nonmetastatic prostate cancer RadOnc-GPT (GPT-4 with RAG on local EHRs and oncology-specific database), Human 1 oncologist, 4 residents, 4 nurses Completeness, Correctness, Clarity, Empathy, Estimated time to respond, Free-text comments N/A Inferences label, semantic similarity score, sentiment scores RadOnc-GPT responses were more positive and generalized, while clinician replies had more balanced sentiment and greater variety. High similarity scores indicated strong content alignment. RadOnc-GPT slightly outperformed the care team in empathy, whereas it had comparable completeness, correctness, and clarity. Key limitations in RadOnc-GPT’s responses were lack of context, insufficient domain-specific knowledge, inability to perform meta-tasks, and hallucination. RadOnc-GPT was estimated to save clinicians time. Hong et al. 40 , 2025 Live EHR, O1, O2, aimed to present a unified evaluation framework with mixed-methods metrics to assess AI for in-basket Framework development and evaluation Not specified 243 patient messages and AI draft pairs to establish and test human evaluation principles 42 patient messages from pilot users, with AI drafts, clinicians’ usage decisions, and final clinician-edited replies for compare qualitative and quantitative evaluations GPT-4, Human + GPT-4 1 clinician reviewer Relevance, Factuality, Completeness, Coherence, Clarity, Brevity, Toxicity N/A Perplexity, DiscoScore, FRE score, Compression ratio, Keyword Matching, ROUGE-N Recall|, BERT Toxicity AI replies exhibit high fluency, clarity, and minimal toxicity, they face challenges with coherence and completeness. Most AI drafts were rated as usable with minor edits. However, reliability and accuracy of AI drafts are inconsistent across message categories. Clinicians’ manual decision to use AI drafts correlates strongly with quantitative metrics. Table 1. Continued. Study Setting & Objective Design Clinical Context Message and/or Response Evaluated Responses by? Participant Outcome Measure Key Findings Expert Patient/ Laypeople Automated Kaur et al. 49 , 2025 Simulated, O1, O3 Cross-sectional survey study Primary care 20 unique patient message-response pairs from the medical center repository and de-identified 10 responses were generated by GPT, and the other 10 were written by real doctors GPT-3.5, Human 8 primary care physicians for the initial review 49 HCPs for the evaluation Accuracy, Readability, Empathy, Relevance, Perception of authorship N/A N/A Compared with real doctors, GPT responses scored significantly higher in empathy and readability. However, no statistically significant difference was observed for relevance and accuracy. Participants correctly identified GPT messages 73% of the time and correctly identified authentic messages 50% of the time. Tse et al. 50 , 2025 Simulated, O1 Cross-sectional study Primary care Patient portal messages were randomly obtained from patients aged 12 to 17 years and their associated proxy users with EHRs (problem list, medications, and lab results) GPT-4 Two pediatricians for review and a third pediatrician for resolving differences Usefulness, Proxy user identification, Protection of confidentiality, Quality (relevance, factual correctness, literacy, conciseness, tone/style) N/A N/A Among the proxy user messages, the AI correctly identified the proxy user 76% of the time. The AI disclosed unsolicited confidential information in fewer than 1% of cases. Most of the responses were in plain language, relevant, factually correct, concise, and 67% were rated as clinical useful. O1: evaluate the content of AI-generated drafts, O2: evaluate the implementation of generative AI and its impact on clinical efficiency, O3: understand user perceptions, preferences, or experiences, O4: examine the effects of prompt engineering, CVD: chronic venous disease, MAs: medical assistants, APPs: advanced practice providers, NPS: net promoter score, A/P: assessment and plan, NASA TLX score: NASA task load index score, PFI-WE: professional fulfillment index-work exhaustion score, PMAR: patient medical advice request, BPH: benign prostatic hyperplasia, RAG: retrieval-augmented generation, HCP: health care professional, FRE: Flesch reading ease. Discussion The synthesis of current research demonstrates a growing interest among health systems in GenAI-assisted replying to patient messages, with various efforts to evaluate draft quality, impacts on clinical efficiency, user perceptions, and the role of prompt engineering. Although varied in study design, scope, and evaluation methods, these early explorations reached some consensus. GenAI-drafted replies were generally perceived as acceptable starting points, especially when enhanced by tailored prompts. Both clinicians and patients recognized its potential to alleviate in-basket burden and enhance patient–provider communication. However, current real-world adoption of GenAI drafts remains limited, and important concerns regarding performance reliability and potential risks persist. In this section, we examine the key methodological and implementation limitations, explore ethical considerations, and outline future directions for advancing the effective and responsible integration of GenAI into high-volume clinical in-basket workflows. Early evaluations and implementations were often constrained in scope, scale, and generalizability. Most studies were conducted at a specific site, within a single health system, or involved relatively small sample sizes of message corpora and participants 32 , 37 – 42 , 45 , 48 – 50 . Several studies focused on clinicians from certain specialties or patient groups with limited demographic diversity, raising concerns about how well the findings translate across clinical settings or populations 31 , 41 , 44 – 47 , 52 – 56 . Evaluations were more commonly conducted in simulated environments rather than in live clinical workflows, with many studies assessing carefully selected single-turn messages or hypothetical inquiries 43 , 44 , 47 – 49 , 52 – 56 . As a result, these findings may not fully capture the complexity of real-world patient–provider messaging interactions, including diverse topics, contextual cues, or evolving patient conditions. Studies employed diverse, often unvalidated evaluation rubrics and relied heavily on human judgment from evaluators with varying levels of clinical training 31 , 36 , 37 , 39 , 47 – 49 . Draft quality assessments were typically conducted by convenient samples of experts 38 , 49 , 50 , and while most studies did not report inter-rater reliability, a few noted low levels of reviewer agreement 38 , 45 . Prompt engineering also differed widely across studies, with many relying on ad hoc or trial-and-error approaches, limiting the reproducibility of these optimizations 39 , 42 , 49 . Additionally, while poor rater agreement suggests diverse communication styles and preferences, current deployments fall short in supporting prompt personalization. This review also noted a lack of transparency around how GenAI tools integrated with EHR systems accessed and utilized medical records. Few studies reported details about incorporating problem lists, clinical notes, or messaging history in draft generation 31 , 48 , 50 , limiting the understanding of how GenAI contextually grounded their responses 49 . Without consistent access to comprehensive, up-to-date clinical information, the risk of generating inaccurate or context-agnostic replies increases. Many early explorations acknowledged that patients were underrepresented 36 , 38 , 39 , 49 , 54 , with none of the current studies involving patients who actually received AI-generated replies during pilot implementations—pointing to a significant gap in which patient-facing impacts and their preferences were often inferred rather than directly assessed 51 . Another limitation in understanding user experiences is that, while a few studies collected and analyzed qualitative user feedback, efforts to explore user perspectives in depth remain absent. Without these explorations, it is difficult to interpret current facilitators and barriers 45 , 56 or reconcile conflicting findings, such as disagreements among evaluators 38 and the misalignment between positive perceptions and low real-world utilization. Differences in model performance and user perceptions across demographic subgroups were also largely unexplored 38 , 39 , 41 , 42 , 45 . Several studies highlighted important ethical and legal considerations surrounding the responsible integration of GenAI into patient–provider messaging. Transparency—specifically whether and how to disclose AI contributions to patients—emerged as a key question to be addressed 37 , 41 , 45 , 56 . Despite findings that disclosure may reduce patient satisfaction 56 , upholding ethical norms in healthcare AI requires supporting patients’ right to be informed when AI is involved in the delivery of their medical information and care 37 , 56 – 58 . Concerns on biased model training, cultural insensitivity, lack of AI attribution, and unequal accessibility also raised important liability and equity implications 38 , 40 – 43 , 45 , 47 , 49 , 51 , 52 , 54 , 55 . However, current studies often flagged but rarely investigated these risks across patient subgroups. The findings underscore the need to align GenAI deployment with responsible principles from the outset across stages, addressing these issues proactively rather than reactively. Finally, in line with broader recognition in healthcare AI 59 , human oversight was consistently emphasized across studies as a key safeguard for ensuring the accountable and safe use of GenAI tools in clinical settings 40 , 42 , 45 , 47 – 50 , 53 , 54 . Given the limitations and implications surfaced in early research, this emerging area presents substantial opportunities for advancement. Future research should build on the efforts of Hong et al. 40 to continuously refine standardized evaluation frameworks that incorporate both human and computational assessments. A core set of validated and scalable measures would enable more reliable benchmarking, improve reproducibility, and inform best evaluation practices across settings and specialties. While the need for standardization is clear, it is also important to acknowledge the subjective nature of patient–provider communication. Future research should explore personalized GenAI prompting to accommodate individual variation and improve clinical relevance 31 , 32 , 38 . Moreover, as GenAI tools become more embedded in routine clinical workflows, the field would benefit from more longitudinal and multi-center trials to enhance the reliability and generalizability of outcomes 31 , 32 , 54 . Future work should include longer-term evaluations that track GenAI impact on reply quality, clinician burnout, patient satisfaction, and clinical outcomes over time to better understand its real-world implications in care delivery. In parallel, greater attention is needed to examine how GenAI systems access and incorporate clinical information from the EHR 31 , 42 , 43 . Studies should also explore how this process might be customized by clinical specialty or patient group to improve draft relevance. Patient perspectives should also be prioritized in future research 31 , 32 , 36 , 38 , 41 , 43 , 49 . While patients are the recipients of AI-generated replies, few studies have directly evaluated their experiences. The linguistic complexity of GenAI outputs may be manageable for clinicians but burdensome for patients with limited health or English literacy 38 . Future studies should assess patient comprehension, preferences, and satisfaction with GenAI-assisted replies and involve patients directly in the co-design of prompt engineering, disclosure strategies, and usability evaluations. Additionally, a few studies reported that GenAI did not generate drafts for all patient messages during pilot deployments 32 , 39 . How the absence of certain drafts may lead to inconsistency in communication style and affect patient experience or trust warrants further investigation. Addressing fairness and mitigating model bias will also be essential to ensure GenAI systems function equitably across diverse patient populations 41 , 60 . To promote ethical and responsible GenAI applications, these considerations should be prioritized and embedded early and across all stages of design, development, evaluation, and implementation. User training represents another critical area for future inquiry. Despite the recognized benefit of human–AI collaboration, clinicians currently work with GenAI drafts with limited guidance or support 31 . Targeted training programs are needed to help clinicians understand GenAI’s capabilities and limitations 55 , enabling them to make informed edits, prevent over-reliance, and ensure adequate oversight. At the same time, researchers, health system leaders, and policymakers should work to establish clear governance frameworks for AI-assisted messaging in response to ongoing calls for responsible AI in healthcare 61 , 62 . This includes addressing automation bias 41 , 43 , 53 , which can influence provider behavior, judgment, and decision-making, and ultimately affect patient outcomes. Additionally, the growing prevalence of billing for patient portal messaging may further complicate the integration of GenAI into patient communication 21 . This makes the need for guidelines and oversight even more pressing to safeguard patient satisfaction and trust. Institutions must develop clear policies on AI disclosure, informed consent, and data privacy to ensure transparency and uphold ethical standards in digitally mediated care. This review contributes to the growing body of knowledge on using GenAI to respond to patient questions and inquiries, particularly within the EHR context, where it integrates medical records to assist patient–provider communication as part of clinical care. The synthesis presented here can be compared with existing reviews that examine GenAI as a tool for patient and medical education 25 , 34 , as well as literature investigating its use as a medical chatbot 63 . These comparisons help reveal the distinct benefits and challenges of GenAI applications across various health information-seeking settings, and underscore the differences between viewing GenAI as a supportive assistant versus as an independent agent. Such distinctions may guide researchers and clinical stakeholders in further identifying appropriate directions and priorities. Additionally, as recent work has also begun to examine the use of LLMs to assist patients in writing efficient messages to their healthcare providers 64 , it is important to consider how AI’s presence on both sides of the communication may create interactive effects and reshape the patient–provider relationship. Limitations All studies included in this review were conducted in English and based in the United States, which limits the generalizability of our findings and does not reflect global efforts relevant to this topic. Generative AI applications in patient communication may differ substantially across countries due to variations in language, digital infrastructure, and regulatory environments. As such, this review may primarily inform contexts with similar health system characteristics. During the screening process, we also identified early studies that evaluated AI-generated replies using public patient inquiries from platforms such as social media or institutional websites, conducted prior to the availability of HIPAA-compliant tools 27 . While these studies provided useful early insights into the feasibility of GenAI-assisted messaging, we excluded them to maintain a focus on evaluations situated within health system or EHR-integrated settings. We were also unable to conduct a formal quality assessment of the included studies due to the exploratory nature and limited methodological detail in many reports. Despite these limitations, this review offers an important synthesis of empirical evaluations and highlights emerging opportunities and challenges in the integration of GenAI into patient–provider communication via portal messaging, addressing the growing interest in this rapidly evolving field. Conclusions This review systematically synthesizes early studies exploring the use of GenAI to draft responses to patient messages, highlighting the promising quality of AI-generated replies and the positive reception from both clinicians and patients. In addition to these encouraging findings, we also identified key limitations in the current evidence base, along with persistent risks and challenges to the effective and safe integration of GenAI into real-world clinical workflows. As these technologies continue to evolve, it is critical to establish shared evaluation standards, develop practical guidelines for disclosure and oversight, and engage diverse stakeholders in shaping responsible implementation. Our findings offer timely insights for health system researchers, leaders, and policymakers aiming to leverage GenAI as a novel tool to address clinician in-basket overload and enhance patient–provider communication via portal messaging. Methods This review followed the 2020 Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 65 guideline for systematic reviews without meta-analysis to ensure methodological transparency and reproducibility. Inclusion and Exclusion Criteria This review focuses on peer-reviewed original research articles that explore the use of GenAI to draft replies to patient messages within the EHR context. Eligible studies were required to present empirical findings from implementations, evaluations, or stakeholder perspectives. As this is an emerging area of research, we also included short reports such as brief communications, research letters, and perspective papers with empirical results. Studies were excluded if they were not published in English or focused solely on non-English outputs. We also excluded studies evaluating GenAI responses to general frequently asked patient questions on institutional websites or to inquiries posted on public platforms outside the EHR or patient portal setting. Patient-facing chatbots developed as standalone health assistants were also not considered. Lastly, we excluded posters and abstracts due to incomplete reporting, along with non-empirical articles such as editorials, opinion pieces, and system design descriptions. Search Strategy A comprehensive literature search was performed on April 5, 2025, using five electronic databases: PubMed, Web of Science, Scopus, IEEE Xplore, and the ACM Digital Library. The search targeted metadata fields (title, abstract, and keywords) and was tailored to each database’s syntax. Search terms included generative artificial intelligence , patient , message , and response as well as their synonyms and variants. To maximize retrieval sensitivity, we also included the terms question and inquiry , which may be used in place of message in this context, to capture studies evaluating GenAI responses to patient messages described with different terminology. A complete list of searching terms are provided in Table 2 . Table 2 Literature searching strategy. Generative AI Terms “generative artificial intelligence” OR “generative AI” OR “AI-generated” OR “AI generated” OR “AI-drafted” OR “AI drafted” OR “artificial intelligence-generated” OR “artificial intelligence generated” OR “artificial intelligence-drafted” OR “artificial intelligence drafted” OR “large language model*” OR llm OR llms OR “transformer model*” OR “pre-trained language model*” OR “generative pre-trained transformer*” OR chatgpt OR gpt AND Action Terms respond* OR response* OR reply* OR replies OR answer* AND Topic Terms patient* AND (messag* OR inquir* OR question*) Table 2 lists the search terms and query logic used in the literature search of this review. Study Screening and Data Extraction Titles and abstracts were independently screened by two of four authors (DH, YG, YZ, LF). At this stage, without reviewing full-text content, we focused on identifying studies that explored GenAI for responding to messages, inquiries, or questions from patients. Disagreements were resolved through consensus discussions involving at least two reviewers. Articles meeting these criteria were retrieved for full-text screening. Each full text was independently reviewed by two of three authors (DH, YG, and YZ), based on the predefined inclusion and exclusion criteria. Any disagreements were resolved through discussion with the senior author (KZ). Following the screening process, a data extraction template was developed to capture study characteristics, context, objectives, design, participants, outcomes, findings, limitations, and implications. Each included study was independently coded by two reviewers (DH and YG). Discrepancies were discussed and resolved with input from the senior author (KZ), ensuring consistency and interpretive rigor across the dataset. Declarations Data availability All data generated or analysed during this study are included in this article. Acknowledgments No specific funding support or contributions to acknowledge. Author Contributions DH conceptualized the study, developed the search strategy, conducted the literature searches, led data screening and extraction, and drafted the manuscript. YG and YZ contributed to refining the inclusion criteria and participated in data screening and extraction. LF participated in the data screening and contributed to consensus discussions. KZ supervised the study, provided methodological guidance, and helped resolve discrepancies during screening and extraction. All authors reviewed and approved the final manuscript. Competing Interests All authors declare no financial or non-financial competing interests. References Lyles, C. R. et al. Using Electronic Health Record Portals to Improve Patient Engagement: Research Priorities and Best Practices. Ann. Intern. Med. 172 , S123–S129 (2020). Lieu, T. A. et al. Primary Care Physicians’ Experiences With and Strategies for Managing Electronic Messages. JAMA Netw. Open 2 , e1918287 (2019). Haun, J. N. et al. Evaluating User Experiences of the Secure Messaging Tool on the Veterans Affairs’ Patient Portal System. J. Med. Internet Res. 16 , e2976 (2014). Sun, S., Zhou, X., Denny, J. C., Rosenbloom, T. S. & Xu, H. Messaging to your doctors: understanding patient-provider communications via a portal system. in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 1739–1748 (ACM, Paris France, 2013). doi:10.1145/2470654.2466230. Rights (OCR), O. for C. HITECH Act Enforcement Interim Final Rule. https://www.hhs.gov/hipaa/for-professionals/special-topics/hitech-act-enforcement-interim-final-rule/index.html (2009). Griffin, A., Skinner, A., Thornhill, J. & Weinberger, M. Patient Portals. Appl. Clin. Inform. 7 , 489–501 (2016). North, F. et al. A Retrospective Analysis of Provider-to-Patient Secure Messages: How Much Are They Increasing, Who Is Doing the Work, and Is the Work Happening After Hours? JMIR Med. Inform. 8 , e16521 (2020). Holmgren, A. J., Apathy, N. C., Crews, J. & Shanafelt, T. National trends in oncology specialists’ EHR inbox work, 2019-2022. JNCI J. Natl. Cancer Inst. djaf052 (2025) doi:10.1093/jnci/djaf052. Holmgren, A. J. et al. Assessing the impact of the COVID-19 pandemic on clinician ambulatory electronic health record use. J. Am. Med. Inform. Assoc. 29 , 453–460 (2022). Adler-Milstein, J., Zhao, W., Willard-Grace, R., Knox, M. & Grumbach, K. Electronic health records and burnout: Time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians. J. Am. Med. Inform. Assoc. JAMIA 27 , 531–538 (2020). Akbar, F. et al. Physicians’ electronic inbox work patterns and factors associated with high inbox work duration. J. Am. Med. Inform. Assoc. JAMIA 28 , 923–930 (2021). Arndt, B. G. et al. Tethered to the EHR: Primary Care Physician Workload Assessment Using EHR Event Log Data and Time-Motion Observations. Ann. Fam. Med. 15 , 419–426 (2017). Akbar, F. et al. Physician Stress During Electronic Health Record Inbox Work: In Situ Measurement With Wearable Sensors. JMIR Med. Inform. 9 , e24014 (2021). Tai-Seale, M. et al. Physicians’ Well-Being Linked To In-Basket Messages Generated By Algorithms In Electronic Health Records. Health Aff. Proj. Hope 38 , 1073–1078 (2019). Hilliard, R. W., Haskell, J. & Gardner, R. L. Are specific elements of electronic health record use associated with clinician burnout more than others? J. Am. Med. Inform. Assoc. JAMIA 27 , 1401 (2020). Ariely, D. & Lanier, W. L. Disturbing Trends in Physician Burnout and Satisfaction With Work-Life Balance: Dealing With Malady Among the Nation’s Healers. Mayo Clin. Proc. 90 , 1593–1596 (2015). Smith, L., Kirk, W., Bennett, M. M., Youens, K. & Ramm, J. From Headache to Handled: Advanced In-Basket Management System in Primary Care Clinics Reduces Provider Workload Burden and Self-Reported Burnout. Appl. Clin. Inform. 15 , 869–876 (2024). Fogg, J. F. & Sinsky, C. A. In-Basket Reduction: A Multiyear Pragmatic Approach to Lessen the Work Burden of Primary Care Physicians. NEJM Catal. 4 , CAT.22.0438 (2023). Cronin, R. M., Fabbri, D., Denny, J. C., Rosenbloom, S. T. & Jackson, G. P. A comparison of rule-based and machine learning approaches for classifying patient portal messages. Int. J. Med. Inf. 105 , 110–120 (2017). Yang, J. et al. Development and evaluation of an artificial intelligence-based workflow for the prioritization of patient portal messages. JAMIA Open 7 , ooae078 (2024). Liu, T., Zhu, Z., Holmgren, A. J. & Ellimoottil, C. National trends in billing patient portal messages as e-visit services in traditional Medicare. Health Aff. Sch. 2 , qxae040 (2024). Holmgren, A. J., Grouse, C. K., Oates, A., O’Brien, J. & Byron, M. E. Changes in Secure Messaging After Implementation of Billing E-Visits by Demographic Group. JAMA Netw. Open 7 , e2427053 (2024). Matulis, J. & McCoy, R. Relief in Sight? Chatbots, In-baskets, and the Overwhelmed Primary Care Clinician. J. Gen. Intern. Med. 38 , 2808–2815 (2023). Waldock, W. J. et al. The Accuracy and Capability of Artificial Intelligence Solutions in Health Care Examinations and Certificates: Systematic Review and Meta-Analysis. J. Med. Internet Res. 26 , e56532 (2024). Zong, H. et al. Large Language Models in Worldwide Medical Exams: Platform Development and Comprehensive Analysis. J. Med. Internet Res. 26 , e66114 (2024). Luo, M., Warren, C. J., Cheng, L., Abdul-Muhsin, H. M. & Banerjee, I. Assessing Empathy in Large Language Models with Real-World Physician-Patient Interactions. in 2024 IEEE International Conference on Big Data (BigData) 6510–6519 (2024). doi:10.1109/BigData62323.2024.10825307. Ayers, J. W. et al. Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Intern. Med. 183 , 589–596 (2023). Armitage, H. AI assists clinicians in responding to patient messages at Stanford Medicine. News Center https://med.stanford.edu/news/all-news/2024/03/ai-patient-messages.html (2024). Introducing Dr. Chatbot. https://today.ucsd.edu/story/introducing-dr-chatbot. Gen AI Saves Nurses Time by Drafting Responses to Patient Messages. EpicShare https://www.epicshare.org/share-and-learn/mayo-ai-message-responses. Garcia, P. et al. Artificial Intelligence–Generated Draft Replies to Patient Inbox Messages. JAMA Netw. Open 7 , e243201 (2024). Tai-Seale, M. et al. AI-Generated Draft Replies Integrated Into Health Records and Physicians’ Electronic Communication. JAMA Netw. Open 7 , e246565 (2024). Singh, G. B., Kumar, R., Ghosh, R. C., Bhakhuni, P. & Sharma, N. Question Answering in Medical Domain Using Natural Language Processing: A Review. in Data Management, Analytics and Innovation (eds. Sharma, N., Goje, A. C., Chakrabarti, A. & Bruckstein, A. M.) 385–397 (Springer Nature, Singapore, 2024). doi:10.1007/978-981-97-3245-6_26. Aydin, S., Karabacak, M., Vlachos, V. & Margetis, K. Large language models in patient education: a scoping review of applications in medicine. Front. Med. 11 , (2024). Wei, Q. et al. Evaluation of ChatGPT-generated medical responses: A systematic review and meta-analysis. J. Biomed. Inform. 151 , 104620 (2024). Afshar, M. et al. Prompt engineering with a large language model to assist providers in responding to patient inquiries: a real-time implementation in the electronic health record. JAMIA Open 7 , ooae080 (2024). English, E., Laughlin, J., Sippel, J., DeCamp, M. & Lin, C.-T. Utility of Artificial Intelligence–Generative Draft Replies to Patient Messages. JAMA Netw. Open 7 , e2438573 (2024). Small, W. R. et al. Large Language Model–Based Responses to Patients’ In-Basket Messages. JAMA Netw. Open 7 , e2422399 (2024). Yan, S. et al. Prompt engineering on leveraging large language models in generating response to InBasket messages. J. Am. Med. Inform. Assoc. 31 , 2263–2270 (2024). Hong, C. et al. Application of unified health large language model evaluation framework to In-Basket message replies: bridging qualitative and quantitative assessments. J. Am. Med. Inform. Assoc. ocaf023 (2025) doi:10.1093/jamia/ocaf023. Nov, O., Singh, N. & Mann, D. Putting ChatGPT’s Medical Advice to the (Turing) Test: Survey Study. JMIR Med. Educ. 9 , e46939 (2023). Baxter, S. L., Longhurst, C. A., Millen, M., Sitapati, A. M. & Tai-Seale, M. Generative artificial intelligence responses to patient messages in the electronic health record: early lessons learned. JAMIA Open 7 , ooae028 (2024). Liu, S. et al. Leveraging large language models for generating responses to patient messages—a subjective analysis. J. Am. Med. Inform. Assoc. 31 , 1367–1379 (2024). Scott, M. et al. Assessing Artificial Intelligence–Generated Responses to Urology Patient In-Basket Messages. Urol. Pract. 11 , 793–798 (2024). Robinson, E. J. et al. Physician vs. AI-generated messages in urology: evaluation of accuracy, completeness, and preference by patients and physicians. World J. Urol. 43 , 48 (2024). Reynolds, K. et al. Comparing the quality of ChatGPT- and physician-generated responses to patients’ dermatology questions in the electronic medical record. Clin. Exp. Dermatol. 49 , 715–718 (2024). Soroudi, D. et al. Comparing Provider and ChatGPT Responses to Breast Reconstruction Patient Questions in the Electronic Health Record. Ann. Plast. Surg. 93 , 541 (2024). Hao, Y. et al. Retrospective Comparative Analysis of Prostate Cancer In-Basket Messages: Responses From Closed-Domain Large Language Models Versus Clinical Teams. Mayo Clin. Proc. Digit. Health 3 , (2025). Kaur, A., Budko, A., Liu, K., Steitz, B. & Johnson, K. Primary Care Providers Acceptance of Generative AI Responses to Patient Portal Messages. Appl. Clin. Inform. 0 , (2025). Tse, G. et al. Large Language Model Responses to Adolescent Patient and Proxy Messages. JAMA Pediatr. 179 , 93–94 (2025). Kim, J. et al. Perspectives on Artificial Intelligence–Generated Responses to Patient Messages. JAMA Netw. Open 7 , e2438535 (2024). Athavale, A., Baier, J., Ross, E. & Fukaya, E. The potential of chatbots in chronic venous disease patient management. JVS-Vasc. Insights 1 , 100019 (2023). Chen, S. et al. The effect of using a large language model to respond to patient messages. Lancet Digit. Health 6 , e379–e381 (2024). Tailor, P. D. et al. A Comparative Study of Responses to Retina Questions from Either Experts, Expert-Edited Large Language Models, or Expert-Edited Large Language Models Alone. Ophthalmol. Sci. 4 , 100485 (2024). Tailor, P. D. et al. Appropriateness of Ophthalmology Recommendations From an Online Chat-Based Artificial Intelligence Model. Mayo Clin. Proc. Digit. Health 2 , 119–128 (2024). Cavalier, J. S. et al. Ethics in Patient Preferences for Artificial Intelligence–Drafted Responses to Electronic Messages. JAMA Netw. Open 8 , e250449 (2025). Jeyaraman, M., Balaji, S., Jeyaraman, N. & Yadav, S. Unraveling the Ethical Enigma: Artificial Intelligence in Healthcare. Cureus 15 , e43262. Park, H. J. Patient perspectives on informed consent for medical AI: A web-based experiment. Digit. Health 10 , 20552076241247938 (2024). De Cremer, D. & Kasparov, G. The ethical AI—paradox: why better technology needs more and not less human responsibility. AI Ethics 2 , 1–4 (2022). Wang, C. et al. Ethical Considerations of Using ChatGPT in Health Care. J. Med. Internet Res. 25 , e48009 (2023). Upadhyay, U. et al. Call for the responsible artificial intelligence in the healthcare. BMJ Health Care Inform. 30 , e100920 (2023). Trocin, C., Mikalef, P., Papamitsiou, Z. & Conboy, K. Responsible AI for Digital Health: a Synthesis and a Research Agenda. Inf. Syst. Front. 25 , 2139–2157 (2023). Giuffrè, M. et al. Systematic review: The use of large language models as medical chatbots in digestive diseases. Aliment. Pharmacol. Ther. 60 , 144–166 (2024). Liu, S. et al. Using large language model to guide patients to create efficient and comprehensive clinical care message. J. Am. Med. Inform. Assoc. 31 , 1665–1670 (2024). Page, M. J. et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. (2021) doi:10.1136/bmj.n71. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 09 Jun, 2025 Reviews received at journal 08 Jun, 2025 Reviews received at journal 27 May, 2025 Reviewers agreed at journal 27 May, 2025 Reviewers agreed at journal 26 May, 2025 Reviewers invited by journal 26 May, 2025 Editor assigned by journal 23 May, 2025 Submission checks completed at journal 23 May, 2025 First submitted to journal 21 May, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {\"props\":{\"pageProps\":{\"initialData\":{\"identity\":\"rs-6713507\",\"acceptedTermsAndConditions\":true,\"allowDirectSubmit\":false,\"archivedVersions\":[],\"articleType\":\"Article\",\"associatedPublications\":[],\"authors\":[{\"id\":462681206,\"identity\":\"6ab718af-db69-4df1-b5e2-0b86f5e16bdd\",\"order_by\":0,\"name\":\"Di Hu\",\"email\":\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAn0lEQVRIiWNgGAWjYDCCwzwMBxIqGBjYgGwJYrUwHnhwhiQtB3iYDz5sg7CJ08J3nPfAgcR5h+X5GJgP3uYhRovkYb6EA4nbDhu2MbAlWxOlxeAwjwFISwIbA4+ZNAla5oC08H8jRUsD2BY24rRIgrQkHEs3bGNmM7acQ4wWvvNnjD/+qLGWl29vfnjjDTFaEICZNOWjYBSMglEwCvABAJQiL9qmssONAAAAAElFTkSuQmCC\",\"orcid\":\"\",\"institution\":\"University of California, Irvine\",\"correspondingAuthor\":true,\"prefix\":\"\",\"firstName\":\"Di\",\"middleName\":\"\",\"lastName\":\"Hu\",\"suffix\":\"\"},{\"id\":462681207,\"identity\":\"5cc70d12-99e6-495f-a5e4-38fa6e607c63\",\"order_by\":1,\"name\":\"Yawen Guo\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"University of California, Irvine\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Yawen\",\"middleName\":\"\",\"lastName\":\"Guo\",\"suffix\":\"\"},{\"id\":462681208,\"identity\":\"19251193-de46-4780-b7d6-8b41daf7b6fb\",\"order_by\":2,\"name\":\"Yiliang Zhou\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"University of California, Irvine\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Yiliang\",\"middleName\":\"\",\"lastName\":\"Zhou\",\"suffix\":\"\"},{\"id\":462681209,\"identity\":\"9db59b6e-631d-4ddc-92cc-266f8713424a\",\"order_by\":3,\"name\":\"Lidia Flores\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"University of California, Irvine\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Lidia\",\"middleName\":\"\",\"lastName\":\"Flores\",\"suffix\":\"\"},{\"id\":462681210,\"identity\":\"6d7cce16-0715-47f9-8c91-9a5f50970bfe\",\"order_by\":4,\"name\":\"Kai Zheng\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"University of California, Irvine\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Kai\",\"middleName\":\"\",\"lastName\":\"Zheng\",\"suffix\":\"\"}],\"badges\":[],\"createdAt\":\"2025-05-21 07:08:25\",\"currentVersionCode\":1,\"declarations\":\"\",\"doi\":\"10.21203/rs.3.rs-6713507/v1\",\"doiUrl\":\"https://doi.org/10.21203/rs.3.rs-6713507/v1\",\"draftVersion\":[],\"editorialEvents\":[],\"editorialNote\":\"\",\"failedWorkflow\":false,\"files\":[{\"id\":83597303,\"identity\":\"a3a6fc29-bc89-4e44-b0ed-f982c4d75ef4\",\"added_by\":\"auto\",\"created_at\":\"2025-05-29 08:10:55\",\"extension\":\"jpg\",\"order_by\":1,\"title\":\"Figure 1\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":70016,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003ePRISMA flow diagram illustrating the study selection process.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"1.jpg\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-6713507/v1/d4721395af6d969befb97289.jpg\"},{\"id\":83597579,\"identity\":\"d115ea5a-3662-47e4-a33b-98d2e716c206\",\"added_by\":\"auto\",\"created_at\":\"2025-05-29 08:18:56\",\"extension\":\"pdf\",\"order_by\":0,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"manuscript-pdf\",\"size\":2482063,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"manuscript.pdf\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-6713507/v1/f2abe554-ea2c-4500-9424-d4da969f55de.pdf\"}],\"financialInterests\":\"No competing interests reported.\",\"formattedTitle\":\"A Systematic Review of Early Evidence on Generative AI for Drafting Responses to Patient Messages\",\"fulltext\":[{\"header\":\"Introduction\",\"content\":\"\\u003cp\\u003ePatient portals, as an integral part of electronic health records (EHRs), are now available in nearly 90% of health systems in the United States, enhancing patient engagement and transforming patient\\u0026ndash;provider communication\\u003csup\\u003e\\u003cspan citationid=\\\"CR1\\\" class=\\\"CitationRef\\\"\\u003e1\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR2\\\" class=\\\"CitationRef\\\"\\u003e2\\u003c/span\\u003e\\u003c/sup\\u003e. Through portal messaging, patients can contact their care teams outside of scheduled visits to ask questions, request medication refills, and follow up on lab-test results\\u003csup\\u003e\\u003cspan citationid=\\\"CR3\\\" class=\\\"CitationRef\\\"\\u003e3\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR4\\\" class=\\\"CitationRef\\\"\\u003e4\\u003c/span\\u003e\\u003c/sup\\u003e. Incentivized by the Health Information Technology for Economic and Clinical Health Act\\u003csup\\u003e\\u003cspan citationid=\\\"CR5\\\" class=\\\"CitationRef\\\"\\u003e5\\u003c/span\\u003e\\u003c/sup\\u003e, messaging has become one of the most frequently used patient portal features\\u003csup\\u003e\\u003cspan citationid=\\\"CR6\\\" class=\\\"CitationRef\\\"\\u003e6\\u003c/span\\u003e\\u003c/sup\\u003e. Over the past decade, message volume has grown substantially\\u003csup\\u003e\\u003cspan citationid=\\\"CR7\\\" class=\\\"CitationRef\\\"\\u003e7\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR8\\\" class=\\\"CitationRef\\\"\\u003e8\\u003c/span\\u003e\\u003c/sup\\u003e, with the COVID-19 pandemic further accelerating this surge, driving a 157% increase compared to pre-pandemic levels. This elevated rate of use has persisted since\\u003csup\\u003e\\u003cspan citationid=\\\"CR9\\\" class=\\\"CitationRef\\\"\\u003e9\\u003c/span\\u003e\\u003c/sup\\u003e. While improved communication is associated with improved patient care, it has also increased the burden on clinicians. The influx of messages has overwhelmed clinicians\\u0026rsquo; \\u0026ldquo;in-baskets\\u0026rdquo;\\u0026mdash;the EHR-based inboxes\\u0026mdash;resulting in a workload that often extends beyond regular work hours\\u003csup\\u003e\\u003cspan citationid=\\\"CR8\\\" class=\\\"CitationRef\\\"\\u003e8\\u003c/span\\u003e,\\u003cspan additionalcitationids=\\\"CR11\\\" citationid=\\\"CR10\\\" class=\\\"CitationRef\\\"\\u003e10\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR12\\\" class=\\\"CitationRef\\\"\\u003e12\\u003c/span\\u003e\\u003c/sup\\u003e. This sustained burden has been linked to clinician burnout, job dissatisfaction, and challenges in maintaining work\\u0026ndash;life balance\\u003csup\\u003e\\u003cspan citationid=\\\"CR10\\\" class=\\\"CitationRef\\\"\\u003e10\\u003c/span\\u003e,\\u003cspan additionalcitationids=\\\"CR14 CR15\\\" citationid=\\\"CR13\\\" class=\\\"CitationRef\\\"\\u003e13\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR16\\\" class=\\\"CitationRef\\\"\\u003e16\\u003c/span\\u003e\\u003c/sup\\u003e.\\u003c/p\\u003e \\u003cp\\u003eHuman, technological, and policy-level strategies have been adopted to address this growing burden. These efforts have included forming designated administrative teams\\u003csup\\u003e\\u003cspan citationid=\\\"CR17\\\" class=\\\"CitationRef\\\"\\u003e17\\u003c/span\\u003e\\u003c/sup\\u003e, refining management workflows\\u003csup\\u003e\\u003cspan citationid=\\\"CR2\\\" class=\\\"CitationRef\\\"\\u003e2\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR18\\\" class=\\\"CitationRef\\\"\\u003e18\\u003c/span\\u003e\\u003c/sup\\u003e, developing artificial intelligence (AI) applications for message triage\\u003csup\\u003e\\u003cspan citationid=\\\"CR19\\\" class=\\\"CitationRef\\\"\\u003e19\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR20\\\" class=\\\"CitationRef\\\"\\u003e20\\u003c/span\\u003e\\u003c/sup\\u003e, and introducing new billing codes for e-visits\\u003csup\\u003e\\u003cspan citationid=\\\"CR21\\\" class=\\\"CitationRef\\\"\\u003e21\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR22\\\" class=\\\"CitationRef\\\"\\u003e22\\u003c/span\\u003e\\u003c/sup\\u003e. Most recently, generative AI (GenAI), particularly generative large language models (LLMs), has emerged as a potential solution for alleviating clinician in-basket overload\\u003csup\\u003e\\u003cspan citationid=\\\"CR23\\\" class=\\\"CitationRef\\\"\\u003e23\\u003c/span\\u003e\\u003c/sup\\u003e. Capable of interpreting complex texts and generating human-like responses, these models have demonstrated the ability to answer medical questions with expert-level knowledge\\u003csup\\u003e\\u003cspan citationid=\\\"CR24\\\" class=\\\"CitationRef\\\"\\u003e24\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR25\\\" class=\\\"CitationRef\\\"\\u003e25\\u003c/span\\u003e\\u003c/sup\\u003e and to respond to patient forum posts in a more empathetic tone than physicians\\u003csup\\u003e\\u003cspan citationid=\\\"CR26\\\" class=\\\"CitationRef\\\"\\u003e26\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR27\\\" class=\\\"CitationRef\\\"\\u003e27\\u003c/span\\u003e\\u003c/sup\\u003e. With such capability, GenAI offers a novel approach to assist clinicians by creating draft replies to patient messages. Several large health systems in the United States have begun implementing Health Insurance Portability and Accountability Act (HIPAA)-compliant GenAI tools for this purpose within their EHR systems\\u003csup\\u003e\\u003cspan additionalcitationids=\\\"CR29\\\" citationid=\\\"CR28\\\" class=\\\"CitationRef\\\"\\u003e28\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR30\\\" class=\\\"CitationRef\\\"\\u003e30\\u003c/span\\u003e\\u003c/sup\\u003e. These early adopters have highlighted GenAI\\u0026rsquo;s potential to generate useful first drafts, reduce clinician exhaustion, and ultimately enable and improve the efficiency of asynchronous care\\u003csup\\u003e\\u003cspan citationid=\\\"CR31\\\" class=\\\"CitationRef\\\"\\u003e31\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR32\\\" class=\\\"CitationRef\\\"\\u003e32\\u003c/span\\u003e\\u003c/sup\\u003e.\\u003c/p\\u003e \\u003cp\\u003eDespite encouraging progress, use of GenAI in facilitating clinical communication is still at an early stage, and the existing literature is highly fragmented. Prior reviews on GenAI in medical question-answering have mainly focused on its performance and efficacy for medical exams, clinical decision support, and patient education, rather than its role in supporting clinicians with in-basket management\\u003csup\\u003e\\u003cspan additionalcitationids=\\\"CR34\\\" citationid=\\\"CR33\\\" class=\\\"CitationRef\\\"\\u003e33\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR35\\\" class=\\\"CitationRef\\\"\\u003e35\\u003c/span\\u003e\\u003c/sup\\u003e. Further, there has been no consensus to date on how to evaluate AI-drafted responses or what lessons can be learned from initial efforts. Current studies vary widely in design, outcome measures, and evaluation methods. Clinical contexts for these studies also differ, spanning various specialties and diverse patient populations, each of which may present nuanced differences in patient\\u0026ndash;provider communication practices. The lack of standardization poses challenges for synthesizing findings. It also emphasizes the need to understand the body of early evidence in order to inform the development and implementation of GenAI tools as they continue to evolve and become better integrated into care delivery.\\u003c/p\\u003e \\u003cp\\u003eThis review addresses these gaps by systematically identifying and synthesizing empirical studies that evaluate GenAI for drafting replies to patient messages through EHR-embedded patient portals. By examining study settings, objectives, designs, outcomes, and key findings, we aimed to provide a timely overview of the current evidence and outline directions for future research. Specifically, this review seeks to answer the following questions:\\u003c/p\\u003e \\u003cp\\u003e \\u003col\\u003e \\u003cspan\\u003e \\u003cli\\u003e \\u003cp\\u003eHow have GenAI tools been studied for drafting responses to patient portal messages and what are the study settings, objectives, and designs?\\u003c/p\\u003e \\u003c/li\\u003e \\u003c/span\\u003e \\u003cspan\\u003e \\u003cli\\u003e \\u003cp\\u003eIn what clinical contexts have these tools been evaluated?\\u003c/p\\u003e \\u003c/li\\u003e \\u003c/span\\u003e \\u003cspan\\u003e \\u003cli\\u003e \\u003cp\\u003eWho are the users participating in these evaluations, and what approaches and outcome measures have been used?\\u003c/p\\u003e \\u003c/li\\u003e \\u003c/span\\u003e \\u003cspan\\u003e \\u003cli\\u003e \\u003cp\\u003eWhat early consensus has emerged from the findings?\\u003c/p\\u003e \\u003c/li\\u003e \\u003c/span\\u003e \\u003c/ol\\u003e \\u003c/p\\u003e \\u003cp\\u003eIn answering these questions, we also discuss the challenges and opportunities surfaced by early studies and highlight ethical considerations that should be prioritized to ensure the responsible use of GenAI in replying to patient messages. Our synthesis offers guidance for future evaluations and implementations of GenAI designed to aid in patient\\u0026ndash;provider communication particularly through patient portals.\\u003c/p\\u003e\"},{\"header\":\"Results\",\"content\":\"\\u003cp\\u003eThis section presents our synthesis of current research on the use of GenAI for drafting responses to patient messages. Table 1 provides an overview of key information extracted from the included studies.\\u003c/p\\u003e\\n\\u003ch2\\u003e\\u003cstrong\\u003eResults of Literature Search and Screening\\u0026nbsp;\\u003c/strong\\u003e\\u003c/h2\\u003e\\n\\u003cp\\u003eFigure 1 illustrates the literature search and screening process following the PRISMA flow diagram format. Our search across five databases (ACM Digital Library, IEEE Xplore, PubMed, Scopus, and Web of Science) resulted in 3,980 potentially relevant papers. After removing 2,003 duplicates, 1,977 articles remained. Screening based on title and abstract further excluded 1,284 additional papers, leaving 693 for full-text review. After reviewing the full-text of these 693 papers, 23 were deemed to meet the inclusion and exclusion criteria (detailed in the Methods section), and were included in the final review.\\u0026nbsp;\\u003c/p\\u003e\\n\\u003ch2\\u003e\\u003cstrong\\u003eStudy Characteristics\\u003c/strong\\u003e\\u003c/h2\\u003e\\n\\u003cp\\u003eAll 23 studies were conducted in the United States and published between 2023 and 2025. The majority of them (n = 16) appeared in medical and informatics journals, including \\u003cem\\u003eJAMA Network Open\\u003c/em\\u003e, \\u003cem\\u003eJournal of the American Medical Informatics Association\\u003c/em\\u003e, \\u003cem\\u003eJAMIA Open\\u003c/em\\u003e, and \\u003cem\\u003eMayo Clinic Proceedings:Digital Health\\u003c/em\\u003e. The remainder (n = 7) were published in clinical specialty-focused venues, such as \\u003cem\\u003eUrology Practice\\u003c/em\\u003e, \\u003cem\\u003eOphthalmology Science\\u003c/em\\u003e, and \\u003cem\\u003eAnnals of Plastic Surgery\\u003c/em\\u003e. In terms of type of publication: 17 were full-length research articles, while the others consisted of three research letters, one brief communication, one perspective, and one commentary.\\u003c/p\\u003e\\n\\u003ch2\\u003e\\u003cstrong\\u003eStudy Setting, Objective, and Design\\u003c/strong\\u003e\\u003c/h2\\u003e\\n\\u003cp\\u003eThe included studies evaluated GenAI for drafting replies to patient messages in two settings: \\u003cstrong\\u003elive EHR systems\\u003c/strong\\u003e (n = 7) and\\u003cstrong\\u003esimulated environments\\u003c/strong\\u003e (n = 16). Across these settings, studies primarily aimed to evaluate: (1) the content of AI-generated drafts (n = 19); (2) the implementation of GenAI and its impact on clinical efficiency (n = 8); (3) user perceptions, preferences, or experiences (n = 12); and (4) the effects of prompt engineering (n = 5). Many studies addressed more than one objective.\\u003c/p\\u003e\\n\\u003cp\\u003eThe live EHR system setting refers to real-world clinical environments where GenAI tools are embedded into existing in-basket workflows to generate draft replies for patient messages. All seven studies in this setting evaluated GenAI tools integrated into the Epic EHR (Epic Systems Corporation, Verona, Wisconsin, USA), using OpenAI\\u0026rsquo;s Generative Pre-Trained Transformer (GPT)-4 for response generation\\u003csup\\u003e31,32,36\\u0026ndash;40\\u003c/sup\\u003e. Several studies explored GenAI implementation and its impact on clinical efficiency and user experience, using measures such as draft utilization\\u003csup\\u003e31,36,37\\u003c/sup\\u003e, extent of edits\\u003csup\\u003e36\\u003c/sup\\u003e, time spent\\u003csup\\u003e31,32\\u003c/sup\\u003e, and clinician burden\\u003csup\\u003e31,39\\u003c/sup\\u003e. Some evaluated user perceptions of AI-drafted content, aiming to understand how clinicians in different roles judged its usefulness\\u003csup\\u003e37\\u003c/sup\\u003e or how linguistic features were associated with perceived empathy\\u003csup\\u003e38\\u003c/sup\\u003e. Three studies primarily examined prompt engineering. Two iteratively modified prompts in live settings and assessed their effect on draft usability and clinician feedback\\u003csup\\u003e36,37\\u003c/sup\\u003e. The third adopted a structured, human-in-the-loop process to refine prompts in a test environment before deploying them in production, evaluating their impact on provider acceptance and patient satisfaction\\u003csup\\u003e39\\u003c/sup\\u003e. A distinct effort sought to bridge qualitative and quantitative evaluation metrics by proposing a unified framework for assessing LLM performance in healthcare, which was then applied to Epic\\u0026rsquo;s in-basket GenAI feature\\u003csup\\u003e40\\u003c/sup\\u003e. All implementations were conducted as pilot-scale efforts, with most framed as quality improvement projects\\u003csup\\u003e31,32,37\\u0026ndash;39\\u003c/sup\\u003e. The majority employed prospective\\u003csup\\u003e31,37,39\\u003c/sup\\u003e or quasi-experimental\\u003csup\\u003e36\\u003c/sup\\u003e designs to evaluate GenAI deployment or prompt modifications in real time. One study used a modified waitlist randomized design to compare outcomes between clinicians with and without access to GenAI drafts\\u003csup\\u003e32\\u003c/sup\\u003e, while another used a cross-sectional survey to capture provider perceptions of draft quality\\u003csup\\u003e38\\u003c/sup\\u003e.\\u0026nbsp;\\u003c/p\\u003e\\n\\u003cp\\u003eIn contrast, studies conducted in the simulated setting evaluated GenAI tools outside of live clinical workflows. These controlled experiments involved generating replies to either de-identified patient messages extracted from the EHR\\u003csup\\u003e41\\u0026ndash;51\\u003c/sup\\u003e or hypothetical inquiries modeled on real patient portal communications\\u003csup\\u003e52\\u0026ndash;56\\u003c/sup\\u003e. The 16 studies conducted in this setting primarily focused on assessing the content quality\\u003csup\\u003e42\\u0026ndash;55\\u003c/sup\\u003e and user perceptions\\u003csup\\u003e41,45,46,49,51,53,56\\u003c/sup\\u003e of AI-drafted replies, using a range of outcome measures detailed later. Many compared GenAI drafts across different models\\u003csup\\u003e43,45,47,51,52,54\\u003c/sup\\u003e or against human-authored responses\\u003csup\\u003e41\\u0026ndash;43,45\\u0026ndash;49,51,53,54,56\\u003c/sup\\u003e. Although not embedded in actual workflows, a few studies examined GenAI\\u0026rsquo;s impact on efficiency through subjective ratings and self-reported time for responding or editing drafts\\u003csup\\u003e47,48,53,54\\u003c/sup\\u003e. While GPT-3.5 and GPT-4 were the most frequently used models, some studies evaluated alternative or customized variants. These included GPT-4-based, specialty-specific retrieval-augmented models\\u003csup\\u003e45,48\\u003c/sup\\u003e, fine-tuned versions of LLaMA adapted for clinical use\\u003csup\\u003e43,52\\u003c/sup\\u003e, and institution-developed GPT-powered tools\\u003csup\\u003e45,51\\u003c/sup\\u003e. One study also compared multiple commercial models, including Bard, Claude, and Bing, alongside GPT variants\\u003csup\\u003e54\\u003c/sup\\u003e. To approximate the EHR context, one study included simulated medical records alongside messages when prompting GPT\\u003csup\\u003e53\\u003c/sup\\u003e, while another connected a customized model to the local EHR system to access clinical notes and patient details in support of drafting\\u003csup\\u003e48\\u003c/sup\\u003e. A few studies also incorporated clinician editing processes to examine the human\\u0026ndash;AI synergistic effect in message drafting\\u003csup\\u003e53,54\\u003c/sup\\u003e as well as to explore patient preferences regarding the disclosure of AI involvement\\u003csup\\u003e56\\u003c/sup\\u003e. Cross-sectional or multi-stage surveys were commonly used to gather user perceptions and assessments, often through randomized and blinded review designs comparing responses by different GenAI models or humans\\u003csup\\u003e41,45\\u0026ndash;47,49,51,53,54,56\\u003c/sup\\u003e. Two studies had distinct objectives and approaches. One developed and fine-tuned a large language model on portal message data and evaluated its performance against baseline GPT and clinician drafts\\u003csup\\u003e43\\u003c/sup\\u003e. The other conducted a retrospective qualitative study focused specifically on GenAI responses to negative patient messages, using thematic analysis to compare content differences between GenAI and clinical care team replies\\u003csup\\u003e42\\u003c/sup\\u003e.\\u003c/p\\u003e\\n\\u003ch2\\u003e\\u003cstrong\\u003eClinical Context, Message Topic, and Participant (Evaluator)\\u003c/strong\\u003e\\u003c/h2\\u003e\\n\\u003cp\\u003eGenAI for responding to patient messages was evaluated across various clinical contexts, with \\u003cstrong\\u003eprimary care\\u003c/strong\\u003e, including internal medicine, family medicine, and pediatrics, being the most common setting (n = 10)\\u003csup\\u003e31,32,36\\u0026ndash;39,43,49\\u0026ndash;51\\u003c/sup\\u003e. Four studies conducted evaluations across primary care and specialties\\u003csup\\u003e31,36,37,51\\u003c/sup\\u003e, while the remaining studies focused on specific specialty domains, including dermatology\\u003csup\\u003e46\\u003c/sup\\u003e, urology\\u003csup\\u003e44,45,48\\u003c/sup\\u003e, ophthalmology\\u003csup\\u003e54,55\\u003c/sup\\u003e, oncology\\u003csup\\u003e53\\u003c/sup\\u003e, and surgery\\u003csup\\u003e47,52\\u003c/sup\\u003e.\\u003c/p\\u003e\\n\\u003cp\\u003eAcross the included studies, GenAI was used to generate draft replies for patient messages covering a variety of topics, ranging from \\u003cstrong\\u003eadministrative inquiries\\u003c/strong\\u003e, such as appointment scheduling and medication refills\\u003csup\\u003e38,43,49,52,56\\u003c/sup\\u003e, to \\u003cstrong\\u003emedical advice requests\\u003c/strong\\u003e related to symptoms, postoperative concerns, and test results\\u003csup\\u003e44,47,48,55,56\\u003c/sup\\u003e. Several studies deliberately varied message seriousness\\u003csup\\u003e56\\u003c/sup\\u003e, complexity\\u003csup\\u003e52\\u003c/sup\\u003e, or level of detail\\u003csup\\u003e55\\u003c/sup\\u003e to include comprehensive test cases, while others focused solely on clinical questions involving decision-making implications or condition-specific issues\\u003csup\\u003e39,41,44\\u0026ndash;48,51,54,55\\u003c/sup\\u003e. Some studies curated representative messages to ensure that evaluations covered the most commonly asked topics in patient portals\\u003csup\\u003e41,43,45,54\\u003c/sup\\u003e, while others randomly selected samples from the in-baskets of participating clinicians or from messages sent by a particular patient group\\u003csup\\u003e38,39,46,50\\u003c/sup\\u003e. To ensure fair GenAI evaluation under simulated conditions without integrated data sources, two studies excluded messages that required access to external information\\u003csup\\u003e38,43\\u003c/sup\\u003e. In addition, some studies collected unique message samples to meet specific research objectives. For example, one study evaluated GenAI using a mix of adolescent patient and proxy messages\\u003csup\\u003e50\\u003c/sup\\u003e, while another tested GenAI in emotionally sensitive scenarios using negative patient messages\\u003csup\\u003e42\\u003c/sup\\u003e.\\u003c/p\\u003e\\n\\u003cp\\u003eParticipants in the studies can be grouped into two high-level categories: \\u003cstrong\\u003eclinical experts\\u0026nbsp;\\u003c/strong\\u003eand\\u003cstrong\\u003e\\u0026nbsp;non-experts\\u003c/strong\\u003e. Over 90% of the studies (n = 21) involved clinical experts across diverse contexts and roles, including physicians\\u003csup\\u003e31,36\\u0026ndash;39,42\\u0026ndash;47,49,50,52\\u0026ndash;55\\u003c/sup\\u003e, advanced practice providers\\u003csup\\u003e31,37,39,47,49\\u003c/sup\\u003e, nurses\\u003csup\\u003e31,36,37,48\\u003c/sup\\u003e, medical trainees (medical students, residents, and fellows)\\u003csup\\u003e47,48\\u003c/sup\\u003e, medical assistants\\u003csup\\u003e37\\u003c/sup\\u003e, and clinical pharmacists\\u003csup\\u003e31\\u003c/sup\\u003e. Clinician participants reviewed GenAI draft replies, shared pilot experiences, and either edited AI drafts or created human-authored counterparts for comparative analysis. In contrast, only a few studies engaged non-expert participants, including patient advisors or laypeople\\u003csup\\u003e39,41,45,51,56\\u003c/sup\\u003e. These participants were typically asked to rate tone, identify the authorship, or share their personal preferences for the given responses, rather than evaluate their clinical quality. While considered laypersons in the medical context, many were active stakeholders, such as long-term patient advisors with extensive experience using patient portals\\u003csup\\u003e39\\u003c/sup\\u003e, participants recruited from institutional research registries\\u003csup\\u003e51\\u003c/sup\\u003e, or pre-screened volunteers with a clinical condition of interest\\u003csup\\u003e45\\u003c/sup\\u003e. Only two studies focused exclusively on non-expert perspectives. One surveyed over 1,400 members of a patient advisory committee to evaluate preferences for AI-generated responses under varying disclosure conditions\\u003csup\\u003e56\\u003c/sup\\u003e. Another recruited a nationally representative sample of laypersons through a crowdsourcing platform to assess their ability to distinguish between human and AI responses as well as their trust in AI\\u0026rsquo;s advice\\u003csup\\u003e41\\u003c/sup\\u003e.\\u003c/p\\u003e\\n\\u003ch2\\u003e\\u003cstrong\\u003eEvaluation Method and Outcome Measure\\u003c/strong\\u003e\\u003c/h2\\u003e\\n\\u003cp\\u003eAll 23 studies incorporated \\u003cstrong\\u003ehuman ratings\\u003c/strong\\u003e using Likert scales to assess GenAI responses and their impact. Ten evaluations also calculated \\u003cstrong\\u003ebasic computational metrics\\u003c/strong\\u003e, including text length, utilization rate, and time changes\\u003csup\\u003e31,32,36\\u0026ndash;38,42,47,51,53,54\\u003c/sup\\u003e, while only six studies employed more \\u003cstrong\\u003eadvanced computational metrics\\u003c/strong\\u003e, such as BERTScore and Flesch-Kincaid grade level\\u003csup\\u003e36,38,40,43,47,48\\u003c/sup\\u003e. Among these six, one evaluation framework study intentionally mapped outcome measures to both human ratings and computational metrics to assess their alignment\\u003csup\\u003e40\\u003c/sup\\u003e. Building on the grouping approach used in prior work\\u003csup\\u003e38\\u003c/sup\\u003e, we categorized the common outcome measures examined in these studies into five groups: (1) information quality, (2) communication quality, (3) user perception, experience, and preference, (4) utilization and efficiency, and (5) composite measures.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003e\\u003cem\\u003eInformation Quality (n = 14).\\u003c/em\\u003e\\u003c/strong\\u003e Information quality was consistently evaluated across the studies, encompassing measures such as accuracy\\u003csup\\u003e38,43\\u0026ndash;47,49\\u003c/sup\\u003e, completeness\\u003csup\\u003e38,40,44,45,48,52\\u003c/sup\\u003e, relevance\\u003csup\\u003e38,40,48\\u0026ndash;50\\u003c/sup\\u003e, and factuality\\u003csup\\u003e40,50\\u003c/sup\\u003e. These measures assess the integrity of the information presented in AI drafts, with accuracy (n = 7) being the most frequently evaluated. While similar or identical terms were often used, they may reflect subtle differences. For instance, some studies evaluated completeness by checking whether drafts lacked essential information needed to answer patients\\u0026rsquo; questions\\u003csup\\u003e38,40,44,48,52\\u003c/sup\\u003e, while others examined whether responses were comprehensive beyond the minimum required content\\u003csup\\u003e45,46\\u003c/sup\\u003e. Relevance or responsiveness was typically rated based on whether AI responses addressed patients\\u0026rsquo; concerns\\u003csup\\u003e38,39,43,49,50\\u003c/sup\\u003e. In contrast, studies using computational methods defined relevance by how well AI drafts inferred patient inquiries\\u003csup\\u003e48\\u003c/sup\\u003e or matched clinician-authored replies\\u003csup\\u003e40\\u003c/sup\\u003e. One study treated information quality as a single dimension, without specifying the subcomponents it included\\u003csup\\u003e51,54\\u003c/sup\\u003e.\\u0026nbsp;\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003e\\u003cem\\u003eCommunication Quality (n = 14)\\u003c/em\\u003e\\u003c/strong\\u003e.Communication quality was another important focus, capturing the patient-centered aspects of responses, with empathy (n = 9) being the most commonly measured aspect\\u003csup\\u003e37,43,45\\u0026ndash;49,51,54\\u003c/sup\\u003e. AI drafts were also evaluated for their tone or style, determining whether their wording and expressions were appropriate for the context of the conversation\\u003csup\\u003e37\\u0026ndash;39,50\\u003c/sup\\u003e. Readability\\u003csup\\u003e46,49\\u003c/sup\\u003e and related subdimensions, such as clarity\\u003csup\\u003e40,48\\u003c/sup\\u003e, understandability\\u003csup\\u003e38,44\\u003c/sup\\u003e, and brevity (or verbosity)\\u003csup\\u003e38,40,50\\u003c/sup\\u003e, were assessed to ensure that responses could be easily comprehended by patients without semantic confusions, literacy challenges, or distractions. Several studies used computational metrics to assess these aspects, including DiscoScore, lexical diversity, and the Flesch reading ease score\\u003csup\\u003e38,40,47\\u003c/sup\\u003e. Two studies analyzed the overall sentiment of the replies\\u003csup\\u003e38,48\\u003c/sup\\u003e, while another calculated BERT Toxicity to detect any pejorative terms or non-inclusive language\\u003csup\\u003e40\\u003c/sup\\u003e.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003e\\u003cem\\u003eUser Perception, Experience, and Preference (n = 12)\\u003c/em\\u003e\\u003c/strong\\u003e\\u003cstrong\\u003e.\\u003c/strong\\u003e A variety of measures were used to evaluate user perceptions, experiences, and preferences regarding the use of GenAI to draft replies to patient messages. A recurring focus was the perception of authorship, specifically whether participants could distinguish between AI-generated and human-written replies\\u003csup\\u003e39,41,45,49\\u003c/sup\\u003e. Some studies also asked participants to indicate their preference or rate their satisfaction with each response\\u003csup\\u003e36,45,46,51,56\\u003c/sup\\u003e. Additionally, patients\\u0026rsquo; trust and perceived level of care from AI-generated replies were assessed\\u003csup\\u003e41,56\\u003c/sup\\u003e. GenAI\\u0026rsquo;s impact on clinician burnout was examined in one study by comparing physician task load and work exhaustion scores\\u003csup\\u003e31\\u003c/sup\\u003e. In parallel, a few other studies captured clinicians\\u0026rsquo; perceptions of reduced cognitive load, time savings, and improved efficiency to evaluate the potential benefits of GenAI assistance\\u003csup\\u003e37,39,53\\u003c/sup\\u003e. The net promoter score was also used as a measure of clinicians\\u0026rsquo; overall support for the GenAI tool\\u003csup\\u003e31,32,37,39\\u003c/sup\\u003e.\\u0026nbsp;\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003e\\u003cem\\u003eUtilization and Efficiency (n = 11)\\u003c/em\\u003e\\u003c/strong\\u003e\\u003cstrong\\u003e.\\u003c/strong\\u003e Measures related to utilization and efficiency often focused on basic characteristics or objective metrics of GenAI drafts and their implementation. Length (n = 7) was the most frequently evaluated characteristic\\u003csup\\u003e32,38,42,47,51,53,54\\u003c/sup\\u003e, as it may affect how efficiently clinicians review AI-generated drafts. Time-related measures included the time spent reading messages, as well as writing or editing replies. Some studies measured time changes before and after GenAI implementations\\u003csup\\u003e31,32\\u003c/sup\\u003e, while others compared the completion time of AI-generated, clinician-written, and clinician-edited replies\\u003csup\\u003e47,48,54\\u003c/sup\\u003e. Three implementation studies reported real-world utilization rates of AI drafts in clinical practices, typically measured by the how often clinicians selected \\u0026ldquo;Start with Draft\\u0026rdquo; instead of \\u0026ldquo;Start Blank Reply.\\u0026rdquo;\\u003csup\\u003e31,36,37\\u003c/sup\\u003e One of these studies also calculated the Damerau-Levenshtein distance between AI-generated drafts and final replies as a metric for estimating the editing effort required before clinical use\\u003csup\\u003e36\\u003c/sup\\u003e.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003e\\u003cem\\u003eComposite Measures (n = 10).\\u0026nbsp;\\u003c/em\\u003e\\u003c/strong\\u003eIn several studies, some measures were framed to assess multiple dimensions of quality through subjective judgment. Appropriateness, treated as a composite concept, involves evaluating whether the tone of the message was suitable and whether the information provided was adequate in a reply\\u003csup\\u003e52,55\\u003c/sup\\u003e. Potential harm associated with AI-generated responses was assessed by considering not only the risk of incorrect content compromising patient safety but also the possibility of communication that appeared unfriendly or perpetuated bias\\u003csup\\u003e37,44,53,54\\u003c/sup\\u003e. Usefulness or acceptability was often rated based on whether clinicians believed that AI-generated responses could be directly sent to patients, used as a starting point, or help improve the quality of a final response\\u003csup\\u003e31,37\\u0026ndash;39,43,44\\u003c/sup\\u003e.\\u003c/p\\u003e\\n\\u003ch2\\u003e\\u003cstrong\\u003eConsensus from Early Findings\\u003c/strong\\u003e\\u003c/h2\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eGenAI responses were generally found to match, and in some cases exceed, the quality of human-authored replies across several dimensions, though risks and limitations remain.\\u0026nbsp;\\u003c/strong\\u003eGenAI drafts were frequently rated as comparable in information qualityand more favorable in communication style compared to clinician-authored responses\\u003csup\\u003e38,43,45,47\\u0026ndash;49,51,54\\u003c/sup\\u003e. Empathy consistently emerged as a notable strength of GenAI drafts\\u003csup\\u003e37,38,43,45,47\\u0026ndash;49,51,54\\u003c/sup\\u003e, with GPT-4 most often recognized as the top-performing model\\u003csup\\u003e43,45,47,52\\u003c/sup\\u003e. However, potential harms associated with these responses, although minimal, were documented\\u003csup\\u003e37,40,42,44,48,50,52\\u0026ndash;55\\u003c/sup\\u003e. Common challenges included hallucinations, incoherent language, and limited contextual understanding\\u003csup\\u003e31,40,48,52,55\\u003c/sup\\u003e. One study found that 7% of AI-generated replies posed a risk of severe harm or death\\u003csup\\u003e53\\u003c/sup\\u003e, while another reported two instances in which AI disclosed unsolicited confidential information in messaging involving proxies\\u003csup\\u003e50\\u003c/sup\\u003e. Additionally, studies noted inconsistent AI performance across message types, with reduced reliability in clinically complex inquiries and an increased risk of escalating emotionally charged conversations\\u003csup\\u003e40\\u0026ndash;42,44,52,55\\u003c/sup\\u003e.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eLaypersons\\u0026mdash;and even clinicians\\u0026mdash;often could not accurately distinguish between AI- and human-authored responses in blinded reviews.\\u003c/strong\\u003e Relatedly, several studies asked participants to identify the authorship of responses and reported only low to modest accuracy, with averages ranging from 24% (among patients) to 73% (among clinicians), suggesting that AI-generated replies closely resemble human communication\\u003csup\\u003e39,41,45,49\\u003c/sup\\u003e. While several blinded evaluations showed that participants, particularly patients, tended to favor AI responses\\u003csup\\u003e45,51,56\\u003c/sup\\u003e, one study found that more empathetic and preferable responses were often attributed to human authorship, even when they were actually generated by AI\\u003csup\\u003e45\\u003c/sup\\u003e. Similarly, another study noted that disclosing AI involvement in reply drafting led to a slight decrease in patient satisfaction, although participants still valued transparency over nondisclosure\\u003csup\\u003e56\\u003c/sup\\u003e.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eDespite positive attitudes, adoption of GenAI for drafting patient message replies in real-world settings remained limited.\\u003c/strong\\u003e While recognizing GenAI\\u0026rsquo;s limitations and the need for edits, clinicians still viewed these tools as helpful aids for managing inbox burden. Several studies highlighted that clinicians found the drafts acceptable or useful as starting points and appreciated features such as templates or pleasantries\\u003csup\\u003e32,38\\u0026ndash;40,42\\u0026ndash;44,46\\u003c/sup\\u003e. Many clinicians also expressed willingness to recommend GenAI tools to colleagues and to retain the tools in future workflows\\u003csup\\u003e31,32,37,39\\u003c/sup\\u003e. However, this enthusiasm did not translate into consistent use. Across pilot implementations, actual utilization of AI-generated drafts was low, with average usage rates no higher than 20%\\u003csup\\u003e31,36,37\\u003c/sup\\u003e. Few studies have explored the disconnect between perceived benefits and limited uptake.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eWhile no strong evidence suggesting time savings, current GenAI implementations were associated with perceived efficiency gains and burnout relief.\\u0026nbsp;\\u003c/strong\\u003eDespite expectations for streamlining workflows, early implementations reported no statistically significant changes in message reply time\\u003csup\\u003e31,32\\u003c/sup\\u003e, and one study even observed an increase in read time following GenAI integration\\u003csup\\u003e32\\u003c/sup\\u003e. Moreover, all studies comparing length found that AI-generated responses were considerably longer than human-written ones, raising concerns about increased burden for draft review and editing\\u003csup\\u003e32,38,42,47,51,53,54\\u003c/sup\\u003e. Nevertheless, survey data revealed that clinicians perceived a meaningful reduction in task and cognitive load, along with decreased work exhaustion\\u003csup\\u003e31,39\\u003c/sup\\u003e. Some also reported a subjective sense of time saved and improved efficiency\\u003csup\\u003e31,37,39,53\\u003c/sup\\u003e, even in the absence of objective evidence for reduced time or workload.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003ePrompt engineering consistently emerged as an effective strategy for enhancing the quality and usability of GenAI drafts.\\u003c/strong\\u003e Across studies, prompt optimization was associated with measurable improvements in response quality, tone, and user acceptance. One study reported a significant increase in clinician acceptance of AI-generated replies after three rounds of prompt refinement, alongside improvements in patient-rated tone and overall message quality\\u003csup\\u003e39\\u003c/sup\\u003e. Another found that a revised prompt led to a reduction in negative clinician feedback on drafts\\u003csup\\u003e36\\u003c/sup\\u003e. Incorporating the most recent assessment and plan into prompts was shown to improve perceived usefulness among clinicians\\u003csup\\u003e37\\u003c/sup\\u003e, while purposely designed prompts helped mitigate inconsistencies between AI- and clinician-authored responses, particularly in relational tone, content relevance, and clinical recommendations\\u003csup\\u003e42\\u003c/sup\\u003e.\\u003c/p\\u003e\\n\\u003cp\\u003eTable 1. Summary of studies included in this review.\\u0026nbsp;\\u003c/p\\u003e\\n\\u003ctable border=\\\"1\\\" cellspacing=\\\"0\\\" cellpadding=\\\"0\\\"\\u003e\\n \\u003ctbody\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 71px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eStudy\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 73px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eSetting \\u0026amp; Objective\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 83px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eDesign\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 80px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eClinical context\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 112px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eMessage and/or Response\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 89px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eEvaluated\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eResponses\\u0026nbsp;\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eby?\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 88px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eParticipant\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd colspan=\\\"3\\\" valign=\\\"top\\\" style=\\\"width: 273px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eOutcome Measure\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 136px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eKey Finding\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 105px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eExpert\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 83px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003ePatient/\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eLaypeople\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 85px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eAutomated\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 71px;\\\"\\u003e\\n \\u003cp\\u003eAthavale et al.\\u003csup\\u003e52\\u003c/sup\\u003e, 2023\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 73px;\\\"\\u003e\\n \\u003cp\\u003eSimulated,\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003eO1\\u0026nbsp;\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 83px;\\\"\\u003e\\n \\u003cp\\u003eExperimental evaluation study\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 80px;\\\"\\u003e\\n \\u003cp\\u003eVascular surgery\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 112px;\\\"\\u003e\\n \\u003cp\\u003eDevised\\u003c/p\\u003e\\n \\u003cp\\u003e20 administrative non-complex and\\u003c/p\\u003e\\n \\u003cp\\u003e20 complex medical questions on CVD based on actual messages via patient portal\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 89px;\\\"\\u003e\\n \\u003cp\\u003eGPT-4, GPT-3.5, Clinical Camel (a healthcare chatbot based on LLaMA)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 88px;\\\"\\u003e\\n \\u003cp\\u003e1 internist and 1 vascular medicine specialist\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 105px;\\\"\\u003e\\n \\u003cp\\u003eAppropriateness,\\u003c/p\\u003e\\n \\u003cp\\u003eCompleteness\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 83px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 85px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 136px;\\\"\\u003e\\n \\u003cp\\u003eChatGPT-4 performed the best across both non-complex and complex question sets (100% and 75% appropriate and complete responses respectively). ChatGPT3.5 ranked second for both sets. Reported one hallucination case from ChatGPT-3.5\\u0026nbsp;\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 71px;\\\"\\u003e\\n \\u003cp\\u003eNov et al.\\u003csup\\u003e41\\u003c/sup\\u003e, 2023\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 73px;\\\"\\u003e\\n \\u003cp\\u003eSimulated,\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003eO3\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 83px;\\\"\\u003e\\n \\u003cp\\u003eCross-sectional survey study\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 80px;\\\"\\u003e\\n \\u003cp\\u003eNot specified\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 112px;\\\"\\u003e\\n \\u003cp\\u003e10 representative, nonadministrative patient\\u0026ndash;provider interactions were extracted from EHRs\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 89px;\\\"\\u003e\\n \\u003cp\\u003eGPT-3.5,\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003eHuman\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 88px;\\\"\\u003e\\n \\u003cp\\u003e392 layerson respondents from a US representative sample\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 105px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 83px;\\\"\\u003e\\n \\u003cp\\u003ePerception of authorship,\\u003c/p\\u003e\\n \\u003cp\\u003eTrust\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 85px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 136px;\\\"\\u003e\\n \\u003cp\\u003eOn average, respondents correctly classified both AI and human responses around 65% of the time, with trust in chatbots being weakly positive but decreasing as task complexity increased.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 71px;\\\"\\u003e\\n \\u003cp\\u003eAfshar et al.\\u003csup\\u003e36\\u003c/sup\\u003e, 2024\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 73px;\\\"\\u003e\\n \\u003cp\\u003eLive EHR,\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003eO2, O3, O4\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 83px;\\\"\\u003e\\n \\u003cp\\u003ePre-post quasi-experimental study\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 80px;\\\"\\u003e\\n \\u003cp\\u003ePrimary care, dermatology, oncology, psychiatry\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 112px;\\\"\\u003e\\n \\u003cp\\u003eAI drafts generated:\\u003c/p\\u003e\\n \\u003cp\\u003e3882 (Pre);\\u003c/p\\u003e\\n \\u003cp\\u003e3723 (Post);\\u003c/p\\u003e\\n \\u003cp\\u003e2573 (Follow-up)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 89px;\\\"\\u003e\\n \\u003cp\\u003eGPT-4\\u003c/p\\u003e\\n \\u003cp\\u003eHuman + GPT-4\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 88px;\\\"\\u003e\\n \\u003cp\\u003ePre \\u0026amp; post: 27 physicians\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003eFollow up: 44 with 17 nurses added\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 105px;\\\"\\u003e\\n \\u003cp\\u003eThumbs up/down feedback\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 83px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 85px;\\\"\\u003e\\n \\u003cp\\u003eUtilization\\u003c/p\\u003e\\n \\u003cp\\u003eEdit distance\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 136px;\\\"\\u003e\\n \\u003cp\\u003eTotal usage: 17.5%. Usage increased in the follow-up: 35.8%. Post prompt engineering, no change for utilization but decreased in \\u0026ldquo;thumb down\\u0026rdquo;. Only 2.6% AI drafts were used without or with minimal provider edits.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003c/tbody\\u003e\\n\\u003c/table\\u003e\\n\\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n\\u003cp\\u003eTable 1. Continued.\\u003c/p\\u003e\\n\\u003ctable border=\\\"1\\\" cellspacing=\\\"0\\\" cellpadding=\\\"0\\\" align=\\\"left\\\" width=\\\"1005\\\"\\u003e\\n \\u003ctbody\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 68px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eStudy\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 74px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eSetting \\u0026amp; Objective\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 90px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eDesign\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 84px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eClinical Context\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 100px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eMessage and/or Response\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 82px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eEvaluated\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eResponses\\u0026nbsp;\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eby?\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 103px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eParticipant\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd colspan=\\\"3\\\" valign=\\\"top\\\" style=\\\"width: 238px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eOutcome Measure\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 166px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eKey Findings\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 85px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eExpert\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 81px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003ePatient/\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eLaypeople\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 72px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eAutomated\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 68px;\\\"\\u003e\\n \\u003cp\\u003eBaxter et al.\\u003csup\\u003e42\\u003c/sup\\u003e, 2024\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 74px;\\\"\\u003e\\n \\u003cp\\u003eSimulated,\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003eO1, O4, to evaluate whether LLMs can help address negative patient messages.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 90px;\\\"\\u003e\\n \\u003cp\\u003eRetrospective qualitative evaluation study\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 84px;\\\"\\u003e\\n \\u003cp\\u003eNot specified\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 100px;\\\"\\u003e\\n \\u003cp\\u003eA random sample of 50 negative sentiment messages and responses extracted from EHRs\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 82px;\\\"\\u003e\\n \\u003cp\\u003eGPT-3.5, Human\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 103px;\\\"\\u003e\\n \\u003cp\\u003eTwo researcher coders. One is an ophthalmologist and another has a master of public health and a doctoral degree\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 85px;\\\"\\u003e\\n \\u003cp\\u003eThemes in response\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 81px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 72px;\\\"\\u003e\\n \\u003cp\\u003eLength\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 166px;\\\"\\u003e\\n \\u003cp\\u003eAI responses were about triple the length of clinicians\\u0026rsquo;. Differences were noted in relational connection, content, and next-step recommendations.\\u003c/p\\u003e\\n \\u003cp\\u003ePrompting mitigated some issues but not all. AI drafts could be helpful starting points but could escalate emotional conversations.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 68px;\\\"\\u003e\\n \\u003cp\\u003eChen et al.\\u003csup\\u003e53\\u003c/sup\\u003e, 2024\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 74px;\\\"\\u003e\\n \\u003cp\\u003eSimulated,\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003eO1, O2, O3\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 90px;\\\"\\u003e\\n \\u003cp\\u003eTwo-stage observational study\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 84px;\\\"\\u003e\\n \\u003cp\\u003eOncology\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 100px;\\\"\\u003e\\n \\u003cp\\u003e100 hypothetical patient messages with simulated EHRs\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 82px;\\\"\\u003e\\n \\u003cp\\u003eGPT-4, Human,\\u003c/p\\u003e\\n \\u003cp\\u003eGPT-4 + Human\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 103px;\\\"\\u003e\\n \\u003cp\\u003eStage 1 \\u0026amp; 2 surveys: 6 attending radiation oncologists\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003e2 additional physicians for content analysis\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 85px;\\\"\\u003e\\n \\u003cp\\u003eHelpfulness,\\u003c/p\\u003e\\n \\u003cp\\u003eRisk/Harm,\\u003c/p\\u003e\\n \\u003cp\\u003eSubjective efficiency,\\u003c/p\\u003e\\n \\u003cp\\u003eContent categories in response\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 81px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 72px;\\\"\\u003e\\n \\u003cp\\u003eLength\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 166px;\\\"\\u003e\\n \\u003cp\\u003eAI/AI-assisted responses were longer than human ones. About 7% of AI drafts posed a risk of severe harm or death. AI responses contained less direct action but provided more extensive education. Physicians reported improved efficiency, and responses became more consistent with AI assistance.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 68px;\\\"\\u003e\\n \\u003cp\\u003eEnglish et al.\\u003csup\\u003e37\\u003c/sup\\u003e, 2024\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 74px;\\\"\\u003e\\n \\u003cp\\u003eLive EHR,\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003eO1, O2, O3, O4\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 90px;\\\"\\u003e\\n \\u003cp\\u003eProspective quality improvement study\\u0026nbsp;\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 84px;\\\"\\u003e\\n \\u003cp\\u003ePrimary care and specialty (non-specified)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 100px;\\\"\\u003e\\n \\u003cp\\u003eAI drafts generated: 21323\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 82px;\\\"\\u003e\\n \\u003cp\\u003eGPT-4\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 103px;\\\"\\u003e\\n \\u003cp\\u003e12 nurses,14 MAs, 93 physicians and APPs\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 85px;\\\"\\u003e\\n \\u003cp\\u003eEmpathy,\\u003c/p\\u003e\\n \\u003cp\\u003eTone,\\u003c/p\\u003e\\n \\u003cp\\u003ePerceived efficiency,\\u003c/p\\u003e\\n \\u003cp\\u003eNPS,\\u003c/p\\u003e\\n \\u003cp\\u003eMinimal risk\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 81px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 72px;\\\"\\u003e\\n \\u003cp\\u003eUtilization\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 166px;\\\"\\u003e\\n \\u003cp\\u003eOverall, 12% utilization rate. Nurses were more likely to recommend the AI tool to others than MAs and clinicians, with more than 90% believing that it improved efficiency, empathy, and tone. Including the last A/P in prompts made some replies useful.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003c/tbody\\u003e\\n\\u003c/table\\u003e\\n\\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n\\u003cp\\u003eTable 1. Continued.\\u003c/p\\u003e\\n\\u003ctable border=\\\"1\\\" cellspacing=\\\"0\\\" cellpadding=\\\"0\\\" align=\\\"left\\\" width=\\\"1005\\\"\\u003e\\n \\u003ctbody\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 64px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eStudy\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eSetting \\u0026amp; Objective\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 77px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eDesign\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 94px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eClinical Context\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 98px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eMessage and/or Response\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 94px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eEvaluated\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eResponses\\u0026nbsp;\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eby?\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 94px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eParticipant\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd colspan=\\\"3\\\" valign=\\\"top\\\" style=\\\"width: 252px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eOutcome Measure\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 154px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eKey Findings\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 91px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eExpert\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 81px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003ePatient/\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eLaypeople\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 80px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eAutomated\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 64px;\\\"\\u003e\\n \\u003cp\\u003eGarcia et al.\\u003csup\\u003e31\\u003c/sup\\u003e, 2024\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003eLive EHR,\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003eO1, O2, O3\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 77px;\\\"\\u003e\\n \\u003cp\\u003eProspective quality improvement study\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 94px;\\\"\\u003e\\n \\u003cp\\u003ePrimary care,\\u003c/p\\u003e\\n \\u003cp\\u003eGastroenterology and hepatology\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 98px;\\\"\\u003e\\n \\u003cp\\u003eAI drafts generated: 9621\\u0026nbsp;\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 94px;\\\"\\u003e\\n \\u003cp\\u003eGPT-4\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 94px;\\\"\\u003e\\n \\u003cp\\u003eTotal: 162\\u003c/p\\u003e\\n \\u003cp\\u003ePrimary care: 83 physicians and APPs, 4 nurses, 8 clinical pharmacists\\u003c/p\\u003e\\n \\u003cp\\u003eGastroenterology and hepatology: 58 physicians and APPs, 10 nurses.\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003eSurveyed: 73\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 91px;\\\"\\u003e\\n \\u003cp\\u003eNASA TLX with a 4-item physician task load score derivative, PFI-WE score,\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003eNPS, Usability (utility, quality, perceived time saved),\\u003c/p\\u003e\\n \\u003cp\\u003eFree-text comment\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 81px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 80px;\\\"\\u003e\\n \\u003cp\\u003eUtilization, Change in reply action time, write time, or read time\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 154px;\\\"\\u003e\\n \\u003cp\\u003eMean AI- draft utilization rate was 20%. No changes in reply action time, write time, or read time between pre-pilot and pilot periods. Task load and work exhaustion scores significantly decreased. Comments identified facilitators such as readiness, utility, and time-saving. Barriers included tone, content relevance, and accuracy.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 64px;\\\"\\u003e\\n \\u003cp\\u003eKim et al.\\u003csup\\u003e51\\u003c/sup\\u003e, 2024\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003eSimulated, O1, O3, O4\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 77px;\\\"\\u003e\\n \\u003cp\\u003eCross-sectional survey study\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 94px;\\\"\\u003e\\n \\u003cp\\u003ePrimary care, Endocrinology, Cardiology\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 98px;\\\"\\u003e\\n \\u003cp\\u003e59 messages selected from PMARs in EHRs\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 94px;\\\"\\u003e\\n \\u003cp\\u003eGPT-4, Stanford Health Care and Stanford School of Medicine GPT,\\u003c/p\\u003e\\n \\u003cp\\u003eHuman\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 94px;\\\"\\u003e\\n \\u003cp\\u003e6 clinicians\\u003c/p\\u003e\\n \\u003cp\\u003e30 survey participants (layperson) recruited through the Stanford Research Registry\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 91px;\\\"\\u003e\\n \\u003cp\\u003eEmpathy, Information quality\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 81px;\\\"\\u003e\\n \\u003cp\\u003eSatisfaction\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 80px;\\\"\\u003e\\n \\u003cp\\u003eLength\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 154px;\\\"\\u003e\\n \\u003cp\\u003ePromoted Stanford GPT was rated best for information quality and empathy. Satisfaction was higher with AI responses than with clinicians\\u0026rsquo; across specialties. Clinician responses were shorter. Satisfaction was not necessarily concordant with clinician-rated information quality and empathy. Clinician response length was associated with satisfaction while AI response length was not.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003c/tbody\\u003e\\n\\u003c/table\\u003e\\n\\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n\\u003cp\\u003eTable 1. Continued.\\u003c/p\\u003e\\n\\u003ctable border=\\\"1\\\" cellspacing=\\\"0\\\" cellpadding=\\\"0\\\" align=\\\"left\\\" width=\\\"1005\\\"\\u003e\\n \\u003ctbody\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 57px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eStudy\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 77px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eSetting \\u0026amp; Objective\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 77px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eDesign\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 65px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eClinical Context\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 143px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eMessage and/or Response\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003e\\u0026nbsp;\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 92px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eEvaluated\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eResponses\\u0026nbsp;\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eby?\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 85px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eParticipant\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd colspan=\\\"3\\\" valign=\\\"top\\\" style=\\\"width: 237px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eOutcome Measure\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 172px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eKey Findings\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 81px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eExpert\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003ePatient/\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eLaypeople\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eAutomated\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 57px;\\\"\\u003e\\n \\u003cp\\u003eLiu et al.\\u003csup\\u003e43\\u003c/sup\\u003e, 2024\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 77px;\\\"\\u003e\\n \\u003cp\\u003eSimulated, O1, aimed to fine-tune a LLM using patient portal interactions as well as evaluate its responses\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 77px;\\\"\\u003e\\n \\u003cp\\u003eModel development and evaluation\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 65px;\\\"\\u003e\\n \\u003cp\\u003ePrimary care\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 143px;\\\"\\u003e\\n \\u003cp\\u003eFine-tune: CLARE-Short (499286 portal message -response pairs)\\u003c/p\\u003e\\n \\u003cp\\u003eCLAIR-Long (the pairs + 5000 open-source patient questions with GPT-4 responses)\\u003c/p\\u003e\\n \\u003cp\\u003eEvaluate set: 10 representative, de-identified, and rephrased patient messages and responses\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 92px;\\\"\\u003e\\n \\u003cp\\u003eGPT-4, GPT-3.5, CLARE-Short,\\u003c/p\\u003e\\n \\u003cp\\u003eCLAIR-Long (based on LLaMA-65B),\\u003c/p\\u003e\\n \\u003cp\\u003eHuman\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 85px;\\\"\\u003e\\n \\u003cp\\u003e4 primary care physicians\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 81px;\\\"\\u003e\\n \\u003cp\\u003eAccuracy, Empathy,\\u003c/p\\u003e\\n \\u003cp\\u003eResponsiveness, Usefulness,\\u003c/p\\u003e\\n \\u003cp\\u003eFree-text comment\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003eBERTScore\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 172px;\\\"\\u003e\\n \\u003cp\\u003eGPT-4 responses were rated the best. ChatGPT models outperformed CLAIR-Long across accuracy, empathy,\\u003c/p\\u003e\\n \\u003cp\\u003eresponsiveness, and usefulness. They all outperformed CLAIR-Short and the provider\\u0026rsquo; responses significantly. ChatGPT 3.5 achieved the highest BERTScore compared to actual provider responses.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 57px;\\\"\\u003e\\n \\u003cp\\u003eReynolds et al.\\u003csup\\u003e46\\u003c/sup\\u003e, 2024\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 77px;\\\"\\u003e\\n \\u003cp\\u003eSimulated,\\u003c/p\\u003e\\n \\u003cp\\u003eO1, O3\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 77px;\\\"\\u003e\\n \\u003cp\\u003eCross-sectional study\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 65px;\\\"\\u003e\\n \\u003cp\\u003eDermatology\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 143px;\\\"\\u003e\\n \\u003cp\\u003e31 patient messages with questions related to dermatological conditions or management and their responses extracted from EHRs\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 92px;\\\"\\u003e\\n \\u003cp\\u003eGPT-3.5, Human\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 85px;\\\"\\u003e\\n \\u003cp\\u003e7 dermatology physicians, 3 nonphysicians\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 81px;\\\"\\u003e\\n \\u003cp\\u003eOverall quality\\u0026sbquo;\\u003c/p\\u003e\\n \\u003cp\\u003eReadability\\u0026sbquo; Accuracy\\u0026sbquo; Thoroughness,\\u003c/p\\u003e\\n \\u003cp\\u003eEmpathy,\\u003c/p\\u003e\\n \\u003cp\\u003ePreference\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 172px;\\\"\\u003e\\n \\u003cp\\u003eBoth physicians and non-physicians preferred physician-generated responses over ChatGPT\\u0026rsquo;s in most cases. Physician responses were rated significantly better in readability, empathy, accuracy, and overall quality. No hallucinations observed.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 57px;\\\"\\u003e\\n \\u003cp\\u003eRobinson et al.\\u003csup\\u003e45\\u003c/sup\\u003e, 2024\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 77px;\\\"\\u003e\\n \\u003cp\\u003eSimulated, O1, O3\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 77px;\\\"\\u003e\\n \\u003cp\\u003eCross-sectional study\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 65px;\\\"\\u003e\\n \\u003cp\\u003eUrology\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 143px;\\\"\\u003e\\n \\u003cp\\u003e20 common BPH-related patient questions from phone or EHR-messaging were pooled, anonymized, and compiled\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 92px;\\\"\\u003e\\n \\u003cp\\u003eGPT-4,\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003eKPGPT (GPT-4-0613)\\u003c/p\\u003e\\n \\u003cp\\u003eSurgiChat (GPT-4-0613 with RAG on BPH literature),\\u003c/p\\u003e\\n \\u003cp\\u003eHuman\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 85px;\\\"\\u003e\\n \\u003cp\\u003e2 urologists,\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003e5 pre-screened non-medical volunteers that relevant to BPH\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 81px;\\\"\\u003e\\n \\u003cp\\u003eAccuracy, Empathy, Comprehensiveness\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003ePerception of authorship,\\u003c/p\\u003e\\n \\u003cp\\u003ePreference,\\u003c/p\\u003e\\n \\u003cp\\u003eEmpathy\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 172px;\\\"\\u003e\\n \\u003cp\\u003eChatbot and urologist responses had similar accuracy, but chatbots rated significantly higher in completeness and empathy. Volunteers identified the correct author 59% and preferred chatbot responses. However, responses labeled as human scored higher in empathy than those labeled as chatbot.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003c/tbody\\u003e\\n\\u003c/table\\u003e\\n\\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n\\u003cp\\u003eTable 1. Continued.\\u003c/p\\u003e\\n\\u003ctable border=\\\"1\\\" cellspacing=\\\"0\\\" cellpadding=\\\"0\\\" align=\\\"left\\\" width=\\\"1006\\\"\\u003e\\n \\u003ctbody\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 69px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eStudy\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 71px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eSetting \\u0026amp; Objective\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 80px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eDesign\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 83px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eClinical Context\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 114px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eMessage and/or Response\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 74px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eEvaluated\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eResponses\\u0026nbsp;\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eby?\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 100px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eParticipant\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd colspan=\\\"3\\\" valign=\\\"top\\\" style=\\\"width: 248px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eOutcome Measure\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 167px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eKey Findings\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 94px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eExpert\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 75px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003ePatient/\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eLaypeople\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 79px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eAutomated\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 69px;\\\"\\u003e\\n \\u003cp\\u003eScott et al.\\u003csup\\u003e44\\u003c/sup\\u003e, 2024\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 71px;\\\"\\u003e\\n \\u003cp\\u003eSimulated, O1\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 80px;\\\"\\u003e\\n \\u003cp\\u003eCross-sectional study\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 83px;\\\"\\u003e\\n \\u003cp\\u003eUrology\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 114px;\\\"\\u003e\\n \\u003cp\\u003e100 patient messages requesting medical advice were collected from the in-basket of a urologist specializing in andrology.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 74px;\\\"\\u003e\\n \\u003cp\\u003eGPT-3.5\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 100px;\\\"\\u003e\\n \\u003cp\\u003e5 urologists\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 94px;\\\"\\u003e\\n \\u003cp\\u003eAccuracy, Helpfulness, Completeness, Harmfulness, Intelligibleness, Acceptability\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 75px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 79px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 167px;\\\"\\u003e\\n \\u003cp\\u003eOverall, ChatGPT was rated to give accurate and intelligible answers, while completeness and helpfulness were rated lower. Harm was minimal. Performance was better on easier questions than harder ones. 47% of responses were considered acceptable.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 69px;\\\"\\u003e\\n \\u003cp\\u003eSmall et al.\\u003csup\\u003e38\\u003c/sup\\u003e, 2024\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 71px;\\\"\\u003e\\n \\u003cp\\u003eLive EHR\\u003c/p\\u003e\\n \\u003cp\\u003eO1, O2, O3\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 80px;\\\"\\u003e\\n \\u003cp\\u003eCross-sectional quality improvement study\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 83px;\\\"\\u003e\\n \\u003cp\\u003ePrimary care\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 114px;\\\"\\u003e\\n \\u003cp\\u003e117 unique HCP and 126 unique AI message-response pairs from pilot users\\u0026rsquo; in-basket\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003eAI drafts from silent validation, not being seen before)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 74px;\\\"\\u003e\\n \\u003cp\\u003eGPT-4,\\u003c/p\\u003e\\n \\u003cp\\u003eHuman\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 100px;\\\"\\u003e\\n \\u003cp\\u003eA convenience sample of 16 primary care physicians\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 94px;\\\"\\u003e\\n \\u003cp\\u003eUsability,\\u003c/p\\u003e\\n \\u003cp\\u003eInformation content quality (completeness, accuracy, relevance)\\u003c/p\\u003e\\n \\u003cp\\u003eCommunication quality (understandable, appropriate, tone; verbosity)\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 75px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 79px;\\\"\\u003e\\n \\u003cp\\u003eLength, Complexity (lexical diversity, Flesch-Kincaid grade level), Sentiment\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 167px;\\\"\\u003e\\n \\u003cp\\u003eBoth AI and HCP responses were rated favorably. AI scored higher in communication style and matched HCPs in information content quality and usable draft proportion. Usable AI responses were seen as more empathetic, possibly due to their subjective and positive tone. They were also longer and more linguistically complex.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 69px;\\\"\\u003e\\n \\u003cp\\u003eSoroudi et al.\\u003csup\\u003e47\\u003c/sup\\u003e, 2024\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 71px;\\\"\\u003e\\n \\u003cp\\u003eSimulated, O1, O2\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 80px;\\\"\\u003e\\n \\u003cp\\u003eCross-sectional survey study\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 83px;\\\"\\u003e\\n \\u003cp\\u003ePlastic surgery\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 114px;\\\"\\u003e\\n \\u003cp\\u003e10 queries from patients undergoing breast reconstruction with highest level of complexity and decision-making implications were extracted from EHRs\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 74px;\\\"\\u003e\\n \\u003cp\\u003eGPT-3,\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003eGPT-4,\\u003c/p\\u003e\\n \\u003cp\\u003eHuman\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 100px;\\\"\\u003e\\n \\u003cp\\u003e2 APPs and 2 plastic surgeons for generated responses\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003e2 medical students and 1 plastic surgeon, and 1 microsurgery fellow for review\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 94px;\\\"\\u003e\\n \\u003cp\\u003eAccuracy (surgeon and fellow)\\u003c/p\\u003e\\n \\u003cp\\u003eEmpathy (medical students)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 75px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 79px;\\\"\\u003e\\n \\u003cp\\u003eLength, FRE score, Time to compilation (self-reported)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 167px;\\\"\\u003e\\n \\u003cp\\u003eCombined provider responses were more readable compared to combined chatbot responses. Empathy scores were higher in chatbot response. No significant differences in accuracy between providers and chatbot responses. Prompts increased readability.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003c/tbody\\u003e\\n\\u003c/table\\u003e\\n\\u003cp\\u003eTable 1. Continued.\\u003c/p\\u003e\\n\\u003ctable border=\\\"1\\\" cellspacing=\\\"0\\\" cellpadding=\\\"0\\\" align=\\\"left\\\" width=\\\"1001\\\"\\u003e\\n \\u003ctbody\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 59px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eStudy\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 72px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eSetting \\u0026amp; Objective\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 72px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eDesign\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eClinical Context\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 102px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eMessage and/or Response\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eEvaluated\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eResponses\\u0026nbsp;\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eby?\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 90px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eParticipant\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd colspan=\\\"3\\\" valign=\\\"top\\\" style=\\\"width: 246px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eOutcome Measure\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 204px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eKey Findings\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 90px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eExpert\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003ePatient/\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eLaypeople\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eAutomated\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 59px;\\\"\\u003e\\n \\u003cp\\u003eTai-Seale et al.\\u003csup\\u003e32\\u003c/sup\\u003e, 2024a\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 72px;\\\"\\u003e\\n \\u003cp\\u003eLive EHR,\\u003c/p\\u003e\\n \\u003cp\\u003eO1, O2, O3\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 72px;\\\"\\u003e\\n \\u003cp\\u003eModified waiting list randomized quality improvement study\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003ePrimary care\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 102px;\\\"\\u003e\\n \\u003cp\\u003e10 679 replies to patient messages were examined\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003eGPT-4\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 90px;\\\"\\u003e\\n \\u003cp\\u003eImmediate activation group: 25 physicians\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003eDelayed activation group: 27\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003eContemporary control group: 70\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 90px;\\\"\\u003e\\n \\u003cp\\u003eNPS\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003eLength, Time spent reading and replying to messages\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 204px;\\\"\\u003e\\n \\u003cp\\u003eAccess to AI drafts was associated with a significant increase in read time, no change in reply time, and significantly longer replies. Physicians\\u0026rsquo; views of AI replies ranged from helpful as starting drafts and for adding empathy, to ineffective, overly focused on visits, and having an overly nice tone. Examples showed physicians kept pleasantries from AI drafts but made substantive edits.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 59px;\\\"\\u003e\\n \\u003cp\\u003eTailor et al.\\u003csup\\u003e55\\u003c/sup\\u003e, 2024a\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 72px;\\\"\\u003e\\n \\u003cp\\u003eSimulated, O1\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 72px;\\\"\\u003e\\n \\u003cp\\u003eCross-sectional study\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003eOphthalmology (across 9 subspecialties)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 102px;\\\"\\u003e\\n \\u003cp\\u003eFor each ophthalmic subspecialty, about 20 clinical questions were generated on the basis of common patient questions received via the clinic or patient portal\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003eGPT-4\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 90px;\\\"\\u003e\\n \\u003cp\\u003e25 subspecialists participated. 22 of them both wrote and graded questions, and 3 only wrote questions.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 90px;\\\"\\u003e\\n \\u003cp\\u003eAppropriateness\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 204px;\\\"\\u003e\\n \\u003cp\\u003eReported robust aggregate appropriateness of an LLM across ophthalmic subspecialties both in the context of a patient information site (56%-100%) and as responses to EHR patient messages (54%-90%). Generally, inappropriate responses were inappropriate recommendations and incorrect or missing information.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 59px;\\\"\\u003e\\n \\u003cp\\u003eTailor et al.\\u003csup\\u003e54\\u003c/sup\\u003e, 2024b\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 72px;\\\"\\u003e\\n \\u003cp\\u003eSimulated, O1, Q2\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 72px;\\\"\\u003e\\n \\u003cp\\u003eRandomized cross-sectional study\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003eOphthalmology\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 102px;\\\"\\u003e\\n \\u003cp\\u003eCreated 21 retina questions similar to common patient inquiries received in clinic or via patient portals\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003eGPT-3.5, GPT-4,\\u003c/p\\u003e\\n \\u003cp\\u003eBard, Claude 2, Bing,\\u003c/p\\u003e\\n \\u003cp\\u003eHuman, Human + GPT-4,\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 90px;\\\"\\u003e\\n \\u003cp\\u003e13 retinal specialists\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 90px;\\\"\\u003e\\n \\u003cp\\u003eInformation quality\\u003c/p\\u003e\\n \\u003cp\\u003eEmpathy, Safety (inappropriate/incorrect/missing content, likelihood of possible harm)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003eLength,\\u003c/p\\u003e\\n \\u003cp\\u003eTime spent (self-reported)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 204px;\\\"\\u003e\\n \\u003cp\\u003eFor quality, Expert+AI performed the best overall while GPT-3.5 was the top performing AI. For empathy, GPT-3.5 got the best score followed by Expert+AI. There were time savings for an Expert+AI response versus expert-created response. ChatGPT-4 performed similarly to Expert for safety metrics.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003c/tbody\\u003e\\n\\u003c/table\\u003e\\n\\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n\\u003cp\\u003eTable 1. Continued.\\u003c/p\\u003e\\n\\u003ctable border=\\\"1\\\" cellspacing=\\\"0\\\" cellpadding=\\\"0\\\" align=\\\"left\\\" width=\\\"1004\\\"\\u003e\\n \\u003ctbody\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 67px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eStudy\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 70px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eSetting \\u0026amp; Objective\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 81px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eDesign\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 63px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eClinical Context\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 136px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eMessage and/or Response\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 76px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eEvaluated\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eResponses\\u0026nbsp;\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eby?\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 99px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eParticipant\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd colspan=\\\"3\\\" valign=\\\"top\\\" style=\\\"width: 269px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eOutcome Measure\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 143px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eKey Findings\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 92px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eExpert\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 99px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003ePatient/\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eLaypeople\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eAutomated\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 67px;\\\"\\u003e\\n \\u003cp\\u003eYan et al.\\u003csup\\u003e39\\u003c/sup\\u003e, 2024\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 70px;\\\"\\u003e\\n \\u003cp\\u003eLive EHR,\\u003c/p\\u003e\\n \\u003cp\\u003eO1, O2, O3, O4\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 81px;\\\"\\u003e\\n \\u003cp\\u003eProspective quality improvement study\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 63px;\\\"\\u003e\\n \\u003cp\\u003ePrimary care\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 136px;\\\"\\u003e\\n \\u003cp\\u003eTrain: 116 random PMARs and responses from 5 pilot physicians\\u0026rsquo; in-basket\\u003c/p\\u003e\\n \\u003cp\\u003eValidation: 200 additional PMARs and responses\\u003c/p\\u003e\\n \\u003cp\\u003eTest: AI drafts to 761 PMARs in production\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003ePatient validation 1\\u0026amp;2:\\u003c/p\\u003e\\n \\u003cp\\u003e250 PMARs and responses including the 116 in train set\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003ePatient validation 3: 250 PMARs and AI responses from production including the 200 in validation\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 76px;\\\"\\u003e\\n \\u003cp\\u003eGPT-4,\\u003c/p\\u003e\\n \\u003cp\\u003eHuman\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 99px;\\\"\\u003e\\n \\u003cp\\u003eTest: 5 primary care physicians, 5 patient advisors\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003eProduction: 69 primary care clinicians (physicians and APPs) including the 5 in test, the same patient advisors\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003ePost production survey: 40 out of 69\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 92px;\\\"\\u003e\\n \\u003cp\\u003eAcceptance (Send, Edit or Reject),\\u003c/p\\u003e\\n \\u003cp\\u003eHelpfulness,\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003eWant to retain the tool?\\u003c/p\\u003e\\n \\u003cp\\u003eRecommend to colleagues?\\u003c/p\\u003e\\n \\u003cp\\u003ePerception of cognitive load reduced,\\u003c/p\\u003e\\n \\u003cp\\u003ePerception of time saved\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 99px;\\\"\\u003e\\n \\u003cp\\u003ePerception of authorship,\\u003c/p\\u003e\\n \\u003cp\\u003eTone,\\u003c/p\\u003e\\n \\u003cp\\u003eOverall quality,\\u003c/p\\u003e\\n \\u003cp\\u003eResponsiveness\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 143px;\\\"\\u003e\\n \\u003cp\\u003eAfter prompt iterations, physician acceptance (send/edit) of AI drafts rose significantly, with 74% rated as helpful. Patients also reported improved tone and overall quality, noting most responses addressed patient questions. Patients were unable to distinguish between humans and AI for 76% of messages. Majority clinicians would like to keep the tool and would recommend it, 72% believe it can reduce cognitive load, and 41% believe it has potential to reduce in-basket time.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 67px;\\\"\\u003e\\n \\u003cp\\u003eCavalier et al.\\u003csup\\u003e56\\u003c/sup\\u003e, 2025\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 70px;\\\"\\u003e\\n \\u003cp\\u003eSimulated, O3\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 81px;\\\"\\u003e\\n \\u003cp\\u003eSurvey-based randomized factorial experiment\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 63px;\\\"\\u003e\\n \\u003cp\\u003eNot specified\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 136px;\\\"\\u003e\\n \\u003cp\\u003eCreated 3 hypothetical patient messages representing low, medium, or high clinical seriousness\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003e3 physicians wrote responses\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003eAI drafts were reviewed and minimally edited by two physician authors\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 76px;\\\"\\u003e\\n \\u003cp\\u003eGPT-3.5 + Human,\\u003c/p\\u003e\\n \\u003cp\\u003eHuman\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 99px;\\\"\\u003e\\n \\u003cp\\u003e1455 respondents from patient advisory committee\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 92px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 99px;\\\"\\u003e\\n \\u003cp\\u003eSatisfaction, Perceived level of care, Usefulness, Preference for disclosure\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 78px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 143px;\\\"\\u003e\\n \\u003cp\\u003eParticipants preferred AI responses over human responses regardless of the disclosure or seriousness of the topic.\\u003c/p\\u003e\\n \\u003cp\\u003eHowever, there was a slight decrease in satisfaction when told AI was involved. Participants preferred the shortest disclosure statement.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003c/tbody\\u003e\\n\\u003c/table\\u003e\\n\\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n\\u003cp\\u003eTable 1. Continued.\\u003c/p\\u003e\\n\\u003ctable border=\\\"1\\\" cellspacing=\\\"0\\\" cellpadding=\\\"0\\\" align=\\\"left\\\" width=\\\"988\\\"\\u003e\\n \\u003ctbody\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 54px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eStudy\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 74px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eSetting \\u0026amp; Objective\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 83px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eDesign\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 68px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eClinical Context\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 142px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eMessage and/or Response\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 88px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eEvaluated\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eResponses\\u0026nbsp;\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eby?\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 75px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eParticipant\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd colspan=\\\"3\\\" valign=\\\"top\\\" style=\\\"width: 242px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eOutcome Measure\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 162px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eKey Findings\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 89px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eExpert\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 74px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003ePatient/\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eLaypeople\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 79px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eAutomated\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 54px;\\\"\\u003e\\n \\u003cp\\u003eHao et al.\\u003csup\\u003e48\\u003c/sup\\u003e, 2025\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 74px;\\\"\\u003e\\n \\u003cp\\u003eSimulated, O1, O2\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 83px;\\\"\\u003e\\n \\u003cp\\u003eRetrospective observational study\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 68px;\\\"\\u003e\\n \\u003cp\\u003eUrology, Radiation oncology\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 142px;\\\"\\u003e\\n \\u003cp\\u003e58 in-basket message interactions, selected from 90 patients with nonmetastatic prostate cancer\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 88px;\\\"\\u003e\\n \\u003cp\\u003eRadOnc-GPT (GPT-4 with RAG on local EHRs and oncology-specific database),\\u003c/p\\u003e\\n \\u003cp\\u003eHuman\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 75px;\\\"\\u003e\\n \\u003cp\\u003e1 oncologist, 4 residents, 4 nurses\\u0026nbsp;\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 89px;\\\"\\u003e\\n \\u003cp\\u003eCompleteness, Correctness, Clarity, Empathy,\\u003c/p\\u003e\\n \\u003cp\\u003eEstimated time to respond,\\u003c/p\\u003e\\n \\u003cp\\u003eFree-text comments\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 74px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 79px;\\\"\\u003e\\n \\u003cp\\u003eInferences label, semantic similarity score, sentiment scores\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 162px;\\\"\\u003e\\n \\u003cp\\u003eRadOnc-GPT responses were more positive and generalized, while clinician replies had more balanced sentiment and greater variety. High similarity scores indicated strong content alignment.\\u003c/p\\u003e\\n \\u003cp\\u003eRadOnc-GPT slightly outperformed the care team in empathy, whereas it had comparable completeness, correctness, and clarity. Key limitations in RadOnc-GPT\\u0026rsquo;s responses were lack of context, insufficient domain-specific knowledge, inability to perform meta-tasks, and hallucination. RadOnc-GPT was estimated to save clinicians time.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 54px;\\\"\\u003e\\n \\u003cp\\u003eHong et al.\\u003csup\\u003e40\\u003c/sup\\u003e, 2025\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 74px;\\\"\\u003e\\n \\u003cp\\u003eLive EHR,\\u003c/p\\u003e\\n \\u003cp\\u003eO1, O2, aimed to present a unified evaluation framework with mixed-methods metrics to assess AI for in-basket\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 83px;\\\"\\u003e\\n \\u003cp\\u003eFramework development and evaluation\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 68px;\\\"\\u003e\\n \\u003cp\\u003eNot specified\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 142px;\\\"\\u003e\\n \\u003cp\\u003e243 patient messages and AI draft pairs to establish and test human evaluation principles\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003e42 patient messages from pilot users, with AI drafts, clinicians\\u0026rsquo; usage decisions, and final clinician-edited replies for compare qualitative and quantitative evaluations\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 88px;\\\"\\u003e\\n \\u003cp\\u003eGPT-4,\\u003c/p\\u003e\\n \\u003cp\\u003eHuman + GPT-4\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 75px;\\\"\\u003e\\n \\u003cp\\u003e1 clinician reviewer\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 89px;\\\"\\u003e\\n \\u003cp\\u003eRelevance, Factuality, Completeness, Coherence, Clarity, Brevity, Toxicity\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 74px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 79px;\\\"\\u003e\\n \\u003cp\\u003ePerplexity,\\u003c/p\\u003e\\n \\u003cp\\u003eDiscoScore,\\u003c/p\\u003e\\n \\u003cp\\u003eFRE score,\\u003c/p\\u003e\\n \\u003cp\\u003eCompression ratio,\\u003c/p\\u003e\\n \\u003cp\\u003eKeyword Matching,\\u003c/p\\u003e\\n \\u003cp\\u003eROUGE-N Recall|,\\u003c/p\\u003e\\n \\u003cp\\u003eBERT Toxicity\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 162px;\\\"\\u003e\\n \\u003cp\\u003eAI replies exhibit high fluency, clarity, and minimal toxicity, they face challenges with coherence and completeness. Most AI drafts were rated as usable with minor edits. However, reliability and accuracy of AI drafts are inconsistent across message categories. Clinicians\\u0026rsquo; manual decision to use AI drafts correlates strongly with quantitative metrics.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003c/tbody\\u003e\\n\\u003c/table\\u003e\\n\\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n\\u003cp\\u003eTable 1. Continued.\\u003c/p\\u003e\\n\\u003ctable border=\\\"1\\\" cellspacing=\\\"0\\\" cellpadding=\\\"0\\\" align=\\\"left\\\" width=\\\"1002\\\"\\u003e\\n \\u003ctbody\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 59px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eStudy\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 77px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eSetting \\u0026amp; Objective\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 90px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eDesign\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 67px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eClinical Context\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 149px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eMessage and/or Response\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 74px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eEvaluated\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eResponses\\u0026nbsp;\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eby?\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 83px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eParticipant\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd colspan=\\\"3\\\" valign=\\\"top\\\" style=\\\"width: 251px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eOutcome Measure\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd rowspan=\\\"2\\\" valign=\\\"top\\\" style=\\\"width: 152px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eKey Findings\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 101px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eExpert\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 75px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003ePatient/\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eLaypeople\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 75px;\\\"\\u003e\\n \\u003cp\\u003e\\u003cstrong\\u003eAutomated\\u003c/strong\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 59px;\\\"\\u003e\\n \\u003cp\\u003eKaur et al.\\u003csup\\u003e49\\u003c/sup\\u003e, 2025\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 77px;\\\"\\u003e\\n \\u003cp\\u003eSimulated, O1, O3\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 90px;\\\"\\u003e\\n \\u003cp\\u003eCross-sectional survey study\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 67px;\\\"\\u003e\\n \\u003cp\\u003ePrimary care\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 149px;\\\"\\u003e\\n \\u003cp\\u003e20 unique patient message-response pairs from the medical center repository and de-identified\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003e10 responses were generated by GPT, and the other 10 were written by real doctors\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 74px;\\\"\\u003e\\n \\u003cp\\u003eGPT-3.5,\\u003c/p\\u003e\\n \\u003cp\\u003eHuman\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 83px;\\\"\\u003e\\n \\u003cp\\u003e8 primary care physicians for the initial review\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003cp\\u003e49 HCPs for the evaluation\\u003c/p\\u003e\\n \\u003cp\\u003e\\u0026nbsp;\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 101px;\\\"\\u003e\\n \\u003cp\\u003eAccuracy, Readability, Empathy, Relevance, Perception of authorship\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 75px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 75px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 152px;\\\"\\u003e\\n \\u003cp\\u003eCompared with real doctors, GPT responses scored significantly higher in empathy and readability. However, no statistically significant difference was observed for relevance and accuracy. Participants correctly identified GPT messages 73% of the time and correctly identified authentic messages 50% of the time.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 59px;\\\"\\u003e\\n \\u003cp\\u003eTse et al.\\u003csup\\u003e50\\u003c/sup\\u003e, 2025\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 77px;\\\"\\u003e\\n \\u003cp\\u003eSimulated, O1\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 90px;\\\"\\u003e\\n \\u003cp\\u003eCross-sectional study\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 67px;\\\"\\u003e\\n \\u003cp\\u003ePrimary care\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 149px;\\\"\\u003e\\n \\u003cp\\u003ePatient portal messages were randomly obtained from patients aged 12 to 17 years and their associated proxy users with EHRs (problem list, medications, and lab results)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 74px;\\\"\\u003e\\n \\u003cp\\u003eGPT-4\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 83px;\\\"\\u003e\\n \\u003cp\\u003eTwo pediatricians for review and a third pediatrician for resolving differences\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 101px;\\\"\\u003e\\n \\u003cp\\u003eUsefulness,\\u003c/p\\u003e\\n \\u003cp\\u003eProxy user identification,\\u003c/p\\u003e\\n \\u003cp\\u003eProtection of confidentiality,\\u003c/p\\u003e\\n \\u003cp\\u003eQuality (relevance, factual correctness, literacy, conciseness, tone/style)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 75px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 75px;\\\"\\u003e\\n \\u003cp\\u003eN/A\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd valign=\\\"top\\\" style=\\\"width: 152px;\\\"\\u003e\\n \\u003cp\\u003eAmong the proxy user messages, the AI correctly identified the proxy user 76% of the time. The AI disclosed unsolicited confidential information in fewer than 1% of cases. Most of the responses were in plain language, relevant, factually correct, concise, and 67% were rated as clinical useful.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003c/tbody\\u003e\\n\\u003c/table\\u003e\\n\\u003cp\\u003eO1: evaluate the content of AI-generated drafts, O2: evaluate the implementation of generative AI and its impact on clinical efficiency, O3: understand user perceptions, preferences, or experiences, O4: examine the effects of prompt engineering, CVD: chronic venous disease, MAs: medical assistants, APPs: advanced practice providers, NPS: net promoter score, A/P: assessment and plan, NASA TLX score: NASA task load index score, PFI-WE: professional fulfillment index-work exhaustion score, PMAR: patient medical advice request, BPH: benign prostatic hyperplasia, RAG: retrieval-augmented generation, HCP: health care professional, FRE: Flesch reading ease.\\u003c/p\\u003e\"},{\"header\":\"Discussion\",\"content\":\"\\u003cp\\u003eThe synthesis of current research demonstrates a growing interest among health systems in GenAI-assisted replying to patient messages, with various efforts to evaluate draft quality, impacts on clinical efficiency, user perceptions, and the role of prompt engineering. Although varied in study design, scope, and evaluation methods, these early explorations reached some consensus. GenAI-drafted replies were generally perceived as acceptable starting points, especially when enhanced by tailored prompts. Both clinicians and patients recognized its potential to alleviate in-basket burden and enhance patient\\u0026ndash;provider communication. However, current real-world adoption of GenAI drafts remains limited, and important concerns regarding performance reliability and potential risks persist. In this section, we examine the key methodological and implementation limitations, explore ethical considerations, and outline future directions for advancing the effective and responsible integration of GenAI into high-volume clinical in-basket workflows.\\u003c/p\\u003e \\u003cp\\u003eEarly evaluations and implementations were often constrained in scope, scale, and generalizability. Most studies were conducted at a specific site, within a single health system, or involved relatively small sample sizes of message corpora and participants\\u003csup\\u003e\\u003cspan citationid=\\\"CR32\\\" class=\\\"CitationRef\\\"\\u003e32\\u003c/span\\u003e,\\u003cspan additionalcitationids=\\\"CR38 CR39 CR40 CR41\\\" citationid=\\\"CR37\\\" class=\\\"CitationRef\\\"\\u003e37\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR42\\\" class=\\\"CitationRef\\\"\\u003e42\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR45\\\" class=\\\"CitationRef\\\"\\u003e45\\u003c/span\\u003e,\\u003cspan additionalcitationids=\\\"CR49\\\" citationid=\\\"CR48\\\" class=\\\"CitationRef\\\"\\u003e48\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR50\\\" class=\\\"CitationRef\\\"\\u003e50\\u003c/span\\u003e\\u003c/sup\\u003e. Several studies focused on clinicians from certain specialties or patient groups with limited demographic diversity, raising concerns about how well the findings translate across clinical settings or populations\\u003csup\\u003e\\u003cspan citationid=\\\"CR31\\\" class=\\\"CitationRef\\\"\\u003e31\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR41\\\" class=\\\"CitationRef\\\"\\u003e41\\u003c/span\\u003e,\\u003cspan additionalcitationids=\\\"CR45 CR46\\\" citationid=\\\"CR44\\\" class=\\\"CitationRef\\\"\\u003e44\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR47\\\" class=\\\"CitationRef\\\"\\u003e47\\u003c/span\\u003e,\\u003cspan additionalcitationids=\\\"CR53 CR54 CR55\\\" citationid=\\\"CR52\\\" class=\\\"CitationRef\\\"\\u003e52\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR56\\\" class=\\\"CitationRef\\\"\\u003e56\\u003c/span\\u003e\\u003c/sup\\u003e. Evaluations were more commonly conducted in simulated environments rather than in live clinical workflows, with many studies assessing carefully selected single-turn messages or hypothetical inquiries\\u003csup\\u003e\\u003cspan citationid=\\\"CR43\\\" class=\\\"CitationRef\\\"\\u003e43\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR44\\\" class=\\\"CitationRef\\\"\\u003e44\\u003c/span\\u003e,\\u003cspan additionalcitationids=\\\"CR48\\\" citationid=\\\"CR47\\\" class=\\\"CitationRef\\\"\\u003e47\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR49\\\" class=\\\"CitationRef\\\"\\u003e49\\u003c/span\\u003e,\\u003cspan additionalcitationids=\\\"CR53 CR54 CR55\\\" citationid=\\\"CR52\\\" class=\\\"CitationRef\\\"\\u003e52\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR56\\\" class=\\\"CitationRef\\\"\\u003e56\\u003c/span\\u003e\\u003c/sup\\u003e. As a result, these findings may not fully capture the complexity of real-world patient\\u0026ndash;provider messaging interactions, including diverse topics, contextual cues, or evolving patient conditions.\\u003c/p\\u003e \\u003cp\\u003eStudies employed diverse, often unvalidated evaluation rubrics and relied heavily on human judgment from evaluators with varying levels of clinical training\\u003csup\\u003e\\u003cspan citationid=\\\"CR31\\\" class=\\\"CitationRef\\\"\\u003e31\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR36\\\" class=\\\"CitationRef\\\"\\u003e36\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR37\\\" class=\\\"CitationRef\\\"\\u003e37\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR39\\\" class=\\\"CitationRef\\\"\\u003e39\\u003c/span\\u003e,\\u003cspan additionalcitationids=\\\"CR48\\\" citationid=\\\"CR47\\\" class=\\\"CitationRef\\\"\\u003e47\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR49\\\" class=\\\"CitationRef\\\"\\u003e49\\u003c/span\\u003e\\u003c/sup\\u003e. Draft quality assessments were typically conducted by convenient samples of experts\\u003csup\\u003e\\u003cspan citationid=\\\"CR38\\\" class=\\\"CitationRef\\\"\\u003e38\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR49\\\" class=\\\"CitationRef\\\"\\u003e49\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR50\\\" class=\\\"CitationRef\\\"\\u003e50\\u003c/span\\u003e\\u003c/sup\\u003e, and while most studies did not report inter-rater reliability, a few noted low levels of reviewer agreement\\u003csup\\u003e\\u003cspan citationid=\\\"CR38\\\" class=\\\"CitationRef\\\"\\u003e38\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR45\\\" class=\\\"CitationRef\\\"\\u003e45\\u003c/span\\u003e\\u003c/sup\\u003e. Prompt engineering also differed widely across studies, with many relying on ad hoc or trial-and-error approaches, limiting the reproducibility of these optimizations\\u003csup\\u003e\\u003cspan citationid=\\\"CR39\\\" class=\\\"CitationRef\\\"\\u003e39\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR42\\\" class=\\\"CitationRef\\\"\\u003e42\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR49\\\" class=\\\"CitationRef\\\"\\u003e49\\u003c/span\\u003e\\u003c/sup\\u003e. Additionally, while poor rater agreement suggests diverse communication styles and preferences, current deployments fall short in supporting prompt personalization. This review also noted a lack of transparency around how GenAI tools integrated with EHR systems accessed and utilized medical records. Few studies reported details about incorporating problem lists, clinical notes, or messaging history in draft generation\\u003csup\\u003e\\u003cspan citationid=\\\"CR31\\\" class=\\\"CitationRef\\\"\\u003e31\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR48\\\" class=\\\"CitationRef\\\"\\u003e48\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR50\\\" class=\\\"CitationRef\\\"\\u003e50\\u003c/span\\u003e\\u003c/sup\\u003e, limiting the understanding of how GenAI contextually grounded their responses\\u003csup\\u003e\\u003cspan citationid=\\\"CR49\\\" class=\\\"CitationRef\\\"\\u003e49\\u003c/span\\u003e\\u003c/sup\\u003e. Without consistent access to comprehensive, up-to-date clinical information, the risk of generating inaccurate or context-agnostic replies increases.\\u003c/p\\u003e \\u003cp\\u003eMany early explorations acknowledged that patients were underrepresented\\u003csup\\u003e\\u003cspan citationid=\\\"CR36\\\" class=\\\"CitationRef\\\"\\u003e36\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR38\\\" class=\\\"CitationRef\\\"\\u003e38\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR39\\\" class=\\\"CitationRef\\\"\\u003e39\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR49\\\" class=\\\"CitationRef\\\"\\u003e49\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR54\\\" class=\\\"CitationRef\\\"\\u003e54\\u003c/span\\u003e\\u003c/sup\\u003e, with none of the current studies involving patients who actually received AI-generated replies during pilot implementations\\u0026mdash;pointing to a significant gap in which patient-facing impacts and their preferences were often inferred rather than directly assessed\\u003csup\\u003e\\u003cspan citationid=\\\"CR51\\\" class=\\\"CitationRef\\\"\\u003e51\\u003c/span\\u003e\\u003c/sup\\u003e. Another limitation in understanding user experiences is that, while a few studies collected and analyzed qualitative user feedback, efforts to explore user perspectives in depth remain absent. Without these explorations, it is difficult to interpret current facilitators and barriers\\u003csup\\u003e\\u003cspan citationid=\\\"CR45\\\" class=\\\"CitationRef\\\"\\u003e45\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR56\\\" class=\\\"CitationRef\\\"\\u003e56\\u003c/span\\u003e\\u003c/sup\\u003e or reconcile conflicting findings, such as disagreements among evaluators\\u003csup\\u003e\\u003cspan citationid=\\\"CR38\\\" class=\\\"CitationRef\\\"\\u003e38\\u003c/span\\u003e\\u003c/sup\\u003e and the misalignment between positive perceptions and low real-world utilization. Differences in model performance and user perceptions across demographic subgroups were also largely unexplored\\u003csup\\u003e\\u003cspan citationid=\\\"CR38\\\" class=\\\"CitationRef\\\"\\u003e38\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR39\\\" class=\\\"CitationRef\\\"\\u003e39\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR41\\\" class=\\\"CitationRef\\\"\\u003e41\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR42\\\" class=\\\"CitationRef\\\"\\u003e42\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR45\\\" class=\\\"CitationRef\\\"\\u003e45\\u003c/span\\u003e\\u003c/sup\\u003e.\\u003c/p\\u003e \\u003cp\\u003eSeveral studies highlighted important ethical and legal considerations surrounding the responsible integration of GenAI into patient\\u0026ndash;provider messaging. Transparency\\u0026mdash;specifically whether and how to disclose AI contributions to patients\\u0026mdash;emerged as a key question to be addressed\\u003csup\\u003e\\u003cspan citationid=\\\"CR37\\\" class=\\\"CitationRef\\\"\\u003e37\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR41\\\" class=\\\"CitationRef\\\"\\u003e41\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR45\\\" class=\\\"CitationRef\\\"\\u003e45\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR56\\\" class=\\\"CitationRef\\\"\\u003e56\\u003c/span\\u003e\\u003c/sup\\u003e. Despite findings that disclosure may reduce patient satisfaction\\u003csup\\u003e\\u003cspan citationid=\\\"CR56\\\" class=\\\"CitationRef\\\"\\u003e56\\u003c/span\\u003e\\u003c/sup\\u003e, upholding ethical norms in healthcare AI requires supporting patients\\u0026rsquo; right to be informed when AI is involved in the delivery of their medical information and care\\u003csup\\u003e\\u003cspan citationid=\\\"CR37\\\" class=\\\"CitationRef\\\"\\u003e37\\u003c/span\\u003e,\\u003cspan additionalcitationids=\\\"CR57\\\" citationid=\\\"CR56\\\" class=\\\"CitationRef\\\"\\u003e56\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR58\\\" class=\\\"CitationRef\\\"\\u003e58\\u003c/span\\u003e\\u003c/sup\\u003e. Concerns on biased model training, cultural insensitivity, lack of AI attribution, and unequal accessibility also raised important liability and equity implications\\u003csup\\u003e\\u003cspan citationid=\\\"CR38\\\" class=\\\"CitationRef\\\"\\u003e38\\u003c/span\\u003e,\\u003cspan additionalcitationids=\\\"CR41 CR42\\\" citationid=\\\"CR40\\\" class=\\\"CitationRef\\\"\\u003e40\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR43\\\" class=\\\"CitationRef\\\"\\u003e43\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR45\\\" class=\\\"CitationRef\\\"\\u003e45\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR47\\\" class=\\\"CitationRef\\\"\\u003e47\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR49\\\" class=\\\"CitationRef\\\"\\u003e49\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR51\\\" class=\\\"CitationRef\\\"\\u003e51\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR52\\\" class=\\\"CitationRef\\\"\\u003e52\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR54\\\" class=\\\"CitationRef\\\"\\u003e54\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR55\\\" class=\\\"CitationRef\\\"\\u003e55\\u003c/span\\u003e\\u003c/sup\\u003e. However, current studies often flagged but rarely investigated these risks across patient subgroups. The findings underscore the need to align GenAI deployment with responsible principles from the outset across stages, addressing these issues proactively rather than reactively. Finally, in line with broader recognition in healthcare AI\\u003csup\\u003e\\u003cspan citationid=\\\"CR59\\\" class=\\\"CitationRef\\\"\\u003e59\\u003c/span\\u003e\\u003c/sup\\u003e, human oversight was consistently emphasized across studies as a key safeguard for ensuring the accountable and safe use of GenAI tools in clinical settings\\u003csup\\u003e\\u003cspan citationid=\\\"CR40\\\" class=\\\"CitationRef\\\"\\u003e40\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR42\\\" class=\\\"CitationRef\\\"\\u003e42\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR45\\\" class=\\\"CitationRef\\\"\\u003e45\\u003c/span\\u003e,\\u003cspan additionalcitationids=\\\"CR48 CR49\\\" citationid=\\\"CR47\\\" class=\\\"CitationRef\\\"\\u003e47\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR50\\\" class=\\\"CitationRef\\\"\\u003e50\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR53\\\" class=\\\"CitationRef\\\"\\u003e53\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR54\\\" class=\\\"CitationRef\\\"\\u003e54\\u003c/span\\u003e\\u003c/sup\\u003e.\\u003c/p\\u003e \\u003cp\\u003eGiven the limitations and implications surfaced in early research, this emerging area presents substantial opportunities for advancement. Future research should build on the efforts of Hong et al.\\u003csup\\u003e\\u003cspan citationid=\\\"CR40\\\" class=\\\"CitationRef\\\"\\u003e40\\u003c/span\\u003e\\u003c/sup\\u003e to continuously refine standardized evaluation frameworks that incorporate both human and computational assessments. A core set of validated and scalable measures would enable more reliable benchmarking, improve reproducibility, and inform best evaluation practices across settings and specialties. While the need for standardization is clear, it is also important to acknowledge the subjective nature of patient\\u0026ndash;provider communication. Future research should explore personalized GenAI prompting to accommodate individual variation and improve clinical relevance\\u003csup\\u003e\\u003cspan citationid=\\\"CR31\\\" class=\\\"CitationRef\\\"\\u003e31\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR32\\\" class=\\\"CitationRef\\\"\\u003e32\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR38\\\" class=\\\"CitationRef\\\"\\u003e38\\u003c/span\\u003e\\u003c/sup\\u003e. Moreover, as GenAI tools become more embedded in routine clinical workflows, the field would benefit from more longitudinal and multi-center trials to enhance the reliability and generalizability of outcomes\\u003csup\\u003e\\u003cspan citationid=\\\"CR31\\\" class=\\\"CitationRef\\\"\\u003e31\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR32\\\" class=\\\"CitationRef\\\"\\u003e32\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR54\\\" class=\\\"CitationRef\\\"\\u003e54\\u003c/span\\u003e\\u003c/sup\\u003e. Future work should include longer-term evaluations that track GenAI impact on reply quality, clinician burnout, patient satisfaction, and clinical outcomes over time to better understand its real-world implications in care delivery. In parallel, greater attention is needed to examine how GenAI systems access and incorporate clinical information from the EHR\\u003csup\\u003e\\u003cspan citationid=\\\"CR31\\\" class=\\\"CitationRef\\\"\\u003e31\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR42\\\" class=\\\"CitationRef\\\"\\u003e42\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR43\\\" class=\\\"CitationRef\\\"\\u003e43\\u003c/span\\u003e\\u003c/sup\\u003e. Studies should also explore how this process might be customized by clinical specialty or patient group to improve draft relevance.\\u003c/p\\u003e \\u003cp\\u003ePatient perspectives should also be prioritized in future research\\u003csup\\u003e\\u003cspan citationid=\\\"CR31\\\" class=\\\"CitationRef\\\"\\u003e31\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR32\\\" class=\\\"CitationRef\\\"\\u003e32\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR36\\\" class=\\\"CitationRef\\\"\\u003e36\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR38\\\" class=\\\"CitationRef\\\"\\u003e38\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR41\\\" class=\\\"CitationRef\\\"\\u003e41\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR43\\\" class=\\\"CitationRef\\\"\\u003e43\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR49\\\" class=\\\"CitationRef\\\"\\u003e49\\u003c/span\\u003e\\u003c/sup\\u003e. While patients are the recipients of AI-generated replies, few studies have directly evaluated their experiences. The linguistic complexity of GenAI outputs may be manageable for clinicians but burdensome for patients with limited health or English literacy\\u003csup\\u003e\\u003cspan citationid=\\\"CR38\\\" class=\\\"CitationRef\\\"\\u003e38\\u003c/span\\u003e\\u003c/sup\\u003e. Future studies should assess patient comprehension, preferences, and satisfaction with GenAI-assisted replies and involve patients directly in the co-design of prompt engineering, disclosure strategies, and usability evaluations. Additionally, a few studies reported that GenAI did not generate drafts for all patient messages during pilot deployments\\u003csup\\u003e\\u003cspan citationid=\\\"CR32\\\" class=\\\"CitationRef\\\"\\u003e32\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR39\\\" class=\\\"CitationRef\\\"\\u003e39\\u003c/span\\u003e\\u003c/sup\\u003e. How the absence of certain drafts may lead to inconsistency in communication style and affect patient experience or trust warrants further investigation. Addressing fairness and mitigating model bias will also be essential to ensure GenAI systems function equitably across diverse patient populations\\u003csup\\u003e\\u003cspan citationid=\\\"CR41\\\" class=\\\"CitationRef\\\"\\u003e41\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR60\\\" class=\\\"CitationRef\\\"\\u003e60\\u003c/span\\u003e\\u003c/sup\\u003e. To promote ethical and responsible GenAI applications, these considerations should be prioritized and embedded early and across all stages of design, development, evaluation, and implementation.\\u003c/p\\u003e \\u003cp\\u003eUser training represents another critical area for future inquiry. Despite the recognized benefit of human\\u0026ndash;AI collaboration, clinicians currently work with GenAI drafts with limited guidance or support\\u003csup\\u003e\\u003cspan citationid=\\\"CR31\\\" class=\\\"CitationRef\\\"\\u003e31\\u003c/span\\u003e\\u003c/sup\\u003e. Targeted training programs are needed to help clinicians understand GenAI\\u0026rsquo;s capabilities and limitations\\u003csup\\u003e\\u003cspan citationid=\\\"CR55\\\" class=\\\"CitationRef\\\"\\u003e55\\u003c/span\\u003e\\u003c/sup\\u003e, enabling them to make informed edits, prevent over-reliance, and ensure adequate oversight. At the same time, researchers, health system leaders, and policymakers should work to establish clear governance frameworks for AI-assisted messaging in response to ongoing calls for responsible AI in healthcare\\u003csup\\u003e\\u003cspan citationid=\\\"CR61\\\" class=\\\"CitationRef\\\"\\u003e61\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR62\\\" class=\\\"CitationRef\\\"\\u003e62\\u003c/span\\u003e\\u003c/sup\\u003e. This includes addressing automation bias\\u003csup\\u003e\\u003cspan citationid=\\\"CR41\\\" class=\\\"CitationRef\\\"\\u003e41\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR43\\\" class=\\\"CitationRef\\\"\\u003e43\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR53\\\" class=\\\"CitationRef\\\"\\u003e53\\u003c/span\\u003e\\u003c/sup\\u003e, which can influence provider behavior, judgment, and decision-making, and ultimately affect patient outcomes. Additionally, the growing prevalence of billing for patient portal messaging may further complicate the integration of GenAI into patient communication\\u003csup\\u003e\\u003cspan citationid=\\\"CR21\\\" class=\\\"CitationRef\\\"\\u003e21\\u003c/span\\u003e\\u003c/sup\\u003e. This makes the need for guidelines and oversight even more pressing to safeguard patient satisfaction and trust. Institutions must develop clear policies on AI disclosure, informed consent, and data privacy to ensure transparency and uphold ethical standards in digitally mediated care.\\u003c/p\\u003e \\u003cp\\u003eThis review contributes to the growing body of knowledge on using GenAI to respond to patient questions and inquiries, particularly within the EHR context, where it integrates medical records to assist patient\\u0026ndash;provider communication as part of clinical care. The synthesis presented here can be compared with existing reviews that examine GenAI as a tool for patient and medical education\\u003csup\\u003e\\u003cspan citationid=\\\"CR25\\\" class=\\\"CitationRef\\\"\\u003e25\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR34\\\" class=\\\"CitationRef\\\"\\u003e34\\u003c/span\\u003e\\u003c/sup\\u003e, as well as literature investigating its use as a medical chatbot\\u003csup\\u003e\\u003cspan citationid=\\\"CR63\\\" class=\\\"CitationRef\\\"\\u003e63\\u003c/span\\u003e\\u003c/sup\\u003e. These comparisons help reveal the distinct benefits and challenges of GenAI applications across various health information-seeking settings, and underscore the differences between viewing GenAI as a supportive assistant versus as an independent agent. Such distinctions may guide researchers and clinical stakeholders in further identifying appropriate directions and priorities. Additionally, as recent work has also begun to examine the use of LLMs to assist patients in writing efficient messages to their healthcare providers\\u003csup\\u003e\\u003cspan citationid=\\\"CR64\\\" class=\\\"CitationRef\\\"\\u003e64\\u003c/span\\u003e\\u003c/sup\\u003e, it is important to consider how AI\\u0026rsquo;s presence on both sides of the communication may create interactive effects and reshape the patient\\u0026ndash;provider relationship.\\u003c/p\\u003e\"},{\"header\":\"Limitations\",\"content\":\"\\u003cp\\u003eAll studies included in this review were conducted in English and based in the United States, which limits the generalizability of our findings and does not reflect global efforts relevant to this topic. Generative AI applications in patient communication may differ substantially across countries due to variations in language, digital infrastructure, and regulatory environments. As such, this review may primarily inform contexts with similar health system characteristics. During the screening process, we also identified early studies that evaluated AI-generated replies using public patient inquiries from platforms such as social media or institutional websites, conducted prior to the availability of HIPAA-compliant tools\\u003csup\\u003e\\u003cspan citationid=\\\"CR27\\\" class=\\\"CitationRef\\\"\\u003e27\\u003c/span\\u003e\\u003c/sup\\u003e. While these studies provided useful early insights into the feasibility of GenAI-assisted messaging, we excluded them to maintain a focus on evaluations situated within health system or EHR-integrated settings. We were also unable to conduct a formal quality assessment of the included studies due to the exploratory nature and limited methodological detail in many reports. Despite these limitations, this review offers an important synthesis of empirical evaluations and highlights emerging opportunities and challenges in the integration of GenAI into patient\\u0026ndash;provider communication via portal messaging, addressing the growing interest in this rapidly evolving field.\\u003c/p\\u003e\"},{\"header\":\"Conclusions\",\"content\":\"\\u003cp\\u003eThis review systematically synthesizes early studies exploring the use of GenAI to draft responses to patient messages, highlighting the promising quality of AI-generated replies and the positive reception from both clinicians and patients. In addition to these encouraging findings, we also identified key limitations in the current evidence base, along with persistent risks and challenges to the effective and safe integration of GenAI into real-world clinical workflows. As these technologies continue to evolve, it is critical to establish shared evaluation standards, develop practical guidelines for disclosure and oversight, and engage diverse stakeholders in shaping responsible implementation. Our findings offer timely insights for health system researchers, leaders, and policymakers aiming to leverage GenAI as a novel tool to address clinician in-basket overload and enhance patient–provider communication via portal messaging.\\u003c/p\\u003e\"},{\"header\":\"Methods\",\"content\":\"\\u003cp\\u003eThis review followed the 2020 Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)\\u003csup\\u003e\\u003cspan citationid=\\\"CR65\\\" class=\\\"CitationRef\\\"\\u003e65\\u003c/span\\u003e\\u003c/sup\\u003e guideline for systematic reviews without meta-analysis to ensure methodological transparency and reproducibility.\\u003c/p\\u003e\\u003ch2\\u003eInclusion and Exclusion Criteria\\u003c/h2\\u003e\\u003cp\\u003eThis review focuses on peer-reviewed original research articles that explore the use of GenAI to draft replies to patient messages within the EHR context. Eligible studies were required to present empirical findings from implementations, evaluations, or stakeholder perspectives. As this is an emerging area of research, we also included short reports such as brief communications, research letters, and perspective papers with empirical results. Studies were excluded if they were not published in English or focused solely on non-English outputs. We also excluded studies evaluating GenAI responses to general frequently asked patient questions on institutional websites or to inquiries posted on public platforms outside the EHR or patient portal setting. Patient-facing chatbots developed as standalone health assistants were also not considered. Lastly, we excluded posters and abstracts due to incomplete reporting, along with non-empirical articles such as editorials, opinion pieces, and system design descriptions.\\u003c/p\\u003e\\u003ch2\\u003eSearch Strategy\\u003c/h2\\u003e\\u003cp\\u003eA comprehensive literature search was performed on April 5, 2025, using five electronic databases: PubMed, Web of Science, Scopus, IEEE Xplore, and the ACM Digital Library. The search targeted metadata fields (title, abstract, and keywords) and was tailored to each database’s syntax. Search terms included \\u003cb\\u003egenerative artificial intelligence\\u003c/b\\u003e, \\u003cb\\u003epatient\\u003c/b\\u003e, \\u003cb\\u003emessage\\u003c/b\\u003e, and \\u003cb\\u003eresponse\\u003c/b\\u003e as well as their synonyms and variants. To maximize retrieval sensitivity, we also included the terms \\u003cb\\u003equestion\\u003c/b\\u003e and \\u003cb\\u003einquiry\\u003c/b\\u003e, which may be used in place of \\u003cb\\u003emessage\\u003c/b\\u003e in this context, to capture studies evaluating GenAI responses to patient messages described with different terminology. A complete list of searching terms are provided in Table\\u0026nbsp;\\u003cspan refid=\\\"Tab3\\\" class=\\\"InternalRef\\\"\\u003e2\\u003c/span\\u003e.\\u003c/p\\u003e\\u003cp\\u003e \\u003c/p\\u003e\\u003cdiv class=\\\"gridtable\\\"\\u003e\\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c1\\\" colnum=\\\"1\\\"\\u003e\\u003c/div\\u003e\\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c2\\\" colnum=\\\"2\\\"\\u003e\\u003c/div\\u003e\\u003ctable float=\\\"Yes\\\" id=\\\"Tab3\\\" border=\\\"1\\\"\\u003e\\u003ccaption language=\\\"En\\\"\\u003e \\u003cdiv class=\\\"CaptionNumber\\\"\\u003eTable 2\\u003c/div\\u003e \\u003cdiv class=\\\"CaptionContent\\\"\\u003e \\u003cp\\u003eLiterature searching strategy.\\u003c/p\\u003e \\u003c/div\\u003e \\u003c/caption\\u003e\\u003ccolgroup cols=\\\"2\\\"\\u003e\\u003c/colgroup\\u003e\\u003cthead\\u003e\\u003ctr\\u003e\\u003cth align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eGenerative AI Terms\\u003c/p\\u003e \\u003c/th\\u003e\\u003cth align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e“generative artificial intelligence” OR “generative AI” OR\\u003c/p\\u003e \\u003cp\\u003e“AI-generated” OR “AI generated” OR\\u003c/p\\u003e \\u003cp\\u003e“AI-drafted” OR “AI drafted” OR\\u003c/p\\u003e \\u003cp\\u003e“artificial intelligence-generated” OR “artificial intelligence generated” OR\\u003c/p\\u003e \\u003cp\\u003e“artificial intelligence-drafted” OR “artificial intelligence drafted” OR\\u003c/p\\u003e \\u003cp\\u003e“large language model*” OR llm OR llms OR\\u003c/p\\u003e \\u003cp\\u003e“transformer model*” OR “pre-trained language model*” OR “generative pre-trained transformer*” OR chatgpt OR gpt\\u003c/p\\u003e \\u003c/th\\u003e\\u003c/tr\\u003e\\u003c/thead\\u003e\\u003ctbody\\u003e\\u003ctr\\u003e\\u003ctd align=\\\"left\\\" colspan=\\\"2\\\" nameend=\\\"c2\\\" namest=\\\"c1\\\"\\u003e \\u003cp\\u003eAND\\u003c/p\\u003e \\u003c/td\\u003e\\u003c/tr\\u003e\\u003ctr\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e\\u003cb\\u003eAction Terms\\u003c/b\\u003e\\u003c/p\\u003e \\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003erespond* OR response* OR reply* OR replies OR answer*\\u003c/p\\u003e \\u003c/td\\u003e\\u003c/tr\\u003e\\u003ctr\\u003e\\u003ctd align=\\\"left\\\" colspan=\\\"2\\\" nameend=\\\"c2\\\" namest=\\\"c1\\\"\\u003e \\u003cp\\u003eAND\\u003c/p\\u003e \\u003c/td\\u003e\\u003c/tr\\u003e\\u003ctr\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e\\u003cb\\u003eTopic Terms\\u003c/b\\u003e\\u003c/p\\u003e \\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003epatient* AND (messag* OR inquir* OR question*)\\u003c/p\\u003e \\u003c/td\\u003e\\u003c/tr\\u003e\\u003c/tbody\\u003e\\u003c/table\\u003e\\u003c/div\\u003e\\u003cp\\u003e\\u003c/p\\u003e\\u003cp\\u003eTable\\u0026nbsp;\\u003cspan refid=\\\"Tab3\\\" class=\\\"InternalRef\\\"\\u003e2\\u003c/span\\u003e lists the search terms and query logic used in the literature search of this review.\\u003c/p\\u003e\\u003ch2\\u003eStudy Screening and Data Extraction\\u003c/h2\\u003e\\u003cp\\u003eTitles and abstracts were independently screened by two of four authors (DH, YG, YZ, LF). At this stage, without reviewing full-text content, we focused on identifying studies that explored GenAI for responding to messages, inquiries, or questions from patients. Disagreements were resolved through consensus discussions involving at least two reviewers. Articles meeting these criteria were retrieved for full-text screening. Each full text was independently reviewed by two of three authors (DH, YG, and YZ), based on the predefined inclusion and exclusion criteria. Any disagreements were resolved through discussion with the senior author (KZ).\\u003c/p\\u003e\\u003cp\\u003eFollowing the screening process, a data extraction template was developed to capture study characteristics, context, objectives, design, participants, outcomes, findings, limitations, and implications. Each included study was independently coded by two reviewers (DH and YG). Discrepancies were discussed and resolved with input from the senior author (KZ), ensuring consistency and interpretive rigor across the dataset.\\u003c/p\\u003e\"},{\"header\":\"Declarations\",\"content\":\"\\u003cp\\u003e\\u003cstrong\\u003eData availability\\u003c/strong\\u003e\\u003c/p\\u003e\\n\\u003cp\\u003eAll data generated or analysed during this study are included in this article.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eAcknowledgments\\u003c/strong\\u003e\\u003c/p\\u003e\\n\\u003cp\\u003eNo specific funding support or contributions to acknowledge.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eAuthor Contributions\\u003c/strong\\u003e\\u003c/p\\u003e\\n\\u003cp\\u003eDH conceptualized the study, developed the search strategy, conducted the literature searches, led data screening and extraction, and drafted the manuscript. YG and YZ contributed to refining the inclusion criteria and participated in data screening and extraction. LF participated in the data screening and contributed to consensus discussions. KZ supervised the study, provided methodological guidance, and helped resolve discrepancies during screening and extraction. All authors reviewed and approved the final manuscript.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eCompeting Interests\\u003c/strong\\u003e\\u003c/p\\u003e\\n\\u003cp\\u003eAll authors declare no financial or non-financial competing interests.\\u003c/p\\u003e\"},{\"header\":\"References\",\"content\":\"\\u003col\\u003e\\n\\u003cli\\u003eLyles, C. R. \\u003cem\\u003eet al.\\u003c/em\\u003e Using Electronic Health Record Portals to Improve Patient Engagement: Research Priorities and Best Practices. \\u003cem\\u003eAnn. Intern. Med. \\u003c/em\\u003e\\u003cstrong\\u003e172\\u003c/strong\\u003e, S123\\u0026ndash;S129 (2020).\\u003c/li\\u003e\\n\\u003cli\\u003eLieu, T. A. \\u003cem\\u003eet al.\\u003c/em\\u003e Primary Care Physicians\\u0026rsquo; Experiences With and Strategies for Managing Electronic Messages. \\u003cem\\u003eJAMA Netw. Open \\u003c/em\\u003e\\u003cstrong\\u003e2\\u003c/strong\\u003e, e1918287 (2019).\\u003c/li\\u003e\\n\\u003cli\\u003eHaun, J. N. \\u003cem\\u003eet al.\\u003c/em\\u003e Evaluating User Experiences of the Secure Messaging Tool on the Veterans Affairs\\u0026rsquo; Patient Portal System. \\u003cem\\u003eJ. Med. Internet Res. \\u003c/em\\u003e\\u003cstrong\\u003e16\\u003c/strong\\u003e, e2976 (2014).\\u003c/li\\u003e\\n\\u003cli\\u003eSun, S., Zhou, X., Denny, J. C., Rosenbloom, T. S. \\u0026amp; Xu, H. Messaging to your doctors: understanding patient-provider communications via a portal system. in \\u003cem\\u003eProceedings of the SIGCHI Conference on Human Factors in Computing Systems\\u003c/em\\u003e 1739\\u0026ndash;1748 (ACM, Paris France, 2013). doi:10.1145/2470654.2466230.\\u003c/li\\u003e\\n\\u003cli\\u003eRights (OCR), O. for C. HITECH Act Enforcement Interim Final Rule. https://www.hhs.gov/hipaa/for-professionals/special-topics/hitech-act-enforcement-interim-final-rule/index.html (2009).\\u003c/li\\u003e\\n\\u003cli\\u003eGriffin, A., Skinner, A., Thornhill, J. \\u0026amp; Weinberger, M. Patient Portals. \\u003cem\\u003eAppl. Clin. Inform. \\u003c/em\\u003e\\u003cstrong\\u003e7\\u003c/strong\\u003e, 489\\u0026ndash;501 (2016).\\u003c/li\\u003e\\n\\u003cli\\u003eNorth, F. \\u003cem\\u003eet al.\\u003c/em\\u003e A Retrospective Analysis of Provider-to-Patient Secure Messages: How Much Are They Increasing, Who Is Doing the Work, and Is the Work Happening After Hours? \\u003cem\\u003eJMIR Med. Inform. \\u003c/em\\u003e\\u003cstrong\\u003e8\\u003c/strong\\u003e, e16521 (2020).\\u003c/li\\u003e\\n\\u003cli\\u003eHolmgren, A. J., Apathy, N. C., Crews, J. \\u0026amp; Shanafelt, T. National trends in oncology specialists\\u0026rsquo; EHR inbox work, 2019-2022. \\u003cem\\u003eJNCI J. Natl. Cancer Inst.\\u003c/em\\u003e djaf052 (2025) doi:10.1093/jnci/djaf052.\\u003c/li\\u003e\\n\\u003cli\\u003eHolmgren, A. J. \\u003cem\\u003eet al.\\u003c/em\\u003e Assessing the impact of the COVID-19 pandemic on clinician ambulatory electronic health record use. \\u003cem\\u003eJ. Am. Med. Inform. Assoc. \\u003c/em\\u003e\\u003cstrong\\u003e29\\u003c/strong\\u003e, 453\\u0026ndash;460 (2022).\\u003c/li\\u003e\\n\\u003cli\\u003eAdler-Milstein, J., Zhao, W., Willard-Grace, R., Knox, M. \\u0026amp; Grumbach, K. Electronic health records and burnout: Time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians. \\u003cem\\u003eJ. Am. Med. Inform. Assoc. JAMIA \\u003c/em\\u003e\\u003cstrong\\u003e27\\u003c/strong\\u003e, 531\\u0026ndash;538 (2020).\\u003c/li\\u003e\\n\\u003cli\\u003eAkbar, F. \\u003cem\\u003eet al.\\u003c/em\\u003e Physicians\\u0026rsquo; electronic inbox work patterns and factors associated with high inbox work duration. \\u003cem\\u003eJ. Am. Med. Inform. Assoc. JAMIA \\u003c/em\\u003e\\u003cstrong\\u003e28\\u003c/strong\\u003e, 923\\u0026ndash;930 (2021).\\u003c/li\\u003e\\n\\u003cli\\u003eArndt, B. G. \\u003cem\\u003eet al.\\u003c/em\\u003e Tethered to the EHR: Primary Care Physician Workload Assessment Using EHR Event Log Data and Time-Motion Observations. \\u003cem\\u003eAnn. Fam. Med. \\u003c/em\\u003e\\u003cstrong\\u003e15\\u003c/strong\\u003e, 419\\u0026ndash;426 (2017).\\u003c/li\\u003e\\n\\u003cli\\u003eAkbar, F. \\u003cem\\u003eet al.\\u003c/em\\u003e Physician Stress During Electronic Health Record Inbox Work: In Situ Measurement With Wearable Sensors. \\u003cem\\u003eJMIR Med. Inform. \\u003c/em\\u003e\\u003cstrong\\u003e9\\u003c/strong\\u003e, e24014 (2021).\\u003c/li\\u003e\\n\\u003cli\\u003eTai-Seale, M. \\u003cem\\u003eet al.\\u003c/em\\u003e Physicians\\u0026rsquo; Well-Being Linked To In-Basket Messages Generated By Algorithms In Electronic Health Records. \\u003cem\\u003eHealth Aff. Proj. Hope \\u003c/em\\u003e\\u003cstrong\\u003e38\\u003c/strong\\u003e, 1073\\u0026ndash;1078 (2019).\\u003c/li\\u003e\\n\\u003cli\\u003eHilliard, R. W., Haskell, J. \\u0026amp; Gardner, R. L. Are specific elements of electronic health record use associated with clinician burnout more than others? \\u003cem\\u003eJ. Am. Med. Inform. Assoc. JAMIA \\u003c/em\\u003e\\u003cstrong\\u003e27\\u003c/strong\\u003e, 1401 (2020).\\u003c/li\\u003e\\n\\u003cli\\u003eAriely, D. \\u0026amp; Lanier, W. L. Disturbing Trends in Physician Burnout and Satisfaction With Work-Life Balance: Dealing With Malady Among the Nation\\u0026rsquo;s Healers. \\u003cem\\u003eMayo Clin. Proc. \\u003c/em\\u003e\\u003cstrong\\u003e90\\u003c/strong\\u003e, 1593\\u0026ndash;1596 (2015).\\u003c/li\\u003e\\n\\u003cli\\u003eSmith, L., Kirk, W., Bennett, M. M., Youens, K. \\u0026amp; Ramm, J. From Headache to Handled: Advanced In-Basket Management System in Primary Care Clinics Reduces Provider Workload Burden and Self-Reported Burnout. \\u003cem\\u003eAppl. Clin. Inform. \\u003c/em\\u003e\\u003cstrong\\u003e15\\u003c/strong\\u003e, 869\\u0026ndash;876 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eFogg, J. F. \\u0026amp; Sinsky, C. A. In-Basket Reduction: A Multiyear Pragmatic Approach to Lessen the Work Burden of Primary Care Physicians. \\u003cem\\u003eNEJM Catal. \\u003c/em\\u003e\\u003cstrong\\u003e4\\u003c/strong\\u003e, CAT.22.0438 (2023).\\u003c/li\\u003e\\n\\u003cli\\u003eCronin, R. M., Fabbri, D., Denny, J. C., Rosenbloom, S. T. \\u0026amp; Jackson, G. P. A comparison of rule-based and machine learning approaches for classifying patient portal messages. \\u003cem\\u003eInt. J. Med. Inf. \\u003c/em\\u003e\\u003cstrong\\u003e105\\u003c/strong\\u003e, 110\\u0026ndash;120 (2017).\\u003c/li\\u003e\\n\\u003cli\\u003eYang, J. \\u003cem\\u003eet al.\\u003c/em\\u003e Development and evaluation of an artificial intelligence-based workflow for the prioritization of patient portal messages. \\u003cem\\u003eJAMIA Open \\u003c/em\\u003e\\u003cstrong\\u003e7\\u003c/strong\\u003e, ooae078 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eLiu, T., Zhu, Z., Holmgren, A. J. \\u0026amp; Ellimoottil, C. National trends in billing patient portal messages as e-visit services in traditional Medicare. \\u003cem\\u003eHealth Aff. Sch. \\u003c/em\\u003e\\u003cstrong\\u003e2\\u003c/strong\\u003e, qxae040 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eHolmgren, A. J., Grouse, C. K., Oates, A., O\\u0026rsquo;Brien, J. \\u0026amp; Byron, M. E. Changes in Secure Messaging After Implementation of Billing E-Visits by Demographic Group. \\u003cem\\u003eJAMA Netw. Open \\u003c/em\\u003e\\u003cstrong\\u003e7\\u003c/strong\\u003e, e2427053 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eMatulis, J. \\u0026amp; McCoy, R. Relief in Sight? Chatbots, In-baskets, and the Overwhelmed Primary Care Clinician. \\u003cem\\u003eJ. Gen. Intern. Med. \\u003c/em\\u003e\\u003cstrong\\u003e38\\u003c/strong\\u003e, 2808\\u0026ndash;2815 (2023).\\u003c/li\\u003e\\n\\u003cli\\u003eWaldock, W. J. \\u003cem\\u003eet al.\\u003c/em\\u003e The Accuracy and Capability of Artificial Intelligence Solutions in Health Care Examinations and Certificates: Systematic Review and Meta-Analysis. \\u003cem\\u003eJ. Med. Internet Res. \\u003c/em\\u003e\\u003cstrong\\u003e26\\u003c/strong\\u003e, e56532 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eZong, H. \\u003cem\\u003eet al.\\u003c/em\\u003e Large Language Models in Worldwide Medical Exams: Platform Development and Comprehensive Analysis. \\u003cem\\u003eJ. Med. Internet Res. \\u003c/em\\u003e\\u003cstrong\\u003e26\\u003c/strong\\u003e, e66114 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eLuo, M., Warren, C. J., Cheng, L., Abdul-Muhsin, H. M. \\u0026amp; Banerjee, I. Assessing Empathy in Large Language Models with Real-World Physician-Patient Interactions. in \\u003cem\\u003e2024 IEEE International Conference on Big Data (BigData)\\u003c/em\\u003e 6510\\u0026ndash;6519 (2024). doi:10.1109/BigData62323.2024.10825307.\\u003c/li\\u003e\\n\\u003cli\\u003eAyers, J. W. \\u003cem\\u003eet al.\\u003c/em\\u003e Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. \\u003cem\\u003eJAMA Intern. Med. \\u003c/em\\u003e\\u003cstrong\\u003e183\\u003c/strong\\u003e, 589\\u0026ndash;596 (2023).\\u003c/li\\u003e\\n\\u003cli\\u003eArmitage, H. AI assists clinicians in responding to patient messages at Stanford Medicine. \\u003cem\\u003eNews Center \\u003c/em\\u003ehttps://med.stanford.edu/news/all-news/2024/03/ai-patient-messages.html (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eIntroducing Dr. Chatbot. https://today.ucsd.edu/story/introducing-dr-chatbot.\\u003c/li\\u003e\\n\\u003cli\\u003eGen AI Saves Nurses Time by Drafting Responses to Patient Messages. \\u003cem\\u003eEpicShare \\u003c/em\\u003ehttps://www.epicshare.org/share-and-learn/mayo-ai-message-responses.\\u003c/li\\u003e\\n\\u003cli\\u003eGarcia, P. \\u003cem\\u003eet al.\\u003c/em\\u003e Artificial Intelligence\\u0026ndash;Generated Draft Replies to Patient Inbox Messages. \\u003cem\\u003eJAMA Netw. Open \\u003c/em\\u003e\\u003cstrong\\u003e7\\u003c/strong\\u003e, e243201 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eTai-Seale, M. \\u003cem\\u003eet al.\\u003c/em\\u003e AI-Generated Draft Replies Integrated Into Health Records and Physicians\\u0026rsquo; Electronic Communication. \\u003cem\\u003eJAMA Netw. Open \\u003c/em\\u003e\\u003cstrong\\u003e7\\u003c/strong\\u003e, e246565 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eSingh, G. B., Kumar, R., Ghosh, R. C., Bhakhuni, P. \\u0026amp; Sharma, N. Question Answering in Medical Domain Using Natural Language Processing: A Review. in \\u003cem\\u003eData Management, Analytics and Innovation\\u003c/em\\u003e (eds. Sharma, N., Goje, A. C., Chakrabarti, A. \\u0026amp; Bruckstein, A. M.) 385\\u0026ndash;397 (Springer Nature, Singapore, 2024). doi:10.1007/978-981-97-3245-6_26.\\u003c/li\\u003e\\n\\u003cli\\u003eAydin, S., Karabacak, M., Vlachos, V. \\u0026amp; Margetis, K. Large language models in patient education: a scoping review of applications in medicine. \\u003cem\\u003eFront. Med. \\u003c/em\\u003e\\u003cstrong\\u003e11\\u003c/strong\\u003e, (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eWei, Q. \\u003cem\\u003eet al.\\u003c/em\\u003e Evaluation of ChatGPT-generated medical responses: A systematic review and meta-analysis. \\u003cem\\u003eJ. Biomed. Inform. \\u003c/em\\u003e\\u003cstrong\\u003e151\\u003c/strong\\u003e, 104620 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eAfshar, M. \\u003cem\\u003eet al.\\u003c/em\\u003e Prompt engineering with a large language model to assist providers in responding to patient inquiries: a real-time implementation in the electronic health record. \\u003cem\\u003eJAMIA Open \\u003c/em\\u003e\\u003cstrong\\u003e7\\u003c/strong\\u003e, ooae080 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eEnglish, E., Laughlin, J., Sippel, J., DeCamp, M. \\u0026amp; Lin, C.-T. Utility of Artificial Intelligence\\u0026ndash;Generative Draft Replies to Patient Messages. \\u003cem\\u003eJAMA Netw. Open \\u003c/em\\u003e\\u003cstrong\\u003e7\\u003c/strong\\u003e, e2438573 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eSmall, W. R. \\u003cem\\u003eet al.\\u003c/em\\u003e Large Language Model\\u0026ndash;Based Responses to Patients\\u0026rsquo; In-Basket Messages. \\u003cem\\u003eJAMA Netw. Open \\u003c/em\\u003e\\u003cstrong\\u003e7\\u003c/strong\\u003e, e2422399 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eYan, S. \\u003cem\\u003eet al.\\u003c/em\\u003e Prompt engineering on leveraging large language models in generating response to InBasket messages. \\u003cem\\u003eJ. Am. Med. Inform. Assoc. \\u003c/em\\u003e\\u003cstrong\\u003e31\\u003c/strong\\u003e, 2263\\u0026ndash;2270 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eHong, C. \\u003cem\\u003eet al.\\u003c/em\\u003e Application of unified health large language model evaluation framework to In-Basket message replies: bridging qualitative and quantitative assessments. \\u003cem\\u003eJ. Am. Med. Inform. Assoc.\\u003c/em\\u003e ocaf023 (2025) doi:10.1093/jamia/ocaf023.\\u003c/li\\u003e\\n\\u003cli\\u003eNov, O., Singh, N. \\u0026amp; Mann, D. Putting ChatGPT\\u0026rsquo;s Medical Advice to the (Turing) Test: Survey Study. \\u003cem\\u003eJMIR Med. Educ. \\u003c/em\\u003e\\u003cstrong\\u003e9\\u003c/strong\\u003e, e46939 (2023).\\u003c/li\\u003e\\n\\u003cli\\u003eBaxter, S. L., Longhurst, C. A., Millen, M., Sitapati, A. M. \\u0026amp; Tai-Seale, M. Generative artificial intelligence responses to patient messages in the electronic health record: early lessons learned. \\u003cem\\u003eJAMIA Open \\u003c/em\\u003e\\u003cstrong\\u003e7\\u003c/strong\\u003e, ooae028 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eLiu, S. \\u003cem\\u003eet al.\\u003c/em\\u003e Leveraging large language models for generating responses to patient messages\\u0026mdash;a subjective analysis. \\u003cem\\u003eJ. Am. Med. Inform. Assoc. \\u003c/em\\u003e\\u003cstrong\\u003e31\\u003c/strong\\u003e, 1367\\u0026ndash;1379 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eScott, M. \\u003cem\\u003eet al.\\u003c/em\\u003e Assessing Artificial Intelligence\\u0026ndash;Generated Responses to Urology Patient In-Basket Messages. \\u003cem\\u003eUrol. Pract. \\u003c/em\\u003e\\u003cstrong\\u003e11\\u003c/strong\\u003e, 793\\u0026ndash;798 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eRobinson, E. J. \\u003cem\\u003eet al.\\u003c/em\\u003e Physician vs. AI-generated messages in urology: evaluation of accuracy, completeness, and preference by patients and physicians. \\u003cem\\u003eWorld J. Urol. \\u003c/em\\u003e\\u003cstrong\\u003e43\\u003c/strong\\u003e, 48 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eReynolds, K. \\u003cem\\u003eet al.\\u003c/em\\u003e Comparing the quality of ChatGPT- and physician-generated responses to patients\\u0026rsquo; dermatology questions in the electronic medical record. \\u003cem\\u003eClin. Exp. Dermatol. \\u003c/em\\u003e\\u003cstrong\\u003e49\\u003c/strong\\u003e, 715\\u0026ndash;718 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eSoroudi, D. \\u003cem\\u003eet al.\\u003c/em\\u003e Comparing Provider and ChatGPT Responses to Breast Reconstruction Patient Questions in the Electronic Health Record. \\u003cem\\u003eAnn. Plast. Surg. \\u003c/em\\u003e\\u003cstrong\\u003e93\\u003c/strong\\u003e, 541 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eHao, Y. \\u003cem\\u003eet al.\\u003c/em\\u003e Retrospective Comparative Analysis of Prostate Cancer In-Basket Messages: Responses From Closed-Domain Large Language Models Versus Clinical Teams. \\u003cem\\u003eMayo Clin. Proc. Digit. Health \\u003c/em\\u003e\\u003cstrong\\u003e3\\u003c/strong\\u003e, (2025).\\u003c/li\\u003e\\n\\u003cli\\u003eKaur, A., Budko, A., Liu, K., Steitz, B. \\u0026amp; Johnson, K. Primary Care Providers Acceptance of Generative AI Responses to Patient Portal Messages. \\u003cem\\u003eAppl. Clin. Inform. \\u003c/em\\u003e\\u003cstrong\\u003e0\\u003c/strong\\u003e, (2025).\\u003c/li\\u003e\\n\\u003cli\\u003eTse, G. \\u003cem\\u003eet al.\\u003c/em\\u003e Large Language Model Responses to Adolescent Patient and Proxy Messages. \\u003cem\\u003eJAMA Pediatr. \\u003c/em\\u003e\\u003cstrong\\u003e179\\u003c/strong\\u003e, 93\\u0026ndash;94 (2025).\\u003c/li\\u003e\\n\\u003cli\\u003eKim, J. \\u003cem\\u003eet al.\\u003c/em\\u003e Perspectives on Artificial Intelligence\\u0026ndash;Generated Responses to Patient Messages. \\u003cem\\u003eJAMA Netw. Open \\u003c/em\\u003e\\u003cstrong\\u003e7\\u003c/strong\\u003e, e2438535 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eAthavale, A., Baier, J., Ross, E. \\u0026amp; Fukaya, E. The potential of chatbots in chronic venous disease patient management. \\u003cem\\u003eJVS-Vasc. Insights \\u003c/em\\u003e\\u003cstrong\\u003e1\\u003c/strong\\u003e, 100019 (2023).\\u003c/li\\u003e\\n\\u003cli\\u003eChen, S. \\u003cem\\u003eet al.\\u003c/em\\u003e The effect of using a large language model to respond to patient messages. \\u003cem\\u003eLancet Digit. Health \\u003c/em\\u003e\\u003cstrong\\u003e6\\u003c/strong\\u003e, e379\\u0026ndash;e381 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eTailor, P. D. \\u003cem\\u003eet al.\\u003c/em\\u003e A Comparative Study of Responses to Retina Questions from Either Experts, Expert-Edited Large Language Models, or Expert-Edited Large Language Models Alone. \\u003cem\\u003eOphthalmol. Sci. \\u003c/em\\u003e\\u003cstrong\\u003e4\\u003c/strong\\u003e, 100485 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eTailor, P. D. \\u003cem\\u003eet al.\\u003c/em\\u003e Appropriateness of Ophthalmology Recommendations From an Online Chat-Based Artificial Intelligence Model. \\u003cem\\u003eMayo Clin. Proc. Digit. Health \\u003c/em\\u003e\\u003cstrong\\u003e2\\u003c/strong\\u003e, 119\\u0026ndash;128 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eCavalier, J. S. \\u003cem\\u003eet al.\\u003c/em\\u003e Ethics in Patient Preferences for Artificial Intelligence\\u0026ndash;Drafted Responses to Electronic Messages. \\u003cem\\u003eJAMA Netw. Open \\u003c/em\\u003e\\u003cstrong\\u003e8\\u003c/strong\\u003e, e250449 (2025).\\u003c/li\\u003e\\n\\u003cli\\u003eJeyaraman, M., Balaji, S., Jeyaraman, N. \\u0026amp; Yadav, S. Unraveling the Ethical Enigma: Artificial Intelligence in Healthcare. \\u003cem\\u003eCureus \\u003c/em\\u003e\\u003cstrong\\u003e15\\u003c/strong\\u003e, e43262.\\u003c/li\\u003e\\n\\u003cli\\u003ePark, H. J. Patient perspectives on informed consent for medical AI: A web-based experiment. \\u003cem\\u003eDigit. Health \\u003c/em\\u003e\\u003cstrong\\u003e10\\u003c/strong\\u003e, 20552076241247938 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eDe Cremer, D. \\u0026amp; Kasparov, G. The ethical AI\\u0026mdash;paradox: why better technology needs more and not less human responsibility. \\u003cem\\u003eAI Ethics \\u003c/em\\u003e\\u003cstrong\\u003e2\\u003c/strong\\u003e, 1\\u0026ndash;4 (2022).\\u003c/li\\u003e\\n\\u003cli\\u003eWang, C. \\u003cem\\u003eet al.\\u003c/em\\u003e Ethical Considerations of Using ChatGPT in Health Care. \\u003cem\\u003eJ. Med. Internet Res. \\u003c/em\\u003e\\u003cstrong\\u003e25\\u003c/strong\\u003e, e48009 (2023).\\u003c/li\\u003e\\n\\u003cli\\u003eUpadhyay, U. \\u003cem\\u003eet al.\\u003c/em\\u003e Call for the responsible artificial intelligence in the healthcare. \\u003cem\\u003eBMJ Health Care Inform. \\u003c/em\\u003e\\u003cstrong\\u003e30\\u003c/strong\\u003e, e100920 (2023).\\u003c/li\\u003e\\n\\u003cli\\u003eTrocin, C., Mikalef, P., Papamitsiou, Z. \\u0026amp; Conboy, K. Responsible AI for Digital Health: a Synthesis and a Research Agenda. \\u003cem\\u003eInf. Syst. Front. \\u003c/em\\u003e\\u003cstrong\\u003e25\\u003c/strong\\u003e, 2139\\u0026ndash;2157 (2023).\\u003c/li\\u003e\\n\\u003cli\\u003eGiuffr\\u0026egrave;, M. \\u003cem\\u003eet al.\\u003c/em\\u003e Systematic review: The use of large language models as medical chatbots in digestive diseases. \\u003cem\\u003eAliment. Pharmacol. Ther. \\u003c/em\\u003e\\u003cstrong\\u003e60\\u003c/strong\\u003e, 144\\u0026ndash;166 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003eLiu, S. \\u003cem\\u003eet al.\\u003c/em\\u003e Using large language model to guide patients to create efficient and comprehensive clinical care message. \\u003cem\\u003eJ. Am. Med. Inform. Assoc. \\u003c/em\\u003e\\u003cstrong\\u003e31\\u003c/strong\\u003e, 1665\\u0026ndash;1670 (2024).\\u003c/li\\u003e\\n\\u003cli\\u003ePage, M. J. \\u003cem\\u003eet al.\\u003c/em\\u003e The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. (2021) doi:10.1136/bmj.n71.\\u003c/li\\u003e\\n\\u003c/ol\\u003e\"}],\"fulltextSource\":\"\",\"fullText\":\"\",\"funders\":[],\"hasAdminPriorityOnWorkflow\":false,\"hasManuscriptDocX\":true,\"hasOptedInToPreprint\":true,\"hasPassedJournalQc\":\"\",\"hasAnyPriority\":false,\"hideJournal\":false,\"highlight\":\"\",\"institution\":\"\",\"isAcceptedByJournal\":true,\"isAuthorSuppliedPdf\":false,\"isDeskRejected\":\"\",\"isHiddenFromSearch\":false,\"isInQc\":false,\"isInWorkflow\":false,\"isPdf\":false,\"isPdfUpToDate\":true,\"isWithdrawnOrRetracted\":false,\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"npj-health-systems\",\"isNatureJournal\":false,\"hasQc\":true,\"allowDirectSubmit\":false,\"externalIdentity\":\"\",\"sideBox\":\"Learn more about [npj Health Systems](https://www.nature.com/npjhealthsyst/)\",\"snPcode\":\"44401\",\"submissionUrl\":\"https://submission.springernature.com/new-submission/44401/3\",\"title\":\"npj Health Systems\",\"twitterHandle\":\"\",\"acdcEnabled\":true,\"dfaEnabled\":true,\"editorialSystem\":\"stoa\",\"reportingPortfolio\":\"NPJ\",\"inReviewEnabled\":true,\"inReviewRevisionsEnabled\":false},\"keywords\":\"\",\"lastPublishedDoi\":\"10.21203/rs.3.rs-6713507/v1\",\"lastPublishedDoiUrl\":\"https://doi.org/10.21203/rs.3.rs-6713507/v1\",\"license\":{\"name\":\"CC BY 4.0\",\"url\":\"https://creativecommons.org/licenses/by/4.0/\"},\"manuscriptAbstract\":\"\\u003cp\\u003eThis systematic review synthesizes currently available empirical evidence on generative artificial intelligence (GenAI) tools for drafting responses to patient messages. Across a total of 23 studies identified, GenAI was found to produce empathetic replies with quality comparable to that of responses drafted by human experts, demonstrating its potential to facilitate patient\\u0026ndash;provider communication and alleviate clinician burnout. Challenges include inconsistent performance, risks to patient safety, and ethical concerns around transparency and oversight. Additionally, utilization of the technology remains limited in real-world settings, and existing evaluation efforts vary greatly in study design and methodological rigor. As this field evolves, there is a critical need to establish robust and standardized evaluation frameworks, develop practical guidelines for disclosure and accountability, and meaningfully engage clinicians, patients, and other stakeholders. This review may provide timely insights into informing future research of GenAI and guiding the responsible integration of this technology into day-to-day clinical work.\\u003c/p\\u003e\",\"manuscriptTitle\":\"A Systematic Review of Early Evidence on Generative AI for Drafting Responses to Patient Messages\",\"msid\":\"\",\"msnumber\":\"\",\"nonDraftVersions\":[{\"code\":1,\"date\":\"2025-05-29 08:10:50\",\"doi\":\"10.21203/rs.3.rs-6713507/v1\",\"editorialEvents\":[{\"type\":\"communityComments\",\"content\":0},{\"type\":\"decision\",\"content\":\"Revision requested\",\"date\":\"2025-06-09T13:40:24+00:00\",\"index\":\"\",\"fulltext\":\"\"},{\"type\":\"editorInvitedReview\",\"content\":\"\",\"date\":\"2025-06-08T21:02:00+00:00\",\"index\":\"hide\",\"fulltext\":\"\"},{\"type\":\"editorInvitedReview\",\"content\":\"\",\"date\":\"2025-05-27T18:10:31+00:00\",\"index\":\"hide\",\"fulltext\":\"\"},{\"type\":\"reviewerAgreed\",\"content\":\"179227377958153825346744706253861897750\",\"date\":\"2025-05-27T12:43:33+00:00\",\"index\":\"hide\",\"fulltext\":\"\"},{\"type\":\"reviewerAgreed\",\"content\":\"260775097211866563134919709326409721007\",\"date\":\"2025-05-26T23:26:39+00:00\",\"index\":\"hide\",\"fulltext\":\"\"},{\"type\":\"reviewersInvited\",\"content\":\"\",\"date\":\"2025-05-26T22:51:46+00:00\",\"index\":\"\",\"fulltext\":\"\"},{\"type\":\"editorAssigned\",\"content\":\"\",\"date\":\"2025-05-24T00:45:09+00:00\",\"index\":\"\",\"fulltext\":\"\"},{\"type\":\"checksComplete\",\"content\":\"\",\"date\":\"2025-05-23T13:25:17+00:00\",\"index\":\"\",\"fulltext\":\"\"},{\"type\":\"submitted\",\"content\":\"npj Health Systems\",\"date\":\"2025-05-21T06:56:50+00:00\",\"index\":\"\",\"fulltext\":\"\"}],\"status\":\"published\",\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"npj-health-systems\",\"isNatureJournal\":false,\"hasQc\":true,\"allowDirectSubmit\":false,\"externalIdentity\":\"\",\"sideBox\":\"Learn more about [npj Health Systems](https://www.nature.com/npjhealthsyst/)\",\"snPcode\":\"44401\",\"submissionUrl\":\"https://submission.springernature.com/new-submission/44401/3\",\"title\":\"npj Health Systems\",\"twitterHandle\":\"\",\"acdcEnabled\":true,\"dfaEnabled\":true,\"editorialSystem\":\"stoa\",\"reportingPortfolio\":\"NPJ\",\"inReviewEnabled\":true,\"inReviewRevisionsEnabled\":false}}],\"origin\":\"\",\"ownerIdentity\":\"cb158199-3f11-4056-be23-bde6832558c1\",\"owner\":[],\"postedDate\":\"May 29th, 2025\",\"published\":true,\"recentEditorialEvents\":[],\"rejectedJournal\":[],\"revision\":\"\",\"amendment\":\"\",\"status\":\"under-review\",\"subjectAreas\":[{\"id\":49128941,\"name\":\"Business and commerce/Information systems and information technology\"},{\"id\":49128942,\"name\":\"Health sciences/Health care\"}],\"tags\":[],\"updatedAt\":\"2025-06-26T13:08:40+00:00\",\"versionOfRecord\":[],\"versionCreatedAt\":\"2025-05-29 08:10:50\",\"video\":\"\",\"vorDoi\":\"\",\"vorDoiUrl\":\"\",\"workflowStages\":[]},\"version\":\"v1\",\"identity\":\"rs-6713507\",\"journalConfig\":\"researchsquare\"},\"__N_SSP\":true},\"page\":\"/article/[identity]/[[...version]]\",\"query\":{\"redirect\":\"/article/rs-6713507\",\"identity\":\"rs-6713507\",\"version\":[\"v1\"]},\"buildId\":\"8U1c8b4HqxoKbykW_rLl7\",\"isFallback\":false,\"isExperimentalCompile\":false,\"dynamicIds\":[84888],\"gssp\":true,\"scriptLoader\":[]}","source_license":"CC-BY-4.0","license_restricted":false}