Development and Prospective Validation of CPX-MATE: An End-to-End Medical Education Platform Integrating Voice-Based Virtual Patient Simulation and Automated Real-time Evaluation

Development and Prospective Validation of CPX-MATE: An End-to-End Medical Education Platform Integrating Voice-Based Virtual Patient Simulation and Automated Real-time Evaluation | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Development and Prospective Validation of CPX-MATE: An End-to-End Medical Education Platform Integrating Voice-Based Virtual Patient Simulation and Automated Real-time Evaluation Ji Woo Song, Minseong Kim, Chanhee Hong, Young Sam Kim, Junho Cho, and 7 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8972105/v1 This work is licensed under a CC BY 4.0 License Status: Under Revision Version 1 posted 12 You are reading this latest preprint version Abstract Background Objective Structured Clinical Examination (OSCE; Clinical Performance Examination [CPX] in South Korea) is a high-stakes assessment of clinical performance, communication, and reasoning during time-limited patient encounters. As AI-enabled virtual standardized patient (VSP) simulation and automated scoring are introduced for OSCE-like training, prospective evidence is needed on how such systems perform and are perceived when embedded in real educational workflows. Methods We developed CPX with Medical students’ Assistant for Training and Evaluation (CPX-MATE), a web-based platform integrating (1) CPX with Virtual Standardized Patient (CPX-VSP), real-time voice dialogue with a VSP using speech-to-speech (STS) models, and (2) CPX with Real-Time Evaluator (CPX-RTE), automated transcription, checklist-based scoring, and feedback from encounter audio using a Speech-to-Text model and a large language model. During an emergency medicine clerkship (Nov 2025-Jan 2026), 60 senior medical students completed two 12-min CPX encounters (VSP with acute pancreatitis; HSP with ureteral stone) with immediate CPX-RTE feedback. For CPX-VSP, students were assigned to either a full-capacity or a resource-limited STS configuration (n = 30 each). Dialogue fidelity was evaluated by turn-by-turn analysis of student–VSP exchanges, classifying responses into clinically meaningful error types (tangential, oversharing, role-breaking, off-script). CPX-RTE performance was assessed by agreement (Gwet’s AC1) with professor real-time and resident video-based ratings using a 45-item checklist. Usability of CPX-VSP and CPX-RTE, with overall system usability scale (SUS), were surveyed, and mean per-session costs for CPX-VSP and CPX-RTE were calculated. Results Across 3,282 dialogue turns, overall error rates were 1.77% versus 9.43% for full-capacity versus resource-limited STS configurations (p < 0.001), driven by fewer tangential and oversharing responses; no off-script errors were observed. The mean per-session cost was $ 0.12 for resource-limited configuration and $ 0.78 for full-capacity configuration. CPX-RTE showed high agreement with human ratings (AC1 = 0.916 vs professor; 0.916 vs resident), with slightly different levels of agreement across four sections, and high usability across all domains (mean scores, 4.65–4.92), with a per-session cost of $ 0.17. CPX-MATE demonstrated good overall usability (median [IQR] of 77.5 [70.0–85.0]). Conclusions Embedded within a prospective clinical clerkship, CPX-MATE demonstrated operational fidelity and human-level checklist agreement as an end-to-end, voice-based AI-assisted OSCE platform. This real-world deployment supports its scalable integration as a complementary assessment tool while highlighting the importance of systematic validation and context-aware implementation in medical education. Biological sciences/Computational biology and bioinformatics Health sciences/Diseases Health sciences/Health care Physical sciences/Mathematics and computing Health sciences/Medical research Figures Figure 1 Figure 2 Figure 3 INTRODUCTION Objective Structured Clinical Examination (OSCE), which is called Clinical Performance Examination (CPX) in South Korea, is a core competency-based assessment designed to evaluate authentic clinical performance of medical students through standardized patient (SP) encounters. 1,2 OSCE is a high-stakes component of undergraduate medical education and licensure, assessing integrated skills such as history taking, communication, clinical reasoning, and initial management within time-limited, face-to-face interactions. Students interact directly with an SP, while evaluators observe in real time from behind a one-way mirror and score student performance using standardized checklists. Despite its strong educational validity, the educational infrastructure underpinning OSCE is inherently resource intensive. It requires substantial financial investment, complex logistical coordination, trained faculty examiners, and the recruitment, training, and compensation of human standardized patients (HSPs). 3 , 4 These demands limit opportunities for repeated practice, timely feedback, and scalable deployment. Such challenges are particularly pronounced in low- and middle-income countries (LMICs), where constrained educational budgets, faculty shortages, and limited access to trained SPs hinder the sustainable delivery of OSCE-based education. 5 , 6 Recent advances in artificial intelligence (AI), particularly large language models (LLMs) and speech-to-speech (STS) models, have generated growing interest in virtual standardized patient (VSP)-based simulations as a means to augment traditional OSCE education. Prior studies have demonstrated the technical feasibility of text-based VSPs and AI-assisted automated scoring systems. 3 , 7 – 16 However, much of the existing literature has focused on whether such systems can be built, rather than on how they should be systematically evaluated and integrated into real-world medical education. In high-stakes clinical assessment, technical performance alone is insufficient; perceived realism, learner trust, and sustained usability are equally critical determinants of educational impact. At the same time, reliance on high-capacity models raises questions about accessibility and cost equity. Another key limitation of prior AI-based OSCE solutions lies in feedback delivery. Although automated scoring modules have been developed, most were designed for retrospective evaluation rather than for real-time, formative feedback during ongoing practice. 17 – 19 Timely feedback is a core principle of experiential learning, enabling learners to identify performance gaps and adjust clinical reasoning while the learning experience remains cognitively salient. Accordingly, we developed and evaluated CPX with Medical students’ Assistant for Training and Evaluation (CPX-MATE), an end-to-end platform integrating voice-based VSP training (CPX with Virtual Standardized Patient, CPX-VSP) and real-time, voice-based evaluation applicable to HSP encounters (CPX with Real-Time Evaluator, CPX-RTE). This study aims to define and empirically examine the dimensions that should be scrutinized as such systems enter routine educational use, with a particular focus on performance fidelity, usability, and deployment within authentic educational settings. METHODS Development of CPX-MATE System Overview CPX-MATE is a web-based system comprising CPX-VSP and CPX-RTE (Fig. 1 ). CPX-VSP enables real-time voice dialogue with a VSP using two STS models ( gpt-realtime and gpt-realtime-mini ). 14 This dual-model design was adopted to examine whether a resource-limited configuration can provide acceptable performance in constrained settings within the rapidly evolving AI landscape, and to quantify the extent of performance gains achievable with a full-capacity model as AI capabilities advance. To simplify presentation, gpt-realtime is hereafter referred to as the full-capacity model, whereas gpt-realtime-mini is referred to as the resource-limited model. CPX-RTE transcribes HSP encounters using speech-to-text (STT) model ( whisper-1 ) 20 , maps transcript evidence to predefined checklist items and generates feedback using an LLM ( GPT-5 ) 21 . CPX-VSP: prompts and interaction rules Two prompts were provided to the STS model: (1) a global SP persona prompt, which was identical across all chief complaints and scenarios, specifying role/tone/constraints and dialogue rules (e.g., disclose information only when asked; answer unspecified questions plausibly without contradicting the scenario; if the student announces a physical exam, respond with corresponding findings), and (2) a scenario-specific prompt which varied by clinical scenario and contained the full clinical case specification (symptoms, history, and physical exam findings). The SP persona prompt was designed as a reusable, chief-complaint–agnostic control layer, whereas the scenario prompt served as a case-specific content layer. Although only a single scenario prompt was used in the present study, this separation was intentionally adopted to ensure scalability to multiple chief complaints and scenarios. Prompts were iteratively refined by an emergency medicine faculty member (CP) and a medical student investigator (JWS). CPX-RTE: scoring and feedback CPX-RTE follows a pipeline consisting of (1) audio transcription, (2) structured checklist-based feedback generation, and (3) narrative feedback generation. The student–patient dialogue is first transcribed using an STT model. For each checklist item, an LLM then reviews the transcript to identify utterances that match a criterion of the item and applies a predefined binary decision rule, classifying the item as “Yes” when one or more relevant utterances are detected and as “No” otherwise. These item-level decisions, together with the matched transcript excerpts, form the structured feedback. Finally, the LLM generates narrative feedback summarizing strengths and areas for improvement based on both the transcript and the structured outputs. Clinical scenarios and checklist Two acute abdominal pain presentations were used as representative, generalizable emergency cases: VSP (CPX-VSP) acute pancreatitis and HSP ureteral stone. Performance was evaluated using a 45-item acute abdominal pain checklist: History Taking (20), Physical Examination (8), Patient Education (6), and Patient–Physician Interaction (11), developed with reference to the Korea Association of Medical Colleges’ The guide to clinical performance, 2nd edition. 22 Validation of CPX-MATE Study design, setting, and participants The study was conducted during an emergency medicine clerkship at Yonsei University College of Medicine between November 2025 and January 2026 and involved year 3–4 medical students. Each participant completed a standardized CPX experience consisting of two clinical encounters followed by a post-session survey. Participants first engaged in a 12-minute CPX encounter with a VSP, during which CPX-VSP was delivered using either a resource-limited STS model (n = 30, first half of the study period) or a full-capacity STS model (n = 30, second half). CPX-RTE generated real-time checklist-based feedback that was shown to students for formative learning during this encounter, but these outputs were not used for performance validation as no independent human scoring was performed for VSP encounters. After a 5-minute intermission, participants completed a second 12-minute CPX encounter with an HSP. These HSP encounters were audio-recorded for automated scoring by CPX-RTE, while video recording was performed as part of routine CPX operations to enable independent human rating. HSP performance was scored in real time by a board-certified emergency medicine attending physician (CP) and later independently scored by an emergency medicine resident (CH) using the same 45-item checklist. CPX-RTE scoring results were also generated and presented to students immediately after the encounter. Following completion of both encounters, participants completed post-session surveys assessing the usability and perceived realism of CPX-VSP, the quality of CPX-RTE feedback, and overall system usability, along with open-ended questions for qualitative feedback. Data collection Collected data included CPX-VSP transcripts and system logs (e.g., turns, duration, token usage), HSP audio (and video for human rating), and post-session surveys (Likert-scale items and free-text responses). All data were de-identified before analysis. Mean per-session costs for CPX-VSP and CPX-RTE were captured through OpenAI’s dashboard. Outcome Measures CPX-VSP performance was assessed at the Minimum Interaction Unit (MIU) level, defined as one student utterance paired with the corresponding VSP response. Each MIU was classified into one of four error types: tangential (plausible content misaligned with question intent), oversharing (unsolicited disclosure beyond what was asked), role-breaking (the VSP speaking as if it were the clinician), and off-script (responses contradicting the predefined scenario prompt). All MIUs were manually reviewed and coded by a medical student investigator (JWS). The primary performance metric was the overall error rate per MIU for each STS model; secondary metrics were error rates per MIU stratified by error type. CPX-VSP usability was assessed using post-session surveys adapted from Cook et al. on a 1-to-6 Likert scale (1 = strongly disagree, 6 = strongly agree). Survey items evaluated two domains: (1) dialogue naturalness (Humanlike, Coherent, Clinically relevant, Dialogue overall) and (2) user experience(UX)/immersion (Realness, Cognitive authenticity, Variability, Involvement, UX overall). One Cook et al. item assessing whether the patient’s “personal preferences” were reflected was excluded to preserve construct alignment with the target assessment context. In the Korean undergraduate CPX setting, checklist-based scoring typically emphasizes standardized history taking, clinical reasoning, and initial management/education under time constraints rather than individualized preference elicitation as a primary learning objective. Accordingly, we focused the usability instrument on dialogue naturalness constructs most relevant to CPX preparation. CPX-RTE performance was evaluated using Gwet’s AC1 for binary checklist items. AC1 was used since checklist items showed marked prevalence imbalance (some items were almost always performed, whereas others were rarely performed), a setting in which kappa-type coefficients can be unstable and underestimate agreement. AC1 is well-known for providing a robust agreement estimate under such imbalanced marginal distributions. The board certified emergency medicine attending physician (CP) scored the encounter in real time using the same 45-item acute abdominal pain checklist, while a second rater (CH, emergency medicine resident) independently scored the video-recorded encounter. Agreement was evaluated across three rater pairs: CPX-RTE vs attending, CPX-RTE vs resident, and attending vs resident. The primary outcome was overall agreement across all 45 checklist items for each pair, with secondary analyses examining section-level agreement (History Taking, Physical Examination, Patient Education, and Patient–Physician Interaction) and item-level agreement for each checklist item. CPX-RTE usability was assessed using post-session surveys adapted from Cook et al. on a 1-to-6 Likert scale across four constructs: Evidence-based, Actionable, Connected, and Balanced. Participants were asked whether they observed biased or stereotyped content (e.g., related to sex, age, race/ethnicity, or socioeconomic status) in either CPX-VSP dialogue or CPX-RTE feedback, and to describe the bias if present. Overall web application usability was assessed using the System Usability Scale (SUS). 23 To promote objective and consistent responses, surveys assessing CPX-VSP realism and user experience and CPX-RTE feedback quality were administered using 1–6 Likert scales accompanied by explicit operational criteria defining each response level. Participants also answered open-ended questions on factors that enhanced or hindered perceived authenticity of CPX-VSP, and on areas for improvement of CPX-RTE. These free-text responses were analyzed using inductive thematic analysis framework suggested by Braun and Clarke. 24 The verbatim survey items are provided in Supplementary Method 1. Statistical analysis Analyses were conducted using a complete-case approach. For CPX-VSP performance, errors were defined at the MIU level. Overall error rates and error rates by subtype were calculated per STS model. CPX-VSP usability was summarized as mean (95% confidence interval [CI]) and full frequency distributions (n, %). Participant-level cluster bootstrap resampling (1,000 iterations) was used to obtain 95% CIs and p-values. Since CPX-VSP performance and usability analysis compared two difference STS models, all multiple comparisons for CPX-VSP were adjusted using the Benjamini–Hochberg false discovery rate procedure. All tests were two-sided with a significance threshold of α = 0.05. For CPX-RTE performance, agreement on binary checklist items was assessed using Gwet’s AC1 to account for marginal imbalance. CPX-RTE usability items were reported as mean (95% CI) and full frequency distributions (n, %). Similarly, participant-level cluster bootstrap resampling (1,000 iterations) was used to obtain 95% CIs. Analyses were performed using Python (ver 3.14.2; Python Software Foundation). Ethics The study was approved by the IRB of Yonsei University College of Medicine (IRB No. [4-2025-1312]). Participation was voluntary, uncompensated, and had no impact on academic evaluation; all recordings/transcripts were de-identified. RESULTS Participants Sixty senior medical students (years 3–4) completed the sequential two-station CPX sessions between November 2025 and January 2026. Participants had a mean age of 26.9 years (SD 0.76) and included 28 female (46.7%) and 32 male (53.3%) students. CPX-VSP was delivered using the resource-limited model (n = 30) or the full-capacity model (n = 30). Detailed demographics are shown in Table 1 . Table 1 Participant demographics by model group Resource-limited model Participants N Male N (%) Age, years mean (SD) 3rd grade N (%) 30 18 (60.0%) 27.2 (0.85) 4 (13.3%) Full-capacity model 30 14 (46.7%) 26.5 (0.35) 16 (53.3%) Total 60 32 (53.3%) 26.9 (0.76) 20 (33.3%) Baseline demographic characteristics of participants stratified by model group. Age is presented as mean (standard deviation), and categorical variables are presented as number (percentage). Performance of CPX-VSP: Dialogue Error Rates per MIU Across CPX-VSP sessions, 3,282 Minimum Interaction Units (MIUs) were identified and manually reviewed (resource-limited model: 1,591 MIUs; full-capacity model: 1,691 MIUs). Figure 2 shows the dialogue error rates per MIU of each model. The overall MIU error rate was higher with the resource-limited model than with the full-capacity model (9.43% [95% CI 7.50–11.25] vs 1.77% [0.80–2.84]), yielding an absolute difference of 7.65 percentage points (resource-limited − full-capacity; 95% CI 5.34–9.74; p < 0.001) (Fig. 1 ). Error subtype analyses showed significantly higher tangential and oversharing error rates for the resource-limited model (tangential 5.28% [3.97–6.69] vs 1.48% [0.66–2.40], Δ = 3.80 pp [2.23–5.40], p < 0.001; oversharing 4.02% [2.86–5.33] vs 0.24% [0.06–0.46], Δ = 3.79 pp [2.60–5.09], p < 0.001). Role-breaking was rare in both groups (0.13% vs 0.06%, Δ = 0.07 pp [− 0.12 to 0.27], p = 0.485), and off-script errors were not observed. Usability of CPX-VSP: Dialogue Naturalness and User Experience As shown in Fig. 3 , the full-capacity model received higher ratings than the resource-limited model for Coherent (4.77 [95% CI 4.43–5.13] vs 4.03 [3.67–4.43]; p = 0.034), Involvement (4.37 [4.03–4.70] vs 3.40 [2.87–3.90]; p = 0.019), and UX overall (4.27 [3.90–4.60] vs 3.27 [2.83–3.63]; p = 0.010). While differences in other usability items did not reach statistical significance, ratings consistently favored the full-capacity model over the resource-limited model across all measures. Detailed results are shown in Supplementary Table S1 . Open-ended responses aligned with quantitative findings. Students most frequently attributed authenticity to content-driven immersion, particularly coherent, clinically plausible, and scenario-consistent responses (Supplementary Table S2 ). Loss of immersion was primarily linked to content-level inconsistencies, including tangential responses and oversharing (Supplementary Table S3 ). Students also emphasized interactional realism (e.g., patient-like tone and affective responsiveness), consistent with higher involvement and overall UX in the full-model group. Once major content- and interaction-level issues were reduced in the full-capacity model, secondary constraints (e.g., latency and screen-mediated awkwardness) became more prominent in qualitative comments. Performance of CPX-RTE: Agreement with Human Ratings Checklist scoring by CPX-RTE was compared with professor real-time ratings and resident video-based ratings using Gwet’s AC1 (Table 2 ). Overall agreement was 0.916 (95% CI 0.902–0.930) for CPX-RTE vs professor, 0.916 (0.903–0.929) for CPX-RTE vs resident, and 0.976 (0.968–0.983) for professor vs resident. Section-level agreement between CPX-RTE and human evaluator was highest for History Taking (CPX-RTE vs professor 0.957 [95% CI 0.944–0.971]; CPX-RTE vs resident 0.958 [0.943–0.972]) and lowest for Patient–Physician Interaction (CPX-RTE vs professor 0.860 [0.823–0.893]; CPX-RTE vs resident 0.847, [0.809–0.881]) Table 2 Inter-rater Agreement (Gwet’s AC1) Between CPX-RTE and Human Raters by Checklist Section Section Gwet’s AC1 (95% CI) CPX-RTE vs Attending CPX-RTE vs Resident Attending vs Resident Overall 0.916 (0.902–0.930) 0.916 (0.903–0.929) 0.976 (0.968–0.983) History 0.957 (0.944–0.971) 0.958 (0.943–0.972) 0.983 (0.974–0.991) Physical Exam 0.905 (0.870–0.935) 0.912 (0.878–0.943) 0.982 (0.966–0.994) Patient Education 0.906 (0.866–0.944) 0.921 (0.881–0.956) 0.965 (0.934–0.990) Patient-Physician Interaction 0.860 (0.823–0.893) 0.847 (0.809–0.881) 0.963 (0.945–0.979) The checklist consisted of four sections: History (20 items), Physical Examination (8 items), Patient Education (6 items), and Patient–Physician Interaction (11 items). Gwet’s AC1 was computed at the case level within each section using checklist items as units of agreement, and section-level summary statistics (mean [95% CI]) are shown. Item-level analyses revealed that the three checklist items with the lowest agreement consistently occurred within the Patient–Physician Interaction domain for both CPX-RTE–professor and CPX-RTE–resident comparisons (Supplementary Table S4). Among these, the item “Explore patient’s deep concern” demonstrated the poorest agreement, with an AC1 of 0.250 for both CPX-RTE versus professor and CPX-RTE versus resident, whereas agreement between the resident and professor raters for the same item remained high (AC1 = 0.937). Usability of CPX-RTE: Feedback Quality and Refinement Suggestions 60 Participants rated CPX-RTE feedback quality (Evidence-based, Actionable, Connected, Balanced) on 1–6 Likert scales. The overall CPX-RTE feedback quality score (mean of the four items) was 4.82 [4.65–4.99] out of 6, indicating generally high perceived feedback quality; detailed item-level ratings are reported in Table 3 . Table 3 Usability Evaluation of CPX-RTE Feedback Item Mean (95% CI) Evidence-Based 4.65 (4.40–4.90) Actionable 4.82 (4.58–5.03) Connected 4.92 (4.70–5.10) Balanced 4.90 (4.72–5.08) Overall (mean of 4 items) 4.82 (4.65–4.99) Evidence-based , Actionable , Connected , and Balanced evaluate whether feedback accurately reflects student performance, provides feasible improvement suggestions, logically links actions to feedback, and maintains an appropriate balance between praise and critique, respectively. For CPX-RTE, open-ended responses highlighted scoring accuracy as the most frequent concern, particularly false-negative detections of performed checklist items (Supplementary Table S5). Despite this, learners consistently reported that the feedback was educationally valuable, emphasizing needs for clearer prioritization, actionable examples, and longitudinal tracking, complementing the high overall usability ratings observed in structured surveys. Bias Assessment, System Usability, and Cost None of the participants reported perceived bias or stereotyped content in CPX-VSP dialogue and/or CPX-RTE feedback. Overall system usability, measured by the System Usability Scale (SUS), had a median score of 77.5 (IQR, 70.0–85.0), indicating generally good usability. The mean per-session cost of CPX-VSP was $ 0.12 for resource-limited configuration and $ 0.78 for full-capacity configuration, reflecting a substantial cost–capability differential. A per-session cost of CPX-RTE was $ 0.17. DISCUSSION This study demonstrates that an end-to-end AI-supported OSCE training loop—voice-based VSP simulation (CPX-VSP) followed by immediate, structured evaluation and feedback (CPX-RTE)—can be deployed within an authentic clerkship workflow with educationally meaningful usability and performance. Across objective and subjective measures, CPX-VSP supported OSCE-oriented practice through sufficiently coherent and immersive dialogue, while CPX-RTE showed high agreement with expert human ratings for checklist-based performance and was perceived by learners as evidence-based, actionable, and balanced. From the learner’s perspective, CPX-MATE functioned as a single continuous cycle—simulation followed by immediate feedback—highlighting the practical premise that OSCE preparation requires not only a realistic encounter but also timely, structured evaluation for deliberate practice. A key contribution of this work is the use of STS models to achieve real-time, voice-based VSP interaction, addressing a persistent limitation of prior AI patient simulators. Prior studies have demonstrated the educational potential of generative AI–based patient simulators. For example, Lavigne et al. showed that a GPT-4–based AI standardized clinical examination simulator can improve OSCE performance, 3 but interaction in that system was entirely text-based, limiting its ability to capture the temporal, prosodic, and turn-taking dynamics that characterize authentic clinical dialogue. Likewise, Luo et al. reported that an LLM-based digital patient system enhanced ophthalmology history-taking skills and incorporated voice interaction via a chained STT–LLM–TTS architecture. 10 However, learners were required to review and correct transcribed text before eliciting a spoken response, introducing an explicit verification step that interrupts conversational flow. By integrating STS models directly into the voice loop, the present work reduces these sequenced bottlenecks and supports more continuous, contextually responsive spoken interaction. Furthermore, since STS models are relatively recent and have not been systematically characterized in medical simulation settings, rigorous and fine-grained evaluation was necessary. Accordingly, we decomposed more than three thousand dialogue turns into MIUs and classified failures into four clinically meaningful error types, enabling identification of characteristic instruction-following and conversational control limitations that are invisible to aggregate realism scores alone. This MIU-level validation framework shifts validation from subjective impressions or downstream outcomes to mechanistic understanding of model behavior and provides a transferable methodology for assessing whether emerging STS models are educationally fit for use in high-stakes clinical skills training. This study also advances prior work on automated OSCE evaluation by prospectively operationalizing real-time, learner-facing assessment within routine clinical education and formally evaluating its usability. Unlike earlier studies that demonstrated feasibility using retrospective or offline transcript analysis, 17–19 CPX-RTE delivered checklist-based scoring and structured narrative feedback to medical students within seconds after each encounter, at scale, during routine clinical education. Importantly, this real-time feedback was not only technically validated against expert raters but also systematically evaluated by survey of usability from sixty end users, providing direct evidence on how immediate AI-generated feedback is perceived, interpreted, and integrated in real educational practice rather than simulated or post hoc settings. At the same time, our findings clarify clear boundaries of what automated evaluation can and cannot replace. Although CPX-RTE achieved high agreement with expert raters for many checklist-based items, residual disagreement was systematically concentrated in Patient–Physician Interaction domains that require nuanced affective and relational judgment and are only weakly inferable from dialogue transcripts. A parallel limitation applies to physical examination assessment: while agreement for physical examination items was acceptable, scoring necessarily relied on verbalized intent rather than whether the examination was performed correctly and appropriately. As a result, both interpersonal competencies and hands-on examination quality remain fundamentally underdetermined by transcript-based automation alone. These results indicate that automated OSCE evaluation should not be positioned as a fully autonomous assessor; instead, it must be embedded within a human-in-the-loop framework in which automated systems cover scalable, standardized components, while human evaluators retain responsibility for competencies that are experiential, relational, or procedural in nature. Looking forward, learner feedback suggests that the most pressing unmet need lies not only in granular checklist scoring but also in higher-level support that helps learners improve , not merely be evaluated . Students consistently requested feedback that explains why certain questions or actions mattered, highlights missed reasoning steps, and makes progress visible across repeated encounters rather than presenting isolated, encounter-level outputs. If implemented as learners envision—combining timely feedback with clinical reasoning guidance and longitudinal progress tracking—AI-powered education tools could enable far more frequent, structured, and scalable deliberate practice than is feasible with faculty time alone. In this sense, continued advances in AI create a plausible optimistic outlook that clinical skills education could become more accessible and less dependent on local instructional resources. 25 However, this democratizing promise should be interpreted with caution, because educational quality remains sensitive to model capability and affordability. As illustrated by our STS-based VSP evaluation, higher-capacity configurations supported more immersive and coherent simulation, whereas resource-limited configurations—though potentially “better than nothing”—were more prone to immersion-breaking conversational errors that can distract from learning. This cost–capability gradient implies an equity risk: even as AI reduces the logistical barriers to simulation-based education, settings that cannot afford sufficiently capable models (including LMICs and other resource-limited programs) may be left with systematically lower-fidelity training, thereby reproducing or widening existing disparities in educational quality. Accordingly, efforts to leverage AI for the broader availability of high-quality clinical education should explicitly define minimum fidelity thresholds and remain attentive to how model choice shapes what learners actually practice. Limitation Several limitations should be considered when interpreting these findings. First, this study was conducted at a single institution and focused on a single chief complaint, requiring future multi-institutional and multi-scenario studies to enhance generalizability. Second, exposure to CPX-VSP and CPX-RTE occurred in a single session, raising the possibility of novelty effects influencing usability and immersion ratings. Third, it did not examine whether CPX-MATE translates into improvements in clinical performance. Fourth, participants were not randomly assigned to model exposure conditions, which may introduce selection or order effects. Despite these limitations, the prospective, workflow-integrated design provides a strong foundation for subsequent studies. The web-based application is currently being further developed to support ubiquitous, on-demand learning, enabling students to engage with the platform independently at any time and location, beyond formal instructional settings, while also expanding the range of clinical presentations. Ongoing work will therefore incorporate longitudinal exposure and evaluate downstream effects on CPX performance and learning transfer through randomized controlled trials. CONCLUSION In conclusion, the development and prospective validation of CPX-MATE within an authentic CPX workflow demonstrated that CPX-VSP enables accurate and immersive VSP simulation, while CPX-RTE achieved checklist-based evaluation performance comparable to that of human faculty during HSP encounters. These findings indicate that end-to-end AI-assisted CPX platforms can be systematically evaluated across both performance and usability dimensions, supporting their potential role as complementary tools within structured CPX training while highlighting the need for careful, context-aware deployment. Declarations Author contributions J.W.S.: Methodology, Software, Formal analysis, Data curation, Validation, Writing – original draft M.K.: Software, Formal analysis, Investigation, Data curation, Validation, Writing – original draft C.H.: Investigation, Data curation, Validation Y.S.K.: Investigation, Resources J.C.: Investigation, Resources J.H.K.: Investigation, Resources J.M.: Investigation, Resources A.C.: Visualization, Resources H.Y.: Visualization, Resources S.G.W.L.: Visualization, Resources S.C.Y.: Conceptualization, Methodology, Writing – review & editing, Supervision C.P.: Conceptualization, Methodology, Investigation, Writing – review & editing, Project administration, Supervision Disclosure of conflicts of interest Dr. You is an inventor, along with J.W.S., M.K., and C.P., on a pending Korean patent application related to the CPX-MATE platform (DP-2026-0104). Outside the submitted work, Dr. You reports grants from Daiichi Sankyo and VUNO, receives compensation as an Associate Editor for JACC, and serves as chief executive officer of PHI Digital Healthcare. The remaining authors declare no competing interests. Data Availability The data supporting the findings of this study are not publicly available due to institutional data governance policies and the inclusion of potentially identifiable educational performance data. De-identified and aggregated data may be made available from the corresponding author upon reasonable request. Code Availability The source code of the CPX-MATE platform is not publicly available due to a pending patent application and institutional intellectual property considerations. De-identified statistical analysis code supporting the findings of this study is available from the corresponding author upon reasonable request. Ethics approval & consent The study was approved by the IRB of Yonsei University College of Medicine (IRB No. [4-2025-1312]). Participation was voluntary and uncompensated, and had no impact on academic evaluation. Written informed consent was obtained from all participants prior to enrollment. All recordings and transcripts were de-identified prior to analysis. The study was conducted in accordance with the Declaration of Helsinki. Funding statement Not applicable. References Ba, H., Zhang, L., He, X. & Li, S. Knowledge mapping and global trends in the field of the objective structured clinical examination: bibliometric and visual analysis (2004-2023). JMIR Medical Education 10 , e57772 (2024). Dewan, P., Khalil, S. & Gupta, P. Objective structured clinical examination for teaching and assessment: Evidence-based critique. Clinical Epidemiology and Global Health 25 , 101477 (2024). Lavigne, E. et al. AI-Standardized Clinical Examination training on OSCE performance. NEJM AI 2 , AIoa2500066 (2025). Patrício, M. F., Julião, M., Fareleira, F. & Carneiro, A. V. Is the OSCE a feasible tool to assess competencies in undergraduate medical education? Medical teacher 35 , 503-514 (2013). Mukurunge, E., Nyoni, C. N. & Hugo, L. Assessment approaches in undergraduate health professions education: towards the development of feasible assessment approaches for low-resource settings. BMC Medical Education 24 , 318 (2024). Ataro, G., Worku, S. & Asaminew, T. Experience and challenges of objective structured clinical examination (OSCE): perspective of students and examiners in a clinical department of Ethiopian University. Ethiopian Journal of Health Sciences 30 (2020). Cook, D. A., Overgaard, J., Pankratz, V. S., Del Fiol, G. & Aakre, C. A. Virtual patients using large language models: Scalable, contextualized simulation of clinician-patient dialogue with feedback. Journal of Medical Internet Research 27 , e68486 (2025). Holderried, F. et al. A language model–powered simulated patient with automated feedback for history taking: Prospective study. JMIR Medical Education 10 , e59213 (2024). Yamamoto, A. et al. Enhancing medical interview skills through AI-Simulated patient interactions: nonrandomized controlled trial. JMIR medical education 10 , e58753 (2024). Luo, M.-J. et al. A large language model digital patient system enhances ophthalmology history taking skills. NPJ Digital Medicine 8 , 502 (2025). Jacobs, C. et al. Application of AI Communication Training Tools in Medical Undergraduate Education: Mixed Methods Feasibility Study Within a Primary Care Context. JMIR Med Educ 11 , e70766 (2025). https://doi.org/10.2196/70766 Pu, Y. et al. in Proc. Interspeech 2025. 26-30. Takata, T., Yamada, R., René, A. O. N., Xu, K. & Fujimoto, M. in 2024 Joint 13th International Conference on Soft Computing and Intelligent Systems and 25th International Symposium on Advanced Intelligent Systems (SCIS&ISIS). 1-5 (IEEE). OpenAI. Introducing gpt-realtime , (2025). Google. Get started with Live API , (2025). Lee, Y. C. Rethinking artificial intelligence in medicine: from tools to agents. Clinical and experimental emergency medicine 12 , 101 (2025). Geathers, J. et al. in International Conference on Artificial Intelligence in Education. 231-245 (Springer). Shakur, A. H. et al. Large Language Models for Medical OSCE Assessment: A Novel Approach to Transcript Analysis. arXiv preprint arXiv:2410.12858 (2024). Holcomb, M. J. et al. Zero-Shot Multimodal Question Answering for Assessment of Medical Student OSCE Physical Exam Videos. medRxiv , 2024.2006. 2005.24308467 (2024). OpenAI. Introducing Whisper , (2025). OpenAI. Introducing GPT-5 , (2025). Roh, H. et al. [Development of guide to clinical performance and basic clinical skills for medical students]. Korean J Med Educ 27 , 309-319 (2015). https://doi.org/10.3946/kjme.2015.27.4.309 Lewis, J. R. The system usability scale: past, present, and future. International Journal of Human–Computer Interaction 34 , 577-590 (2018). Ahmed, S. K. et al. Using thematic analysis in qualitative research. Journal of Medicine, Surgery, and Public Health 6 , 100198 (2025). Khan, S. Sal Khan’s 2023 TED Talk: AI in the classroom can transform education , (2023). Additional Declarations Competing interest reported. Dr. You is an inventor, along with J.W.S., M.K., and C.P., on a pending Korean patent application related to the CPX-MATE platform (DP-2026-0104). Outside the submitted work, Dr. You reports grants from Daiichi Sankyo and VUNO, receives compensation as an Associate Editor for JACC, and serves as chief executive officer of PHI Digital Healthcare. The remaining authors declare no competing interests. Supplementary Files 3.CPXMATESupplementary260223.docx 4.miclearllm.docx 5.STROBEchecklistv4.docx Cite Share Download PDF Status: Under Revision Version 1 posted Editorial decision: Revision requested 18 May, 2026 Reviews received at journal 17 May, 2026 Reviews received at journal 12 May, 2026 Reviewers agreed at journal 20 Apr, 2026 Reviews received at journal 09 Apr, 2026 Reviewers agreed at journal 09 Apr, 2026 Reviewers agreed at journal 20 Mar, 2026 Reviewers agreed at journal 19 Mar, 2026 Reviewers invited by journal 02 Mar, 2026 Editor assigned by journal 26 Feb, 2026 Submission checks completed at journal 26 Feb, 2026 First submitted to journal 25 Feb, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8972105","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":599624391,"identity":"2efd8e61-00da-4867-b14f-5e5569cb05b8","order_by":0,"name":"Ji Woo Song","email":"","orcid":"","institution":"Yonsei University College of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Ji","middleName":"Woo","lastName":"Song","suffix":""},{"id":599624392,"identity":"7fe8a11e-ef5b-4147-bf7e-9a8986cf8359","order_by":1,"name":"Minseong Kim","email":"","orcid":"","institution":"Yonsei University College of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Minseong","middleName":"","lastName":"Kim","suffix":""},{"id":599624393,"identity":"3697810b-a1e7-459a-adcd-a8a8f3d75f59","order_by":2,"name":"Chanhee Hong","email":"","orcid":"","institution":"Severance Hospital, Yonsei University College of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Chanhee","middleName":"","lastName":"Hong","suffix":""},{"id":599624394,"identity":"aea8ea27-0a3d-40fb-b465-8f5877e4bb22","order_by":3,"name":"Young Sam Kim","email":"","orcid":"","institution":"Severance Hospital, Yonsei University College of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Young","middleName":"Sam","lastName":"Kim","suffix":""},{"id":599624395,"identity":"245089cc-c882-4e30-b574-7e8965c6cb8e","order_by":4,"name":"Junho Cho","email":"","orcid":"","institution":"Severance Hospital, Yonsei University College of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Junho","middleName":"","lastName":"Cho","suffix":""},{"id":599624396,"identity":"19d8c205-46d6-4eb9-80aa-56fe875b1da1","order_by":5,"name":"Ji Hoon Kim","email":"","orcid":"","institution":"Severance Hospital, Yonsei University College of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Ji","middleName":"Hoon","lastName":"Kim","suffix":""},{"id":599624397,"identity":"d495c2d9-cac6-4bb4-a6f7-2e215942cff4","order_by":6,"name":"Jinwoo Myung","email":"","orcid":"","institution":"Severance Hospital, Yonsei University College of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Jinwoo","middleName":"","lastName":"Myung","suffix":""},{"id":599624398,"identity":"4bbb8bad-3fb7-4ead-9052-274a46ad24bf","order_by":7,"name":"Arom Choi","email":"","orcid":"","institution":"Severance Hospital, Yonsei University College of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Arom","middleName":"","lastName":"Choi","suffix":""},{"id":599624399,"identity":"8a1ecc0f-b2c3-44b8-aa45-5c9fae906618","order_by":8,"name":"Hanna Yoon","email":"","orcid":"","institution":"Severance Hospital, Yonsei University College of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Hanna","middleName":"","lastName":"Yoon","suffix":""},{"id":599624400,"identity":"2b766254-01c1-42eb-9996-9b961d2530fa","order_by":9,"name":"Stephen Gyung Won Lee","email":"","orcid":"","institution":"Severance Hospital, Yonsei University College of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Stephen","middleName":"Gyung Won","lastName":"Lee","suffix":""},{"id":599624401,"identity":"8acd22fc-dd11-43d6-83b4-b6a4eab15481","order_by":10,"name":"Seng Chan You","email":"","orcid":"","institution":"Yonsei University College of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Seng","middleName":"Chan","lastName":"You","suffix":""},{"id":599624402,"identity":"dce7d402-9cf9-4f88-9530-bd6eb87a630b","order_by":11,"name":"Chaeryoung Park","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA9ElEQVRIiWNgGAWjYDACCRTeARt0EcJa0kjXcpiwFv7ZzQ8fV7bZJG7nX3vwc8WZ84kbbrc/YPhRg8eSO8eMDc+2pSXunPEuWfLMjduJG+6cMWDsOYZbi4FEgplkY9vhxA03zhhINnwAarmRw8DA24BPS/r3n1Atxj8bPpwDMtIfMP7FqyXHjBGs5XyPmWTDjQNALQkGzPhskbiRUyzZcC7NeMMNvjTLhjPJxjNv5BgclsHjF/4Z6Rs/NpTZyG44f/bwzYZjdrJ9N9IfPnyDJ8TAgJENZF8OmO0IctIBAhqA4A/IvjNgpj1h1aNgFIyCUTDSAAB1dGNe7UlO6wAAAABJRU5ErkJggg==","orcid":"","institution":"Severance Hospital, Yonsei University College of Medicine","correspondingAuthor":true,"prefix":"","firstName":"Chaeryoung","middleName":"","lastName":"Park","suffix":""}],"badges":[],"createdAt":"2026-02-26 01:38:15","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8972105/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8972105/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":104179215,"identity":"585ff368-d989-4e36-a6f4-33a6d5c7a097","added_by":"auto","created_at":"2026-03-08 17:03:16","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":127417,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eOverview of the CPX-Mate Platform\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e(A) System architecture of CPX-Mate for CPX with VSP and CPX with Real-Time Evaluator which containsvirtual standardized patientssimulation using STS (speech-to-speech) model and automatic, checklist-based assessment using STT (speech-to-text) model large language models (LLMs). (B) Mobile Web-app workflow of CPX-Mate. Users select a chief complaint, interact with a virtual patient through real-time dialogue (CPX-VSP), and receive checklist-based performance evaluation and feedback (CPX-RTE).\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-8972105/v1/074ebb7192a70df389f08fe1.png"},{"id":104179219,"identity":"51b25a3e-b194-4b9a-9cad-d4b9e6acb81b","added_by":"auto","created_at":"2026-03-08 17:03:17","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":101542,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eCPX-VSP MIU error rates across error subtypes\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTangential answers refer to responses that are factually correct but misaligned with the student’s question intent; oversharing answers disclose unsolicited information beyond what was asked; off-script answers contradict the predefined scenario prompts.\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-8972105/v1/5ea6a6060524ca02ee574c01.png"},{"id":104404185,"identity":"124557d3-8a7e-4322-8a95-2460fd627609","added_by":"auto","created_at":"2026-03-11 12:19:47","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":58743,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eComparison of CPX-VSP Usability and User Experience Scores Between Models\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eHumanlike\u003c/em\u003e, \u003cem\u003eCoherent\u003c/em\u003e, \u003cem\u003eClinically Relevant\u003c/em\u003e, and \u003cem\u003eDialogue Overall\u003c/em\u003eassess the naturalness, consistency, clinical usefulness, and overall realism of the dialogue, respectively, while \u003cem\u003eRealness\u003c/em\u003e, \u003cem\u003eCognitiveAuthenticity\u003c/em\u003e, \u003cem\u003eVariability\u003c/em\u003e, \u003cem\u003eInvolvement\u003c/em\u003e, and \u003cem\u003eUX Overall\u003c/em\u003eevaluate realism, cognitive engagement, spontaneity, immersion, and overall user experience, respectively.\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-8972105/v1/e27a79af826ae27f541460ef.png"},{"id":104408801,"identity":"cc44c7db-a3a2-468a-a619-2a1909fe80f7","added_by":"auto","created_at":"2026-03-11 12:43:27","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1271041,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8972105/v1/0ee74ec4-c3fb-4a06-9532-d2d551f6b482.pdf"},{"id":104179216,"identity":"9c5e48a7-d594-4890-8850-d4ee491dae7a","added_by":"auto","created_at":"2026-03-08 17:03:17","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":88703,"visible":true,"origin":"","legend":"","description":"","filename":"3.CPXMATESupplementary260223.docx","url":"https://assets-eu.researchsquare.com/files/rs-8972105/v1/51da3460cfa71bb0aac3efd2.docx"},{"id":104179217,"identity":"a041d93d-0fe7-4d35-ae2a-cf9b84c30239","added_by":"auto","created_at":"2026-03-08 17:03:17","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":30667,"visible":true,"origin":"","legend":"","description":"","filename":"4.miclearllm.docx","url":"https://assets-eu.researchsquare.com/files/rs-8972105/v1/74f47dc6462ae3928d51df96.docx"},{"id":104404058,"identity":"dbb91075-0e50-479f-8c3d-70863fabdf40","added_by":"auto","created_at":"2026-03-11 12:19:40","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":38109,"visible":true,"origin":"","legend":"","description":"","filename":"5.STROBEchecklistv4.docx","url":"https://assets-eu.researchsquare.com/files/rs-8972105/v1/43378f23f2e14b3f16373a4b.docx"}],"financialInterests":"Competing interest reported. Dr. You is an inventor, along with J.W.S., M.K., and C.P., on a pending Korean patent application related to the CPX-MATE platform (DP-2026-0104). Outside the submitted work, Dr. You reports grants from Daiichi Sankyo and VUNO, receives compensation as an Associate Editor for JACC, and serves as chief executive officer of PHI Digital Healthcare. The remaining authors declare no competing interests.","formattedTitle":"Development and Prospective Validation of CPX-MATE: An End-to-End Medical Education Platform Integrating Voice-Based Virtual Patient Simulation and Automated Real-time Evaluation","fulltext":[{"header":"INTRODUCTION","content":"\u003cp\u003eObjective Structured Clinical Examination (OSCE), which is called Clinical Performance Examination (CPX) in South Korea, is a core competency-based assessment designed to evaluate authentic clinical performance of medical students through standardized patient (SP) encounters. \u003csup\u003e1,2\u003c/sup\u003e OSCE is a high-stakes component of undergraduate medical education and licensure, assessing integrated skills such as history taking, communication, clinical reasoning, and initial management within time-limited, face-to-face interactions. Students interact directly with an SP, while evaluators observe in real time from behind a one-way mirror and score student performance using standardized checklists.\u003c/p\u003e \u003cp\u003eDespite its strong educational validity, the educational infrastructure underpinning OSCE is inherently resource intensive. It requires substantial financial investment, complex logistical coordination, trained faculty examiners, and the recruitment, training, and compensation of human standardized patients (HSPs).\u003csup\u003e\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e,\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u003c/sup\u003e These demands limit opportunities for repeated practice, timely feedback, and scalable deployment. Such challenges are particularly pronounced in low- and middle-income countries (LMICs), where constrained educational budgets, faculty shortages, and limited access to trained SPs hinder the sustainable delivery of OSCE-based education.\u003csup\u003e\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e,\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eRecent advances in artificial intelligence (AI), particularly large language models (LLMs) and speech-to-speech (STS) models, have generated growing interest in virtual standardized patient (VSP)-based simulations as a means to augment traditional OSCE education. Prior studies have demonstrated the technical feasibility of text-based VSPs and AI-assisted automated scoring systems.\u003csup\u003e\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e,\u003cspan additionalcitationids=\"CR8 CR9 CR10 CR11 CR12 CR13 CR14 CR15\" citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e\u003c/sup\u003e However, much of the existing literature has focused on whether such systems can be built, rather than on how they should be systematically evaluated and integrated into real-world medical education. In high-stakes clinical assessment, technical performance alone is insufficient; perceived realism, learner trust, and sustained usability are equally critical determinants of educational impact. At the same time, reliance on high-capacity models raises questions about accessibility and cost equity.\u003c/p\u003e \u003cp\u003eAnother key limitation of prior AI-based OSCE solutions lies in feedback delivery. Although automated scoring modules have been developed, most were designed for retrospective evaluation rather than for real-time, formative feedback during ongoing practice.\u003csup\u003e\u003cspan additionalcitationids=\"CR18\" citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e Timely feedback is a core principle of experiential learning, enabling learners to identify performance gaps and adjust clinical reasoning while the learning experience remains cognitively salient.\u003c/p\u003e \u003cp\u003eAccordingly, we developed and evaluated CPX with Medical students\u0026rsquo; Assistant for Training and Evaluation (CPX-MATE), an end-to-end platform integrating voice-based VSP training (CPX with Virtual Standardized Patient, CPX-VSP) and real-time, voice-based evaluation applicable to HSP encounters (CPX with Real-Time Evaluator, CPX-RTE). This study aims to define and empirically examine the dimensions that should be scrutinized as such systems enter routine educational use, with a particular focus on performance fidelity, usability, and deployment within authentic educational settings.\u003c/p\u003e"},{"header":"METHODS","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eDevelopment of CPX-MATE\u003c/h2\u003e \u003cdiv id=\"Sec4\" class=\"Section3\"\u003e \u003ch2\u003eSystem Overview\u003c/h2\u003e \u003cp\u003eCPX-MATE is a web-based system comprising CPX-VSP and CPX-RTE (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). CPX-VSP enables real-time voice dialogue with a VSP using two STS models (\u003cem\u003egpt-realtime\u003c/em\u003e and \u003cem\u003egpt-realtime-mini\u003c/em\u003e).\u003csup\u003e\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e\u003c/sup\u003e This dual-model design was adopted to examine whether a resource-limited configuration can provide acceptable performance in constrained settings within the rapidly evolving AI landscape, and to quantify the extent of performance gains achievable with a full-capacity model as AI capabilities advance. To simplify presentation, \u003cem\u003egpt-realtime\u003c/em\u003e is hereafter referred to as the full-capacity model, whereas \u003cem\u003egpt-realtime-mini\u003c/em\u003e is referred to as the resource-limited model. CPX-RTE transcribes HSP encounters using speech-to-text (STT) model (\u003cem\u003ewhisper-1\u003c/em\u003e)\u003csup\u003e20\u003c/sup\u003e, maps transcript evidence to predefined checklist items and generates feedback using an LLM (\u003cem\u003eGPT-5\u003c/em\u003e)\u003csup\u003e21\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003c/div\u003e\n\u003ch3\u003eCPX-VSP: prompts and interaction rules\u003c/h3\u003e\n\u003cp\u003eTwo prompts were provided to the STS model: (1) a global SP persona prompt, which was identical across all chief complaints and scenarios, specifying role/tone/constraints and dialogue rules (e.g., disclose information only when asked; answer unspecified questions plausibly without contradicting the scenario; if the student announces a physical exam, respond with corresponding findings), and (2) a scenario-specific prompt which varied by clinical scenario and contained the full clinical case specification (symptoms, history, and physical exam findings). The SP persona prompt was designed as a reusable, chief-complaint\u0026ndash;agnostic control layer, whereas the scenario prompt served as a case-specific content layer. Although only a single scenario prompt was used in the present study, this separation was intentionally adopted to ensure scalability to multiple chief complaints and scenarios. Prompts were iteratively refined by an emergency medicine faculty member (CP) and a medical student investigator (JWS).\u003c/p\u003e\n\u003ch3\u003eCPX-RTE: scoring and feedback\u003c/h3\u003e\n\u003cp\u003eCPX-RTE follows a pipeline consisting of (1) audio transcription, (2) structured checklist-based feedback generation, and (3) narrative feedback generation. The student\u0026ndash;patient dialogue is first transcribed using an STT model. For each checklist item, an LLM then reviews the transcript to identify utterances that match a criterion of the item and applies a predefined binary decision rule, classifying the item as \u0026ldquo;Yes\u0026rdquo; when one or more relevant utterances are detected and as \u0026ldquo;No\u0026rdquo; otherwise. These item-level decisions, together with the matched transcript excerpts, form the structured feedback. Finally, the LLM generates narrative feedback summarizing strengths and areas for improvement based on both the transcript and the structured outputs.\u003c/p\u003e\n\u003ch3\u003eClinical scenarios and checklist\u003c/h3\u003e\n\u003cp\u003eTwo acute abdominal pain presentations were used as representative, generalizable emergency cases: VSP (CPX-VSP) acute pancreatitis and HSP ureteral stone. Performance was evaluated using a 45-item acute abdominal pain checklist: History Taking (20), Physical Examination (8), Patient Education (6), and Patient\u0026ndash;Physician Interaction (11), developed with reference to the Korea Association of Medical Colleges\u0026rsquo; The guide to clinical performance, 2nd edition.\u003csup\u003e\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eValidation of CPX-MATE\u003c/h2\u003e \u003cdiv id=\"Sec9\" class=\"Section3\"\u003e \u003ch2\u003eStudy design, setting, and participants\u003c/h2\u003e \u003cp\u003eThe study was conducted during an emergency medicine clerkship at Yonsei University College of Medicine between November 2025 and January 2026 and involved year 3\u0026ndash;4 medical students. Each participant completed a standardized CPX experience consisting of two clinical encounters followed by a post-session survey.\u003c/p\u003e \u003cp\u003eParticipants first engaged in a 12-minute CPX encounter with a VSP, during which CPX-VSP was delivered using either a resource-limited STS model (n\u0026thinsp;=\u0026thinsp;30, first half of the study period) or a full-capacity STS model (n\u0026thinsp;=\u0026thinsp;30, second half). CPX-RTE generated real-time checklist-based feedback that was shown to students for formative learning during this encounter, but these outputs were not used for performance validation as no independent human scoring was performed for VSP encounters.\u003c/p\u003e \u003cp\u003eAfter a 5-minute intermission, participants completed a second 12-minute CPX encounter with an HSP. These HSP encounters were audio-recorded for automated scoring by CPX-RTE, while video recording was performed as part of routine CPX operations to enable independent human rating. HSP performance was scored in real time by a board-certified emergency medicine attending physician (CP) and later independently scored by an emergency medicine resident (CH) using the same 45-item checklist. CPX-RTE scoring results were also generated and presented to students immediately after the encounter.\u003c/p\u003e \u003cp\u003eFollowing completion of both encounters, participants completed post-session surveys assessing the usability and perceived realism of CPX-VSP, the quality of CPX-RTE feedback, and overall system usability, along with open-ended questions for qualitative feedback.\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e\n\u003ch3\u003eData collection\u003c/h3\u003e\n\u003cp\u003eCollected data included CPX-VSP transcripts and system logs (e.g., turns, duration, token usage), HSP audio (and video for human rating), and post-session surveys (Likert-scale items and free-text responses). All data were de-identified before analysis. Mean per-session costs for CPX-VSP and CPX-RTE were captured through OpenAI\u0026rsquo;s dashboard.\u003c/p\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003eOutcome Measures\u003c/h2\u003e \u003cp\u003eCPX-VSP performance was assessed at the Minimum Interaction Unit (MIU) level, defined as one student utterance paired with the corresponding VSP response. Each MIU was classified into one of four error types: tangential (plausible content misaligned with question intent), oversharing (unsolicited disclosure beyond what was asked), role-breaking (the VSP speaking as if it were the clinician), and off-script (responses contradicting the predefined scenario prompt). All MIUs were manually reviewed and coded by a medical student investigator (JWS). The primary performance metric was the overall error rate per MIU for each STS model; secondary metrics were error rates per MIU stratified by error type.\u003c/p\u003e \u003cp\u003eCPX-VSP usability was assessed using post-session surveys adapted from Cook et al. on a 1-to-6 Likert scale (1\u0026thinsp;=\u0026thinsp;strongly disagree, 6\u0026thinsp;=\u0026thinsp;strongly agree). Survey items evaluated two domains: (1) dialogue naturalness (Humanlike, Coherent, Clinically relevant, Dialogue overall) and (2) user experience(UX)/immersion (Realness, Cognitive authenticity, Variability, Involvement, UX overall). One Cook et al. item assessing whether the patient\u0026rsquo;s \u0026ldquo;personal preferences\u0026rdquo; were reflected was excluded to preserve construct alignment with the target assessment context. In the Korean undergraduate CPX setting, checklist-based scoring typically emphasizes standardized history taking, clinical reasoning, and initial management/education under time constraints rather than individualized preference elicitation as a primary learning objective. Accordingly, we focused the usability instrument on dialogue naturalness constructs most relevant to CPX preparation.\u003c/p\u003e \u003cp\u003eCPX-RTE performance was evaluated using Gwet\u0026rsquo;s AC1 for binary checklist items. AC1 was used since checklist items showed marked prevalence imbalance (some items were almost always performed, whereas others were rarely performed), a setting in which kappa-type coefficients can be unstable and underestimate agreement. AC1 is well-known for providing a robust agreement estimate under such imbalanced marginal distributions.\u003c/p\u003e \u003cp\u003eThe board certified emergency medicine attending physician (CP) scored the encounter in real time using the same 45-item acute abdominal pain checklist, while a second rater (CH, emergency medicine resident) independently scored the video-recorded encounter. Agreement was evaluated across three rater pairs: CPX-RTE vs attending, CPX-RTE vs resident, and attending vs resident. The primary outcome was overall agreement across all 45 checklist items for each pair, with secondary analyses examining section-level agreement (History Taking, Physical Examination, Patient Education, and Patient\u0026ndash;Physician Interaction) and item-level agreement for each checklist item.\u003c/p\u003e \u003cp\u003eCPX-RTE usability was assessed using post-session surveys adapted from Cook et al. on a 1-to-6 Likert scale across four constructs: Evidence-based, Actionable, Connected, and Balanced.\u003c/p\u003e \u003cp\u003eParticipants were asked whether they observed biased or stereotyped content (e.g., related to sex, age, race/ethnicity, or socioeconomic status) in either CPX-VSP dialogue or CPX-RTE feedback, and to describe the bias if present. Overall web application usability was assessed using the System Usability Scale (SUS).\u003csup\u003e\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eTo promote objective and consistent responses, surveys assessing CPX-VSP realism and user experience and CPX-RTE feedback quality were administered using 1\u0026ndash;6 Likert scales accompanied by explicit operational criteria defining each response level. Participants also answered open-ended questions on factors that enhanced or hindered perceived authenticity of CPX-VSP, and on areas for improvement of CPX-RTE. These free-text responses were analyzed using inductive thematic analysis framework suggested by Braun and Clarke.\u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eThe verbatim survey items are provided in Supplementary Method 1.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003eStatistical analysis\u003c/h2\u003e \u003cp\u003eAnalyses were conducted using a complete-case approach. For CPX-VSP performance, errors were defined at the MIU level. Overall error rates and error rates by subtype were calculated per STS model. CPX-VSP usability was summarized as mean (95% confidence interval [CI]) and full frequency distributions (n, %). Participant-level cluster bootstrap resampling (1,000 iterations) was used to obtain 95% CIs and p-values. Since CPX-VSP performance and usability analysis compared two difference STS models, all multiple comparisons for CPX-VSP were adjusted using the Benjamini\u0026ndash;Hochberg false discovery rate procedure. All tests were two-sided with a significance threshold of α\u0026thinsp;=\u0026thinsp;0.05.\u003c/p\u003e \u003cp\u003eFor CPX-RTE performance, agreement on binary checklist items was assessed using Gwet\u0026rsquo;s AC1 to account for marginal imbalance. CPX-RTE usability items were reported as mean (95% CI) and full frequency distributions (n, %). Similarly, participant-level cluster bootstrap resampling (1,000 iterations) was used to obtain 95% CIs. Analyses were performed using Python (ver 3.14.2; Python Software Foundation).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003eEthics\u003c/h2\u003e \u003cp\u003e The study was approved by the IRB of Yonsei University College of Medicine (IRB No. [4-2025-1312]). Participation was voluntary, uncompensated, and had no impact on academic evaluation; all recordings/transcripts were de-identified.\u003c/p\u003e \u003c/div\u003e"},{"header":"RESULTS","content":"\u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003eParticipants\u003c/h2\u003e \u003cp\u003eSixty senior medical students (years 3\u0026ndash;4) completed the sequential two-station CPX sessions between November 2025 and January 2026. Participants had a mean age of 26.9 years (SD 0.76) and included 28 female (46.7%) and 32 male (53.3%) students. CPX-VSP was delivered using the resource-limited model (n\u0026thinsp;=\u0026thinsp;30) or the full-capacity model (n\u0026thinsp;=\u0026thinsp;30). Detailed demographics are shown in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eParticipant demographics by model group\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e\u003cb\u003eResource-limited model\u003c/b\u003e\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eParticipants\u003c/p\u003e \u003cp\u003eN\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMale\u003c/p\u003e \u003cp\u003eN (%)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAge, years\u003c/p\u003e \u003cp\u003emean (SD)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003e3rd grade\u003c/p\u003e \u003cp\u003eN (%)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003e30\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003e18 (60.0%)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003e27.2 (0.85)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003e4 (13.3%)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eFull-capacity model\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e30\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e14 (46.7%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e26.5 (0.35)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e16 (53.3%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eTotal\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e60\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e32 (53.3%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e26.9 (0.76)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e20 (33.3%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"5\"\u003eBaseline demographic characteristics of participants stratified by model group. Age is presented as mean (standard deviation), and categorical variables are presented as number (percentage).\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003ePerformance of CPX-VSP: Dialogue Error Rates per MIU\u003c/h2\u003e \u003cp\u003eAcross CPX-VSP sessions, 3,282 Minimum Interaction Units (MIUs) were identified and manually reviewed (resource-limited model: 1,591 MIUs; full-capacity model: 1,691 MIUs). Figure\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e shows the dialogue error rates per MIU of each model. The overall MIU error rate was higher with the resource-limited model than with the full-capacity model (9.43% [95% CI 7.50\u0026ndash;11.25] vs 1.77% [0.80\u0026ndash;2.84]), yielding an absolute difference of 7.65 percentage points (resource-limited\u0026thinsp;\u0026minus;\u0026thinsp;full-capacity; 95% CI 5.34\u0026ndash;9.74; p\u0026thinsp;\u0026lt;\u0026thinsp;0.001) (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Error subtype analyses showed significantly higher tangential and oversharing error rates for the resource-limited model (tangential 5.28% [3.97\u0026ndash;6.69] vs 1.48% [0.66\u0026ndash;2.40], Δ\u0026thinsp;=\u0026thinsp;3.80 pp [2.23\u0026ndash;5.40], p\u0026thinsp;\u0026lt;\u0026thinsp;0.001; oversharing 4.02% [2.86\u0026ndash;5.33] vs 0.24% [0.06\u0026ndash;0.46], Δ\u0026thinsp;=\u0026thinsp;3.79 pp [2.60\u0026ndash;5.09], p\u0026thinsp;\u0026lt;\u0026thinsp;0.001). Role-breaking was rare in both groups (0.13% vs 0.06%, Δ\u0026thinsp;=\u0026thinsp;0.07 pp [\u0026minus;\u0026thinsp;0.12 to 0.27], p\u0026thinsp;=\u0026thinsp;0.485), and off-script errors were not observed.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec17\" class=\"Section2\"\u003e \u003ch2\u003eUsability of CPX-VSP: Dialogue Naturalness and User Experience\u003c/h2\u003e \u003cp\u003eAs shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e, the full-capacity model received higher ratings than the resource-limited model for Coherent (4.77 [95% CI 4.43\u0026ndash;5.13] vs 4.03 [3.67\u0026ndash;4.43]; p\u0026thinsp;=\u0026thinsp;0.034), Involvement (4.37 [4.03\u0026ndash;4.70] vs 3.40 [2.87\u0026ndash;3.90]; p\u0026thinsp;=\u0026thinsp;0.019), and UX overall (4.27 [3.90\u0026ndash;4.60] vs 3.27 [2.83\u0026ndash;3.63]; p\u0026thinsp;=\u0026thinsp;0.010). While differences in other usability items did not reach statistical significance, ratings consistently favored the full-capacity model over the resource-limited model across all measures. Detailed results are shown in Supplementary Table \u003cspan refid=\"MOESM1\" class=\"InternalRef\"\u003eS1\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eOpen-ended responses aligned with quantitative findings. Students most frequently attributed authenticity to content-driven immersion, particularly coherent, clinically plausible, and scenario-consistent responses (Supplementary Table \u003cspan refid=\"MOESM2\" class=\"InternalRef\"\u003eS2\u003c/span\u003e). Loss of immersion was primarily linked to content-level inconsistencies, including tangential responses and oversharing (Supplementary Table \u003cspan refid=\"MOESM3\" class=\"InternalRef\"\u003eS3\u003c/span\u003e). Students also emphasized interactional realism (e.g., patient-like tone and affective responsiveness), consistent with higher involvement and overall UX in the full-model group. Once major content- and interaction-level issues were reduced in the full-capacity model, secondary constraints (e.g., latency and screen-mediated awkwardness) became more prominent in qualitative comments.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec18\" class=\"Section2\"\u003e \u003ch2\u003ePerformance of CPX-RTE: Agreement with Human Ratings\u003c/h2\u003e \u003cp\u003eChecklist scoring by CPX-RTE was compared with professor real-time ratings and resident video-based ratings using Gwet\u0026rsquo;s AC1 (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). Overall agreement was 0.916 (95% CI 0.902\u0026ndash;0.930) for CPX-RTE vs professor, 0.916 (0.903\u0026ndash;0.929) for CPX-RTE vs resident, and 0.976 (0.968\u0026ndash;0.983) for professor vs resident. Section-level agreement between CPX-RTE and human evaluator was highest for History Taking (CPX-RTE vs professor 0.957 [95% CI 0.944\u0026ndash;0.971]; CPX-RTE vs resident 0.958 [0.943\u0026ndash;0.972]) and lowest for Patient\u0026ndash;Physician Interaction (CPX-RTE vs professor 0.860 [0.823\u0026ndash;0.893]; CPX-RTE vs resident 0.847, [0.809\u0026ndash;0.881])\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eInter-rater Agreement (Gwet\u0026rsquo;s AC1) Between CPX-RTE and Human Raters by Checklist Section\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eSection\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c4\" namest=\"c2\"\u003e \u003cp\u003eGwet\u0026rsquo;s AC1 (95% CI)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCPX-RTE vs Attending\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCPX-RTE vs Resident\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAttending vs Resident\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOverall\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.916 (0.902\u0026ndash;0.930)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.916 (0.903\u0026ndash;0.929)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.976 (0.968\u0026ndash;0.983)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHistory\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.957 (0.944\u0026ndash;0.971)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.958 (0.943\u0026ndash;0.972)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.983 (0.974\u0026ndash;0.991)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePhysical Exam\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.905 (0.870\u0026ndash;0.935)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.912 (0.878\u0026ndash;0.943)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.982 (0.966\u0026ndash;0.994)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePatient Education\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.906 (0.866\u0026ndash;0.944)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.921 (0.881\u0026ndash;0.956)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.965 (0.934\u0026ndash;0.990)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePatient-Physician Interaction\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.860 (0.823\u0026ndash;0.893)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.847 (0.809\u0026ndash;0.881)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.963 (0.945\u0026ndash;0.979)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"4\"\u003eThe checklist consisted of four sections: History (20 items), Physical Examination (8 items), Patient Education (6 items), and Patient\u0026ndash;Physician Interaction (11 items). Gwet\u0026rsquo;s AC1 was computed at the case level within each section using checklist items as units of agreement, and section-level summary statistics (mean [95% CI]) are shown.\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eItem-level analyses revealed that the three checklist items with the lowest agreement consistently occurred within the Patient\u0026ndash;Physician Interaction domain for both CPX-RTE\u0026ndash;professor and CPX-RTE\u0026ndash;resident comparisons (Supplementary Table S4). Among these, the item \u0026ldquo;Explore patient\u0026rsquo;s deep concern\u0026rdquo; demonstrated the poorest agreement, with an AC1 of 0.250 for both CPX-RTE versus professor and CPX-RTE versus resident, whereas agreement between the resident and professor raters for the same item remained high (AC1\u0026thinsp;=\u0026thinsp;0.937).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec19\" class=\"Section2\"\u003e \u003ch2\u003eUsability of CPX-RTE: Feedback Quality and Refinement Suggestions\u003c/h2\u003e \u003cp\u003e 60 Participants rated CPX-RTE feedback quality (Evidence-based, Actionable, Connected, Balanced) on 1\u0026ndash;6 Likert scales. The overall CPX-RTE feedback quality score (mean of the four items) was 4.82 [4.65\u0026ndash;4.99] out of 6, indicating generally high perceived feedback quality; detailed item-level ratings are reported in Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eUsability Evaluation of CPX-RTE Feedback\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eItem\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMean (95% CI)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEvidence-Based\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e4.65 (4.40\u0026ndash;4.90)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eActionable\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e4.82 (4.58\u0026ndash;5.03)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eConnected\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e4.92 (4.70\u0026ndash;5.10)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBalanced\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e4.90 (4.72\u0026ndash;5.08)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eOverall (mean of 4 items)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e4.82 (4.65\u0026ndash;4.99)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"2\"\u003e\u003cem\u003eEvidence-based\u003c/em\u003e, \u003cem\u003eActionable\u003c/em\u003e, \u003cem\u003eConnected\u003c/em\u003e, and \u003cem\u003eBalanced\u003c/em\u003e evaluate whether feedback accurately reflects student performance, provides feasible improvement suggestions, logically links actions to feedback, and maintains an appropriate balance between praise and critique, respectively.\u003c/td\u003e\u003c/tr\u003e\u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eFor CPX-RTE, open-ended responses highlighted scoring accuracy as the most frequent concern, particularly false-negative detections of performed checklist items (Supplementary Table S5). Despite this, learners consistently reported that the feedback was educationally valuable, emphasizing needs for clearer prioritization, actionable examples, and longitudinal tracking, complementing the high overall usability ratings observed in structured surveys.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec20\" class=\"Section2\"\u003e \u003ch2\u003eBias Assessment, System Usability, and Cost\u003c/h2\u003e \u003cp\u003eNone of the participants reported perceived bias or stereotyped content in CPX-VSP dialogue and/or CPX-RTE feedback. Overall system usability, measured by the System Usability Scale (SUS), had a median score of 77.5 (IQR, 70.0\u0026ndash;85.0), indicating generally good usability. The mean per-session cost of CPX-VSP was \u003cspan\u003e$\u003c/span\u003e0.12 for resource-limited configuration and \u003cspan\u003e$\u003c/span\u003e0.78 for full-capacity configuration, reflecting a substantial cost\u0026ndash;capability differential. A per-session cost of CPX-RTE was \u003cspan\u003e$\u003c/span\u003e0.17.\u003c/p\u003e \u003c/div\u003e"},{"header":"DISCUSSION","content":"\u003cp\u003eThis study demonstrates that an end-to-end AI-supported OSCE training loop\u0026mdash;voice-based VSP simulation (CPX-VSP) followed by immediate, structured evaluation and feedback (CPX-RTE)\u0026mdash;can be deployed within an authentic clerkship workflow with educationally meaningful usability and performance. Across objective and subjective measures, CPX-VSP supported OSCE-oriented practice through sufficiently coherent and immersive dialogue, while CPX-RTE showed high agreement with expert human ratings for checklist-based performance and was perceived by learners as evidence-based, actionable, and balanced. From the learner\u0026rsquo;s perspective, CPX-MATE functioned as a single continuous cycle\u0026mdash;simulation followed by immediate feedback\u0026mdash;highlighting the practical premise that OSCE preparation requires not only a realistic encounter but also timely, structured evaluation for deliberate practice.\u003c/p\u003e \u003cp\u003eA key contribution of this work is the use of STS models to achieve real-time, voice-based VSP interaction, addressing a persistent limitation of prior AI patient simulators. Prior studies have demonstrated the educational potential of generative AI\u0026ndash;based patient simulators. For example, Lavigne et al. showed that a GPT-4\u0026ndash;based AI standardized clinical examination simulator can improve OSCE performance, \u003csup\u003e3\u003c/sup\u003e but interaction in that system was entirely text-based, limiting its ability to capture the temporal, prosodic, and turn-taking dynamics that characterize authentic clinical dialogue. Likewise, Luo et al. reported that an LLM-based digital patient system enhanced ophthalmology history-taking skills and incorporated voice interaction via a chained STT\u0026ndash;LLM\u0026ndash;TTS architecture.\u003csup\u003e\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u003c/sup\u003e However, learners were required to review and correct transcribed text before eliciting a spoken response, introducing an explicit verification step that interrupts conversational flow. By integrating STS models directly into the voice loop, the present work reduces these sequenced bottlenecks and supports more continuous, contextually responsive spoken interaction.\u003c/p\u003e \u003cp\u003eFurthermore, since STS models are relatively recent and have not been systematically characterized in medical simulation settings, rigorous and fine-grained evaluation was necessary. Accordingly, we decomposed more than three thousand dialogue turns into MIUs and classified failures into four clinically meaningful error types, enabling identification of characteristic instruction-following and conversational control limitations that are invisible to aggregate realism scores alone. This MIU-level validation framework shifts validation from subjective impressions or downstream outcomes to mechanistic understanding of model behavior and provides a transferable methodology for assessing whether emerging STS models are educationally fit for use in high-stakes clinical skills training.\u003c/p\u003e \u003cp\u003eThis study also advances prior work on automated OSCE evaluation by prospectively operationalizing real-time, learner-facing assessment within routine clinical education and formally evaluating its usability. Unlike earlier studies that demonstrated feasibility using retrospective or offline transcript analysis,\u003csup\u003e17\u0026ndash;19\u003c/sup\u003e CPX-RTE delivered checklist-based scoring and structured narrative feedback to medical students within seconds after each encounter, at scale, during routine clinical education. Importantly, this real-time feedback was not only technically validated against expert raters but also systematically evaluated by survey of usability from sixty end users, providing direct evidence on how immediate AI-generated feedback is perceived, interpreted, and integrated in real educational practice rather than simulated or post hoc settings.\u003c/p\u003e \u003cp\u003eAt the same time, our findings clarify clear boundaries of what automated evaluation can and cannot replace. Although CPX-RTE achieved high agreement with expert raters for many checklist-based items, residual disagreement was systematically concentrated in Patient\u0026ndash;Physician Interaction domains that require nuanced affective and relational judgment and are only weakly inferable from dialogue transcripts. A parallel limitation applies to physical examination assessment: while agreement for physical examination items was acceptable, scoring necessarily relied on verbalized intent rather than whether the examination was performed correctly and appropriately. As a result, both interpersonal competencies and hands-on examination quality remain fundamentally underdetermined by transcript-based automation alone. These results indicate that automated OSCE evaluation should not be positioned as a fully autonomous assessor; instead, it must be embedded within a human-in-the-loop framework in which automated systems cover scalable, standardized components, while human evaluators retain responsibility for competencies that are experiential, relational, or procedural in nature.\u003c/p\u003e \u003cp\u003eLooking forward, learner feedback suggests that the most pressing unmet need lies not only in granular checklist scoring but also in higher-level support that helps learners \u003cem\u003eimprove\u003c/em\u003e, not merely \u003cem\u003ebe evaluated\u003c/em\u003e. Students consistently requested feedback that explains why certain questions or actions mattered, highlights missed reasoning steps, and makes progress visible across repeated encounters rather than presenting isolated, encounter-level outputs. If implemented as learners envision\u0026mdash;combining timely feedback with clinical reasoning guidance and longitudinal progress tracking\u0026mdash;AI-powered education tools could enable far more frequent, structured, and scalable deliberate practice than is feasible with faculty time alone. In this sense, continued advances in AI create a plausible optimistic outlook that clinical skills education could become more accessible and less dependent on local instructional resources.\u003csup\u003e\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eHowever, this democratizing promise should be interpreted with caution, because educational quality remains sensitive to model capability and affordability. As illustrated by our STS-based VSP evaluation, higher-capacity configurations supported more immersive and coherent simulation, whereas resource-limited configurations\u0026mdash;though potentially \u0026ldquo;better than nothing\u0026rdquo;\u0026mdash;were more prone to immersion-breaking conversational errors that can distract from learning. This cost\u0026ndash;capability gradient implies an equity risk: even as AI reduces the logistical barriers to simulation-based education, settings that cannot afford sufficiently capable models (including LMICs and other resource-limited programs) may be left with systematically lower-fidelity training, thereby reproducing or widening existing disparities in educational quality. Accordingly, efforts to leverage AI for the broader availability of high-quality clinical education should explicitly define minimum fidelity thresholds and remain attentive to how model choice shapes what learners actually practice.\u003c/p\u003e \u003cdiv id=\"Sec22\" class=\"Section2\"\u003e \u003ch2\u003eLimitation\u003c/h2\u003e \u003cp\u003eSeveral limitations should be considered when interpreting these findings. First, this study was conducted at a single institution and focused on a single chief complaint, requiring future multi-institutional and multi-scenario studies to enhance generalizability. Second, exposure to CPX-VSP and CPX-RTE occurred in a single session, raising the possibility of novelty effects influencing usability and immersion ratings. Third, it did not examine whether CPX-MATE translates into improvements in clinical performance. Fourth, participants were not randomly assigned to model exposure conditions, which may introduce selection or order effects. Despite these limitations, the prospective, workflow-integrated design provides a strong foundation for subsequent studies. The web-based application is currently being further developed to support ubiquitous, on-demand learning, enabling students to engage with the platform independently at any time and location, beyond formal instructional settings, while also expanding the range of clinical presentations. Ongoing work will therefore incorporate longitudinal exposure and evaluate downstream effects on CPX performance and learning transfer through randomized controlled trials.\u003c/p\u003e \u003c/div\u003e"},{"header":"CONCLUSION","content":"\u003cp\u003eIn conclusion, the development and prospective validation of CPX-MATE within an authentic CPX workflow demonstrated that CPX-VSP enables accurate and immersive VSP simulation, while CPX-RTE achieved checklist-based evaluation performance comparable to that of human faculty during HSP encounters. These findings indicate that end-to-end AI-assisted CPX platforms can be systematically evaluated across both performance and usability dimensions, supporting their potential role as complementary tools within structured CPX training while highlighting the need for careful, context-aware deployment.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eAuthor contributions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eJ.W.S.: Methodology, Software, Formal analysis, Data curation, Validation, Writing\u0026nbsp;– original draft\u003c/p\u003e\n\u003cp\u003eM.K.: Software, Formal analysis, Investigation, Data curation, Validation, Writing\u0026nbsp;– original draft\u003c/p\u003e\n\u003cp\u003eC.H.: Investigation, Data curation, Validation\u003c/p\u003e\n\u003cp\u003eY.S.K.: Investigation, Resources\u003c/p\u003e\n\u003cp\u003eJ.C.: Investigation, Resources\u003c/p\u003e\n\u003cp\u003eJ.H.K.: Investigation, Resources\u003c/p\u003e\n\u003cp\u003eJ.M.: Investigation, Resources\u003c/p\u003e\n\u003cp\u003eA.C.: Visualization, Resources\u003c/p\u003e\n\u003cp\u003eH.Y.: Visualization, Resources\u003c/p\u003e\n\u003cp\u003eS.G.W.L.: Visualization, Resources\u003c/p\u003e\n\u003cp\u003eS.C.Y.: Conceptualization, Methodology, Writing\u0026nbsp;– review \u0026amp; editing, Supervision\u003c/p\u003e\n\u003cp\u003eC.P.: Conceptualization, Methodology, Investigation, Writing – review \u0026amp; editing, Project administration, Supervision\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDisclosure of conflicts of interest\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eDr. You is an inventor, along with J.W.S., M.K., and C.P., on a pending Korean patent application related to the CPX-MATE platform (DP-2026-0104). Outside the submitted work, Dr. You reports grants from Daiichi Sankyo and VUNO, receives compensation as an Associate Editor for JACC, and serves as chief executive officer of PHI Digital Healthcare. The remaining authors declare no competing interests.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData Availability\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe data supporting the findings of this study are not publicly available due to institutional data governance policies and the inclusion of potentially identifiable educational performance data. De-identified and aggregated data may be made available from the corresponding author upon reasonable request.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCode Availability\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe source code of the CPX-MATE platform is not publicly available due to a pending patent application and institutional intellectual property considerations. De-identified statistical analysis code supporting the findings of this study is available from the corresponding author upon reasonable request.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEthics approval \u0026amp; consent\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe study was approved by the IRB of Yonsei University College of Medicine (IRB No. [4-2025-1312]). Participation was voluntary and uncompensated, and had no impact on academic evaluation. Written informed consent was obtained from all participants prior to enrollment. All recordings and transcripts were de-identified prior to analysis. The study was conducted in accordance with the Declaration of Helsinki.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding statement\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eBa, H., Zhang, L., He, X. \u0026amp; Li, S. Knowledge mapping and global trends in the field of the objective structured clinical examination: bibliometric and visual analysis (2004-2023). \u003cem\u003eJMIR Medical Education\u003c/em\u003e \u003cstrong\u003e10\u003c/strong\u003e, e57772 (2024).\u003c/li\u003e\n\u003cli\u003eDewan, P., Khalil, S. \u0026amp; Gupta, P. Objective structured clinical examination for teaching and assessment: Evidence-based critique. \u003cem\u003eClinical Epidemiology and Global Health\u003c/em\u003e \u003cstrong\u003e25\u003c/strong\u003e, 101477 (2024).\u003c/li\u003e\n\u003cli\u003eLavigne, E.\u003cem\u003e et al.\u003c/em\u003e AI-Standardized Clinical Examination training on OSCE performance. \u003cem\u003eNEJM AI\u003c/em\u003e \u003cstrong\u003e2\u003c/strong\u003e, AIoa2500066 (2025).\u003c/li\u003e\n\u003cli\u003ePatr\u0026iacute;cio, M. F., Juli\u0026atilde;o, M., Fareleira, F. \u0026amp; Carneiro, A. V. Is the OSCE a feasible tool to assess competencies in undergraduate medical education? \u003cem\u003eMedical teacher\u003c/em\u003e \u003cstrong\u003e35\u003c/strong\u003e, 503-514 (2013).\u003c/li\u003e\n\u003cli\u003eMukurunge, E., Nyoni, C. N. \u0026amp; Hugo, L. Assessment approaches in undergraduate health professions education: towards the development of feasible assessment approaches for low-resource settings. \u003cem\u003eBMC Medical Education\u003c/em\u003e \u003cstrong\u003e24\u003c/strong\u003e, 318 (2024).\u003c/li\u003e\n\u003cli\u003eAtaro, G., Worku, S. \u0026amp; Asaminew, T. Experience and challenges of objective structured clinical examination (OSCE): perspective of students and examiners in a clinical department of Ethiopian University. \u003cem\u003eEthiopian Journal of Health Sciences\u003c/em\u003e \u003cstrong\u003e30\u003c/strong\u003e (2020).\u003c/li\u003e\n\u003cli\u003eCook, D. A., Overgaard, J., Pankratz, V. S., Del Fiol, G. \u0026amp; Aakre, C. A. Virtual patients using large language models: Scalable, contextualized simulation of clinician-patient dialogue with feedback. \u003cem\u003eJournal of Medical Internet Research\u003c/em\u003e \u003cstrong\u003e27\u003c/strong\u003e, e68486 (2025).\u003c/li\u003e\n\u003cli\u003eHolderried, F.\u003cem\u003e et al.\u003c/em\u003e A language model\u0026ndash;powered simulated patient with automated feedback for history taking: Prospective study. \u003cem\u003eJMIR Medical Education\u003c/em\u003e \u003cstrong\u003e10\u003c/strong\u003e, e59213 (2024).\u003c/li\u003e\n\u003cli\u003eYamamoto, A.\u003cem\u003e et al.\u003c/em\u003e Enhancing medical interview skills through AI-Simulated patient interactions: nonrandomized controlled trial. \u003cem\u003eJMIR medical education\u003c/em\u003e \u003cstrong\u003e10\u003c/strong\u003e, e58753 (2024).\u003c/li\u003e\n\u003cli\u003eLuo, M.-J.\u003cem\u003e et al.\u003c/em\u003e A large language model digital patient system enhances ophthalmology history taking skills. \u003cem\u003eNPJ Digital Medicine\u003c/em\u003e \u003cstrong\u003e8\u003c/strong\u003e, 502 (2025).\u003c/li\u003e\n\u003cli\u003eJacobs, C.\u003cem\u003e et al.\u003c/em\u003e Application of AI Communication Training Tools in Medical Undergraduate Education: Mixed Methods Feasibility Study Within a Primary Care Context. \u003cem\u003eJMIR Med Educ\u003c/em\u003e \u003cstrong\u003e11\u003c/strong\u003e, e70766 (2025). https://doi.org/10.2196/70766\u003c/li\u003e\n\u003cli\u003ePu, Y.\u003cem\u003e et al.\u003c/em\u003e in \u003cem\u003eProc. Interspeech 2025.\u003c/em\u003e 26-30.\u003c/li\u003e\n\u003cli\u003eTakata, T., Yamada, R., Ren\u0026eacute;, A. O. N., Xu, K. \u0026amp; Fujimoto, M. in \u003cem\u003e2024 Joint 13th International Conference on Soft Computing and Intelligent Systems and 25th International Symposium on Advanced Intelligent Systems (SCIS\u0026amp;ISIS).\u003c/em\u003e 1-5 (IEEE).\u003c/li\u003e\n\u003cli\u003eOpenAI. \u003cem\u003eIntroducing gpt-realtime\u003c/em\u003e, \u0026lt;https://openai.com/ko-KR/index/introducing-gpt-realtime/\u0026gt; (2025).\u003c/li\u003e\n\u003cli\u003eGoogle. \u003cem\u003eGet started with Live API\u003c/em\u003e, \u0026lt;https://ai.google.dev/gemini-api/docs/live?hl=ko\u0026amp;example=mic-stream\u0026gt; (2025).\u003c/li\u003e\n\u003cli\u003eLee, Y. C. Rethinking artificial intelligence in medicine: from tools to agents. \u003cem\u003eClinical and experimental emergency medicine\u003c/em\u003e \u003cstrong\u003e12\u003c/strong\u003e, 101 (2025).\u003c/li\u003e\n\u003cli\u003eGeathers, J.\u003cem\u003e et al.\u003c/em\u003e in \u003cem\u003eInternational Conference on Artificial Intelligence in Education.\u003c/em\u003e 231-245 (Springer).\u003c/li\u003e\n\u003cli\u003eShakur, A. H.\u003cem\u003e et al.\u003c/em\u003e Large Language Models for Medical OSCE Assessment: A Novel Approach to Transcript Analysis. \u003cem\u003earXiv preprint arXiv:2410.12858\u003c/em\u003e (2024).\u003c/li\u003e\n\u003cli\u003eHolcomb, M. J.\u003cem\u003e et al.\u003c/em\u003e Zero-Shot Multimodal Question Answering for Assessment of Medical Student OSCE Physical Exam Videos. \u003cem\u003emedRxiv\u003c/em\u003e, 2024.2006. 2005.24308467 (2024).\u003c/li\u003e\n\u003cli\u003eOpenAI. \u003cem\u003eIntroducing Whisper\u003c/em\u003e, \u0026lt;https://openai.com/ko-KR/index/whisper/\u0026gt; (2025).\u003c/li\u003e\n\u003cli\u003eOpenAI. \u003cem\u003eIntroducing GPT-5\u003c/em\u003e, \u0026lt;https://openai.com/ko-KR/index/introducing-gpt-5/\u0026gt; (2025).\u003c/li\u003e\n\u003cli\u003eRoh, H.\u003cem\u003e et al.\u003c/em\u003e [Development of guide to clinical performance and basic clinical skills for medical students]. \u003cem\u003eKorean J Med Educ\u003c/em\u003e \u003cstrong\u003e27\u003c/strong\u003e, 309-319 (2015). https://doi.org/10.3946/kjme.2015.27.4.309\u003c/li\u003e\n\u003cli\u003eLewis, J. R. The system usability scale: past, present, and future. \u003cem\u003eInternational Journal of Human\u0026ndash;Computer Interaction\u003c/em\u003e \u003cstrong\u003e34\u003c/strong\u003e, 577-590 (2018).\u003c/li\u003e\n\u003cli\u003eAhmed, S. K.\u003cem\u003e et al.\u003c/em\u003e Using thematic analysis in qualitative research. \u003cem\u003eJournal of Medicine, Surgery, and Public Health\u003c/em\u003e \u003cstrong\u003e6\u003c/strong\u003e, 100198 (2025).\u003c/li\u003e\n\u003cli\u003eKhan, S. \u003cem\u003eSal Khan\u0026rsquo;s 2023 TED Talk: AI in the classroom can transform education\u003c/em\u003e, \u0026lt;https://blog.khanacademy.org/sal-khans-2023-ted-talk-ai-in-the-classroom-can-transform-education/\u0026gt; (2023).\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"npj-digital-medicine","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"npjdigitalmed","sideBox":"Learn more about [npj Digital Medicine](http://www.nature.com/npjdigitalmed/)","snPcode":"41746","submissionUrl":"https://submission.springernature.com/new-submission/41746/3","title":"npj Digital Medicine","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-8972105/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8972105/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e \u003cp\u003eObjective Structured Clinical Examination (OSCE; Clinical Performance Examination [CPX] in South Korea) is a high-stakes assessment of clinical performance, communication, and reasoning during time-limited patient encounters. As AI-enabled virtual standardized patient (VSP) simulation and automated scoring are introduced for OSCE-like training, prospective evidence is needed on how such systems perform and are perceived when embedded in real educational workflows.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eWe developed CPX with Medical students\u0026rsquo; Assistant for Training and Evaluation (CPX-MATE), a web-based platform integrating (1) CPX with Virtual Standardized Patient (CPX-VSP), real-time voice dialogue with a VSP using speech-to-speech (STS) models, and (2) CPX with Real-Time Evaluator (CPX-RTE), automated transcription, checklist-based scoring, and feedback from encounter audio using a Speech-to-Text model and a large language model. During an emergency medicine clerkship (Nov 2025-Jan 2026), 60 senior medical students completed two 12-min CPX encounters (VSP with acute pancreatitis; HSP with ureteral stone) with immediate CPX-RTE feedback. For CPX-VSP, students were assigned to either a full-capacity or a resource-limited STS configuration (n\u0026thinsp;=\u0026thinsp;30 each). Dialogue fidelity was evaluated by turn-by-turn analysis of student\u0026ndash;VSP exchanges, classifying responses into clinically meaningful error types (tangential, oversharing, role-breaking, off-script). CPX-RTE performance was assessed by agreement (Gwet\u0026rsquo;s AC1) with professor real-time and resident video-based ratings using a 45-item checklist. Usability of CPX-VSP and CPX-RTE, with overall system usability scale (SUS), were surveyed, and mean per-session costs for CPX-VSP and CPX-RTE were calculated.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eAcross 3,282 dialogue turns, overall error rates were 1.77% versus 9.43% for full-capacity versus resource-limited STS configurations (p\u0026thinsp;\u0026lt;\u0026thinsp;0.001), driven by fewer tangential and oversharing responses; no off-script errors were observed. The mean per-session cost was \u003cspan\u003e$\u003c/span\u003e0.12 for resource-limited configuration and \u003cspan\u003e$\u003c/span\u003e0.78 for full-capacity configuration. CPX-RTE showed high agreement with human ratings (AC1\u0026thinsp;=\u0026thinsp;0.916 vs professor; 0.916 vs resident), with slightly different levels of agreement across four sections, and high usability across all domains (mean scores, 4.65\u0026ndash;4.92), with a per-session cost of \u003cspan\u003e$\u003c/span\u003e0.17. CPX-MATE demonstrated good overall usability (median [IQR] of 77.5 [70.0\u0026ndash;85.0]).\u003c/p\u003e\u003ch2\u003eConclusions\u003c/h2\u003e \u003cp\u003eEmbedded within a prospective clinical clerkship, CPX-MATE demonstrated operational fidelity and human-level checklist agreement as an end-to-end, voice-based AI-assisted OSCE platform. This real-world deployment supports its scalable integration as a complementary assessment tool while highlighting the importance of systematic validation and context-aware implementation in medical education.\u003c/p\u003e","manuscriptTitle":"Development and Prospective Validation of CPX-MATE: An End-to-End Medical Education Platform Integrating Voice-Based Virtual Patient Simulation and Automated Real-time Evaluation","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-03-08 17:03:12","doi":"10.21203/rs.3.rs-8972105/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-05-18T15:17:46+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-17T09:37:29+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-12T18:01:51+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"152870472484347592553117740475247597078","date":"2026-04-20T06:27:38+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-09T18:18:17+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"214876834263194536765343198504009948601","date":"2026-04-09T18:06:18+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"332412254252362374629278931092339196913","date":"2026-03-20T11:35:16+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"164164433384043107432512720638935539840","date":"2026-03-20T02:13:54+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-03-03T00:22:14+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-02-27T01:38:24+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-02-26T08:54:22+00:00","index":"","fulltext":""},{"type":"submitted","content":"npj Digital Medicine","date":"2026-02-26T01:33:45+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"npj-digital-medicine","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"npjdigitalmed","sideBox":"Learn more about [npj Digital Medicine](http://www.nature.com/npjdigitalmed/)","snPcode":"41746","submissionUrl":"https://submission.springernature.com/new-submission/41746/3","title":"npj Digital Medicine","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"3c6ff58e-4ac3-422f-83f9-d912470e3143","owner":[],"postedDate":"March 8th, 2026","published":true,"recentEditorialEvents":[{"type":"decision","content":"Revision requested","date":"2026-05-18T15:17:46+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-17T09:37:29+00:00","index":55,"fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-12T18:01:51+00:00","index":54,"fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"in-revision","subjectAreas":[{"id":63805589,"name":"Biological sciences/Computational biology and bioinformatics"},{"id":63805590,"name":"Health sciences/Diseases"},{"id":63805591,"name":"Health sciences/Health care"},{"id":63805592,"name":"Physical sciences/Mathematics and computing"},{"id":63805593,"name":"Health sciences/Medical research"}],"tags":[],"updatedAt":"2026-05-18T15:25:33+00:00","versionOfRecord":[],"versionCreatedAt":"2026-03-08 17:03:12","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8972105","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8972105","identity":"rs-8972105","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}