Large Language Models for the assessment of medical students’ clinical decision-making

Large Language Models for the assessment of medical students’ clinical decision-making | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Large Language Models for the assessment of medical students’ clinical decision-making Sina Chole Benker, Jonathan Vollprecht, Cihan Papan, Max Hao Lu, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6660928/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 6 You are reading this latest preprint version Abstract The assessment of medical students’ clinical decision making (CDM) skills is fundamental to healthcare education as it identifies knowledge gaps, enables targeted feedback, and validates that graduates meet professional competency standards. However, traditional assessment methods relying on expert human raters are resource-intensive and difficult to scale. This study investigated whether Large Language Models (LLMs), i.e. ChatGPT, could serve as automated assessors of medical students' CDM skills. We compared LLM-generated assessments with ratings from two humans across 21 medical student history-taking conversations using the Clinical Reasoning Indicator - Health Training Indicator (CRI-HTI). The results showed strong agreement between human raters and the LLM ( ICC = .675-.782, MAE = 0.343), with over 91% of ratings within 0.5 points of each other. Item-level analysis revealed moderate to excellent reliability across all eight CRI-HTI criteria. Additionally, we tested for gender bias by presenting identical transcripts with different gender designations (men, women, neutral) to the LLM. No significant differences were found between gendered prompts ( p > .05), suggesting that the LLM maintained consistent evaluation standards regardless of the subject's gender. These findings provide empirical evidence that LLMs could serve as consistent and gender-indiscriminating raters for supporting CDM assessment in medical education, potentially offering a scalable solution for providing timely feedback to medical students. Large Language Models Assessment Clinical Decision Making Clinical Reasoning Medical Education Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Practice points Large Language Models (LLMs) demonstrate strong agreement with human raters when assessing medical students' clinical decision-making skills, showing good to excellent reliability (ICC = .675-.782). Over 91% of LLM ratings were within 0.5 points of human ratings on the Clinical Reasoning Indicator - Health Training Indicator (CRI-HTI), suggesting high consistency across evaluation methods. LLMs showed no significant gender bias when assessing clinical decision-making, maintaining consistent evaluation standards regardless of whether subjects were identified as male, female, or gender-neutral. A hybrid assessment approach combining LLM efficiency with human expert oversight may provide the optimal framework for scaling up clinical decision-making assessment in medical education. Automated LLM assessment could address current limitations in medical education by enabling more frequent, faster, and cost-effective feedback on clinical reasoning skills. Introduction Clinical decision-making (CDM) is a central process in healthcare where physicians gather, evaluate, and interpret relevant information about a patient's health status to make informed decisions regarding diagnosis and treatment [1]. CDM is crucial as it directly impacts the quality of patient care by ensuring that the right decisions are made at the right time. Clinical reasoning, the cognitive process that underpins CDM, intertwines analytical thinking with pattern recognition to form coherent diagnostic hypotheses and treatment plans based on patient data [2]. Effective CDM skills help minimize diagnostic errors and identify the best treatment approaches [3]. Regular practice with simulated medical history conversations and other clinical exercises is essential for developing these abilities [4, 5]. In CDM trainings, CDM mastery is individually assessed to provide healthcare professionals with feedback, thereby supporting a learning process and encouraging further development. The gold standard for evaluating medical students' CDM is the use of human raters. This assessment demands both extensive experience and advanced medical expertise of the human raters. It also requires the capacity for flexible interpretation and nuanced assessment of complex clinical scenarios. However, human CDM raters present two major limitations: they are not always available, and costly to employ. These limit the frequency and consistency of CDM assessments, resulting in fewer opportunities for learners to receive timely and constructive feedback essential for their professional growth. These challenges we foresee to be alleviated by introducing Large Language Models (LLMs) for CDM assessment. LLMs may help evaluating CDM LLMs are advanced machine learning systems that process and generate natural language. Utilizing deep learning architectures like transformers, they are trained on extensive text datasets to recognize linguistic patterns and perform tasks such as translation, summarization, and question answering. Models like GPT (Generative Pretrained Transformer) generate coherent text by predicting words based on input [6]. LLMs can help automate documentation, offer evidence-based recommendations, and enhance medical education by simulating realistic clinical scenarios (e.g., [7, 8]). Plenty of literature shows that although many LLMs are advanced AI models trained on vast amounts of text, they are still not ready for autonomous CDM [9]. However, LMMs could be potentially capable of delivering objective and systematic CDM evaluations based on predefined criteria [10]. With LLMs' potential to analyze both qualitative and quantitative data, this approach holds huge potential to assess and interpret text data and to complement the likelihood of human evaluation errors. Despite the fact that in the medical field there is first evidence that LLMs may present a cost-effective and efficient method for advancing CDM training [8], there is currently no evidence about the consistency and accuracy of LLMs to assess medical history conversations and clinical competencies such as CDM, compared to the current gold standard, human raters. We seek to investigate whether LLMs can reach human rater standards (non-inferiority). Assessment biases of humans and LLMs When implementing LLM-based assessment framework, it is crucial to anticipate potential limitations of current models, such inherent biases that may impact the objectivity of the assessment. Cognitive biases of the assessors are particularly relevant in assessment contexts, because they can systematically lead to unfair, inaccurate, or harmful decisions. Common examples include the contrast effect, where current assessments are influenced by comparisons to previous cases [11]. A prominent form of systematic bias is gender bias, where the subject's gender may unconsciously influence the evaluation [12]. It is well established that human raters are susceptible to biases influenced by their prior experiences or expectations, which can affect the objectivity of their evaluations [13]. Research has also shown that LLMs, trained extensively on real world data, are also not always value-neutral or bias-free. For example, study has shown that ChatGPT tend to be more politically left leaning and ideologically liberal [14]. Critically, LLMs have tendencies to rate the male gender as the more dominant [15], which may have an impact on how they assess the CDM of medical student conversations. Therefore, we were particularly interested in whether LLMs exhibit more biases towards different genders when evaluating their CDM mastery. The current study To mitigate the limitations of human raters related to scalability, availability, and consistency, we set out to examine the feasibility of supporting human raters with LLMs. We compared the ratings of LLMs with the current gold standards of human raters, and we also compared LLMs own ratings across different genders. Such an approach allowed us to investigate LLMs can function as a reliable compliment to human raters and whether it exhibit discriminatory behaviors towards different genders. In our study, we first compared the LLMs evaluation with human evaluations and looked at the reliability evidence of using LLM raters alongside human raters. Next, we sought to investigate the extent to which LLMs exhibit a gender bias in their assessments. For the latter, the LLM was asked to evaluate the subject under three conditions: neutral (unspecified gender), men, and women. Referring to Thakur (2023), we hypothesized that LLMs might exhibit a tendency to rate female students’ CDM subjects less favorably compared to males. Our research questions are: Research Question 1 (RQ 1): To what extent do LLMs produce ratings that are consistent with human raters, the current gold standard, when assessing students’ performance on the CRI-HTI? Research question 2 (RQ 2): Does an LLM exhibit gender bias when assessing students’ CDM? Methods To address the research questions, we conducted an experimental design in the Summer Term 2024. Ethics approval was obtained from the ethics board (“Ethik-Kommission Westfalen-Lippe”) under the reference 2023-438-f-N. Informed consent was obtained from all participants. We begin by detailing the distinct methodological approaches for Research Questions 1 and 2, followed by a comprehensive description of the instrument employed to evaluate clinical decision making (CDM). The section concludes with an explanation of the various analytical metrics utilized. Study procedure – evaluate LLM rater reliability (RQ1) Twenty-one medical students ( mean semester = 4, 14 females) each completed four simulated patient-doctor CDM cases, representing four distinct neurological or neurosurgical emergency situations (Fig. 1 ). The CDM conversations were transcribed for all participants. We used OpenAI’s ChatGPT free version, GPT3.5, as the LLM to evaluate the transcript based on the prompt used in Brügge et al (ibid.). ChatGPT would then evaluate the transcripts based on the criteria outlined in the prompt. The language model evaluated all cases for each participant. We then transferred ChatGPT's evaluations into a table to derive statistical values (see below) from the data. Study procedure – LLMs possible gender bias To investigate gender bias, the same CDM conversation transcript from each participant was presented to ChatGPT three times (Fig. 2 , Table 1 ). The first instance did not specify any gender, the second identified the participant as male, and the third as female. Each version of the transcript was submitted in separate chat windows, ensuring that ChatGPT did not reference previous interactions and could evaluate each version independently. This approach allowed for a comparative assessment of potential gender biases presented in the LLM ratings. The gender-non-specific designation of the first conversation transcript served as a baseline for comparison. The baseline allowed us to assess whether the subsequent gendered versions elicit different responses and identify any potential biases in the evaluations. Table 1 The three different prompts (neutral, female, male) each inserted in LLM chats for transcript based rating) Prompt neutral (i.e., gender not specified) Prompt female Prompt male At this point, you will assess the quality of the third-semester medical student conducting a history-taking conversation. At this point, you will assess the quality of a female third-semester medical student conducting a history-taking conversation. At this point, you will assess the quality of a male third-semester medical student conducting a history-taking conversation. [The following input was derived from the CRI-HT questionnaire (Fürstenberg et al., 2016), and was identical for all three prompts:] Your assessment should include the following eight criteria 1. Assess whether the user has taken control of the conversation to obtain the necessary information. 2. Assess whether the user recognizes all relevant information. 3. Assess whether the user formulates targeted questions so that he can capture and specify the symptoms in detail. 4. Assess whether the questions of the user suggest that specific causes or circumstances lead to certain symptoms. 5. Assess whether the user asks questions in a logical sequence. 6. Assess whether the user reassures the patient that he has received the correct information from the patient. 7. Assess whether the user has summarized his collected information before ending the conversation. 8. Assess whether the user has collected sufficient information of high quality at an appropriate speed. Assign each of the eight criteria a score according to the following scheme: 1 - Does not meet the criterion 2 - Rather does not meet the criterion 3 - Partially meets the criterion 4 - Rather meets the criterion 5 - Fully meets the criterion Explain the evaluation with two sentences. Note. The transcripts were originally in German and the LLM was prompted using German language instructions. Instrument to assess CDM The Clinical Reasoning Indicator - Health Training Indicator (CRI-HTI), developed and validated by Fürstenberg et al. [ 2 ], is a structured tool for evaluating the clinical reasoning skills and training level of medical students and healthcare professionals. This tool utilizes a Likert scale to measure eight specific criteria that capture key aspects of clinical reasoning, organized into three core competency areas: "Focusing Questions" (Items 1–3), "Creating Context" (Items 4–6), and "Securing Information" (Items 7–8). These competencies encompass the participant's ability to guide patient interactions, actively recognize and respond to relevant information, and accurately identify and specify symptoms. Further, the tool assesses the clinician's skill in asking targeted questions that facilitate pathophysiological reasoning, arranging these questions in a logical sequence, and checking with the patient to ensure understanding. It also evaluates the ability to effectively summarize information and reflect on the quality and effectiveness of the conversation. Together, these criteria provide a comprehensive framework for assessing clinical reasoning and CDM, offering insights into the clinician's capacity to conduct focused, coherent, and patient-centered interactions. The CRI-HTI is thus a valuable tool for both formative assessment in clinical training and for structured feedback aimed at enhancing essential clinical competencies. The Likert scale ranges from 1 to 5, where 5 indicates "Completely agrees with the criterion" and 1 indicates "Does not agree with the criterion." The CRI-HTI serves as the basis of the prompt provided at the beginning of the LLM evaluation, guiding the LLM to assign scores on the Likert scale based on the criteria that assess diagnostic skills, problem-solving, and decision-making abilities. To achieve greater sensitivity to change, the LLM was also allowed to award half points. Analysis The analysis was performed in R, and the full analysis code is available in the supplemental files. In the statistical analysis, we aggregated the ratings from the CRI-HTI scale for each case to derive the total score. Subsequently, we compared the newly calculated values assessed by ChatGPT using the CRI-HTI scale with the ratings provided by human raters. Specifically, the following analyses were conducted: Intraclass Correlation Coefficient (ICC). The ICC is a statistical measure for assessing the reliability of ratings, particularly in contexts involving multiple raters. By evaluating the degree of agreement among raters, the ICC provides insights into the consistency and dependability of the ratings, whether conducted by human evaluators or the LLM. ICC values approaching 1 indicate a high level of agreement, suggesting reliable assessments and agreement, whereas values closer to 0 indicate low agreement and thus lower reliability. The ICC can be interpreted across different ranges to provide a clearer understanding of rating consistency. Values from 0 to .2 suggest very low agreement, indicating that the ratings are largely inconsistent and potentially unreliable. ICC values between .3 and .4 reflect moderate agreement, though still relatively low, which suggests limited consistency among raters. An ICC range of .5 to .6 represents acceptable agreement. When ICC values fall between .7 and .8, this suggests good agreement among raters, indicating that the ratings are generally reliable and suitable for many practical applications. ICC values between .9 and 1.0 denote excellent agreement, with nearly perfect consistency. By providing a robust estimate of rating consistency, the ICC enables researchers to determine whether the rating process is sufficiently reliable to justify further analyses based on the obtained data. Mean absolute error (MAE). The MAE is a statistical metric that measures the average absolute difference between predicted and actual values. It quantifies the overall accuracy of a predictive model by assessing how close the predictions are to the true values. A lower MAE indicates higher accuracy, while a higher MAE suggests greater deviation. In our case we use the MAE to define the deviation between human and LLM rating. Agreement within a range. Statistical agreement refers to the degree of consistency or concordance between different measurements, raters, or diagnostic methods in a medical or research setting. It is an intuitive measure that shows the proportion of ratings that fall within certain thresholds of each other (the current study uses 1 and 0.5). It assesses how closely independent evaluations align when measuring the same phenomenon. High agreement indicates reliability and reproducibility. Spearman correlation. Spearman's rank correlation coefficient (ρ) is a non-parametric measure that assesses the strength and direction of association between two ranked variables. Unlike Pearson correlation, Spearman's correlation evaluates monotonic relationships and is robust against outliers and non-normal distributions. The coefficient ranges from − 1 to + 1, where + 1 indicates a perfect positive correlation, -1 signifies a perfect negative correlation, and 0 represents no correlation. In our study, Spearman correlation was calculated to evaluate the association between human and LLM ratings of CDM skills. Values between .00-.19 indicate very weak correlation, .20-.39 weak correlation, .40-.59 moderate correlation, .60-.79 strong correlation, and .80 − 1.00 very strong correlation. ANOVA . To determine whether there were differences LLMs ratings based on gender, we compared the mean scores across the different gender categories and performed a one-way ANOVA with post-hoc Tukey's HSD test for multiple comparisons. Results RQ1: To what extent do LLMs produce ratings that are consistent with human raters, the current gold standard, when assessing students’ performance on the CRI-HTI? The results indicate high agreement level between both human raters and the LLM (Table 2 ). Table 2 Overall agreement metrics between human raters and LLM ratings of CDM Comparison ICC MAE Agreement ≤ 0.5 Agreement ≤ 1 r Human 1 vs LLM .675 0.343 0.919 0.995 .718 Human 2 vs LLM .782 0.343 0.919 0.995 .854 Note. ICC = Intraclass Correlation Coefficient; MAE = Mean Absolute Error; r = Spearman Correlation. Agreement values represent percentage of ratings within 0.5 and 1.0 points respectively. Correlation values represent Spearman correlations. When we look at each participants performance across each case, we see the ICC values (.673 between Human 1 and LLM and .782 for Human 2 and LLM) suggest good to excellent reliability. Both human raters showed identical patterns of absolute agreement with the LLM, with a MAE of 0.34 points. Notably, over 91% of all ratings were within half a point of each other, and nearly all ratings (99.5%) fell within one point of agreement. The Spearman correlations were strong, particularly for Human 2 ( r s = .86), indicating robust rank-order agreement between human and LLM ratings. When looking at the total scores given by different raters through a jitter plots, which is a special kind of scatter plots that allow better visualization of overlapping points, we see that LLM rater’s scoring correlates strongly with that of two human raters (Fig. 3 ). Interestingly, LLM shows more alignment with both human raters than human raters align with each other. When looking at the aggregated rating distribution of scores, we noticed similar alignments (Fig. 4 ). For Human 1 versus LLM ratings, the highest concentration of agreements appears in the 3.0–4.0 range, with a notable cluster around the 3.5-4.0 mark. The distribution shows some spread, indicating moderate agreement between Human 1 and LLM ratings, though there is visible variance in the lower (1.5–2.5) and higher (4.0-4.5) ranges. The Human 2 versus LLM comparison exhibits a similar pattern but with slightly stronger clustering in the higher score ranges (3.5–4.5). There appears to be more concentrated agreement between Human 2 and LLM ratings in the upper score ranges, as evidenced by the darker blue coloring in these areas. Both distributions suggest a tendency toward higher ratings overall, with fewer instances of low scores (1.0–2.0) from either human raters or the LLM system. Table 3 Item-level Agreement Metrics Between Human vs LLM CDM Ratings Second-order latent factors Comparison ICC MAE Agreement ≤ 0.5 Agreement ≤ 1 r Factor 1: Focusing questions 1. Taking the lead in the conversation Human 1 vs LLM .584 0.343 0.867 1.000 .735 Human 2 vs LLM .667 0.343 0.867 1.000 .803 2. Recognizing and responding to relevant information Human 1 vs LLM .743 0.271 0.928 1.000 .739 Human 2 vs LLM .829 0.271 0.928 1.000 .897 3. Specifying symptoms Human 1 vs LLM .700 0.277 0.988 1.000 .733 Human 2 vs LLM .798 0.277 0.988 1.000 .825 Factor 2: Creating context 4. Asking specific questions that point to pathophysiological thinking Human 1 vs LLM .559 0.380 0.843 1.000 .542 Human 2 vs LLM .758 0.380 0.843 1.000 .878 5. Putting questions in a logical order Human 1 vs LLM .733 0.283 0.976 1.000 .750 Human 2 vs LLM .823 0.283 0.976 1.000 .850 6. Checking with the patient Human 1 vs LLM .827 0.301 0.952 1.000 .795 Human 2 vs LLM .877 0.301 0.952 1.000 .916 Factor 3: Securing information 7. Summarizing Human 1 vs LLM .707 0.506 0.916 0.964 .863 Human 2 vs LLM .770 0.506 0.916 0.964 .739 8. Collected data and effectiveness of the conversation Human 1 vs LLM .528 0.373 0.880 1.000 .580 Human 2 vs LLM .735 0.373 0.880 1.000 .938 Note. ICC = Intraclass Correlation Coefficient; MAE = Mean Absolute Error; r = Spearman Correlation. Agreement values represent percentage of ratings within 0.5 and 1.0 points respectively. A deep dive into each item and how they correlated across three raters across all participants and all four cases, we see the results indicated strong agreement between human raters and LLM across all CDM items (Table 3 ). ICC values ranged from moderate to excellent, with Human 2 consistently showing higher agreement with LLM compared to Human 1. The lowest ICC was observed for Item 8 (collected data and effectiveness of the conversation) between Human 1 and LLM ( ICC = 0.528), while the highest agreement was found for Item 6 (checking with the patient) between Human 2 and LLM ( ICC = 0.877). MAE remained consistently low across all items, ranging from 0.271 to 0.506, indicating minimal deviation between human and LLM ratings. Items 2 and 3 showed the lowest MAE (0.271 and 0.277, respectively), while Item 7 had the highest MAE (0.506). The proportion of ratings within 0.5 points of each other was notably high across all items, ranging from 84.3–98.8%. Perfect agreement (within 1.0 point) was achieved for all items except Item 7 (summarizing), which still maintained a high agreement rate of 96.4%. Correlation coefficients further support the strong agreement, with values ranging from moderate ( r = 0.542 for Human 1 vs. LLM on Item 4) to very strong ( r = 0.938 for Human 2 vs. LLM on Item 8). Human 2 generally showed higher correlations with LLM ratings compared to Human 1, with particularly strong correlations for Items 6 ( r = 0.916) and 8 ( r = 0.938). RQ2: Does an LLM exhibit gender bias when assessing students’ CDM? An ANOVA was conducted to examine the effect of gendered transcripts on LLM-rated CDM scores. Post-hoc comparisons using Tukey's HSD test were performed to investigate specific differences between prompts. The results revealed no statistically significant differences between any of the prompt variations. Prompts using women-oriented language showed marginally higher scores compared to men-oriented language ( M diff = 0.12, 95% CI [-0.66, 0.91], p = .928). Neutral language prompts scored slightly lower than male-oriented prompts ( M diff = -0.07, 95% CI [-0.86, 0.71], p = .975) and women-oriented prompts ( M diff = -0.19, 95% CI [-0.98, 0.59], p = .831). The small differences between conditions and high p -values suggest that the gendering of prompts does not have a meaningful effect on LLM scores in this dataset. This analysis indicates that the three prompt variations (men-oriented, women-oriented, and neutral [i.e., non-gender specific]) yielded similar LLM-rated CDM scores, with any observed differences being statistically and practically insignificant. The substantial overlap in the confidence intervals further supports the conclusion that the gendering of prompts is not a determining factor in LLM score outcomes. Discussion Traditional assessment of CDM by human experts is resource-intensive, time-consuming, and potentially inconsistent. Integrating LLMs into this process could provide a cost-effective, scalable, and consistent complement to human raters, enabling more frequent and faster feedback for students. This is particularly important in medical education, as the ability to make clinical decisions is crucial for patient care. Automated LLM-based assessment of students’ CDM is possible The results of this empirical study demonstrate a high level of agreement between human evaluations and those of the LLM, particularly with an ICC ranging from 0.675 to 0.782. These values suggest good to excellent reliability. An interesting pattern emerges in the distribution of ratings: while the LLM replicates middle values well, statements at the extreme ends of the scale are less consistent. This could indicate that the model tends to favor moderate ratings and avoid extreme judgments. Such a tendency suggests that while LLMs can offer valuable support, they should not act as the sole evaluative instance in every situation. LLM-based CDM assessment did not show gender discriminations Another objective of the study was to examine whether the LLM exhibits gender-specific biases in its assessments. The results show no significant differences between neutral, male, or female gendered prompts in the test conditions. This contrasts with existing studies that have identified biases in LLM-driven decision-making processes (e.g., [16–21]). One possible explanation is the standardized assessment methodology, based on fixed criteria, which minimizes potential biases. Furthermore, advancements in model architecture may have contributed to the reduction of bias. Toward automated assessment? Practical implications for medical education Integrating LLMs into medical education offers transformative potential for the training and assessment CDM. For example, students could receive regular evaluations and targeted feedback to improve their decision-making skills. Automating assessments allows for more frequent use without additional resource expenditure on human staff. A particularly promising approach is a hybrid assessment model where LLMs provide preliminary evaluations that human experts subsequently validate or adjust. This approach would combine the efficiency and scalability of LLM with the nuanced judgment and contextual understanding of experienced raters. Such a system could significantly enhance both the quality and quantity of feedback while maintaining high standards of evaluation integrity. Study Limitations Several limitations of the study should still be considered. We operated with only one LLM (ChatGPT 3.5), other models may yield different results. Additionally, the study was limited to two human raters, which could affect the generalizability of the findings. The sample size was relatively small, so replication with a larger sample would be necessary. Furthermore, the study only examined one type of bias (gender), while other factors such as age or cultural differences may also play a role. Finally, the analysis was not conducted in real-time but based on transcribed conversations, which could limit its applicability to practical scenarios. The implementation of LLM-based language models in real patient interactions remains an exciting field for future research. Conclusion The results suggest that LLMs could be a promising addition to the assessment of CDM in medical education. They offer an efficient, cost-effective, and relatively objective alternative to human evaluation. However, full automation is currently not recommended due to existing discrepancies and limitations. A combination of human expertise and LLM-driven assessment could represent the optimal approach to further enhancing medical education. Abbreviations CDM = Clinical Decision-Making CRI-HT = Clinical Reasoning Indicator - Health Training Indicator ICC = Intraclass Correlation Coefficient LLM = Large Language Model MAE = Mean Absolute Error Declarations Ethics approval and consent to participate Ethics approval was obtained from the ethics board (“Ethik-Kommission Westfalen-Lippe”) under the reference 2023-438-f-N Consent for publication All authors consent for publication. Availability of data and materials Data is available from the corresponding author upon reasonable request. Competing Interests The authors declare no competing interest. Funding This study received no specific funding. Authors’ contributions SB and DD collected and analyzed the data. SB, JV, CP, MHL and DD wrote the manuscript. Registration details for clinical trials Not applicable Clinical trial number Not applicable AI disclosure: Claude v. 3.5 Sonnet has been used for language editing. All content and ideas remain the original work of the authors, with AI assistance to improve linguistic clarity. References Tanner CA. Thinking like a nurse: A research-based model of clinical judgment in nursing. J Nurs Educ. 2006;45(6):204–11. https://doi.org/10.3928/01484834-20060601-04 . Fürstenberg S, Helm T, Prediger S, Kadmon M, Berberat PO, Harendza S. Assessing clinical reasoning in undergraduate medical students during history taking with an empirically derived scale for clinical reasoning indicators. BMC Med Educ. 2020;20(1):368. https://doi.org/10.1186/s12909-020-02260-9 . Patton DD. Introduction to clinical decision making. Semin Nucl Med. 1978;8(4):273–82. https://doi.org/10.1016/S0001-2998(78)80013-0 . McGregor CA, Paton C, Thomson C, Chandratilake M, Scott H. Preparing medical students for clinical decision making: a pilot study exploring how students make decisions and the perceived impact of a clinical decision making teaching intervention. Med Teach. 2012;34(7):e508–17. https://doi.org/10.3109/0142159X.2012.670323 . Hoffmann TC, Bennett S, Tomsett C, Del Mar C. Brief training of student clinicians in shared decision making: A single-blind randomized controlled trial. J Gen Intern Med. 2014;29(6):844–9. https://doi.org/10.1007/s11606-014-2765-5 . Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med. 2023;29(8):1930–40. https://doi.org/10.1038/s41591-023-02448-8 . Arora A, Arora A. The promise of large language models in health care. Lancet. 2023;401(10377):641. https://doi.org/10.1016/S0140-6736(23)00216-7 . Brügge E, Ricchizzi S, Arenbeck M, Keller MN, Schur L, Stummer W, Holling M, Lu MH, Darici D. Large language models improve clinical decision making of medical students through patient simulation and structured feedback: A randomized controlled trial. BMC Med Educ. 2024;24(1):1391. https://doi.org/10.1186/s12909-024-06399-7 . Hager P, Jungmann F, Holland R, Bhagat K, Hubrecht I, Knauer M, Vielhauer J, Makowski M, Braren R, Kaissis G, Rueckert D. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat Med. 2024;30(9):2613–22. https://doi.org/10.1038/s41591-024-03097-1 . Fabiyi SD. What can ChatGPT not do in education? Evaluating its effectiveness in assessing educational learning outcomes. Innovations Educ Teach Int. 2024;1–15. https://doi.org/10.1080/14703297.2024.2333395 . Gerken M, Beebe JR. Knowledge in and out of Contrast. Noûs. 2016;50(1):133–64. https://doi.org/10.1111/nous.12064 . Koch AJ, D'Mello SD, Sackett PR. A meta-analysis of gender stereotypes and bias in experimental simulations of employment decision making. J Appl Psychol. 2015;100(1):128–61. https://doi.org/10.1037/a0036734 . Gingerich A, Regehr G, Eva KW. Rater-based assessments as social judgments: Rethinking the etiology of rater errors. Acad Medicine: J Association Am Med Colleges. 2011;86(10 Suppl). https://doi.org/10.1097/ACM.0b013e31822a6cf8 . S1-7. Motoki F, Pinho Neto V, Rodrigues V. More human than human: Measuring ChatGPT political bias. Public Choice. 2024;198(1–2):3–23. https://doi.org/10.1007/s11127-023-01097-2 . Thakur V. Unveiling gender bias in terms of profession across LLMs: Analyzing and addressing sociological implications. 2023. https://doi.org/10.48550/arXiv.2307.09162 Klein R, Julian KA, Snyder ED, Koch J, Ufere NN, Volerman A, Vandenberg AE, Schaeffer S, Palamara K. Gender bias in resident assessment in graduate medical education: Review of the literature. J Gen Intern Med. 2019;34(5):712–9. https://doi.org/10.1007/s11606-019-04884-0 . Chapman EN, Kaatz A, Carnes M. Physicians and implicit bias: How doctors may unwittingly perpetuate health care disparities. J Gen Intern Med. 2013;28(11):1504–10. https://doi.org/10.1007/s11606-013-2441-1 . Darici D, Masthoff M, Rischen R, Schmitz M, Ohlenburg H, Missler M. Medical imaging training with eye movement modeling examples: A randomized controlled study. Med Teach. 2023;45(8):918–24. https://doi.org/10.1080/0142159X.2023.2189538 . Darici D, Reissner C, Missler M. Webcam-based eye-tracking to measure visual expertise of medical students during online histology training. GMS J Med Educ. 2023;40(5):Doc60. https://doi.org/10.3205/zma001642 . Otto N, Böckers A, Shiozawa T, Brunk I, Schumann S, Kugelmann D, Missler M, Darici D. Profiling learning strategies of medical students: A person-centered approach. Med Educ. 2024;58(11):1304–14. https://doi.org/10.1111/medu.15388 . Additional Declarations No competing interests reported. Supplementary Files supplfiles.docx Cite Share Download PDF Status: Under Review Version 1 posted Reviewers agreed at journal 02 Jul, 2025 Reviewers invited by journal 13 Jun, 2025 Editor invited by journal 22 May, 2025 Editor assigned by journal 20 May, 2025 Submission checks completed at journal 20 May, 2025 First submitted to journal 14 May, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6660928","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":472203971,"identity":"bd309ce2-0ec6-4ae6-a589-2bee9e52bb11","order_by":0,"name":"Sina Chole Benker","email":"","orcid":"","institution":"University of Münster","correspondingAuthor":false,"prefix":"","firstName":"Sina","middleName":"Chole","lastName":"Benker","suffix":""},{"id":472203972,"identity":"1833bfc7-e4b1-4fdf-8804-8a536f456a91","order_by":1,"name":"Jonathan Vollprecht","email":"","orcid":"","institution":"University of Heidelberg","correspondingAuthor":false,"prefix":"","firstName":"Jonathan","middleName":"","lastName":"Vollprecht","suffix":""},{"id":472203973,"identity":"994e868e-df14-4135-9fa0-d4cd4f09080f","order_by":2,"name":"Cihan Papan","email":"","orcid":"","institution":"University Hospital Bonn","correspondingAuthor":false,"prefix":"","firstName":"Cihan","middleName":"","lastName":"Papan","suffix":""},{"id":472203975,"identity":"5246cb93-096c-47ad-a238-c7a83ba2b084","order_by":3,"name":"Max Hao Lu","email":"","orcid":"","institution":"Harvard Graduate School of Education","correspondingAuthor":false,"prefix":"","firstName":"Max","middleName":"Hao","lastName":"Lu","suffix":""},{"id":472203976,"identity":"e3085c4c-756d-4d63-8580-a646c5fb6cb6","order_by":4,"name":"Dogus Darici","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA6klEQVRIiWNgGAWjYDCCAyBkYMPAwIwsykNYSxpCCw8xWoDgMJoF+LTwHT/78MCPgvOJ29kZWDfzVNyxt2fvfcDwpgK3Fskz6QYHewxuJ+5sZmC7zXPmWWIPz3EDxjlncGsxOJAGdBVQy4bD/N9u87YdTuCRSGNg5m3Do+X8M5CWc0AtQFt4/x2255F/BtTyD4+WG2BbDkC1NBxm7JFgA2ppwOOXG88YgH5JNgZpuTnn2OHEnjNpDAfnHMOthe98GvOHH3/sZDecP8B2403NYXv29mOMD97U4NaCHRwgVcMoGAWjYBSMAlQAANKGVNr0v77OAAAAAElFTkSuQmCC","orcid":"","institution":"University of Münster","correspondingAuthor":true,"prefix":"","firstName":"Dogus","middleName":"","lastName":"Darici","suffix":""}],"badges":[],"createdAt":"2025-05-14 06:23:24","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6660928/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6660928/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":84806542,"identity":"75bca2bb-423e-44a9-a1dd-801788d0b2b1","added_by":"auto","created_at":"2025-06-17 14:10:04","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":85234,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eIllustration of study procedure regarding inter-rater agreement (RQ1)\u003c/strong\u003e Simulated student-patient conversations were given to both human raters and LLM to assess students’ CDM.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-6660928/v1/36712c992c54adc0909be26a.png"},{"id":84806978,"identity":"2a4f5d52-9a2a-488a-9697-b5516c0354cf","added_by":"auto","created_at":"2025-06-17 14:18:04","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":78994,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eIllustration of study procedure regarding gender bias (RQ2)\u003c/strong\u003e Simulated student-patient conversations were analyzed by LLM using either gender neutral, female or male prompts.\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-6660928/v1/82b5753d18688ee1ec7c2f66.png"},{"id":84806544,"identity":"1c1b3890-7932-4efe-adb5-3a31d6754a76","added_by":"auto","created_at":"2025-06-17 14:10:04","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":103383,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003ePairwise jitter plots showing the different strength of correlation between two human raters and AI rater (=LLM)\u003c/strong\u003e. Each point represents the participants’ performance for one case; therefore, each participant has four dots in each plot.\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-6660928/v1/cda87e95a3b3365ed72738d3.png"},{"id":84806545,"identity":"116910a6-e36e-455d-9857-e888f78cc9bd","added_by":"auto","created_at":"2025-06-17 14:10:04","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":41864,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eComparison of CDM rating distributions between AI scores (= LLM) and two different human raters (Human 1 and Human 2).\u003c/strong\u003e The heat maps illustrate the relationship between human-assigned CDM scores (x-axis, ranging from 1 to 5) and LLM-assigned CDM scores (y-axis, ranging from 1 to 5), with color intensity indicating the percentage of ratings falling into each combination (darker blue representing higher percentages, up to 15%).\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-6660928/v1/9d7a621f87bdc55ba8e39bf8.png"},{"id":84806548,"identity":"b55b9841-88ab-461c-a959-0aa11f74f68a","added_by":"auto","created_at":"2025-06-17 14:10:04","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":26598,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eFigure 4. Distribution of LLM scores across different gendered prompt conditions (m = men-oriented, w = women-oriented, n = neutral). \u003c/strong\u003eThe boxplots show the median (horizontal line), interquartile range (box), and minimum/maximum values within 1.5 times the interquartile range (whiskers). Individual points represent outliers. All three conditions show similar median scores around 31 points (range 0-40 points), with comparable distributions and spread. The presence of outliers in all conditions suggests occasional extreme scores, particularly on the higher end (around 40 points).\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-6660928/v1/109470bf32330bbca883293c.png"},{"id":84809336,"identity":"278f82ff-854b-4371-8761-4d78ff3fbcb5","added_by":"auto","created_at":"2025-06-17 14:42:09","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1466482,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6660928/v1/95b54a82-bb25-4ab8-8b05-e62eeeccb15a.pdf"},{"id":84806547,"identity":"0580226b-c2a6-41dc-a1c6-abe9742086f9","added_by":"auto","created_at":"2025-06-17 14:10:04","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":23530,"visible":true,"origin":"","legend":"","description":"","filename":"supplfiles.docx","url":"https://assets-eu.researchsquare.com/files/rs-6660928/v1/d39998e1419f69d24df07fcc.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Large Language Models for the assessment of medical students’ clinical decision-making","fulltext":[{"header":"Practice points","content":"\u003cul class=\"decimal_type\"\u003e\n \u003cli\u003eLarge Language Models (LLMs) demonstrate strong agreement with human raters when assessing medical students\u0026apos; clinical decision-making skills, showing good to excellent reliability (ICC = .675-.782).\u003c/li\u003e\n \u003cli\u003eOver 91% of LLM ratings were within 0.5 points of human ratings on the Clinical Reasoning Indicator - Health Training Indicator (CRI-HTI), suggesting high consistency across evaluation methods.\u003c/li\u003e\n \u003cli\u003eLLMs showed no significant gender bias when assessing clinical decision-making, maintaining consistent evaluation standards regardless of whether subjects were identified as male, female, or gender-neutral.\u003c/li\u003e\n \u003cli\u003eA hybrid assessment approach combining LLM efficiency with human expert oversight may provide the optimal framework for scaling up clinical decision-making assessment in medical education.\u003c/li\u003e\n \u003cli\u003eAutomated LLM assessment could address current limitations in medical education by enabling more frequent, faster, and cost-effective feedback on clinical reasoning skills.\u003c/li\u003e\n\u003c/ul\u003e"},{"header":"Introduction","content":"\u003cp\u003eClinical decision-making (CDM) is a central process in healthcare where physicians gather, evaluate, and interpret relevant information about a patient's health status to make informed decisions regarding diagnosis and treatment [1]. CDM is crucial as it directly impacts the quality of patient care by ensuring that the right decisions are made at the right time. Clinical reasoning, the cognitive process that underpins CDM, intertwines analytical thinking with pattern recognition to form coherent diagnostic hypotheses and treatment plans based on patient data [2]. Effective CDM skills help minimize diagnostic errors and identify the best treatment approaches [3]. Regular practice with simulated medical history conversations and other clinical exercises is essential for developing these abilities [4, 5].\u003c/p\u003e\n\u003cp\u003eIn CDM trainings, CDM mastery is individually assessed to provide healthcare professionals with feedback, thereby supporting a learning process and encouraging further development. The gold standard for evaluating medical students' CDM is the use of human raters. This assessment demands both extensive experience and advanced medical expertise of the human raters. It also requires the capacity for flexible interpretation and nuanced assessment of complex clinical scenarios.\u003c/p\u003e\n\u003cp\u003eHowever, human CDM raters present two major limitations: they are not always available, and costly to employ. These limit the frequency and consistency of CDM assessments, resulting in fewer opportunities for learners to receive timely and constructive feedback essential for their professional growth. These challenges we foresee to be alleviated by introducing Large Language Models (LLMs) for CDM assessment.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eLLMs may help evaluating CDM\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eLLMs are advanced machine learning systems that process and generate natural language. Utilizing deep learning architectures like transformers, they are trained on extensive text datasets to recognize linguistic patterns and perform tasks such as translation, summarization, and question answering. Models like GPT (Generative Pretrained Transformer) generate coherent text by predicting words based on input [6]. LLMs can help automate documentation, offer evidence-based recommendations, and enhance medical education by simulating realistic clinical scenarios (e.g., [7, 8]).\u003c/p\u003e\n\u003cp\u003ePlenty of literature shows that although many LLMs are advanced AI models trained on vast amounts of text, they are still not ready for autonomous CDM [9]. However, LMMs could be potentially capable of delivering objective and systematic CDM evaluations based on predefined criteria [10]. With LLMs' potential to analyze both qualitative and quantitative data, this approach holds huge potential to assess and interpret text data and to complement the likelihood of human evaluation errors.\u003c/p\u003e\n\u003cp\u003eDespite the fact that in the medical field there is first evidence that LLMs may present a cost-effective and efficient method for advancing CDM training [8], there is currently no evidence about the consistency and accuracy of LLMs to assess medical history conversations and clinical competencies such as CDM, compared to the current gold standard, human raters. We seek to investigate whether LLMs can reach human rater standards (non-inferiority).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAssessment biases of humans and LLMs\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWhen implementing LLM-based assessment framework, it is crucial to anticipate potential limitations of current models, such inherent biases that may impact the objectivity of the assessment. Cognitive biases of the assessors are particularly relevant in assessment contexts, because they can systematically lead to unfair, inaccurate, or harmful decisions. Common examples include the contrast effect, where current assessments are influenced by comparisons to previous cases [11]. A prominent form of systematic bias is gender bias, where the subject's gender may unconsciously influence the evaluation [12].\u003c/p\u003e\n\u003cp\u003eIt is well established that human raters are susceptible to biases influenced by their prior experiences or expectations, which can affect the objectivity of their evaluations [13]. Research has also shown that LLMs, trained extensively on real world data, are also not always value-neutral or bias-free. For example, study has shown that ChatGPT tend to be more politically left leaning and ideologically liberal [14]. Critically, LLMs have tendencies to rate the male gender as the more dominant [15], which may have an impact on how they assess the CDM of medical student conversations. Therefore, we were particularly interested in whether LLMs exhibit more biases towards different genders when evaluating their CDM mastery.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eThe current study\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTo mitigate the limitations of human raters related to scalability, availability, and consistency, we set out to examine the feasibility of supporting human raters with LLMs. We compared the ratings of LLMs with the current gold standards of human raters, and we also compared LLMs own ratings across different genders. Such an approach allowed us to investigate LLMs can function as a reliable compliment to human raters and whether it exhibit discriminatory behaviors towards different genders.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eIn our study, we first compared the LLMs evaluation with human evaluations and looked at the reliability evidence of using LLM raters alongside human raters. Next, we sought to investigate the extent to which LLMs exhibit a gender bias in their assessments. For the latter, the LLM was asked to evaluate the subject under three conditions: neutral (unspecified gender), men, and women. Referring to Thakur (2023), we hypothesized that LLMs might exhibit a tendency to rate female students’ CDM subjects less favorably compared to males.\u003c/p\u003e\n\u003cp\u003eOur research questions are:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResearch Question 1 (RQ 1):\u0026nbsp;\u003c/strong\u003eTo what extent do LLMs produce ratings that are consistent with human raters, the current gold standard, when assessing students’ performance on the CRI-HTI?\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResearch question 2 (RQ 2):\u0026nbsp;\u003c/strong\u003eDoes an LLM exhibit gender bias when assessing students’ CDM?\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003eTo address the research questions, we conducted an experimental design in the Summer Term 2024. Ethics approval was obtained from the ethics board (\u0026ldquo;Ethik-Kommission Westfalen-Lippe\u0026rdquo;) under the reference 2023-438-f-N. Informed consent was obtained from all participants.\u003c/p\u003e \u003cp\u003eWe begin by detailing the distinct methodological approaches for Research Questions 1 and 2, followed by a comprehensive description of the instrument employed to evaluate clinical decision making (CDM). The section concludes with an explanation of the various analytical metrics utilized.\u003c/p\u003e\n\u003ch3\u003eStudy procedure – evaluate LLM rater reliability (RQ1)\u003c/h3\u003e\n\u003cp\u003eTwenty-one medical students (\u003cem\u003emean\u003c/em\u003e semester\u0026thinsp;=\u0026thinsp;4, 14 females) each completed four simulated patient-doctor CDM cases, representing four distinct neurological or neurosurgical emergency situations (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). The CDM conversations were transcribed for all participants. We used OpenAI\u0026rsquo;s ChatGPT free version, GPT3.5, as the LLM to evaluate the transcript based on the prompt used in Br\u0026uuml;gge et al (ibid.). ChatGPT would then evaluate the transcripts based on the criteria outlined in the prompt. The language model evaluated all cases for each participant. We then transferred ChatGPT's evaluations into a table to derive statistical values (see below) from the data.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\n\u003ch3\u003eStudy procedure – LLMs possible gender bias\u003c/h3\u003e\n\u003cp\u003e To investigate gender bias, the same CDM conversation transcript from each participant was presented to ChatGPT three times (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e, Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). The first instance did not specify any gender, the second identified the participant as male, and the third as female. Each version of the transcript was submitted in separate chat windows, ensuring that ChatGPT did not reference previous interactions and could evaluate each version independently. This approach allowed for a comparative assessment of potential gender biases presented in the LLM ratings.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThe gender-non-specific designation of the first conversation transcript served as a baseline for comparison. The baseline allowed us to assess whether the subsequent gendered versions elicit different responses and identify any potential biases in the evaluations.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eThe three different prompts (neutral, female, male) each inserted in LLM chats for transcript based rating)\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePrompt neutral (i.e., gender not specified)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePrompt female\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003ePrompt male\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAt this point, you will assess the quality of the third-semester medical student conducting a history-taking conversation.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAt this point, you will assess the quality of \u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003ea female\u003c/span\u003e third-semester medical student conducting a history-taking conversation.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAt this point, you will assess the quality of \u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003ea male\u003c/span\u003e third-semester medical student conducting a history-taking conversation.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"3\" nameend=\"c3\" namest=\"c1\"\u003e \u003cp\u003e\u003cspan type=\"ItalicUnderline\" class=\"ItalicUnderline\" name=\"Emphasis\"\u003e[The following input was derived from the CRI-HT questionnaire (F\u0026uuml;rstenberg et al., 2016), and was identical for all three prompts:]\u003c/span\u003e\u003c/p\u003e \u003cp\u003eYour assessment should include the following eight criteria\u003c/p\u003e \u003cp\u003e1. Assess whether the user has taken control of the conversation to obtain the necessary information.\u003c/p\u003e \u003cp\u003e2. Assess whether the user recognizes all relevant information.\u003c/p\u003e \u003cp\u003e3. Assess whether the user formulates targeted questions so that he can capture and specify the symptoms in detail.\u003c/p\u003e \u003cp\u003e4. Assess whether the questions of the user suggest that specific causes or circumstances lead to certain symptoms.\u003c/p\u003e \u003cp\u003e5. Assess whether the user asks questions in a logical sequence.\u003c/p\u003e \u003cp\u003e6. Assess whether the user reassures the patient that he has received the correct information from the patient.\u003c/p\u003e \u003cp\u003e7. Assess whether the user has summarized his collected information before ending the conversation.\u003c/p\u003e \u003cp\u003e8. Assess whether the user has collected sufficient information of high quality at an appropriate speed.\u003c/p\u003e \u003cp\u003eAssign each of the eight criteria a score according to the following scheme:\u003c/p\u003e \u003cp\u003e1 - Does not meet the criterion 2 - Rather does not meet the criterion 3 - Partially meets the criterion 4 - Rather meets the criterion 5 - Fully meets the criterion\u003c/p\u003e \u003cp\u003eExplain the evaluation with two sentences.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"3\" nameend=\"c3\" namest=\"c1\"\u003e \u003cp\u003e\u003cem\u003eNote. The transcripts were originally in German and the LLM was prompted using German language instructions.\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eInstrument to assess CDM\u003c/h2\u003e \u003cp\u003eThe Clinical Reasoning Indicator - Health Training Indicator (CRI-HTI), developed and validated by F\u0026uuml;rstenberg et al. [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e], is a structured tool for evaluating the clinical reasoning skills and training level of medical students and healthcare professionals. This tool utilizes a Likert scale to measure eight specific criteria that capture key aspects of clinical reasoning, organized into three core competency areas: \"Focusing Questions\" (Items 1\u0026ndash;3), \"Creating Context\" (Items 4\u0026ndash;6), and \"Securing Information\" (Items 7\u0026ndash;8).\u003c/p\u003e \u003cp\u003eThese competencies encompass the participant's ability to guide patient interactions, actively recognize and respond to relevant information, and accurately identify and specify symptoms. Further, the tool assesses the clinician's skill in asking targeted questions that facilitate pathophysiological reasoning, arranging these questions in a logical sequence, and checking with the patient to ensure understanding. It also evaluates the ability to effectively summarize information and reflect on the quality and effectiveness of the conversation.\u003c/p\u003e \u003cp\u003eTogether, these criteria provide a comprehensive framework for assessing clinical reasoning and CDM, offering insights into the clinician's capacity to conduct focused, coherent, and patient-centered interactions. The CRI-HTI is thus a valuable tool for both formative assessment in clinical training and for structured feedback aimed at enhancing essential clinical competencies.\u003c/p\u003e \u003cp\u003eThe Likert scale ranges from 1 to 5, where 5 indicates \"Completely agrees with the criterion\" and 1 indicates \"Does not agree with the criterion.\" The CRI-HTI serves as the basis of the prompt provided at the beginning of the LLM evaluation, guiding the LLM to assign scores on the Likert scale based on the criteria that assess diagnostic skills, problem-solving, and decision-making abilities. To achieve greater sensitivity to change, the LLM was also allowed to award half points.\u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003eAnalysis\u003c/h3\u003e\n\u003cp\u003eThe analysis was performed in R, and the full analysis code is available in the supplemental files. In the statistical analysis, we aggregated the ratings from the CRI-HTI scale for each case to derive the total score. Subsequently, we compared the newly calculated values assessed by ChatGPT using the CRI-HTI scale with the ratings provided by human raters. Specifically, the following analyses were conducted:\u003c/p\u003e \u003cp\u003e\u003cem\u003eIntraclass Correlation Coefficient (ICC).\u003c/em\u003e The \u003cem\u003eICC\u003c/em\u003e is a statistical measure for assessing the reliability of ratings, particularly in contexts involving multiple raters. By evaluating the degree of agreement among raters, the \u003cem\u003eICC\u003c/em\u003e provides insights into the consistency and dependability of the ratings, whether conducted by human evaluators or the LLM. \u003cem\u003eICC\u003c/em\u003e values approaching 1 indicate a high level of agreement, suggesting reliable assessments and agreement, whereas values closer to 0 indicate low agreement and thus lower reliability. The \u003cem\u003eICC\u003c/em\u003e can be interpreted across different ranges to provide a clearer understanding of rating consistency. Values from 0 to .2 suggest very low agreement, indicating that the ratings are largely inconsistent and potentially unreliable. \u003cem\u003eICC\u003c/em\u003e values between .3 and .4 reflect moderate agreement, though still relatively low, which suggests limited consistency among raters. An \u003cem\u003eICC\u003c/em\u003e range of .5 to .6 represents acceptable agreement. When \u003cem\u003eICC\u003c/em\u003e values fall between .7 and .8, this suggests good agreement among raters, indicating that the ratings are generally reliable and suitable for many practical applications. \u003cem\u003eICC\u003c/em\u003e values between .9 and 1.0 denote excellent agreement, with nearly perfect consistency. By providing a robust estimate of rating consistency, the \u003cem\u003eICC\u003c/em\u003e enables researchers to determine whether the rating process is sufficiently reliable to justify further analyses based on the obtained data.\u003c/p\u003e \u003cp\u003e \u003cem\u003eMean absolute error (MAE).\u003c/em\u003e The \u003cem\u003eMAE\u003c/em\u003e is a statistical metric that measures the average absolute difference between predicted and actual values. It quantifies the overall accuracy of a predictive model by assessing how close the predictions are to the true values. A lower \u003cem\u003eMAE\u003c/em\u003e indicates higher accuracy, while a higher MAE suggests greater deviation. In our case we use the \u003cem\u003eMAE\u003c/em\u003e to define the deviation between human and LLM rating.\u003c/p\u003e \u003cp\u003e \u003cem\u003eAgreement within a range.\u003c/em\u003e Statistical agreement refers to the degree of consistency or concordance between different measurements, raters, or diagnostic methods in a medical or research setting. It is an intuitive measure that shows the proportion of ratings that fall within certain thresholds of each other (the current study uses 1 and 0.5). It assesses how closely independent evaluations align when measuring the same phenomenon. High agreement indicates reliability and reproducibility.\u003c/p\u003e \u003cp\u003e \u003cem\u003eSpearman correlation.\u003c/em\u003e Spearman's rank correlation coefficient (ρ) is a non-parametric measure that assesses the strength and direction of association between two ranked variables. Unlike Pearson correlation, Spearman's correlation evaluates monotonic relationships and is robust against outliers and non-normal distributions. The coefficient ranges from \u0026minus;\u0026thinsp;1 to +\u0026thinsp;1, where +\u0026thinsp;1 indicates a perfect positive correlation, -1 signifies a perfect negative correlation, and 0 represents no correlation. In our study, Spearman correlation was calculated to evaluate the association between human and LLM ratings of CDM skills. Values between .00-.19 indicate very weak correlation, .20-.39 weak correlation, .40-.59 moderate correlation, .60-.79 strong correlation, and .80\u0026thinsp;\u0026minus;\u0026thinsp;1.00 very strong correlation.\u003c/p\u003e \u003cp\u003e \u003cem\u003eANOVA\u003c/em\u003e. To determine whether there were differences LLMs ratings based on gender, we compared the mean scores across the different gender categories and performed a one-way ANOVA with post-hoc Tukey's HSD test for multiple comparisons.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003e \u003cb\u003eRQ1: To what extent do LLMs produce ratings that are consistent with human raters, the current gold standard, when assessing students\u0026rsquo; performance on the CRI-HTI?\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThe results indicate high agreement level between both human raters and the LLM (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eOverall agreement metrics between human raters and LLM ratings of CDM\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"6\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eComparison\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003eICC\u003c/em\u003e\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eMAE\u003c/em\u003e\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAgreement\u0026thinsp;\u0026le;\u0026thinsp;0.5\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eAgreement\u0026thinsp;\u0026le;\u0026thinsp;1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003er\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHuman 1 vs LLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e.675\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.343\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.919\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.995\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e.718\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHuman 2 vs LLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e.782\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.343\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.919\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.995\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e.854\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"6\" nameend=\"c6\" namest=\"c1\"\u003e \u003cp\u003e\u003cem\u003eNote. ICC\u003c/em\u003e\u0026thinsp;=\u0026thinsp;Intraclass Correlation Coefficient; \u003cem\u003eMAE\u003c/em\u003e\u0026thinsp;=\u0026thinsp;Mean Absolute Error; \u003cem\u003er\u003c/em\u003e\u0026thinsp;=\u0026thinsp;Spearman Correlation. Agreement values represent percentage of ratings within 0.5 and 1.0 points respectively. Correlation values represent Spearman correlations.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eWhen we look at each participants performance across each case, we see the \u003cem\u003eICC\u003c/em\u003e values (.673 between Human 1 and LLM and .782 for Human 2 and LLM) suggest good to excellent reliability. Both human raters showed identical patterns of absolute agreement with the LLM, with a \u003cem\u003eMAE\u003c/em\u003e of 0.34 points. Notably, over 91% of all ratings were within half a point of each other, and nearly all ratings (99.5%) fell within one point of agreement. The Spearman correlations were strong, particularly for Human 2 (\u003cem\u003er\u003c/em\u003es\u0026thinsp;=\u0026thinsp;.86), indicating robust rank-order agreement between human and LLM ratings.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eWhen looking at the total scores given by different raters through a jitter plots, which is a special kind of scatter plots that allow better visualization of overlapping points, we see that LLM rater\u0026rsquo;s scoring correlates strongly with that of two human raters (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). Interestingly, LLM shows more alignment with both human raters than human raters align with each other.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eWhen looking at the aggregated rating distribution of scores, we noticed similar alignments (Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e4\u003c/span\u003e). For Human 1 versus LLM ratings, the highest concentration of agreements appears in the 3.0\u0026ndash;4.0 range, with a notable cluster around the 3.5-4.0 mark. The distribution shows some spread, indicating moderate agreement between Human 1 and LLM ratings, though there is visible variance in the lower (1.5\u0026ndash;2.5) and higher (4.0-4.5) ranges.\u003c/p\u003e \u003cp\u003eThe Human 2 versus LLM comparison exhibits a similar pattern but with slightly stronger clustering in the higher score ranges (3.5\u0026ndash;4.5). There appears to be more concentrated agreement between Human 2 and LLM ratings in the upper score ranges, as evidenced by the darker blue coloring in these areas. Both distributions suggest a tendency toward higher ratings overall, with fewer instances of low scores (1.0\u0026ndash;2.0) from either human raters or the LLM system.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u003cem\u003eItem-level Agreement Metrics Between Human vs LLM CDM Ratings\u003c/em\u003e\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"7\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSecond-order latent factors\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eComparison\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eICC\u003c/em\u003e\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cem\u003eMAE\u003c/em\u003e\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eAgreement \u003c/p\u003e \u003cp\u003e\u0026le; 0.5\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAgreement \u003c/p\u003e \u003cp\u003e\u0026le; 1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cem\u003er\u003c/em\u003e\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"8\" rowspan=\"9\"\u003e \u003cp\u003eFactor 1: Focusing questions\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e1. Taking the lead in the conversation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHuman 1 vs LLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e.584\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.343\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.867\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e.735\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHuman 2 vs LLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e.667\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.343\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.867\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e.803\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2. Recognizing and responding to relevant information\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHuman 1 vs LLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e.743\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.271\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.928\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e.739\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHuman 2 vs LLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e.829\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.271\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.928\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e.897\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e3. Specifying symptoms\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHuman 1 vs LLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e.700\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.277\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.988\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e.733\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHuman 2 vs LLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e.798\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.277\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.988\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e.825\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"8\" rowspan=\"9\"\u003e \u003cp\u003eFactor 2:\u003c/p\u003e \u003cp\u003eCreating context\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e4. Asking specific questions that point to pathophysiological thinking\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHuman 1 vs LLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e.559\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.380\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.843\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e.542\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHuman 2 vs LLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e.758\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.380\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.843\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e.878\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e5. Putting questions in a logical order\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHuman 1 vs LLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e.733\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.283\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.976\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e.750\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHuman 2 vs LLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e.823\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.283\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.976\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e.850\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e6. Checking with the patient\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHuman 1 vs LLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e.827\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.301\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.952\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e.795\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHuman 2 vs LLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e.877\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.301\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.952\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e.916\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"5\" rowspan=\"6\"\u003e \u003cp\u003eFactor 3: Securing information\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e7. Summarizing\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHuman 1 vs LLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e.707\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.506\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.916\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.964\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e.863\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHuman 2 vs LLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e.770\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.506\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.916\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.964\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e.739\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e8. Collected data and effectiveness of the conversation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHuman 1 vs LLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e.528\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.373\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.880\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e.580\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHuman 2 vs LLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e.735\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.373\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.880\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1.000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e.938\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"7\"\u003e\u003cem\u003eNote. ICC\u003c/em\u003e\u0026thinsp;=\u0026thinsp;Intraclass Correlation Coefficient; \u003cem\u003eMAE\u003c/em\u003e\u0026thinsp;=\u0026thinsp;Mean Absolute Error; \u003cem\u003er\u003c/em\u003e\u0026thinsp;=\u0026thinsp;Spearman Correlation. Agreement values represent percentage of ratings within 0.5 and 1.0 points respectively.\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eA deep dive into each item and how they correlated across three raters across all participants and all four cases, we see the results indicated strong agreement between human raters and LLM across all CDM items (Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). ICC values ranged from moderate to excellent, with Human 2 consistently showing higher agreement with LLM compared to Human 1. The lowest ICC was observed for Item 8 (collected data and effectiveness of the conversation) between Human 1 and LLM (\u003cem\u003eICC\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.528), while the highest agreement was found for Item 6 (checking with the patient) between Human 2 and LLM (\u003cem\u003eICC\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.877).\u003c/p\u003e \u003cp\u003eMAE remained consistently low across all items, ranging from 0.271 to 0.506, indicating minimal deviation between human and LLM ratings. Items 2 and 3 showed the lowest \u003cem\u003eMAE\u003c/em\u003e (0.271 and 0.277, respectively), while Item 7 had the highest \u003cem\u003eMAE\u003c/em\u003e (0.506).\u003c/p\u003e \u003cp\u003eThe proportion of ratings within 0.5 points of each other was notably high across all items, ranging from 84.3\u0026ndash;98.8%. Perfect agreement (within 1.0 point) was achieved for all items except Item 7 (summarizing), which still maintained a high agreement rate of 96.4%.\u003c/p\u003e \u003cp\u003eCorrelation coefficients further support the strong agreement, with values ranging from moderate (\u003cem\u003er\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.542 for Human 1 vs. LLM on Item 4) to very strong (\u003cem\u003er\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.938 for Human 2 vs. LLM on Item 8). Human 2 generally showed higher correlations with LLM ratings compared to Human 1, with particularly strong correlations for Items 6 (\u003cem\u003er\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.916) and 8 (\u003cem\u003er\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.938).\u003c/p\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003eRQ2: Does an LLM exhibit gender bias when assessing students\u0026rsquo; CDM?\u003c/h2\u003e \u003cp\u003eAn ANOVA was conducted to examine the effect of gendered transcripts on LLM-rated CDM scores. Post-hoc comparisons using Tukey's HSD test were performed to investigate specific differences between prompts. The results revealed no statistically significant differences between any of the prompt variations. Prompts using women-oriented language showed marginally higher scores compared to men-oriented language (\u003cem\u003eM\u003c/em\u003e\u003csub\u003e\u003cem\u003ediff\u003c/em\u003e\u003c/sub\u003e = 0.12, 95% \u003cem\u003eCI\u003c/em\u003e [-0.66, 0.91], \u003cem\u003ep\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.928). Neutral language prompts scored slightly lower than male-oriented prompts (\u003cem\u003eM\u003c/em\u003e\u003csub\u003e\u003cem\u003ediff\u003c/em\u003e\u003c/sub\u003e = -0.07, 95% \u003cem\u003eCI\u003c/em\u003e [-0.86, 0.71], \u003cem\u003ep\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.975) and women-oriented prompts (\u003cem\u003eM\u003c/em\u003e\u003csub\u003e\u003cem\u003ediff\u003c/em\u003e\u003c/sub\u003e = -0.19, 95% \u003cem\u003eCI\u003c/em\u003e [-0.98, 0.59], \u003cem\u003ep\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.831). The small differences between conditions and high \u003cem\u003ep\u003c/em\u003e-values suggest that the gendering of prompts does not have a meaningful effect on LLM scores in this dataset.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThis analysis indicates that the three prompt variations (men-oriented, women-oriented, and neutral [i.e., non-gender specific]) yielded similar LLM-rated CDM scores, with any observed differences being statistically and practically insignificant. The substantial overlap in the confidence intervals further supports the conclusion that the gendering of prompts is not a determining factor in LLM score outcomes.\u003c/p\u003e \u003c/div\u003e"},{"header":"Discussion","content":"\u003cp\u003eTraditional assessment of CDM by human experts is resource-intensive, time-consuming, and potentially inconsistent. Integrating LLMs into this process could provide a cost-effective, scalable, and consistent complement to human raters, enabling more frequent and faster feedback for students. This is particularly important in medical education, as the ability to make clinical decisions is crucial for patient care.\u003c/p\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003eAutomated LLM-based assessment of students\u0026rsquo; CDM is possible\u003c/h2\u003e \u003cp\u003eThe results of this empirical study demonstrate a high level of agreement between human evaluations and those of the LLM, particularly with an \u003cem\u003eICC\u003c/em\u003e ranging from 0.675 to 0.782. These values suggest good to excellent reliability. An interesting pattern emerges in the distribution of ratings: while the LLM replicates middle values well, statements at the extreme ends of the scale are less consistent. This could indicate that the model tends to favor moderate ratings and avoid extreme judgments. Such a tendency suggests that while LLMs can offer valuable support, they should not act as the sole evaluative instance in every situation.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003eLLM-based CDM assessment did not show gender discriminations\u003c/h2\u003e \u003cp\u003eAnother objective of the study was to examine whether the LLM exhibits gender-specific biases in its assessments. The results show no significant differences between neutral, male, or female gendered prompts in the test conditions. This contrasts with existing studies that have identified biases in LLM-driven decision-making processes (e.g., [16\u0026ndash;21]). One possible explanation is the standardized assessment methodology, based on fixed criteria, which minimizes potential biases. Furthermore, advancements in model architecture may have contributed to the reduction of bias.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003eToward automated assessment? Practical implications for medical education\u003c/h2\u003e \u003cp\u003eIntegrating LLMs into medical education offers transformative potential for the training and assessment CDM. For example, students could receive regular evaluations and targeted feedback to improve their decision-making skills. Automating assessments allows for more frequent use without additional resource expenditure on human staff.\u003c/p\u003e \u003cp\u003eA particularly promising approach is a hybrid assessment model where LLMs provide preliminary evaluations that human experts subsequently validate or adjust. This approach would combine the efficiency and scalability of LLM with the nuanced judgment and contextual understanding of experienced raters. Such a system could significantly enhance both the quality and quantity of feedback while maintaining high standards of evaluation integrity.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003eStudy Limitations\u003c/h2\u003e \u003cp\u003eSeveral limitations of the study should still be considered. We operated with only one LLM (ChatGPT 3.5), other models may yield different results. Additionally, the study was limited to two human raters, which could affect the generalizability of the findings. The sample size was relatively small, so replication with a larger sample would be necessary.\u003c/p\u003e \u003cp\u003eFurthermore, the study only examined one type of bias (gender), while other factors such as age or cultural differences may also play a role. Finally, the analysis was not conducted in real-time but based on transcribed conversations, which could limit its applicability to practical scenarios. The implementation of LLM-based language models in real patient interactions remains an exciting field for future research.\u003c/p\u003e \u003c/div\u003e"},{"header":"Conclusion","content":"\u003cp\u003eThe results suggest that LLMs could be a promising addition to the assessment of CDM in medical education. They offer an efficient, cost-effective, and relatively objective alternative to human evaluation. However, full automation is currently not recommended due to existing discrepancies and limitations. A combination of human expertise and LLM-driven assessment could represent the optimal approach to further enhancing medical education.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cp\u003eCDM = Clinical Decision-Making\u003c/p\u003e\n\u003cp\u003eCRI-HT =\u0026nbsp;Clinical Reasoning Indicator - Health Training Indicator\u003c/p\u003e\n\u003cp\u003eICC = Intraclass Correlation Coefficient\u003c/p\u003e\n\u003cp\u003eLLM = Large Language Model\u003c/p\u003e\n\u003cp\u003eMAE = Mean Absolute Error\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eEthics approval and consent to participate\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eEthics approval was obtained from the ethics board (\u0026ldquo;Ethik-Kommission Westfalen-Lippe\u0026rdquo;) under the reference 2023-438-f-N\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent for publication\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAll authors consent for publication.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAvailability of data and materials\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eData is available from the corresponding author upon reasonable request.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting Interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors declare no competing interest.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis study received no specific funding.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthors\u0026rsquo; contributions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eSB and DD collected and analyzed the data. SB, JV, CP, MHL and DD wrote the manuscript.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eRegistration details for clinical trials\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eClinical trial number\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e\u003cp\u003e\u003cstrong\u003eAI disclosure:\u003c/strong\u003e Claude v. 3.5 Sonnet has been used for language editing. All content and ideas remain the original work of the authors, with AI assistance to improve linguistic clarity.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eTanner CA. Thinking like a nurse: A research-based model of clinical judgment in nursing. J Nurs Educ. 2006;45(6):204\u0026ndash;11. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3928/01484834-20060601-04\u003c/span\u003e\u003cspan address=\"10.3928/01484834-20060601-04\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eF\u0026uuml;rstenberg S, Helm T, Prediger S, Kadmon M, Berberat PO, Harendza S. Assessing clinical reasoning in undergraduate medical students during history taking with an empirically derived scale for clinical reasoning indicators. BMC Med Educ. 2020;20(1):368. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1186/s12909-020-02260-9\u003c/span\u003e\u003cspan address=\"10.1186/s12909-020-02260-9\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePatton DD. Introduction to clinical decision making. Semin Nucl Med. 1978;8(4):273\u0026ndash;82. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/S0001-2998(78)80013-0\u003c/span\u003e\u003cspan address=\"10.1016/S0001-2998(78)80013-0\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMcGregor CA, Paton C, Thomson C, Chandratilake M, Scott H. Preparing medical students for clinical decision making: a pilot study exploring how students make decisions and the perceived impact of a clinical decision making teaching intervention. Med Teach. 2012;34(7):e508\u0026ndash;17. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3109/0142159X.2012.670323\u003c/span\u003e\u003cspan address=\"10.3109/0142159X.2012.670323\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHoffmann TC, Bennett S, Tomsett C, Del Mar C. Brief training of student clinicians in shared decision making: A single-blind randomized controlled trial. J Gen Intern Med. 2014;29(6):844\u0026ndash;9. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s11606-014-2765-5\u003c/span\u003e\u003cspan address=\"10.1007/s11606-014-2765-5\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eThirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med. 2023;29(8):1930\u0026ndash;40. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41591-023-02448-8\u003c/span\u003e\u003cspan address=\"10.1038/s41591-023-02448-8\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eArora A, Arora A. The promise of large language models in health care. Lancet. 2023;401(10377):641. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/S0140-6736(23)00216-7\u003c/span\u003e\u003cspan address=\"10.1016/S0140-6736(23)00216-7\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBr\u0026uuml;gge E, Ricchizzi S, Arenbeck M, Keller MN, Schur L, Stummer W, Holling M, Lu MH, Darici D. Large language models improve clinical decision making of medical students through patient simulation and structured feedback: A randomized controlled trial. BMC Med Educ. 2024;24(1):1391. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1186/s12909-024-06399-7\u003c/span\u003e\u003cspan address=\"10.1186/s12909-024-06399-7\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHager P, Jungmann F, Holland R, Bhagat K, Hubrecht I, Knauer M, Vielhauer J, Makowski M, Braren R, Kaissis G, Rueckert D. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat Med. 2024;30(9):2613\u0026ndash;22. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41591-024-03097-1\u003c/span\u003e\u003cspan address=\"10.1038/s41591-024-03097-1\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFabiyi SD. What can ChatGPT not do in education? Evaluating its effectiveness in assessing educational learning outcomes. Innovations Educ Teach Int. 2024;1\u0026ndash;15. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1080/14703297.2024.2333395\u003c/span\u003e\u003cspan address=\"10.1080/14703297.2024.2333395\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGerken M, Beebe JR. Knowledge in and out of Contrast. No\u0026ucirc;s. 2016;50(1):133\u0026ndash;64. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1111/nous.12064\u003c/span\u003e\u003cspan address=\"10.1111/nous.12064\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKoch AJ, D'Mello SD, Sackett PR. A meta-analysis of gender stereotypes and bias in experimental simulations of employment decision making. J Appl Psychol. 2015;100(1):128\u0026ndash;61. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1037/a0036734\u003c/span\u003e\u003cspan address=\"10.1037/a0036734\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGingerich A, Regehr G, Eva KW. Rater-based assessments as social judgments: Rethinking the etiology of rater errors. Acad Medicine: J Association Am Med Colleges. 2011;86(10 Suppl). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1097/ACM.0b013e31822a6cf8\u003c/span\u003e\u003cspan address=\"10.1097/ACM.0b013e31822a6cf8\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. S1-7.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMotoki F, Pinho Neto V, Rodrigues V. More human than human: Measuring ChatGPT political bias. Public Choice. 2024;198(1\u0026ndash;2):3\u0026ndash;23. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s11127-023-01097-2\u003c/span\u003e\u003cspan address=\"10.1007/s11127-023-01097-2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eThakur V. Unveiling gender bias in terms of profession across LLMs: Analyzing and addressing sociological implications. 2023. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.2307.09162\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2307.09162\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKlein R, Julian KA, Snyder ED, Koch J, Ufere NN, Volerman A, Vandenberg AE, Schaeffer S, Palamara K. Gender bias in resident assessment in graduate medical education: Review of the literature. J Gen Intern Med. 2019;34(5):712\u0026ndash;9. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s11606-019-04884-0\u003c/span\u003e\u003cspan address=\"10.1007/s11606-019-04884-0\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChapman EN, Kaatz A, Carnes M. Physicians and implicit bias: How doctors may unwittingly perpetuate health care disparities. J Gen Intern Med. 2013;28(11):1504\u0026ndash;10. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s11606-013-2441-1\u003c/span\u003e\u003cspan address=\"10.1007/s11606-013-2441-1\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDarici D, Masthoff M, Rischen R, Schmitz M, Ohlenburg H, Missler M. Medical imaging training with eye movement modeling examples: A randomized controlled study. Med Teach. 2023;45(8):918\u0026ndash;24. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1080/0142159X.2023.2189538\u003c/span\u003e\u003cspan address=\"10.1080/0142159X.2023.2189538\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDarici D, Reissner C, Missler M. Webcam-based eye-tracking to measure visual expertise of medical students during online histology training. GMS J Med Educ. 2023;40(5):Doc60. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3205/zma001642\u003c/span\u003e\u003cspan address=\"10.3205/zma001642\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOtto N, B\u0026ouml;ckers A, Shiozawa T, Brunk I, Schumann S, Kugelmann D, Missler M, Darici D. Profiling learning strategies of medical students: A person-centered approach. Med Educ. 2024;58(11):1304\u0026ndash;14. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1111/medu.15388\u003c/span\u003e\u003cspan address=\"10.1111/medu.15388\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-education","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"meed","sideBox":"Learn more about [BMC Medical Education](http://bmcmededuc.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/meed/default.aspx","title":"BMC Medical Education","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Large Language Models, Assessment, Clinical Decision Making, Clinical Reasoning, Medical Education","lastPublishedDoi":"10.21203/rs.3.rs-6660928/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6660928/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe assessment of medical students’ clinical decision making (CDM) skills is fundamental to healthcare education as it identifies knowledge gaps, enables targeted feedback, and validates that graduates meet professional competency standards. However, traditional assessment methods relying on expert human raters are resource-intensive and difficult to scale. This study investigated whether Large Language Models (LLMs), i.e. ChatGPT, could serve as automated assessors of medical students' CDM skills. We compared LLM-generated assessments with ratings from two humans across 21 medical student history-taking conversations using the Clinical Reasoning Indicator - Health Training Indicator (CRI-HTI). The results showed strong agreement between human raters and the LLM (\u003cem\u003eICC\u003c/em\u003e= .675-.782, \u003cem\u003eMAE\u003c/em\u003e = 0.343), with over 91% of ratings within 0.5 points of each other. Item-level analysis revealed moderate to excellent reliability across all eight CRI-HTI criteria. Additionally, we tested for gender bias by presenting identical transcripts with different gender designations (men, women, neutral) to the LLM. No significant differences were found between gendered prompts (\u003cem\u003ep\u003c/em\u003e \u0026gt; .05), suggesting that the LLM maintained consistent evaluation standards regardless of the subject's gender. These findings provide empirical evidence that LLMs could serve as consistent and gender-indiscriminating raters for supporting CDM assessment in medical education, potentially offering a scalable solution for providing timely feedback to medical students.\u003c/p\u003e","manuscriptTitle":"Large Language Models for the assessment of medical students’ clinical decision-making","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-06-17 14:09:59","doi":"10.21203/rs.3.rs-6660928/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"reviewerAgreed","content":"231258369701125765214057830644140944052","date":"2025-07-02T13:04:08+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-06-13T16:05:10+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2025-05-22T12:24:11+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-05-20T12:15:05+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-05-20T12:14:25+00:00","index":"","fulltext":""},{"type":"submitted","content":"BMC Medical Education","date":"2025-05-14T06:12:43+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-education","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"meed","sideBox":"Learn more about [BMC Medical Education](http://bmcmededuc.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/meed/default.aspx","title":"BMC Medical Education","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"f63678c7-6c7a-4435-8678-a398e6658d36","owner":[],"postedDate":"June 17th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2025-06-17T14:10:00+00:00","versionOfRecord":[],"versionCreatedAt":"2025-06-17 14:09:59","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6660928","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6660928","identity":"rs-6660928","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}