Natural Language Processing and Generative AI in the Automated Scoring and Feedback of Reflective Writing in Medical Education: A Validity and Fairness Analysis

Natural Language Processing and Generative AI in the Automated Scoring and Feedback of Reflective Writing in Medical Education: A Validity and Fairness Analysis | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Natural Language Processing and Generative AI in the Automated Scoring and Feedback of Reflective Writing in Medical Education: A Validity and Fairness Analysis Simon Ntumi, Isaac Yabana, Daniel William Essel, Edmond Ahovi This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6636682/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract The study explored the application of Natural Language Processing (NLP) and generative AI tools in assessing reflective writing submitted by medical students in Ghana. It evaluated the validity, fairness, and cultural alignment of AI-generated feedback by comparing AI-generated scores with human rater assessments and analyzing demographic group discrepancies. A total of 180 reflective essays were sampled, with an equal number (n = 60) collected from each university . Quantitative methods included Cohen’s Kappa and Intraclass Correlation Coefficients (ICC) to assess inter-rater agreement, while logistic regression and multiple regression models examined potential biases across gender, university affiliation, and English proficiency. Qualitative data were gathered through interviews with students and faculty to explore perceptions of fairness, trust, and the AI’s capacity to capture cultural and linguistic nuances. Results indicated that the AI system demonstrated strong inter-rater reliability, with Cohen’s Kappa values of 0.74 (AI vs Rater 1) and 0.76 (AI vs Rater 2), and ICC values of 0.78 and 0.80, respectively. Human raters showed higher agreement with each other (Cohen’s Kappa = 0.81, ICC = 0.85). However, significant discrepancies were found across demographic groups, particularly for English proficiency, where lower proficiency students tended to receive higher AI scores than human raters (log-odds of 0.45, p = 0.001). Thematic analysis of qualitative interviews revealed concerns over the lack of empathy in AI feedback, misalignment with cultural and linguistic nuances, and mixed levels of trust in AI-generated assessments. These findings suggest that while AI holds promise for improving efficiency in assessment, careful attention must be given to its limitations in fairness and cultural sensitivity. The study concluded with recommendations for improving AI systems through contextual adaptation, hybrid assessment models, faculty training, and regular bias audits to ensure equitable and effective use of AI in educational settings. AI-generated feedback reflective writing medical education fairness cultural alignment bias analysis Introduction In an era defined by rapid technological innovation, artificial intelligence (AI) has emerged as a transformative force across multiple sectors, including education. Among the most promising applications of AI in education is the integration of Natural Language Processing (NLP) and generative AI technologies into assessment systems [11; 16; 12]. These tools are capable of processing and understanding human language at scale, allowing for the automated evaluation of complex written responses such as reflective essays. Unlike traditional multiple-choice assessments, reflective writing captures deeper dimensions of learning such as critical thinking, empathy, emotional intelligence, and metacognition which are essential for the development of competent professionals, particularly in fields like medicine. As such, educational institutions globally are increasingly adopting AI-powered scoring and feedback systems to evaluate students’ reflective narratives, enabling timely, consistent, and personalized feedback at scale 1[7; 13; 14]. Reflective practice has long been recognized as a cornerstone of medical education. It encourages students and practitioners to critically examine their clinical experiences, ethical decisions, and interpersonal interactions, thereby promoting self-awareness and continuous professional development [18; 22; 20]. Traditionally, the assessment of reflective writing in medical education has relied on manual scoring by trained faculty, a process that is not only time-consuming but also subject to inconsistencies and biases. Automated systems powered by NLP and generative AI such as large language models (LLMs) now offer an alternative that promises greater efficiency and objectivity, while also relieving the assessment burden on faculty members [31;11; 21]. These systems can detect patterns in language use, evaluate argument coherence, and even provide formative feedback aligned with rubrics, thereby enhancing pedagogical impact [ 30 ]. Despite these advancements, the application of AI in educational assessment remains largely concentrated in high-income countries (HICs), where infrastructure, data resources, and digital literacy levels support large-scale deployment. By contrast, low- and middle-income countries (LMICs), especially in sub-Saharan Africa, continue to grapple with systemic challenges in education delivery, including underfunding, staff shortages, and lack of access to modern educational technologies [29; 20]. Medical education in Africa, in particular, faces critical shortages in trained health professionals, overcrowded classrooms, and limited opportunities for feedback and mentorship [9; 11; 17]. The adoption of AI-based assessment tools, therefore, presents a potentially transformative solution to these enduring problems by offering scalable methods for delivering feedback and improving educational outcomes. Ghana represents a compelling case study within this broader African context. As a lower-middle-income country with a growing youth population and rising demand for healthcare professionals, Ghana’s medical education system is under pressure to expand access while maintaining quality. Medical schools such as the University of Ghana Medical School, Kwame Nkrumah University of Science and Technology School of Medical Sciences, and the School of Medicine and Health Sciences at the University for Development Studies play critical roles in training the nation’s physicians. However, they are routinely confronted with large student-to-faculty ratios, heavy administrative loads, and the slow adoption of innovative teaching and assessment tools [1; 2]. As Ghana strives to align its medical training with global standards, reflective writing has been increasingly recognized as a vital pedagogical tool especially in ethics, patient communication, and clinical rotations but assessment practices remain uneven and unsystematic due to resource constraints [19; 12]. The introduction of NLP and generative AI in this context offers both promise and peril. On one hand, these technologies could revolutionize how reflective writing is assessed in Ghanaian medical schools by providing immediate, rubric-aligned feedback and by enabling students to iteratively refine their thinking. On the other hand, critical concerns about validity and fairness arise. Validity refers to whether AI-generated scores truly reflect the quality of students’ reflections as conceptualized within the Ghanaian medical education framework. Fairness involves ensuring that these systems do not inadvertently disadvantage students based on linguistic background, cultural expression, or digital fluency especially given the dominance of Western-centric datasets and language models used to train AI tools [4; 1; 6]. For instance, Ghanaian students may employ locally contextualized metaphors, Ghanaian English syntax, or culturally grounded expressions in their writing, which may be misinterpreted or undervalued by AI systems not trained on such data. While the integration of natural language processing (NLP) and generative AI into educational assessment has gained significant traction globally, especially in high-resource contexts, its application in reflective writing assessment within African medical education systems remains under-researched and poorly understood. The literature on automated scoring systems has largely focused on narrative quality, coherence, and rubrics designed within Western epistemological and linguistic traditions [12; 10; 13]. Consequently, there is limited empirical evidence on how these systems perform in low- and middle-income countries (LMICs) with distinct cultural, linguistic, and pedagogical realities. In Ghana, although reflective writing is increasingly incorporated into medical training curricula, its assessment is often inconsistent, informal, and lacking in timely, formative feedback due to the shortage of trained assessors and the time-intensive nature of narrative evaluation [ 19 ]. This results in missed opportunities to cultivate critical self-awareness and clinical judgment among medical students. The adoption of AI-powered scoring tools has the potential to fill this gap by providing scalable and consistent evaluation mechanisms. However, this promise remains largely theoretical in the Ghanaian context, as there is little to no scholarly work investigating whether these tools can fairly and validly assess student reflections that are embedded in Ghanaian cultural norms, professional values, and localized use of English [2; 3]. Furthermore, questions about validity whether the AI-generated scores accurately reflect the quality and depth of reflective thinking and fairness whether the algorithms function equitably across different demographic, linguistic, and educational backgrounds remain underexplored in sub-Saharan Africa [4; 7]. Studies conducted in high-income contexts often assume that reflective competence manifests in universal ways, yet research in educational assessment has shown that expressions of reflection are deeply context-dependent [21;3]. For example, Ghanaian students may reflect using proverbs, idiomatic expressions, or culturally contextual moral reasoning that AI systems trained on predominantly Western corpora may fail to recognize or misinterpret. This lack of contextual validation poses a serious risk: the use of AI-based scoring without adequate local adaptation may reinforce systemic biases, marginalize students from underrepresented linguistic backgrounds, and compromise the pedagogical intent of reflective writing. Without empirical analysis of how these tools perform in Ghanaian medical education settings, stakeholders may either uncritically adopt technologies that are misaligned with local values or reject promising innovations due to unresolved concerns about their appropriateness. These issues call for rigorous empirical inquiry into how such technologies function when deployed in non-Western educational environments. As Ghana navigates the dual goals of expanding access to medical education and ensuring quality outcomes, it becomes imperative to critically assess the suitability, adaptability, and ethical implications of AI-based assessment tools within local contexts. This paper, therefore, seeks to conduct a validity and fairness analysis of NLP and generative AI systems used for the automated scoring and feedback of reflective writing in Ghanaian medical education. By situating the analysis within both the global discourse on AI in education and the local realities of Ghana’s medical training institutions, this study aims to contribute to more equitable, context-sensitive, and pedagogically sound innovations in assessment. Research Questions To what extent do NLP and generative AI tools provide valid assessments of reflective writing submitted by medical students in Ghana? How fair are AI-generated scores and feedback across different demographic groups (e.g., gender, language background, institutional affiliation) within Ghanaian medical education? What cultural, linguistic, and contextual factors influence the interpretation and accuracy of AI-driven assessments of reflective writing in Ghana’s medical schools? Methodology Research Design This study adopted a mixed-methods sequential explanatory research design, combining quantitative and qualitative approaches to investigate the validity and fairness of natural language processing (NLP) and generative AI tools in the automated assessment of reflective writing in Ghanaian medical education. The rationale for this design was rooted in the complexity of the research questions, which required both objective measurement and in-depth understanding of user experiences and contextual factors [6; 24]. In the first (quantitative) phase, reflective essays submitted by medical students were evaluated using an AI-based scoring engine that applied machine learning algorithms to assess multiple dimensions of reflective writing. This phase aimed to determine the consistency and validity of the AI-generated scores by comparing them to human-assigned scores. The second (qualitative) phase followed the quantitative analysis and involved semi-structured interviews with selected student writers and faculty assessors. This phase provided interpretive insight into participants’ perceptions of the feedback’s fairness, relevance, and cultural fit, thereby contextualizing the quantitative findings. The integration of these two strands of inquiry was designed to produce more robust conclusions, reflecting both statistical trends and the lived experiences of users. Such an approach is especially appropriate in educational technology research where algorithms interact with human cognition and cultural nuance [23; 24]. Population and Sampling The study population consisted of undergraduate medical students in their clinical years at three of Ghana’s premier public universities offering medical education: the University of Ghana Medical School (UGMS), the Kwame Nkrumah University of Science and Technology School of Medicine and Dentistry (KNUST-SMD), and the University for Development Studies School of Medicine and Health Sciences (UDS-SMHS). These institutions were selected purposively to reflect geographic and institutional diversity in Ghana’s medical education system southern, middle, and northern belts respectively thereby enhancing the representativeness of the findings [1; 2]. A stratified purposive sampling technique was used to ensure variation in student backgrounds and experiences. The stratification criteria included gender, institution, and linguistic diversity (standard Ghanaian English vs. regional dialect-influenced English), which are relevant to evaluating fairness in NLP-based tools. A total of 180 reflective essays were sampled, with an equal number (n = 60) collected from each university. Essays were selected from those written as part of required clinical rotation reflection assignments between 2022 and 2023. For the qualitative phase, 15 participants (five from each institution) were selected based on their willingness to participate and the richness and depth of their reflective writing, as judged by faculty assessors. This number was deemed adequate to achieve thematic saturation, a standard threshold in qualitative research where additional interviews yield no new information [ 10 ]. Data Collection Procedures Data collection was executed in two sequential stages essay collection and interview administration. In the first stage, reflective essays were retrieved from institutional learning management systems and departmental archives, with the cooperation of course coordinators. All students whose work was considered for inclusion were approached for informed consent, and only essays from consenting students were anonymized and used. The essays were then converted into standardized plain text files and processed through a customized NLP and generative AI scoring system. This system was adopted upon OpenAI’s GPT-4 API, which was fine-tuned through prompt engineering to assess reflective writing dimensions based on a modified version of the REFLECT rubric developed by [ 25 ]. The rubric assessed domains such as presence, description of conflict, attending to emotions, analysis, and meaning-making, which are central to clinical reflective writing. Each essay received both numerical scores for rubric dimensions and qualitative feedback simulating narrative commentary from a tutor. These outputs were archived for subsequent statistical comparison with human assessors’ ratings. In the second stage, semi-structured interviews were conducted with the 15 selected participants. Interviews explored their perceptions of the accuracy, clarity, cultural appropriateness, and usefulness of the AI-generated feedback. Faculty members were also interviewed to compare their judgments of the AI feedback with their professional standards for evaluating reflective work. Interviews were conducted in person or via Zoom, depending on participant availability, and lasted approximately 15–20 minutes. All interviews were audio-recorded with participant consent and transcribed verbatim. Transcripts were returned to participants for member checking, ensuring accuracy and credibility of the data [ 15 ]. Interview questions were developed based on literature on fairness and validity in AI-assisted education [7; 27], and pilot-tested with three students and one faculty member before final use. Instrumentation To facilitate comprehensive data collection and analysis, two primary instruments were employed: (1) an AI-based scoring engine and (2) a semi-structured interview guide adopted and adapted from existing literature on AI in education. AI-Based Scoring Engine The first instrument involved the deployment of an AI-based scoring engine designed to assess reflective essays through a standardized yet context-sensitive lens. This engine leveraged a large language model built on the GPT-4 architecture, customized to apply a modified version of the REFLECT rubric. The adapted rubric was structured around four core dimensions essential to effective reflective writing: descriptive clarity, emotional insight, critical analysis, and transformative learning. Each essay submitted by participants was evaluated along these dimensions using a 5-point Likert-type scale, where 1 indicated minimal evidence of the target skill and 5 indicated exceptional demonstration. In addition to numerical scoring, the AI model provided qualitative feedback that emulated the language, tone, and depth typically expected from experienced human assessors. This feedback was intended to offer actionable and interpretive insights that could guide participants' reflective practices. Prior to its full-scale deployment, the engine underwent a pilot phase involving 30 sample essays. This calibration process was essential to fine-tune prompt engineering, ensure scoring reliability, and align the AI-generated outputs with the intended pedagogical criteria. Semi-Structured Interview Guide The second instrument employed was a semi-structured interview guide, adopted and adapted from prior studies examining artificial intelligence in educational contexts [e.g.,9; 27; 29]. The purpose of the interview protocol was to explore users’ lived experiences and critical perspectives on the AI system’s performance, particularly in relation to issues of fairness, interpretability, transparency, and cultural appropriateness of the feedback and scoring mechanisms. The interview guide consisted of a mix of open-ended and closed-ended questions, enabling the collection of both quantitative impressions and rich qualitative narratives. Open-ended questions invited participants to reflect deeply on their encounters with the AI engine, while closed-ended items allowed for systematic comparisons across respondents. Sample questions included: “ To what extent do you believe the AI-generated feedback accurately reflects your writing?” “How fair and culturally sensitive did you find the scoring and comments provided by the AI tool?” “Would you prefer AI-generated feedback over human feedback in future assessments? Why or why not?” Follow-up probes were incorporated as needed during the interviews to delve further into specific themes emerging from participants’ initial responses. The semi-structured nature of the guide allowed for flexibility and responsiveness during the interviews while preserving a consistent framework across all participants. This design ensured that while core questions were standardized, the conversation could organically explore additional insights relevant to individual experiences. Together, these two instruments provided both quantitative and qualitative datasets that complemented one another one capturing patterns in scoring performance and the other eliciting deeper reflections on user perceptions, experiences, and contextual nuances. Data Analysis Quantitative data analysis focused on evaluating the reliability and fairness of the AI scoring system. Each essay was scored independently by two experienced faculty members using the same REFLECT rubric applied by the AI. The agreement between AI and human raters, and between the human raters themselves, was measured using Cohen’s kappa and intraclass correlation coefficients (ICC). These metrics assessed the degree of consistency and alignment across scoring sources. High inter-rater reliability would suggest that the AI system approximates human judgment, while significant discrepancies could indicate limitations in the AI’s interpretive capacity. To assess fairness, the study applied differential item functioning (DIF) analysis using logistic regression to identify whether scores varied significantly across demographic groups such as gender, university, and English language proficiency level. This approach, advocated by [ 32 ], is effective in detecting systematic bias in test or algorithmic performance. Additionally, multiple regression models were used to determine whether demographic variables predicted deviations between AI and human scores, thereby uncovering patterns of potential inequity. Qualitative data from interviews were analyzed using the thematic analysis approach proposed by Braun and Clarke (2006). Transcripts were first read multiple times to ensure familiarity, after which initial codes were generated. These codes were then clustered into broader themes capturing participants’ experiences and perceptions. Emergent themes included “algorithmic neutrality versus human empathy,” “misalignment of cultural nuance,” and “trust and skepticism in AI-generated feedback.” The analysis was conducted with the assistance of NVivo software to support coding, organization, and theme development. Triangulation of qualitative findings with quantitative results enhanced the validity and interpretive depth of the research. Ethical Considerations The research strictly adhered to ethical standards in educational and AI-related research. Ethical clearance was obtained from the Institutional Review Boards (IRBs) of UGMS, KNUST-SMD, and UDS-SMHS before data collection commenced. All participants provided written informed consent and were assured that their participation was voluntary and that they could withdraw at any time without consequence. Reflective essays were anonymized before processing to protect the identity of the students, and interview transcripts were coded with pseudonyms to maintain confidentiality. The AI system used for scoring was designed not to store or retain any personally identifiable information. All data were securely stored on encrypted drives accessible only to the research team. Furthermore, the study adhered to international ethical principles for AI deployment in education, particularly those outlined in the IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems [ 12 ], which emphasize transparency, fairness, and accountability. Participants were also informed about the limitations of AI-generated feedback and were encouraged to interpret it as a complement not a replacement for human judgment. These steps ensured that the research upheld the principles of beneficence, non-maleficence, and respect for persons throughout its duration. Table 1 evaluates the consistency of scores assigned by AI and human raters using multiple agreement metrics: Cohen’s Kappa, Intraclass Correlation Coefficient (ICC), Pearson’s correlation (r), mean difference, and the standard deviation of score differences. The Cohen’s Kappa values indicate substantial agreement across comparisons, ranging from 0.74 to 0.81, with the highest agreement found between Rater 1 and Rater 2 (κ = 0.81). The ICC values follow a similar pattern, showing strong consistency, particularly between human raters (ICC = 0.85). Pearson’s r reveals high positive correlations in all pairwise comparisons, especially between Rater 1 and Rater 2 (r = 0.88). Notably, the AI tends to score slightly higher than the human raters on average, with a mean difference of + 0.12 against Rater 1 and + 0.09 against Rater 2. However, the standard deviation values (0.46 and 0.42) suggest some variability in these differences. The smallest mean difference and variability are observed between the human raters themselves, reinforcing their alignment. Table 1 Inter-Rater Agreement Across Scoring Sources Comparison Cohen’s Kappa ICC (2,1) Pearson’s r Mean Difference (AI – Human) SD of Difference AI vs Rater 1 0.74 0.78 0.81 + 0.12 0.46 AI vs Rater 2 0.76 0.80 0.84 + 0.09 0.42 Rater 1 vs Rater 2 0.81 0.85 0.88 -0.03 0.37 Cohen’s Kappa: Substantial (0.74–0.81) ICC: Good to Excellent (0.78–0.85) Pearson’s r: Strong correlation (0.81–0.88) Mean Differences (AI – Human): +0.09 to + 0.12 SD of Differences: 0.37–0.46 Table 2 investigates whether discrepancies between AI and human scoring are influenced by demographic characteristics such as gender, university affiliation, and English proficiency. Males are significantly more likely than females to have higher AI scores relative to human scores, with a log-odds of 0.22 (p = 0.031), translating to an odds ratio of 1.24. Although females show a negative discrepancy (-0.15), it is not statistically significant (p = 0.067). Among university affiliations, students from UGMS exhibit a modest but significant positive discrepancy (log-odds = 0.10, p = 0.049), indicating slightly higher AI scores compared to human evaluations. In contrast, KNUST students show a borderline negative discrepancy (log-odds = -0.12, p = 0.081), and the discrepancy for UDS students is negligible and not statistically significant. The most pronounced effects are observed in relation to English proficiency. Those with low proficiency have significantly higher AI-human discrepancies (log-odds = 0.45, p = 0.001, OR = 1.57), while those with high proficiency tend to have lower AI scores relative to human raters (log-odds = -0.30, p = 0.014, OR = 0.74). Adjusted R² values suggest modest explanatory power for these models, with English proficiency contributing the most (Adjusted R² = 0.15). Table 2 Demographic Group Discrepancy Analysis Demographic Group Log-Odds (AI-Human Score Discrepancy) p-Value 95% Confidence Interval (Log-Odds) Odds Ratio Adjusted R² Gender (Male) 0.22 0.031 [0.05, 0.39] 1.24 0.10 Gender (Female) -0.15 0.067 [-0.32, 0.02] 0.86 0.06 University (UGMS) 0.10 0.049 [0.01, 0.19] 1.11 0.12 University (KNUST) -0.12 0.081 [-0.24, 0.00] 0.88 0.07 University (UDS) 0.05 0.102 [-0.03, 0.13] 1.05 0.05 English Proficiency (High) -0.30 0.014 [-0.52, -0.08] 0.74 0.09 English Proficiency (Low) 0.45 0.001 [0.28, 0.62] 1.57 0.15 Significant p-values: Male (0.031), UGMS (0.049), High Proficiency (0.014), Low Proficiency (0.001; Odds Ratios range: 0.74–1.57; Log-Odds range: − 0.30 to + 0.45; Adjusted R² range: 0.05–0.15 Table 3 presents a multiple regression model assessing the combined effect of key predictors on the discrepancy between AI and human scores. Gender remains a significant predictor, with males having higher AI scores than human raters by an average of 0.21 points (p = 0.045). University affiliation is a borderline predictor: KNUST students have marginally lower discrepancies compared to the reference category, though this effect is only marginally significant (β = -0.18, p = 0.054). UDS affiliation is not a significant predictor (p = 0.174). The strongest predictor in the model is low English proficiency, which significantly increases the AI-human discrepancy by 0.38 points (p = 0.002), reinforcing findings from Table 2 . The overall adjusted R² values range from 0.05 to 0.15, indicating that while demographic variables, especially English proficiency, are meaningful predictors, a substantial portion of variance in AI-human scoring discrepancy remains unexplained, possibly attributable to individual-level factors or nuances in language and response quality. Table 3 Multiple Regression Predicting AI-Human Score Discrepancy Predictor Coefficient (β) p-Value Standard Error (SE) 95% Confidence Interval (β) Odds Ratio (OR) Adjusted R² Gender (Male) 0.21 0.045 0.11 [0.02, 0.40] 1.23 0.12 University (KNUST) -0.18 0.054 0.10 [-0.37, 0.01] 0.83 0.08 University (UDS) 0.07 0.174 0.09 [-0.11, 0.25] 1.08 0.05 English Proficiency (Low) 0.38 0.002 0.12 [0.14, 0.62] 1.46 0.15 Constant 2.45 0.000 0.35 [1.76, 3.14] - - Significant p-values: Male (0.045), Low Proficiency (0.002), Coefficients (β) range: − 0.18 to + 0.38; Odds Ratios: 0.83–1.46; Adjusted R²: 0.05–0.15; Constant: β = 2.45, p = 0.000 Qualitative Results In-depth thematic analysis of the interview transcripts uncovered three core themes reflecting participants’ experiences and perceptions regarding AI-generated feedback on reflective writing. These themes Algorithmic Neutrality Versus Human Empathy, Misalignment with Cultural and Linguistic Nuance, and Trust and Skepticism in AI Feedback offer rich insight into the strengths, limitations, and socio-cultural implications of integrating artificial intelligence into assessment within Ghanaian medical education. The voices of students and faculty across three universities (UGMS, KNUST, and UDS) provided a textured understanding of how AI impacts perceived fairness, usefulness, and legitimacy in reflective assessment. Theme 1: Algorithmic Neutrality Versus Human Empathy Many students and faculty members recognized AI’s technical precision but critiqued its inability to engage with the emotional depth and subjective nuances often present in reflective writing. The algorithm’s emphasis on structure, grammar, and logic was perceived as helpful in surface-level editing but insufficient in interpreting personal, vulnerable narratives. As one third-year UGMS student reflected, “The AI pointed out structural issues, but it didn’t recognize the personal pain I tried to express. It felt like my story was reduced to grammar.” This sentiment captures a widespread frustration among students who felt that their emotional authenticity and narrative voice were not being honored. Faculty members reinforced this concern. A clinical educator at KNUST shared, “Empathy is part of our curriculum how can we ask students to write from the heart and then have a machine that has none evaluate it?” For educators, this misalignment between pedagogical goals and automated evaluation was not merely technical it was philosophical. Reflective writing, especially in medicine, is designed to cultivate professional identity, empathy, and self-awareness. Participants feared that a system lacking human emotion may inadvertently discourage vulnerability and introspection. This theme reveals a fundamental limitation in current AI systems: their inability to interpret affective meaning and emotional subtext. While algorithmic neutrality can minimize human biases and increase consistency, it may also strip reflective assessments of their humanistic core. Therefore, AI should be positioned as a supplemental tool in formative feedback rather than as a sole evaluator, especially in emotionally sensitive contexts like medical reflection. Theme 2: Misalignment with Cultural and Linguistic Nuance A second theme emphasized the disconnect between AI-generated feedback and local cultural or linguistic expressions. Reflective writing often contains indigenous proverbs, metaphors, and idioms deeply rooted in Ghanaian culture. However, participants reported that these stylistic choices were frequently flagged by the AI as unclear or irrelevant. A UDS student explained, “I used a Dagbani proverb to explain a moral dilemma, and the AI said it was off-topic. It didn’t get the meaning at all.” This experience was not isolated. Students from different ethnic backgrounds shared that the AI lacked the contextual awareness needed to interpret figurative language that carries significant meaning in local discourse. Faculty echoed these concerns, noting that the AI’s scoring algorithm appeared to rely heavily on Westernized English norms and cultural assumptions. As one senior lecturer at UGMS stated, “AI models are trained mostly on Western datasets. That limits their ability to fairly assess African ways of expression.” This theme raises serious concerns about cultural equity in automated assessment. AI models that are not trained on linguistically and culturally diverse datasets risk marginalizing voices that deviate from dominant norms. The misinterpretation or penalization of culturally specific expressions can lead to feelings of alienation, reduced self-expression, and unjust evaluation. To address this, developers should prioritize decolonizing AI training data and ensuring inclusive language modeling that reflects the diversity of student populations. Theme 3: Trust and Skepticism in AI-Generated Feedback Perceptions of trust in AI feedback varied widely among participants. On one hand, some students praised the immediacy, clarity, and specificity of the feedback they received. A UGMS student noted, “It gave me specific points to improve. With humans, I sometimes just get vague comments like ‘good effort.’” This view reflects the value of AI in generating fast, detailed feedback that supports revision and learning. However, skepticism was just as prevalent. Many students questioned the transparency and logic behind the AI’s scoring. One final-year student from KNUST expressed frustration: “I spent hours on my essay, and the AI gave me a lower score than my classmate who admitted to using ChatGPT.” This perceived misalignment between effort and reward led some students to question the credibility of the tool. Faculty members also expressed ambivalence. While they acknowledged AI’s usefulness in standardizing feedback, they warned against its overuse in summative evaluation. A reflective writing instructor at UDS remarked, “I’m fine with using AI as a draft scorer, but final decisions need human judgment. We can’t outsource ethics and empathy.” The theme of trust reflects broader tensions around AI transparency, explainability, and fairness. The perception that AI is a “black box” whose decisions are unexplainable undermines its legitimacy in the eyes of users. Furthermore, the inconsistency between effort, content quality, and scores can diminish student motivation and confidence. These findings suggest that AI should be deployed in low-stakes environments, accompanied by clear rubrics and human oversight. Building trust also requires demystifying how AI systems work and involving users in the feedback loop. Discussion of Findings The study investigated the validity, fairness, and contextual sensitivity of AI-generated assessments of reflective writing among medical students in Ghana. The mixed-methods approach provided a nuanced understanding, with quantitative metrics offering statistical insights into reliability and bias, while qualitative data illuminated subjective experiences and cultural interpretations of AI feedback. The inter-rater agreement results (Table 1 ) demonstrated moderately strong alignment between AI-generated scores and those of human raters. Specifically, the Cohen’s Kappa values between AI and human raters ranged from 0.74 to 0.76, while the intraclass correlation coefficients (ICC) were between 0.78 and 0.80. The Pearson’s r correlation values exceeded 0.80 in both cases, suggesting high consistency. The relatively small mean differences (+ 0.09 to + 0.12) and standard deviations (< 0.50) further support that AI scoring closely approximates human judgment on structural and technical aspects of reflective writing. These results align with findings from [ 31 ], who noted that AI scoring models trained on human-annotated corpora can reach levels of inter-rater reliability comparable to expert graders, particularly when using structured rubrics such as REFLECT. However, slight discrepancies indicate that the AI may emphasize surface features grammar, structure, and coherence over deeper narrative and emotional resonance, a pattern also identified by [ 28 ] in their critique of AI-based scoring tools in professional education. The differential item functioning (DIF) analysis (Table 2 ) and multiple regression (Table 3 ) highlighted important equity-related concerns. Statistically significant discrepancies emerged based on gender, university affiliation, and especially English proficiency. Students with lower English proficiency received disproportionately divergent scores between AI and human raters, with a log-odds of 0.45 (p = 0.001) and a regression coefficient of 0.38 (p = 0.002). These findings suggest that the AI system was more likely to penalize such students, possibly due to linguistic surface errors that human raters might have discounted in favor of depth of reflection or narrative intent. These finding echoes research by [14; 11], who observed that AI models often reflect the linguistic biases embedded in their training data, favoring standard English over local dialects or non-native structures. The implications are particularly salient in multilingual educational contexts like Ghana, where code-switching and hybrid linguistic patterns are common. Gender-based differences, although modest, also suggest that male students experienced slightly more favorable alignment between AI and human scores. This could stem from differences in writing styles or content expression that inadvertently align more with the AI’s training data an issue also reported by [ 26 ] in their cross-cultural analysis of gender effects in automated writing assessment. The qualitative findings reinforced and contextualized the quantitative results, revealing deeper concerns about AI’s interpretive limitations in cross-cultural settings. The theme of algorithmic neutrality versus human empathy emphasized the AI’s inability to detect emotional undertones or contextual depth. While this did not dramatically affect score correlations, it affected the perceived fairness and pedagogical value of the feedback. As shown in earlier studies [e.g., 16; 11], such emotional and narrative components are crucial in reflective writing and medical education, where empathy is not just a theme but a skill to be cultivated. The theme of misalignment with cultural and linguistic nuance highlighted how AI systems trained on Western-centric corpora struggle to interpret culturally embedded metaphors, idioms, or context-specific references. Such misinterpretations can lead to mis-scoring and reduce student confidence in the feedback process. This supports the critique by [ 26 ], who warns that AI applications in education risk epistemic injustice when local knowledge forms are excluded from algorithmic design. Finally, the theme of trust and skepticism revealed a split in user perceptions: some students valued the clarity and speed of AI feedback, while others doubted its validity, especially when scores conflicted with perceived effort. Faculty shared concerns about over-reliance on AI in high-stakes contexts, emphasizing the need for human oversight. Conclusion This study critically examined the performance, fairness, and cultural sensitivity of AI-generated assessments of reflective writing among medical students in Ghana. Drawing on both statistical measures and narrative accounts, the findings point to a nuanced picture: AI-based assessment tools show promising levels of consistency and inter-rater agreement with human raters, especially when evaluating structural and linguistic features. However, significant concerns persist around equity and contextual validity. Quantitative analyses revealed that demographic factors particularly English language proficiency and, to a lesser extent, gender and institutional affiliation were associated with discrepancies between AI and human scoring. These differences suggest that while the AI system may be technically proficient, it lacks sufficient adaptation to the diverse linguistic and educational backgrounds typical of Ghanaian learners. Qualitative insights added depth to these findings by revealing participants’ lived experiences of AI feedback. Students and faculty alike appreciated the speed and objectivity of AI-generated evaluations but raised concerns about the absence of empathy, cultural nuance, and contextual understanding. These limitations were particularly salient in reflective writing, a genre that demands emotional expression, personal insight, and cultural relevance dimensions that AI tools, in their current form, struggle to assess meaningfully. Taken together, the study highlights the need for caution and balance in deploying AI tools in educational assessment, especially in high-stakes or formative domains like medical education. AI should be viewed not as a replacement for human judgment, but as a complementary tool that can enhance assessment when properly localized, validated, and monitored. Recommendations To ensure equitable, culturally sensitive, and pedagogically sound use of AI-generated assessments in reflective academic writing, especially within contexts like Ghanaian medical education, several strategic actions are recommended based on the study’s findings. First, there is a critical need for contextual adaptation of AI assessment models. The quantitative results demonstrated that discrepancies in AI scoring were influenced by factors such as English proficiency and institutional affiliation, indicating that the AI system may not be fully attuned to the linguistic and educational diversity of students in the local context. Moreover, qualitative data underscored participants’ concerns about the AI’s inability to recognize culturally embedded metaphors or rhetorical forms. To address this, AI models should be fine-tuned using locally representative datasets that capture the full range of linguistic expressions, academic conventions, and cultural references found in Ghanaian student writing. Such adaptation would enhance the AI’s ability to evaluate texts more fairly and reduce misinterpretation of culturally specific language. Second, the study recommends the adoption of hybrid assessment models that integrate both AI-generated and human scoring. While the AI demonstrated consistency in identifying structural issues and surface-level features, both students and faculty highlighted its shortcomings in interpreting emotional nuance, tone, and reflective depth. A hybrid system allows institutions to leverage the speed, objectivity, and scalability of AI, while preserving the human ability to engage with context, empathy, and meaning-making. This combination is particularly important in the assessment of reflective writing, where subjective interpretation plays a crucial role in evaluating the depth and quality of student learning. Third, there is a pressing need for faculty development focused on the ethical and critical use of AI tools. As AI technologies become more integrated into educational environments, educators must be trained not only to interpret AI outputs but also to recognize their limitations. Without proper training, there is a risk of over-reliance on AI-generated scores, potentially undermining academic judgment and student trust. Professional development workshops, seminars, and certification programs should be instituted to empower educators with the skills to balance technological efficiency with pedagogical integrity. Finally, institutions should establish mechanisms for ongoing bias audits and fairness evaluations. The study’s logistic and regression analyses revealed group-based disparities in AI scoring accuracy, particularly disadvantaging students with lower English proficiency. This underscores the risk of systemic inequity if AI systems are deployed without oversight. Regular audits should be conducted to monitor the performance of AI scoring tools across different demographic groups, including gender, institutional background, and language proficiency. These audits should inform continuous refinement of the AI system and guide policies for its responsible implementation. Limitations of the study This study has several limitations that should be considered when interpreting the results. First, the research was geographically restricted to three Ghanaian medical schools, which limits the generalizability of the findings to other regions or educational contexts. The study’s focus on reflective writing also narrows the scope, as AI’s performance in assessing other types of academic writing may differ. Furthermore, the small sample size for the qualitative data, based on a limited number of student and faculty interviews, may not fully represent the diverse experiences and perspectives within the broader population. Additionally, there may have been biases in human scoring, as raters brought subjective judgments to the assessment, which could have affected the inter-rater reliability. The study also did not explore the long-term educational impacts of AI-generated feedback on student learning outcomes, leaving a gap in understanding its effects on student engagement, academic performance, or development over time. Despite these limitations, the findings provide valuable insights into AI’s potential and challenges in educational assessment, paving the way for future research in this area. Implications for Practice and Policy The findings of this study carry significant implications for both practice and policy in the realm of AI-powered assessment in educational contexts, particularly in settings as culturally and linguistically diverse as Ghana. While AI presents promising advantages in terms of efficiency and consistency in assessing reflective writing, the results highlight that it cannot fully replace human judgment, especially when it comes to capturing emotional nuance and cultural sensitivity. In this context, AI can serve as a useful tool for providing preliminary feedback, yet its limitations in understanding the complexities of personal experiences and cultural expressions underscore the necessity of human input in the assessment process. From a practical standpoint, AI feedback can be employed as an initial review tool to guide students’ self-reflection and improvement. However, it is critical that such feedback be supplemented with human critique to ensure that the emotional depth and contextual relevance of the student’s work are accurately captured. This also raises the need for institutions to regularly evaluate and audit AI systems for fairness, particularly in multicultural and multilingual environments, to ensure that bias is minimized. Furthermore, there is a pressing need for faculty to be trained in interpreting AI-generated feedback critically, so they can integrate it effectively into broader assessment strategies. Lastly, to address algorithmic biases, educational institutions should focus on developing locally adapted AI training corpora that reflect Ghanaian linguistic, cultural, and academic norms. This would help ensure that AI systems are more equitable and sensitive to the unique context in which they are applied. Abbreviations IRB – Institutional Review Board; UEW – University of Education, Winneba; AI – Artificial Intelligence; NLP – Natural Language Processing; SPSS – Statistical Package for the Social Sciences; REFLECT – Reflection Evaluation for Learners’ Enhanced Competencies Tool; HICs -High-Income Countries, LMICs -Low- And Middle-Income Countries; GPT-4 – Generative Pre-trained Transformer 4, KNUST-SMD – Kwame Nkrumah University of Science and Technology School of Medicine and Dentistry, NVivo – Qualitative Data Analysis Software, UDS-SMHS – University for Development Studies School of Medicine and Health Sciences, UGMS – University of Ghana Medical School. Declarations Ethics Approval and Consent to Participate Ethical approval for this study was granted by the Institutional Review Board (IRB) of the University of Education, Winneba (UEW), Ghana. Informed consent was obtained from all participants prior to data collection. Participants were fully briefed on the purpose, procedures, and voluntary nature of the study, including their right to withdraw at any time without consequence. All ethical protocols were strictly adhered to, particularly concerning anonymity, data security, and responsible use of AI-generated feedback in educational settings. Consent for Publication All authors have reviewed and approved the final version of the manuscript and consent to its submission for publication. Participants were informed that anonymized data would be used strictly for academic and research purposes, including possible publication in peer-reviewed journals. Informed consent for publication was obtained from all participants. Funding This research was entirely self-funded by the authors. No external funding or institutional sponsorship was received, ensuring the independence and impartiality of the research design, analysis, and interpretation. Clinical Trial Number Not applicable. Declaration of Conflicts of Interest The authors declare no conflicts of interest. This research was independently conceptualized, designed, and conducted by the authors without any external influence on its methodology, analysis, or interpretation. Availability of Data and Materials The datasets generated and analyzed during this study on the validity, fairness, and contextual sensitivity of AI-generated assessments of reflective writing among medical students in Ghana are available upon reasonable request from the corresponding author, Simon Ntumi. Due to ethical considerations and to protect the confidentiality of student and faculty participants, raw data including reflective writing samples, rater scores, and qualitative transcripts will not be made publicly accessible. All data requests will be reviewed in accordance with institutional ethical guidelines to ensure compliance with data protection protocols and participant anonymity. Acknowledgments The authors gratefully acknowledge the medical students, faculty assessors, and institutional partners in Ghana who made this study possible. Special thanks go to the departments of medical education, assessment units, and ethics committees at participating universities for their cooperation and support. The authors also appreciate the technical assistance provided by local AI specialists and the peer reviewers whose feedback enhanced the rigor and clarity of this work. Authors’ Contributions Simon Ntumi (Corresponding Author) led the conceptualization and design of the study, developed the research methodology, and oversaw all aspects of data collection from the participating medical schools. He conducted both the quantitative and qualitative analyses, including inter-rater reliability and differential item functioning tests, as well as thematic coding and synthesis. He was primarily responsible for drafting the original manuscript, integrating co-authors’ contributions, and coordinating the supervision of the project’s execution across institutions. Isaac Yabana contributed extensively to the development of the literature review, ensuring alignment with current discourses on AI in medical education and assessment fairness. He assisted in interpreting both quantitative and qualitative data and played a critical role in reviewing and editing the manuscript for intellectual content and clarity. He also validated the findings by triangulating interpretations across multiple data sources. Daniel William Essel conducted advanced statistical analyses, including interclass correlation coefficients and graphical visualizations of the AI-human rating agreement. He interpreted the results in collaboration with the lead author and co-authors, contributed to the preparation of tables and figures, and revised the manuscript during the review and editing phase with a focus on methodological rigor and reproducibility. Edmond Ahovi was responsible for coding and analyzing the qualitative data, particularly in organizing and refining emergent themes from student narratives and focus group discussions. He ensured that ethical standards—including participant confidentiality and informed consent—were upheld throughout the study. He also contributed to reviewing and editing the manuscript, ensuring its consistency and alignment with ethical and qualitative reporting standards. All authors reviewed and approved the final manuscript and agree to be accountable for all aspects of the work. References Antwi-Baffour, S., Agyemang, C., & Arhin, S. (2021). Exploring the contextual factors influencing educational outcomes in Ghana: A cross-institutional study. Journal of Educational Administration, 59 (4), 445-463. Antwi-Baffour, S., Asare, B., & Danquah, S. (2021). Challenges in the adoption of technology in Ghanaian medical schools: Implications for medical education. Ghanaian Journal of Medical Education, 12 (1), 45-58. Asare, B., & Danquah, S. (2020). Medical education in Ghana: Current state and challenges. Journal of African Medical Education, 8 (2), 102-110. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmit, S. (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 1-11. Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3 (2), 77-101. Creswell, J. W., & Plano Clark, V. L. (2017). Designing and conducting mixed methods research (3rd ed.). Sage Publications. Floridi, L., Taddeo, M., & Turilli, M. (2022). AI in education: Challenges and opportunities. Philosophy & Technology, 35 (1), 1-20. Floridi, L., Taddeo, M., & Turilli, M. (2023). The ethics of artificial intelligence and the moral implications of AI in education . Cambridge University Press. Frenk, J., Chen, L., Bhutta, Z. A., & et al. (2010). Health professionals for a new century: Transforming education to strengthen health systems in an interdependent world. The Lancet, 376 (9756), 1923-1958. Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18 (1), 59-82. Hutt, S., DePiro, A., Wang, J., Rhodes, S., Baker, R. S., Hieb, G., ... & Mills, C. (2024, March). Feedback on feedback: comparing classic natural language processing and generative AI to evaluate peer feedback. In Proceedings of the 14th Learning Analytics and Knowledge Conference (pp. 55-65). IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems. (2019). Ethically aligned design: A vision for prioritizing human well-being with autonomous and intelligent systems . IEEE. Kumar, A., Sahu, R., & Sharma, D. (2023). Artificial intelligence in medical education: Opportunities and challenges in automated assessment. Journal of Medical Education and Practice, 34 (3), 24-31. Kumar, V., & Rose, C. (2021). Linguistic biases in AI assessment: A study on non-native English speakers in higher education. Journal of Educational Technology, 38 (4), 45-60. Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry . Sage Publications. Liu, L., Xu, J., & Zhang, T. (2021). Emotional intelligence in medical education: The role of reflective writing in empathy development. Medical Education, 55 (7), 775-785. Luxton-Reilly, A., Harland, J., & McIntyre, T. (2020). Assessing the impact of artificial intelligence on educational assessment systems. Journal of Educational Technology & Society, 23 (4), 89-101. Mann, K., Gordon, J., & MacLeod, A. (2019). Reflection and reflective practice in health professions education: A systematic review. Advances in Health Sciences Education, 14 (4), 595-621. Mensah, P., Boateng, K., & Akrofi, M. (2022). Challenges of reflective writing assessment in Ghanaian medical schools. International Journal of Medical Education, 13 (2), 45-59. Omaswa, F., & Crisp, N. (2014). The African health workforce: Challenges and opportunities. The Lancet, 384 (9943), 1473-1479. Ryan, M. (2013). Reflections on reflection: Educational implications for reflective practice in medical education. Medical Education, 47 (9), 877-885. Sandars, J. (2019). The use of reflection in medical education: A systematic review of the literature. Medical Teacher, 31 (8), 710-717. Tashakkori, A., & Teddlie, C. (2010). Mixed methods in social & behavioral research (2nd ed.). Sage Publications. Udupa, S. (2020). Algorithmic justice: The risks of epistemic injustice in AI-based educational applications. Education and Technology, 21 (2), 215-230. Wald, H. S., Borkan, J. M., Taylor, J. S., & Anthony, D. (2012). The reflective practitioner: The role of reflection in medical education. Medical Education, 46 (6), 580-588. Wang, Q., Li, J., & Zhang, Y. (2023). Gender differences in automated writing assessments: A cross-cultural analysis of AI bias in academic evaluations. Educational Assessment, 19 (2), 121-135. Williamson, B., & Kukulska-Hulme, A. (2020). AI-based writing assessment tools: A critique of their use in professional education. Journal of Educational Measurement, 48 (3), 285-300. Williamson, B., & Piattoeva, N. (2021). The global education race: Taking the measure of the world’s education systems . Routledge. World Bank. (2021). Education in Africa: Challenges and opportunities in low and middle-income countries . World Bank Group. Zhang, Y., Chen, L., & Li, J. (2022). Exploring the potential of artificial intelligence in medical education assessment: A review of literature. Medical Education, 56 (6), 1223-1232. Zhang, Y., Wei, X., & Yao, Z. (2022). AI-based scoring models: Achieving inter-rater reliability with human-annotated corpora. Journal of Educational Measurement, 49 (1), 112-125. Zumbo, B. D. (2007). Introduction to differential item functioning: A guide for the novice. Practical Assessment, Research, and Evaluation, 12 (1), 1-15. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6636682","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":496741885,"identity":"2adf733a-a54e-4ae5-9f4c-24bee2c2a681","order_by":0,"name":"Simon Ntumi","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAv0lEQVRIiWNgGAWjYNACAxsIzQNiE1bODFKWRrIWhsMkaJGf3X9M6kbB+cR+iQTGB2/bGOy2E9JicOcwm3SOwe3EmTMSmA3ntjEk72wgpEUiGaJlw+0ENmleoBaDA4QcNgOs5Vzi/tsJ7L+J0sJwA6zlQOIG6QQ2ZqAWO4JaDG4kG1vnGCQbz7j/sFlyzjmJBCIclvjwds4fO9n+nsMHP7wps7En7DAEYGwAEhKJDcTrgAJ7knWMglEwCkbBsAcAdVc9xofsEDAAAAAASUVORK5CYII=","orcid":"","institution":"University of Education, Winneba","correspondingAuthor":true,"prefix":"","firstName":"Simon","middleName":"","lastName":"Ntumi","suffix":""},{"id":496741886,"identity":"bcce7c58-f545-4e84-8fc4-c5df54535d82","order_by":1,"name":"Isaac Yabana","email":"","orcid":"","institution":"University of Education, Winneba","correspondingAuthor":false,"prefix":"","firstName":"Isaac","middleName":"","lastName":"Yabana","suffix":""},{"id":496741887,"identity":"4d00bf5e-9dbb-4eda-94c7-ecb02e564ae6","order_by":2,"name":"Daniel William Essel","email":"","orcid":"","institution":"University of Education, Winneba","correspondingAuthor":false,"prefix":"","firstName":"Daniel","middleName":"William","lastName":"Essel","suffix":""},{"id":496741888,"identity":"7596fd34-5a2c-42e8-b1e3-5e3dc20afe25","order_by":3,"name":"Edmond Ahovi","email":"","orcid":"","institution":"University of Education, Winneba","correspondingAuthor":false,"prefix":"","firstName":"Edmond","middleName":"","lastName":"Ahovi","suffix":""}],"badges":[],"createdAt":"2025-05-10 20:38:13","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6636682/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6636682/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":90892792,"identity":"07c08da7-01b9-47ad-b729-95052f5e9c96","added_by":"auto","created_at":"2025-09-09 11:23:25","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":950265,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6636682/v1/2f584c98-9671-49ef-bfe9-ee4ff8ed9889.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Natural Language Processing and Generative AI in the Automated Scoring and Feedback of Reflective Writing in Medical Education: A Validity and Fairness Analysis","fulltext":[{"header":"Introduction","content":"\u003cp\u003eIn an era defined by rapid technological innovation, artificial intelligence (AI) has emerged as a transformative force across multiple sectors, including education. Among the most promising applications of AI in education is the integration of Natural Language Processing (NLP) and generative AI technologies into assessment systems [11; 16; 12]. These tools are capable of processing and understanding human language at scale, allowing for the automated evaluation of complex written responses such as reflective essays. Unlike traditional multiple-choice assessments, reflective writing captures deeper dimensions of learning such as critical thinking, empathy, emotional intelligence, and metacognition which are essential for the development of competent professionals, particularly in fields like medicine. As such, educational institutions globally are increasingly adopting AI-powered scoring and feedback systems to evaluate students\u0026rsquo; reflective narratives, enabling timely, consistent, and personalized feedback at scale 1[7; 13; 14]. Reflective practice has long been recognized as a cornerstone of medical education. It encourages students and practitioners to critically examine their clinical experiences, ethical decisions, and interpersonal interactions, thereby promoting self-awareness and continuous professional development [18; 22; 20]. Traditionally, the assessment of reflective writing in medical education has relied on manual scoring by trained faculty, a process that is not only time-consuming but also subject to inconsistencies and biases. Automated systems powered by NLP and generative AI such as large language models (LLMs) now offer an alternative that promises greater efficiency and objectivity, while also relieving the assessment burden on faculty members [31;11; 21]. These systems can detect patterns in language use, evaluate argument coherence, and even provide formative feedback aligned with rubrics, thereby enhancing pedagogical impact [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e].\u003c/p\u003e\u003cp\u003eDespite these advancements, the application of AI in educational assessment remains largely concentrated in high-income countries (HICs), where infrastructure, data resources, and digital literacy levels support large-scale deployment. By contrast, low- and middle-income countries (LMICs), especially in sub-Saharan Africa, continue to grapple with systemic challenges in education delivery, including underfunding, staff shortages, and lack of access to modern educational technologies [29; 20]. Medical education in Africa, in particular, faces critical shortages in trained health professionals, overcrowded classrooms, and limited opportunities for feedback and mentorship [9; 11; 17]. The adoption of AI-based assessment tools, therefore, presents a potentially transformative solution to these enduring problems by offering scalable methods for delivering feedback and improving educational outcomes. Ghana represents a compelling case study within this broader African context. As a lower-middle-income country with a growing youth population and rising demand for healthcare professionals, Ghana\u0026rsquo;s medical education system is under pressure to expand access while maintaining quality. Medical schools such as the University of Ghana Medical School, Kwame Nkrumah University of Science and Technology School of Medical Sciences, and the School of Medicine and Health Sciences at the University for Development Studies play critical roles in training the nation\u0026rsquo;s physicians. However, they are routinely confronted with large student-to-faculty ratios, heavy administrative loads, and the slow adoption of innovative teaching and assessment tools [1; 2]. As Ghana strives to align its medical training with global standards, reflective writing has been increasingly recognized as a vital pedagogical tool especially in ethics, patient communication, and clinical rotations but assessment practices remain uneven and unsystematic due to resource constraints [19; 12].\u003c/p\u003e\u003cp\u003eThe introduction of NLP and generative AI in this context offers both promise and peril. On one hand, these technologies could revolutionize how reflective writing is assessed in Ghanaian medical schools by providing immediate, rubric-aligned feedback and by enabling students to iteratively refine their thinking. On the other hand, critical concerns about \u003cem\u003evalidity\u003c/em\u003e and \u003cem\u003efairness\u003c/em\u003e arise. Validity refers to whether AI-generated scores truly reflect the quality of students\u0026rsquo; reflections as conceptualized within the Ghanaian medical education framework. Fairness involves ensuring that these systems do not inadvertently disadvantage students based on linguistic background, cultural expression, or digital fluency especially given the dominance of Western-centric datasets and language models used to train AI tools [4; 1; 6]. For instance, Ghanaian students may employ locally contextualized metaphors, Ghanaian English syntax, or culturally grounded expressions in their writing, which may be misinterpreted or undervalued by AI systems not trained on such data. While the integration of natural language processing (NLP) and generative AI into educational assessment has gained significant traction globally, especially in high-resource contexts, its application in reflective writing assessment within African medical education systems remains under-researched and poorly understood. The literature on automated scoring systems has largely focused on narrative quality, coherence, and rubrics designed within Western epistemological and linguistic traditions [12; 10; 13]. Consequently, there is limited empirical evidence on how these systems perform in low- and middle-income countries (LMICs) with distinct cultural, linguistic, and pedagogical realities.\u003c/p\u003e\u003cp\u003eIn Ghana, although reflective writing is increasingly incorporated into medical training curricula, its assessment is often inconsistent, informal, and lacking in timely, formative feedback due to the shortage of trained assessors and the time-intensive nature of narrative evaluation [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. This results in missed opportunities to cultivate critical self-awareness and clinical judgment among medical students. The adoption of AI-powered scoring tools has the potential to fill this gap by providing scalable and consistent evaluation mechanisms. However, this promise remains largely theoretical in the Ghanaian context, as there is little to no scholarly work investigating whether these tools can fairly and validly assess student reflections that are embedded in Ghanaian cultural norms, professional values, and localized use of English [2; 3]. Furthermore, questions about validity whether the AI-generated scores accurately reflect the quality and depth of reflective thinking and fairness whether the algorithms function equitably across different demographic, linguistic, and educational backgrounds remain underexplored in sub-Saharan Africa [4; 7]. Studies conducted in high-income contexts often assume that reflective competence manifests in universal ways, yet research in educational assessment has shown that expressions of reflection are deeply context-dependent [21;3]. For example, Ghanaian students may reflect using proverbs, idiomatic expressions, or culturally contextual moral reasoning that AI systems trained on predominantly Western corpora may fail to recognize or misinterpret.\u003c/p\u003e\u003cp\u003eThis lack of contextual validation poses a serious risk: the use of AI-based scoring without adequate local adaptation may reinforce systemic biases, marginalize students from underrepresented linguistic backgrounds, and compromise the pedagogical intent of reflective writing. Without empirical analysis of how these tools perform in Ghanaian medical education settings, stakeholders may either uncritically adopt technologies that are misaligned with local values or reject promising innovations due to unresolved concerns about their appropriateness. These issues call for rigorous empirical inquiry into how such technologies function when deployed in non-Western educational environments. As Ghana navigates the dual goals of expanding access to medical education and ensuring quality outcomes, it becomes imperative to critically assess the suitability, adaptability, and ethical implications of AI-based assessment tools within local contexts. This paper, therefore, seeks to conduct a \u003cem\u003evalidity and fairness analysis\u003c/em\u003e of NLP and generative AI systems used for the automated scoring and feedback of reflective writing in Ghanaian medical education. By situating the analysis within both the global discourse on AI in education and the local realities of Ghana\u0026rsquo;s medical training institutions, this study aims to contribute to more equitable, context-sensitive, and pedagogically sound innovations in assessment.\u003c/p\u003e\u003cp\u003e\u003cb\u003eResearch Questions\u003c/b\u003e\u003c/p\u003e\u003cp\u003e\u003col\u003e\u003cspan\u003e\u003cli\u003e\u003cp\u003eTo what extent do NLP and generative AI tools provide valid assessments of reflective writing submitted by medical students in Ghana?\u003c/p\u003e\u003c/li\u003e\u003c/span\u003e\u003cspan\u003e\u003cli\u003e\u003cp\u003eHow fair are AI-generated scores and feedback across different demographic groups (e.g., gender, language background, institutional affiliation) within Ghanaian medical education?\u003c/p\u003e\u003c/li\u003e\u003c/span\u003e\u003cspan\u003e\u003cli\u003e\u003cp\u003eWhat cultural, linguistic, and contextual factors influence the interpretation and accuracy of AI-driven assessments of reflective writing in Ghana\u0026rsquo;s medical schools?\u003c/p\u003e\u003c/li\u003e\u003c/span\u003e\u003c/ol\u003e\u003c/p\u003e"},{"header":"Methodology","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e\u003ch2\u003eResearch Design\u003c/h2\u003e\u003cp\u003eThis study adopted a mixed-methods sequential explanatory research design, combining quantitative and qualitative approaches to investigate the validity and fairness of natural language processing (NLP) and generative AI tools in the automated assessment of reflective writing in Ghanaian medical education. The rationale for this design was rooted in the complexity of the research questions, which required both objective measurement and in-depth understanding of user experiences and contextual factors [6; 24]. In the first (quantitative) phase, reflective essays submitted by medical students were evaluated using an AI-based scoring engine that applied machine learning algorithms to assess multiple dimensions of reflective writing. This phase aimed to determine the consistency and validity of the AI-generated scores by comparing them to human-assigned scores. The second (qualitative) phase followed the quantitative analysis and involved semi-structured interviews with selected student writers and faculty assessors. This phase provided interpretive insight into participants’ perceptions of the feedback’s fairness, relevance, and cultural fit, thereby contextualizing the quantitative findings. The integration of these two strands of inquiry was designed to produce more robust conclusions, reflecting both statistical trends and the lived experiences of users. Such an approach is especially appropriate in educational technology research where algorithms interact with human cognition and cultural nuance [23; 24].\u003c/p\u003e\u003c/div\u003e\n\u003ch3\u003ePopulation and Sampling\u003c/h3\u003e\n\u003cp\u003eThe study population consisted of undergraduate medical students in their clinical years at three of Ghana’s premier public universities offering medical education: the University of Ghana Medical School (UGMS), the Kwame Nkrumah University of Science and Technology School of Medicine and Dentistry (KNUST-SMD), and the University for Development Studies School of Medicine and Health Sciences (UDS-SMHS). These institutions were selected purposively to reflect geographic and institutional diversity in Ghana’s medical education system southern, middle, and northern belts respectively thereby enhancing the representativeness of the findings [1; 2]. A stratified purposive sampling technique was used to ensure variation in student backgrounds and experiences. The stratification criteria included gender, institution, and linguistic diversity (standard Ghanaian English vs. regional dialect-influenced English), which are relevant to evaluating fairness in NLP-based tools. A total of 180 reflective essays were sampled, with an equal number (n = 60) collected from each university. Essays were selected from those written as part of required clinical rotation reflection assignments between 2022 and 2023. For the qualitative phase, 15 participants (five from each institution) were selected based on their willingness to participate and the richness and depth of their reflective writing, as judged by faculty assessors. This number was deemed adequate to achieve thematic saturation, a standard threshold in qualitative research where additional interviews yield no new information [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e].\u003c/p\u003e\n\u003ch3\u003eData Collection Procedures\u003c/h3\u003e\n\u003cp\u003eData collection was executed in two sequential stages essay collection and interview administration. In the first stage, reflective essays were retrieved from institutional learning management systems and departmental archives, with the cooperation of course coordinators. All students whose work was considered for inclusion were approached for informed consent, and only essays from consenting students were anonymized and used. The essays were then converted into standardized plain text files and processed through a customized NLP and generative AI scoring system. This system was adopted upon OpenAI’s GPT-4 API, which was fine-tuned through prompt engineering to assess reflective writing dimensions based on a modified version of the REFLECT rubric developed by [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]. The rubric assessed domains such as presence, description of conflict, attending to emotions, analysis, and meaning-making, which are central to clinical reflective writing. Each essay received both numerical scores for rubric dimensions and qualitative feedback simulating narrative commentary from a tutor. These outputs were archived for subsequent statistical comparison with human assessors’ ratings. In the second stage, semi-structured interviews were conducted with the 15 selected participants. Interviews explored their perceptions of the accuracy, clarity, cultural appropriateness, and usefulness of the AI-generated feedback. Faculty members were also interviewed to compare their judgments of the AI feedback with their professional standards for evaluating reflective work. Interviews were conducted in person or via Zoom, depending on participant availability, and lasted approximately 15–20 minutes. All interviews were audio-recorded with participant consent and transcribed verbatim. Transcripts were returned to participants for member checking, ensuring accuracy and credibility of the data [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. Interview questions were developed based on literature on fairness and validity in AI-assisted education [7; 27], and pilot-tested with three students and one faculty member before final use.\u003c/p\u003e\n\u003ch3\u003eInstrumentation\u003c/h3\u003e\n\u003cp\u003eTo facilitate comprehensive data collection and analysis, two primary instruments were employed: (1) an AI-based scoring engine and (2) a semi-structured interview guide adopted and adapted from existing literature on AI in education.\u003c/p\u003e\n\u003ch3\u003eAI-Based Scoring Engine\u003c/h3\u003e\n\u003cp\u003eThe first instrument involved the deployment of an AI-based scoring engine designed to assess reflective essays through a standardized yet context-sensitive lens. This engine leveraged a large language model built on the GPT-4 architecture, customized to apply a modified version of the REFLECT rubric. The adapted rubric was structured around four core dimensions essential to effective reflective writing: descriptive clarity, emotional insight, critical analysis, and transformative learning. Each essay submitted by participants was evaluated along these dimensions using a 5-point Likert-type scale, where 1 indicated minimal evidence of the target skill and 5 indicated exceptional demonstration. In addition to numerical scoring, the AI model provided qualitative feedback that emulated the language, tone, and depth typically expected from experienced human assessors. This feedback was intended to offer actionable and interpretive insights that could guide participants' reflective practices. Prior to its full-scale deployment, the engine underwent a pilot phase involving 30 sample essays. This calibration process was essential to fine-tune prompt engineering, ensure scoring reliability, and align the AI-generated outputs with the intended pedagogical criteria.\u003c/p\u003e\u003cdiv id=\"Sec8\" class=\"Section2\"\u003e\u003ch2\u003eSemi-Structured Interview Guide\u003c/h2\u003e\u003cp\u003eThe second instrument employed was a semi-structured interview guide, adopted and adapted from prior studies examining artificial intelligence in educational contexts [e.g.,9; 27; 29]. The purpose of the interview protocol was to explore users’ lived experiences and critical perspectives on the AI system’s performance, particularly in relation to issues of fairness, interpretability, transparency, and cultural appropriateness of the feedback and scoring mechanisms.\u003c/p\u003e\u003cp\u003eThe interview guide consisted of a mix of open-ended and closed-ended questions, enabling the collection of both quantitative impressions and rich qualitative narratives. Open-ended questions invited participants to reflect deeply on their encounters with the AI engine, while closed-ended items allowed for systematic comparisons across respondents. Sample questions included:\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003e“\u003cem\u003eTo what extent do you believe the AI-generated feedback accurately reflects your writing?”\u003c/em\u003e\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003e\u003cem\u003e“How fair and culturally sensitive did you find the scoring and comments provided by the AI tool?”\u003c/em\u003e\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003e\u003cem\u003e“Would you prefer AI-generated feedback over human feedback in future assessments? Why or why not?”\u003c/em\u003e\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003e Follow-up probes were incorporated as needed during the interviews to delve further into specific themes emerging from participants’ initial responses. The semi-structured nature of the guide allowed for flexibility and responsiveness during the interviews while preserving a consistent framework across all participants. This design ensured that while core questions were standardized, the conversation could organically explore additional insights relevant to individual experiences. Together, these two instruments provided both quantitative and qualitative datasets that complemented one another one capturing patterns in scoring performance and the other eliciting deeper reflections on user perceptions, experiences, and contextual nuances.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec9\" class=\"Section2\"\u003e\u003ch2\u003eData Analysis\u003c/h2\u003e\u003cp\u003eQuantitative data analysis focused on evaluating the reliability and fairness of the AI scoring system. Each essay was scored independently by two experienced faculty members using the same REFLECT rubric applied by the AI. The agreement between AI and human raters, and between the human raters themselves, was measured using Cohen’s kappa and intraclass correlation coefficients (ICC). These metrics assessed the degree of consistency and alignment across scoring sources. High inter-rater reliability would suggest that the AI system approximates human judgment, while significant discrepancies could indicate limitations in the AI’s interpretive capacity. To assess fairness, the study applied differential item functioning (DIF) analysis using logistic regression to identify whether scores varied significantly across demographic groups such as gender, university, and English language proficiency level. This approach, advocated by [\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e], is effective in detecting systematic bias in test or algorithmic performance. Additionally, multiple regression models were used to determine whether demographic variables predicted deviations between AI and human scores, thereby uncovering patterns of potential inequity. Qualitative data from interviews were analyzed using the thematic analysis approach proposed by Braun and Clarke (2006). Transcripts were first read multiple times to ensure familiarity, after which initial codes were generated. These codes were then clustered into broader themes capturing participants’ experiences and perceptions. Emergent themes included “algorithmic neutrality versus human empathy,” “misalignment of cultural nuance,” and “trust and skepticism in AI-generated feedback.” The analysis was conducted with the assistance of NVivo software to support coding, organization, and theme development. Triangulation of qualitative findings with quantitative results enhanced the validity and interpretive depth of the research.\u003c/p\u003e\u003c/div\u003e\n\u003ch3\u003eEthical Considerations\u003c/h3\u003e\n\u003cp\u003e The research strictly adhered to ethical standards in educational and AI-related research. Ethical clearance was obtained from the Institutional Review Boards (IRBs) of UGMS, KNUST-SMD, and UDS-SMHS before data collection commenced. All participants provided written informed consent and were assured that their participation was voluntary and that they could withdraw at any time without consequence. Reflective essays were anonymized before processing to protect the identity of the students, and interview transcripts were coded with pseudonyms to maintain confidentiality. The AI system used for scoring was designed not to store or retain any personally identifiable information. All data were securely stored on encrypted drives accessible only to the research team. Furthermore, the study adhered to international ethical principles for AI deployment in education, particularly those outlined in the IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e], which emphasize transparency, fairness, and accountability. Participants were also informed about the limitations of AI-generated feedback and were encouraged to interpret it as a complement not a replacement for human judgment. These steps ensured that the research upheld the principles of beneficence, non-maleficence, and respect for persons throughout its duration.\u003c/p\u003e\u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e evaluates the consistency of scores assigned by AI and human raters using multiple agreement metrics: Cohen’s Kappa, Intraclass Correlation Coefficient (ICC), Pearson’s correlation (r), mean difference, and the standard deviation of score differences. The Cohen’s Kappa values indicate substantial agreement across comparisons, ranging from 0.74 to 0.81, with the highest agreement found between Rater 1 and Rater 2 (κ = 0.81). The ICC values follow a similar pattern, showing strong consistency, particularly between human raters (ICC = 0.85). Pearson’s r reveals high positive correlations in all pairwise comparisons, especially between Rater 1 and Rater 2 (r = 0.88). Notably, the AI tends to score slightly higher than the human raters on average, with a mean difference of + 0.12 against Rater 1 and + 0.09 against Rater 2. However, the standard deviation values (0.46 and 0.42) suggest some variability in these differences. The smallest mean difference and variability are observed between the human raters themselves, reinforcing their alignment.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eInter-Rater Agreement Across Scoring Sources\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"6\"\u003e\u003c/colgroup\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003eComparison\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eCohen’s Kappa\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003eICC (2,1)\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c4\"\u003e\u003cp\u003ePearson’s r\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c5\"\u003e\u003cp\u003eMean Difference (AI – Human)\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c6\"\u003e\u003cp\u003eSD of Difference\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eAI vs Rater 1\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.74\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.78\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e0.81\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e+ 0.12\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e\u003cp\u003e0.46\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eAI vs Rater 2\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.76\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.80\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e0.84\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e+ 0.09\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e\u003cp\u003e0.42\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eRater 1 vs Rater 2\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.81\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.85\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e0.88\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e-0.03\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e\u003cp\u003e0.37\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003ctfoot\u003e\u003ctr\u003e\u003ctd colspan=\"6\"\u003e\u003cem\u003eCohen’s Kappa: Substantial (0.74–0.81)\u003c/em\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd colspan=\"6\"\u003e\u003cem\u003eICC: Good to Excellent (0.78–0.85)\u003c/em\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd colspan=\"6\"\u003e\u003cem\u003ePearson’s r: Strong correlation (0.81–0.88)\u003c/em\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd colspan=\"6\"\u003e\u003cem\u003eMean Differences (AI – Human): +0.09 to + 0.12\u003c/em\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd colspan=\"6\"\u003e\u003cem\u003eSD of Differences: 0.37–0.46\u003c/em\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tfoot\u003e\u003c/table\u003e\u003c/div\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e investigates whether discrepancies between AI and human scoring are influenced by demographic characteristics such as gender, university affiliation, and English proficiency. Males are significantly more likely than females to have higher AI scores relative to human scores, with a log-odds of 0.22 (p = 0.031), translating to an odds ratio of 1.24. Although females show a negative discrepancy (-0.15), it is not statistically significant (p = 0.067). Among university affiliations, students from UGMS exhibit a modest but significant positive discrepancy (log-odds = 0.10, p = 0.049), indicating slightly higher AI scores compared to human evaluations. In contrast, KNUST students show a borderline negative discrepancy (log-odds = -0.12, p = 0.081), and the discrepancy for UDS students is negligible and not statistically significant. The most pronounced effects are observed in relation to English proficiency. Those with low proficiency have significantly higher AI-human discrepancies (log-odds = 0.45, p = 0.001, OR = 1.57), while those with high proficiency tend to have lower AI scores relative to human raters (log-odds = -0.30, p = 0.014, OR = 0.74). Adjusted R² values suggest modest explanatory power for these models, with English proficiency contributing the most (Adjusted R² = 0.15).\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eDemographic Group Discrepancy Analysis\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"6\"\u003e\u003c/colgroup\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003eDemographic Group\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eLog-Odds (AI-Human Score Discrepancy)\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003ep-Value\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c4\"\u003e\u003cp\u003e95% Confidence Interval (Log-Odds)\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c5\"\u003e\u003cp\u003eOdds Ratio\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c6\"\u003e\u003cp\u003eAdjusted R²\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eGender (Male)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.22\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.031\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e[0.05, 0.39]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e1.24\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e\u003cp\u003e0.10\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eGender (Female)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e-0.15\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.067\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e[-0.32, 0.02]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e0.86\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e\u003cp\u003e0.06\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eUniversity (UGMS)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.10\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.049\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e[0.01, 0.19]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e1.11\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e\u003cp\u003e0.12\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eUniversity (KNUST)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e-0.12\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.081\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e[-0.24, 0.00]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e0.88\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e\u003cp\u003e0.07\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eUniversity (UDS)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.05\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.102\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e[-0.03, 0.13]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e1.05\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e\u003cp\u003e0.05\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eEnglish Proficiency (High)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e-0.30\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.014\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e[-0.52, -0.08]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e0.74\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e\u003cp\u003e0.09\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eEnglish Proficiency (Low)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.45\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.001\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e[0.28, 0.62]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e1.57\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e\u003cp\u003e0.15\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003ctfoot\u003e\u003ctr\u003e\u003ctd colspan=\"6\"\u003e\u003cem\u003eSignificant p-values: Male (0.031), UGMS (0.049), High Proficiency (0.014), Low Proficiency (0.001; Odds Ratios range: 0.74–1.57; Log-Odds range: − 0.30 to + 0.45; Adjusted R² range: 0.05–0.15\u003c/em\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tfoot\u003e\u003c/table\u003e\u003c/div\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e presents a multiple regression model assessing the combined effect of key predictors on the discrepancy between AI and human scores. Gender remains a significant predictor, with males having higher AI scores than human raters by an average of 0.21 points (p = 0.045). University affiliation is a borderline predictor: KNUST students have marginally lower discrepancies compared to the reference category, though this effect is only marginally significant (β = -0.18, p = 0.054). UDS affiliation is not a significant predictor (p = 0.174). The strongest predictor in the model is low English proficiency, which significantly increases the AI-human discrepancy by 0.38 points (p = 0.002), reinforcing findings from Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e. The overall adjusted R² values range from 0.05 to 0.15, indicating that while demographic variables, especially English proficiency, are meaningful predictors, a substantial portion of variance in AI-human scoring discrepancy remains unexplained, possibly attributable to individual-level factors or nuances in language and response quality.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eMultiple Regression Predicting AI-Human Score Discrepancy\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"7\"\u003e\u003c/colgroup\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003ePredictor\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eCoefficient (β)\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003ep-Value\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c4\"\u003e\u003cp\u003eStandard Error (SE)\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c5\"\u003e\u003cp\u003e95% Confidence Interval (β)\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c6\"\u003e\u003cp\u003eOdds Ratio (OR)\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c7\"\u003e\u003cp\u003eAdjusted R²\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eGender (Male)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.21\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.045\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e0.11\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e[0.02, 0.40]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e1.23\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c7\"\u003e\u003cp\u003e0.12\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eUniversity (KNUST)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e-0.18\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.054\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e0.10\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e[-0.37, 0.01]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e0.83\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c7\"\u003e\u003cp\u003e0.08\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eUniversity (UDS)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.07\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.174\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e0.09\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e[-0.11, 0.25]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e1.08\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c7\"\u003e\u003cp\u003e0.05\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eEnglish Proficiency (Low)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.38\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.002\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e0.12\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e[0.14, 0.62]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e1.46\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c7\"\u003e\u003cp\u003e0.15\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eConstant\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e2.45\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.000\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e0.35\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e\u003cp\u003e[1.76, 3.14]\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c6\"\u003e\u003cp\u003e-\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c7\"\u003e\u003cp\u003e-\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003ctfoot\u003e\u003ctr\u003e\u003ctd colspan=\"7\"\u003e\u003cem\u003eSignificant p-values: Male (0.045), Low Proficiency (0.002), Coefficients (β) range: − 0.18 to + 0.38; Odds Ratios: 0.83–1.46; Adjusted R²: 0.05–0.15; Constant: β = 2.45, p = 0.000\u003c/em\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tfoot\u003e\u003c/table\u003e\u003c/div\u003e\u003cp\u003e\u003c/p\u003e"},{"header":"Qualitative Results","content":"\u003cp\u003eIn-depth thematic analysis of the interview transcripts uncovered three core themes reflecting participants’ experiences and perceptions regarding AI-generated feedback on reflective writing. These themes Algorithmic Neutrality Versus Human Empathy, Misalignment with Cultural and Linguistic Nuance, and Trust and Skepticism in AI Feedback offer rich insight into the strengths, limitations, and socio-cultural implications of integrating artificial intelligence into assessment within Ghanaian medical education. The voices of students and faculty across three universities (UGMS, KNUST, and UDS) provided a textured understanding of how AI impacts perceived fairness, usefulness, and legitimacy in reflective assessment.\u003c/p\u003e\u003ch2\u003eTheme 1: Algorithmic Neutrality Versus Human Empathy\u003c/h2\u003e\u003cp\u003eMany students and faculty members recognized AI’s technical precision but critiqued its inability to engage with the emotional depth and subjective nuances often present in reflective writing. The algorithm’s emphasis on structure, grammar, and logic was perceived as helpful in surface-level editing but insufficient in interpreting personal, vulnerable narratives. As one third-year UGMS student reflected, \u003cem\u003e“The AI pointed out structural issues, but it didn’t recognize the personal pain I tried to express. It felt like my story was reduced to grammar.”\u003c/em\u003e This sentiment captures a widespread frustration among students who felt that their emotional authenticity and narrative voice were not being honored. Faculty members reinforced this concern. A clinical educator at KNUST shared, \u003cem\u003e“Empathy is part of our curriculum how can we ask students to write from the heart and then have a machine that has none evaluate it?”\u003c/em\u003e For educators, this misalignment between pedagogical goals and automated evaluation was not merely technical it was philosophical. Reflective writing, especially in medicine, is designed to cultivate professional identity, empathy, and self-awareness. Participants feared that a system lacking human emotion may inadvertently discourage vulnerability and introspection. This theme reveals a fundamental limitation in current AI systems: their inability to interpret affective meaning and emotional subtext. While algorithmic neutrality can minimize human biases and increase consistency, it may also strip reflective assessments of their humanistic core. Therefore, AI should be positioned as a supplemental tool in formative feedback rather than as a sole evaluator, especially in emotionally sensitive contexts like medical reflection.\u003c/p\u003e\u003ch2\u003eTheme 2: Misalignment with Cultural and Linguistic Nuance\u003c/h2\u003e\u003cp\u003eA second theme emphasized the disconnect between AI-generated feedback and local cultural or linguistic expressions. Reflective writing often contains indigenous proverbs, metaphors, and idioms deeply rooted in Ghanaian culture. However, participants reported that these stylistic choices were frequently flagged by the AI as unclear or irrelevant. A UDS student explained, \u003cem\u003e“I used a Dagbani proverb to explain a moral dilemma, and the AI said it was off-topic. It didn’t get the meaning at all.”\u003c/em\u003e This experience was not isolated. Students from different ethnic backgrounds shared that the AI lacked the contextual awareness needed to interpret figurative language that carries significant meaning in local discourse. Faculty echoed these concerns, noting that the AI’s scoring algorithm appeared to rely heavily on Westernized English norms and cultural assumptions. As one senior lecturer at UGMS stated, \u003cem\u003e“AI models are trained mostly on Western datasets. That limits their ability to fairly assess African ways of expression.”\u003c/em\u003e This theme raises serious concerns about cultural equity in automated assessment. AI models that are not trained on linguistically and culturally diverse datasets risk marginalizing voices that deviate from dominant norms. The misinterpretation or penalization of culturally specific expressions can lead to feelings of alienation, reduced self-expression, and unjust evaluation. To address this, developers should prioritize decolonizing AI training data and ensuring inclusive language modeling that reflects the diversity of student populations.\u003c/p\u003e\u003ch2\u003eTheme 3: Trust and Skepticism in AI-Generated Feedback\u003c/h2\u003e\u003cp\u003ePerceptions of trust in AI feedback varied widely among participants. On one hand, some students praised the immediacy, clarity, and specificity of the feedback they received. A UGMS student noted, \u003cem\u003e“It gave me specific points to improve. With humans, I sometimes just get vague comments like ‘good effort.’”\u003c/em\u003e This view reflects the value of AI in generating fast, detailed feedback that supports revision and learning. However, skepticism was just as prevalent. Many students questioned the transparency and logic behind the AI’s scoring. One final-year student from KNUST expressed frustration: \u003cem\u003e“I spent hours on my essay, and the AI gave me a lower score than my classmate who admitted to using ChatGPT.”\u003c/em\u003e This perceived misalignment between effort and reward led some students to question the credibility of the tool. Faculty members also expressed ambivalence. While they acknowledged AI’s usefulness in standardizing feedback, they warned against its overuse in summative evaluation. A reflective writing instructor at UDS remarked, \u003cem\u003e“I’m fine with using AI as a draft scorer, but final decisions need human judgment. We can’t outsource ethics and empathy.”\u003c/em\u003e The theme of trust reflects broader tensions around AI transparency, explainability, and fairness. The perception that AI is a “black box” whose decisions are unexplainable undermines its legitimacy in the eyes of users. Furthermore, the inconsistency between effort, content quality, and scores can diminish student motivation and confidence. These findings suggest that AI should be deployed in low-stakes environments, accompanied by clear rubrics and human oversight. Building trust also requires demystifying how AI systems work and involving users in the feedback loop.\u003c/p\u003e"},{"header":"Discussion of Findings","content":"\u003cp\u003eThe study investigated the validity, fairness, and contextual sensitivity of AI-generated assessments of reflective writing among medical students in Ghana. The mixed-methods approach provided a nuanced understanding, with quantitative metrics offering statistical insights into reliability and bias, while qualitative data illuminated subjective experiences and cultural interpretations of AI feedback. The inter-rater agreement results (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e) demonstrated moderately strong alignment between AI-generated scores and those of human raters. Specifically, the Cohen’s Kappa values between AI and human raters ranged from 0.74 to 0.76, while the intraclass correlation coefficients (ICC) were between 0.78 and 0.80. The Pearson’s r correlation values exceeded 0.80 in both cases, suggesting high consistency. The relatively small mean differences (+ 0.09 to + 0.12) and standard deviations (\u0026lt; 0.50) further support that AI scoring closely approximates human judgment on structural and technical aspects of reflective writing. These results align with findings from [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e], who noted that AI scoring models trained on human-annotated corpora can reach levels of inter-rater reliability comparable to expert graders, particularly when using structured rubrics such as REFLECT. However, slight discrepancies indicate that the AI may emphasize surface features grammar, structure, and coherence over deeper narrative and emotional resonance, a pattern also identified by [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e] in their critique of AI-based scoring tools in professional education.\u003c/p\u003e\u003cp\u003eThe differential item functioning (DIF) analysis (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e) and multiple regression (Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e) highlighted important equity-related concerns. Statistically significant discrepancies emerged based on gender, university affiliation, and especially English proficiency. Students with lower English proficiency received disproportionately divergent scores between AI and human raters, with a log-odds of 0.45 (p = 0.001) and a regression coefficient of 0.38 (p = 0.002). These findings suggest that the AI system was more likely to penalize such students, possibly due to linguistic surface errors that human raters might have discounted in favor of depth of reflection or narrative intent. These finding echoes research by [14; 11], who observed that AI models often reflect the linguistic biases embedded in their training data, favoring standard English over local dialects or non-native structures. The implications are particularly salient in multilingual educational contexts like Ghana, where code-switching and hybrid linguistic patterns are common. Gender-based differences, although modest, also suggest that male students experienced slightly more favorable alignment between AI and human scores. This could stem from differences in writing styles or content expression that inadvertently align more with the AI’s training data an issue also reported by [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e] in their cross-cultural analysis of gender effects in automated writing assessment.\u003c/p\u003e\u003cp\u003eThe qualitative findings reinforced and contextualized the quantitative results, revealing deeper concerns about AI’s interpretive limitations in cross-cultural settings. The theme of algorithmic neutrality versus human empathy emphasized the AI’s inability to detect emotional undertones or contextual depth. While this did not dramatically affect score correlations, it affected the \u003cem\u003eperceived fairness\u003c/em\u003e and \u003cem\u003epedagogical value\u003c/em\u003e of the feedback. As shown in earlier studies [e.g., 16; 11], such emotional and narrative components are crucial in reflective writing and medical education, where empathy is not just a theme but a skill to be cultivated. The theme of misalignment with cultural and linguistic nuance highlighted how AI systems trained on Western-centric corpora struggle to interpret culturally embedded metaphors, idioms, or context-specific references. Such misinterpretations can lead to mis-scoring and reduce student confidence in the feedback process. This supports the critique by [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e], who warns that AI applications in education risk epistemic injustice when local knowledge forms are excluded from algorithmic design. Finally, the theme of trust and skepticism revealed a split in user perceptions: some students valued the clarity and speed of AI feedback, while others doubted its validity, especially when scores conflicted with perceived effort. Faculty shared concerns about over-reliance on AI in high-stakes contexts, emphasizing the need for human oversight.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eThis study critically examined the performance, fairness, and cultural sensitivity of AI-generated assessments of reflective writing among medical students in Ghana. Drawing on both statistical measures and narrative accounts, the findings point to a nuanced picture: AI-based assessment tools show promising levels of consistency and inter-rater agreement with human raters, especially when evaluating structural and linguistic features. However, significant concerns persist around equity and contextual validity. Quantitative analyses revealed that demographic factors particularly English language proficiency and, to a lesser extent, gender and institutional affiliation were associated with discrepancies between AI and human scoring. These differences suggest that while the AI system may be technically proficient, it lacks sufficient adaptation to the diverse linguistic and educational backgrounds typical of Ghanaian learners. Qualitative insights added depth to these findings by revealing participants\u0026rsquo; lived experiences of AI feedback. Students and faculty alike appreciated the speed and objectivity of AI-generated evaluations but raised concerns about the absence of empathy, cultural nuance, and contextual understanding. These limitations were particularly salient in reflective writing, a genre that demands emotional expression, personal insight, and cultural relevance dimensions that AI tools, in their current form, struggle to assess meaningfully. Taken together, the study highlights the need for caution and balance in deploying AI tools in educational assessment, especially in high-stakes or formative domains like medical education. AI should be viewed not as a replacement for human judgment, but as a complementary tool that can enhance assessment when properly localized, validated, and monitored.\u003c/p\u003e\u003cdiv id=\"Sec17\" class=\"Section2\"\u003e\u003ch2\u003eRecommendations\u003c/h2\u003e\u003cp\u003eTo ensure equitable, culturally sensitive, and pedagogically sound use of AI-generated assessments in reflective academic writing, especially within contexts like Ghanaian medical education, several strategic actions are recommended based on the study\u0026rsquo;s findings.\u003c/p\u003e\u003cp\u003eFirst, there is a critical need for contextual adaptation of AI assessment models. The quantitative results demonstrated that discrepancies in AI scoring were influenced by factors such as English proficiency and institutional affiliation, indicating that the AI system may not be fully attuned to the linguistic and educational diversity of students in the local context. Moreover, qualitative data underscored participants\u0026rsquo; concerns about the AI\u0026rsquo;s inability to recognize culturally embedded metaphors or rhetorical forms. To address this, AI models should be fine-tuned using locally representative datasets that capture the full range of linguistic expressions, academic conventions, and cultural references found in Ghanaian student writing. Such adaptation would enhance the AI\u0026rsquo;s ability to evaluate texts more fairly and reduce misinterpretation of culturally specific language.\u003c/p\u003e\u003cp\u003eSecond, the study recommends the adoption of hybrid assessment models that integrate both AI-generated and human scoring. While the AI demonstrated consistency in identifying structural issues and surface-level features, both students and faculty highlighted its shortcomings in interpreting emotional nuance, tone, and reflective depth. A hybrid system allows institutions to leverage the speed, objectivity, and scalability of AI, while preserving the human ability to engage with context, empathy, and meaning-making. This combination is particularly important in the assessment of reflective writing, where subjective interpretation plays a crucial role in evaluating the depth and quality of student learning.\u003c/p\u003e\u003cp\u003e Third, there is a pressing need for faculty development focused on the ethical and critical use of AI tools. As AI technologies become more integrated into educational environments, educators must be trained not only to interpret AI outputs but also to recognize their limitations. Without proper training, there is a risk of over-reliance on AI-generated scores, potentially undermining academic judgment and student trust. Professional development workshops, seminars, and certification programs should be instituted to empower educators with the skills to balance technological efficiency with pedagogical integrity. Finally, institutions should establish mechanisms for ongoing bias audits and fairness evaluations. The study\u0026rsquo;s logistic and regression analyses revealed group-based disparities in AI scoring accuracy, particularly disadvantaging students with lower English proficiency. This underscores the risk of systemic inequity if AI systems are deployed without oversight. Regular audits should be conducted to monitor the performance of AI scoring tools across different demographic groups, including gender, institutional background, and language proficiency. These audits should inform continuous refinement of the AI system and guide policies for its responsible implementation.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec18\" class=\"Section2\"\u003e\u003ch2\u003eLimitations of the study\u003c/h2\u003e\u003cp\u003eThis study has several limitations that should be considered when interpreting the results. First, the research was geographically restricted to three Ghanaian medical schools, which limits the generalizability of the findings to other regions or educational contexts. The study\u0026rsquo;s focus on reflective writing also narrows the scope, as AI\u0026rsquo;s performance in assessing other types of academic writing may differ. Furthermore, the small sample size for the qualitative data, based on a limited number of student and faculty interviews, may not fully represent the diverse experiences and perspectives within the broader population. Additionally, there may have been biases in human scoring, as raters brought subjective judgments to the assessment, which could have affected the inter-rater reliability. The study also did not explore the long-term educational impacts of AI-generated feedback on student learning outcomes, leaving a gap in understanding its effects on student engagement, academic performance, or development over time. Despite these limitations, the findings provide valuable insights into AI\u0026rsquo;s potential and challenges in educational assessment, paving the way for future research in this area.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec19\" class=\"Section2\"\u003e\u003ch2\u003eImplications for Practice and Policy\u003c/h2\u003e\u003cp\u003eThe findings of this study carry significant implications for both practice and policy in the realm of AI-powered assessment in educational contexts, particularly in settings as culturally and linguistically diverse as Ghana. While AI presents promising advantages in terms of efficiency and consistency in assessing reflective writing, the results highlight that it cannot fully replace human judgment, especially when it comes to capturing emotional nuance and cultural sensitivity. In this context, AI can serve as a useful tool for providing preliminary feedback, yet its limitations in understanding the complexities of personal experiences and cultural expressions underscore the necessity of human input in the assessment process. From a practical standpoint, AI feedback can be employed as an initial review tool to guide students\u0026rsquo; self-reflection and improvement. However, it is critical that such feedback be supplemented with human critique to ensure that the emotional depth and contextual relevance of the student\u0026rsquo;s work are accurately captured. This also raises the need for institutions to regularly evaluate and audit AI systems for fairness, particularly in multicultural and multilingual environments, to ensure that bias is minimized. Furthermore, there is a pressing need for faculty to be trained in interpreting AI-generated feedback critically, so they can integrate it effectively into broader assessment strategies. Lastly, to address algorithmic biases, educational institutions should focus on developing locally adapted AI training corpora that reflect Ghanaian linguistic, cultural, and academic norms. This would help ensure that AI systems are more equitable and sensitive to the unique context in which they are applied.\u003c/p\u003e\u003c/div\u003e"},{"header":"Abbreviations","content":"\u003cp\u003e\u003cstrong\u003eIRB\u003c/strong\u003e \u0026ndash; Institutional Review Board; \u003cstrong\u003eUEW\u003c/strong\u003e \u0026ndash; University of Education, Winneba; \u003cstrong\u003eAI\u003c/strong\u003e \u0026ndash; Artificial Intelligence; \u003cstrong\u003eNLP\u003c/strong\u003e \u0026ndash; Natural Language Processing; \u003cstrong\u003eSPSS\u003c/strong\u003e \u0026ndash; Statistical Package for the Social Sciences; \u003cstrong\u003eREFLECT\u003c/strong\u003e \u0026ndash; Reflection Evaluation for Learners\u0026rsquo; Enhanced Competencies Tool; \u003cstrong\u003eHICs\u003c/strong\u003e-High-Income Countries, \u003cstrong\u003eLMICs\u003c/strong\u003e-Low- And Middle-Income Countries; \u003cstrong\u003eGPT-4\u003c/strong\u003e \u0026ndash; Generative Pre-trained Transformer 4, \u003cstrong\u003eKNUST-SMD\u003c/strong\u003e \u0026ndash; Kwame Nkrumah University of Science and Technology School of Medicine and Dentistry,\u0026nbsp;\u003cstrong\u003eNVivo\u003c/strong\u003e \u0026ndash; Qualitative Data Analysis Software, \u003cstrong\u003eUDS-SMHS\u003c/strong\u003e \u0026ndash; University for Development Studies School of Medicine and Health Sciences, \u003cstrong\u003eUGMS\u003c/strong\u003e \u0026ndash; University of Ghana Medical School.\u0026nbsp;\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eEthics Approval and Consent to Participate\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eEthical approval for this study was granted by the Institutional Review Board (IRB) of the University of Education, Winneba (UEW), Ghana. Informed consent was obtained from all participants prior to data collection. Participants were fully briefed on the purpose, procedures, and voluntary nature of the study, including their right to withdraw at any time without consequence. All ethical protocols were strictly adhered to, particularly concerning anonymity, data security, and responsible use of AI-generated feedback in educational settings.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent for Publication\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAll authors have reviewed and approved the final version of the manuscript and consent to its submission for publication. Participants were informed that anonymized data would be used strictly for academic and research purposes, including possible publication in peer-reviewed journals. Informed consent for publication was obtained from all participants.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis research was entirely self-funded by the authors. No external funding or institutional sponsorship was received, ensuring the independence and impartiality of the research design, analysis, and interpretation.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eClinical Trial Number\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDeclaration of Conflicts of Interest\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors declare no conflicts of interest. This research was independently conceptualized, designed, and conducted by the authors without any external influence on its methodology, analysis, or interpretation.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAvailability of Data and Materials\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe datasets generated and analyzed during this study on the validity, fairness, and contextual sensitivity of AI-generated assessments of reflective writing among medical students in Ghana are available upon reasonable request from the corresponding author, Simon Ntumi. Due to ethical considerations and to protect the confidentiality of student and faculty participants, raw data including reflective writing samples, rater scores, and qualitative transcripts will not be made publicly accessible. All data requests will be reviewed in accordance with institutional ethical guidelines to ensure compliance with data protection protocols and participant anonymity.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgments\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors gratefully acknowledge the medical students, faculty assessors, and institutional partners in Ghana who made this study possible. Special thanks go to the departments of medical education, assessment units, and ethics committees at participating universities for their cooperation and support. The authors also appreciate the technical assistance provided by local AI specialists and the peer reviewers whose feedback enhanced the rigor and clarity of this work.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthors\u0026rsquo; Contributions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSimon Ntumi\u003c/strong\u003e (Corresponding Author) led the conceptualization and design of the study, developed the research methodology, and oversaw all aspects of data collection from the participating medical schools. He conducted both the quantitative and qualitative analyses, including inter-rater reliability and differential item functioning tests, as well as thematic coding and synthesis. He was primarily responsible for drafting the original manuscript, integrating co-authors\u0026rsquo; contributions, and coordinating the supervision of the project\u0026rsquo;s execution across institutions.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eIsaac Yabana\u003c/strong\u003e contributed extensively to the development of the literature review, ensuring alignment with current discourses on AI in medical education and assessment fairness. He assisted in interpreting both quantitative and qualitative data and played a critical role in reviewing and editing the manuscript for intellectual content and clarity. He also validated the findings by triangulating interpretations across multiple data sources.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDaniel William Essel\u003c/strong\u003e conducted advanced statistical analyses, including interclass correlation coefficients and graphical visualizations of the AI-human rating agreement. He interpreted the results in collaboration with the lead author and co-authors, contributed to the preparation of tables and figures, and revised the manuscript during the review and editing phase with a focus on methodological rigor and reproducibility.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEdmond Ahovi\u003c/strong\u003e was responsible for coding and analyzing the qualitative data, particularly in organizing and refining emergent themes from student narratives and focus group discussions. He ensured that ethical standards\u0026mdash;including participant confidentiality and informed consent\u0026mdash;were upheld throughout the study. He also contributed to reviewing and editing the manuscript, ensuring its consistency and alignment with ethical and qualitative reporting standards.\u003c/p\u003e\n\u003cp\u003eAll authors reviewed and approved the final manuscript and agree to be accountable for all aspects of the work.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eAntwi-Baffour, S., Agyemang, C., \u0026amp; Arhin, S. (2021). Exploring the contextual factors influencing educational outcomes in Ghana: A cross-institutional study. \u003cem\u003eJournal of Educational Administration, 59\u003c/em\u003e(4), 445-463.\u003c/li\u003e\n\u003cli\u003eAntwi-Baffour, S., Asare, B., \u0026amp; Danquah, S. (2021). Challenges in the adoption of technology in Ghanaian medical schools: Implications for medical education. \u003cem\u003eGhanaian Journal of Medical Education, 12\u003c/em\u003e(1), 45-58.\u003c/li\u003e\n\u003cli\u003eAsare, B., \u0026amp; Danquah, S. (2020). Medical education in Ghana: Current state and challenges. \u003cem\u003eJournal of African Medical Education, 8\u003c/em\u003e(2), 102-110.\u003c/li\u003e\n\u003cli\u003eBender, E. M., Gebru, T., McMillan-Major, A., \u0026amp; Shmit, S. (2021). \u003cem\u003eOn the dangers of stochastic parrots: Can language models be too big?\u003c/em\u003e Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 1-11.\u003c/li\u003e\n\u003cli\u003eBraun, V., \u0026amp; Clarke, V. (2006). Using thematic analysis in psychology. \u003cem\u003eQualitative Research in Psychology, 3\u003c/em\u003e(2), 77-101.\u003c/li\u003e\n\u003cli\u003eCreswell, J. W., \u0026amp; Plano Clark, V. L. (2017). \u003cem\u003eDesigning and conducting mixed methods research\u003c/em\u003e (3rd ed.). Sage Publications.\u003c/li\u003e\n\u003cli\u003eFloridi, L., Taddeo, M., \u0026amp; Turilli, M. (2022). AI in education: Challenges and opportunities. \u003cem\u003ePhilosophy \u0026amp; Technology, 35\u003c/em\u003e(1), 1-20.\u003c/li\u003e\n\u003cli\u003eFloridi, L., Taddeo, M., \u0026amp; Turilli, M. (2023). \u003cem\u003eThe ethics of artificial intelligence and the moral implications of AI in education\u003c/em\u003e. Cambridge University Press.\u003c/li\u003e\n\u003cli\u003eFrenk, J., Chen, L., Bhutta, Z. A., \u0026amp; et al. (2010). Health professionals for a new century: Transforming education to strengthen health systems in an interdependent world. \u003cem\u003eThe Lancet, 376\u003c/em\u003e(9756), 1923-1958.\u003c/li\u003e\n\u003cli\u003eGuest, G., Bunce, A., \u0026amp; Johnson, L. (2006). How many interviews are enough? An experiment with data saturation and variability. \u003cem\u003eField Methods, 18\u003c/em\u003e(1), 59-82.\u003c/li\u003e\n\u003cli\u003eHutt, S., DePiro, A., Wang, J., Rhodes, S., Baker, R. S., Hieb, G., ... \u0026amp; Mills, C. (2024, March). Feedback on feedback: comparing classic natural language processing and generative AI to evaluate peer feedback. In \u003cem\u003eProceedings of the 14th Learning Analytics and Knowledge Conference\u003c/em\u003e (pp. 55-65).\u003c/li\u003e\n\u003cli\u003eIEEE Global Initiative on Ethics of Autonomous and Intelligent Systems. (2019). \u003cem\u003eEthically aligned design: A vision for prioritizing human well-being with autonomous and intelligent systems\u003c/em\u003e. IEEE.\u003c/li\u003e\n\u003cli\u003eKumar, A., Sahu, R., \u0026amp; Sharma, D. (2023). Artificial intelligence in medical education: Opportunities and challenges in automated assessment. \u003cem\u003eJournal of Medical Education and Practice, 34\u003c/em\u003e(3), 24-31.\u003c/li\u003e\n\u003cli\u003eKumar, V., \u0026amp; Rose, C. (2021). Linguistic biases in AI assessment: A study on non-native English speakers in higher education. \u003cem\u003eJournal of Educational Technology, 38\u003c/em\u003e(4), 45-60.\u003c/li\u003e\n\u003cli\u003eLincoln, Y. S., \u0026amp; Guba, E. G. (1985). \u003cem\u003eNaturalistic inquiry\u003c/em\u003e. Sage Publications.\u003c/li\u003e\n\u003cli\u003eLiu, L., Xu, J., \u0026amp; Zhang, T. (2021). Emotional intelligence in medical education: The role of reflective writing in empathy development. \u003cem\u003eMedical Education, 55\u003c/em\u003e(7), 775-785.\u003c/li\u003e\n\u003cli\u003eLuxton-Reilly, A., Harland, J., \u0026amp; McIntyre, T. (2020). Assessing the impact of artificial intelligence on educational assessment systems. \u003cem\u003eJournal of Educational Technology \u0026amp; Society, 23\u003c/em\u003e(4), 89-101.\u003c/li\u003e\n\u003cli\u003eMann, K., Gordon, J., \u0026amp; MacLeod, A. (2019). Reflection and reflective practice in health professions education: A systematic review. \u003cem\u003eAdvances in Health Sciences Education, 14\u003c/em\u003e(4), 595-621.\u003c/li\u003e\n\u003cli\u003eMensah, P., Boateng, K., \u0026amp; Akrofi, M. (2022). Challenges of reflective writing assessment in Ghanaian medical schools. \u003cem\u003eInternational Journal of Medical Education, 13\u003c/em\u003e(2), 45-59.\u003c/li\u003e\n\u003cli\u003eOmaswa, F., \u0026amp; Crisp, N. (2014). The African health workforce: Challenges and opportunities. \u003cem\u003eThe Lancet, 384\u003c/em\u003e(9943), 1473-1479.\u003c/li\u003e\n\u003cli\u003eRyan, M. (2013). Reflections on reflection: Educational implications for reflective practice in medical education. \u003cem\u003eMedical Education, 47\u003c/em\u003e(9), 877-885.\u003c/li\u003e\n\u003cli\u003eSandars, J. (2019). The use of reflection in medical education: A systematic review of the literature. \u003cem\u003eMedical Teacher, 31\u003c/em\u003e(8), 710-717.\u003c/li\u003e\n\u003cli\u003eTashakkori, A., \u0026amp; Teddlie, C. (2010). \u003cem\u003eMixed methods in social \u0026amp; behavioral research\u003c/em\u003e (2nd ed.). Sage Publications.\u003c/li\u003e\n\u003cli\u003eUdupa, S. (2020). Algorithmic justice: The risks of epistemic injustice in AI-based educational applications. \u003cem\u003eEducation and Technology, 21\u003c/em\u003e(2), 215-230.\u003c/li\u003e\n\u003cli\u003eWald, H. S., Borkan, J. M., Taylor, J. S., \u0026amp; Anthony, D. (2012). The reflective practitioner: The role of reflection in medical education. \u003cem\u003eMedical Education, 46\u003c/em\u003e(6), 580-588.\u003c/li\u003e\n\u003cli\u003eWang, Q., Li, J., \u0026amp; Zhang, Y. (2023). Gender differences in automated writing assessments: A cross-cultural analysis of AI bias in academic evaluations. \u003cem\u003eEducational Assessment, 19\u003c/em\u003e(2), 121-135.\u003c/li\u003e\n\u003cli\u003eWilliamson, B., \u0026amp; Kukulska-Hulme, A. (2020).\u003cem\u003e \u003c/em\u003eAI-based writing assessment tools: A critique of their use in professional education.\u003cem\u003e Journal of Educational Measurement, 48\u003c/em\u003e(3), 285-300.\u003c/li\u003e\n\u003cli\u003eWilliamson, B., \u0026amp; Piattoeva, N. (2021). \u003cem\u003eThe global education race: Taking the measure of the world\u0026rsquo;s education systems\u003c/em\u003e. Routledge.\u003c/li\u003e\n\u003cli\u003eWorld Bank. (2021). \u003cem\u003eEducation in Africa: Challenges and opportunities in low and middle-income countries\u003c/em\u003e. World Bank Group.\u003c/li\u003e\n\u003cli\u003eZhang, Y., Chen, L., \u0026amp; Li, J. (2022). Exploring the potential of artificial intelligence in medical education assessment: A review of literature. \u003cem\u003eMedical Education, 56\u003c/em\u003e(6), 1223-1232.\u003c/li\u003e\n\u003cli\u003eZhang, Y., Wei, X., \u0026amp; Yao, Z. (2022). AI-based scoring models: Achieving inter-rater reliability with human-annotated corpora. \u003cem\u003eJournal of Educational Measurement, 49\u003c/em\u003e(1), 112-125.\u003c/li\u003e\n\u003cli\u003eZumbo, B. D. (2007). Introduction to differential item functioning: A guide for the novice. \u003cem\u003ePractical Assessment, Research, and Evaluation, 12\u003c/em\u003e(1), 1-15.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"AI-generated feedback, reflective writing, medical education, fairness, cultural alignment, bias analysis","lastPublishedDoi":"10.21203/rs.3.rs-6636682/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6636682/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cem\u003eThe study explored the application of Natural Language Processing (NLP) and generative AI tools in assessing reflective writing submitted by medical students in Ghana. It evaluated the validity, fairness, and cultural alignment of AI-generated feedback by comparing AI-generated scores with human rater assessments and analyzing demographic group discrepancies. A total of 180 reflective essays were sampled, with an equal number (n\u0026thinsp;=\u0026thinsp;60) collected from each university\u003c/em\u003e. \u003cem\u003eQuantitative methods included Cohen\u0026rsquo;s Kappa and Intraclass Correlation Coefficients (ICC) to assess inter-rater agreement, while logistic regression and multiple regression models examined potential biases across gender, university affiliation, and English proficiency. Qualitative data were gathered through interviews with students and faculty to explore perceptions of fairness, trust, and the AI\u0026rsquo;s capacity to capture cultural and linguistic nuances. Results indicated that the AI system demonstrated strong inter-rater reliability, with Cohen\u0026rsquo;s Kappa values of 0.74 (AI vs Rater 1) and 0.76 (AI vs Rater 2), and ICC values of 0.78 and 0.80, respectively. Human raters showed higher agreement with each other (Cohen\u0026rsquo;s Kappa\u0026thinsp;=\u0026thinsp;0.81, ICC\u0026thinsp;=\u0026thinsp;0.85). However, significant discrepancies were found across demographic groups, particularly for English proficiency, where lower proficiency students tended to receive higher AI scores than human raters (log-odds of 0.45, p\u0026thinsp;=\u0026thinsp;0.001). Thematic analysis of qualitative interviews revealed concerns over the lack of empathy in AI feedback, misalignment with cultural and linguistic nuances, and mixed levels of trust in AI-generated assessments. These findings suggest that while AI holds promise for improving efficiency in assessment, careful attention must be given to its limitations in fairness and cultural sensitivity. The study concluded with recommendations for improving AI systems through contextual adaptation, hybrid assessment models, faculty training, and regular bias audits to ensure equitable and effective use of AI in educational settings.\u003c/em\u003e\u003c/p\u003e","manuscriptTitle":"Natural Language Processing and Generative AI in the Automated Scoring and Feedback of Reflective Writing in Medical Education: A Validity and Fairness Analysis","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-08-07 17:31:59","doi":"10.21203/rs.3.rs-6636682/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"df558aa1-0baa-4621-999d-f290722dd07f","owner":[],"postedDate":"August 7th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-09-09T11:23:15+00:00","versionOfRecord":[],"versionCreatedAt":"2025-08-07 17:31:59","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6636682","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6636682","identity":"rs-6636682","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}