Evaluating LingualAI: A Prospective Validation of AI-Based Real- Time Translation Against Certified Human Interpreters

Evaluating LingualAI: A Prospective Validation of AI-Based Real- Time Translation Against Certified Human Interpreters | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Evaluating LingualAI: A Prospective Validation of AI-Based Real- Time Translation Against Certified Human Interpreters Uday P. Singh, Carlos A. Jaimes Garcia, Gabriel M. Aisenberg, and 11 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8205991/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 9 You are reading this latest preprint version Abstract Background : Limited English proficiency (LEP) affects >25 million people in the United States and is linked to health disparities in safety, quality, and outcomes. While professional interpreters remain the standard, access is often constrained. Real-time AI translation systems are increasingly available, yet their clinical performance relative to certified interpreters is uncertain. Objective : To evaluate whether an in-house AI application (LingualAI) achieves non-inferior translation quality compared with certified interpreters in English–Spanish otorhinolaryngology encounters. Design, Setting, and Participants : Prospective, within-subject comparison using three standardized outpatient scenarios (33 lines: 18 clinicians, 15 patients) enacted by two pairs of native speakers. Each line was translated by LingualAI and by two certified medical interpreters. Nine bilingual clinicians, blinded to source but given scenario context, independently rated anonymized audio clips. Main Measures : Twelve domains on 5-point Likert scales: primary ( terminology accuracy , adequacy of meaning ), secondary ( completeness , grammar , vocabulary , cultural appropriateness ), voice-related ( fluency , clarity , prosody , pacing ), and conclusive ( overall quality , clinical confidence ). Non-inferiority margin prespecified at 0.30 points (Δ = Human − AI). Analyses used paired tests and mixed-effects models with random intercepts for line; inter-rater reliability via Krippendorff’s α. Results : Across models, LingualAI was non-inferior for adequacy of meaning and terminology accuracy; completeness also met the criterion. Human interpreters scored higher on delivery-related and linguistic-mechanics domains, including clarity/intelligibility (Δ≈0.50), fluency (Δ≈1.1), prosody (Δ≈0.6), pacing (Δ≈0.4), grammar, vocabulary, and cultural appropriateness. Conclusive ratings favored humans for overall quality (Δ≈0.6) and clinical confidence (Δ≈0.6). Findings were consistent in direction-specific contrasts (English→Spanish clinician lines; Spanish→English patient lines). Inter-rater reliability was modest (α=0.31), reflecting first-impression scoring. In exploratory system metrics, mean end-to-end translation latency was ~9.7s with substantially lower estimated per-session costs than phone/video interpreter services. Conclusions : LingualAI preserves core meaning and terminology at near-interpreter levels but lags in speech naturalness and delivery ( fluency , prosody , pacing ), leading to lower overall quality and clinical confidence. AI translation may serve as a useful aid when interpreters are unavailable; however, its use today should remain aligned with professional standards and ideally follow an interpreter-in-the-loop model rather than replacement. Continued refinement of voice and delivery features potentially will improve perceived speech naturalness and delivery and thus, in the long-run applications such as Lingual AI, will more closely approximate the performance of human interpreters on all measures. Technical work, plus clinical validation is necessary for the safe and effective deployment of applications such as Lingual AI in real-world settings (Fig. 1). Health sciences/Health care Health sciences/Medical research Artificial intelligence Medical translation Clinical communication Limited English proficiency Interpreter services Otorhinolaryngology Figures Figure 1 Figure 2 Figure 3 INTRODUCTION Effective communication is fundamental to safe and equitable healthcare. In countries with substantial linguistic diversity, such as the United States, a growing proportion of clinical encounters involve patients with limited English proficiency (LEP), estimated at approximately 25 million people, or about 8% of the population aged 5 years and older who report speaking English less than “very well”. ( 1 , 2 ) LEP patients face barriers to understanding diagnoses, treatment instructions, and follow-up care, contributing to disparities in safety, quality, and health outcomes. ( 3 , 4 ) A recent systematic review found that when LEP patients receive language-concordant care, clinical outcomes are improved in most settings. ( 1 ) Although professional medical interpreters are the standard of care, access is often limited, particularly in primary care, rural settings, and time-sensitive encounters, leaving many patients without reliable language support. ( 5 , 6 ) Recent advances in artificial intelligence (AI) have enabled speech- and language-model–based systems capable of real-time multilingual translation. ( 7 , 8 ) Several mobile or app-based tools are now available at the point of care; yet systematic evidence regarding their performance, accuracy, and clinical appropriateness remains scarce. In particular, little is known about how AI-based translations compare with certified human interpreters when assessed against clinical communication standards such as terminology accuracy, adequacy of meaning, cultural appropriateness, and speech fluency. Notably, in a recent study evaluating three commercially available applications for bidirectional clinician–patient dialogue, none were found to be suitable for safe two-way clinical communication when compared with professional interpreters. ( 9 ) This evidence gap complicates safe integration into health systems. The LingualAI application was created to reduce language barriers in clinical settings through real-time voice translation. Developed at UTHealth Houston, its purpose it help clinical team members and patients communicate seamlessly through a continuous flow of voice capture, transcription, translation, and synthesis. Its interface is designed to be intuitive: users can select patient and clinician languages, record conversations linked to each patient, and view interactive bubbles on screen with real-time transcription and translation. In addition, the app offers translation quality feedback, conversation history, secure device synchronization, and an offline mode that ensures reliability even in low-connectivity environments. However, its effectiveness relative to certified interpreters has not been formally evaluated. We conducted a prospective, within-subject comparative study using scripted bilingual dialogues to evaluate LingualAI’s performance in English–Spanish interactions in an outpatient otorhinolaryngology setting. We assess whether LingualAI’s audio translations are non-inferior to certified human interpreters across multiple domains of translation quality, including terminology accuracy, adequacy of meaning, clarity/fluency, cultural appropriateness, and overall confidence for clinical use. By applying a structured validation framework, this work provides early evidence on the role of AI-based translation tools in supporting equitable, multilingual healthcare delivery. METHODS We conducted a prospective, within-subject comparison of translations generated by the LingualAI application versus certified human interpreters (Fig. 2 ). Two translation directions were evaluated: English → Spanish (clinician utterances) and Spanish → English (patient utterances). Each input phrase was translated by both arms, enabling paired comparisons with evaluators blinded to translation source but not scenario context. Similar prospective, paired or randomized designs have been used in prior translation and interpreter studies that have compared Machine or AI Translation with human translators and bilingual clinical evaluators ( 10 , 11 ). Clinical scenarios and audio collection Three standardized otorhinolaryngology scenarios were developed to represent typical care interactions. Together, the scripts included 18 dialogue paragraphs (33 lines) covering history-taking, clinical instructions, and empathic statements. Each scenario was enacted by two pairs of native English and Spanish speakers to reduce bias from voice quality or pronunciation differences. As a result, each phrase was recorded twice, yielding two independent input sets. Use of standardized scripted encounters with native speakers mirrors prior interpreter research and simulation-based translation evaluations ( 12 , 13 ). The full text of the three standardized scenarios is provided in Supplementary Material SM1. Translation pipelines In the AI arm, recordings were processed through LingualAI, which integrates automatic speech recognition (ASR), neural machine translation (NMT), and text-to-speech synthesis. Each phrase was submitted twice using the two independent source recordings, generating two AI-translated audio files per phrase in each direction. Outputs were pooled and randomized in blocks of 2 using a fixed random seed to ensure reproducibility, and evaluators were blinded to the translation source (AI vs. human). They were, however, provided with scenario context, line order, and speaker role (clinician vs. patient) so that ratings reflected the intended clinical meaning. Modular pipelines combining ASR, neural MT, and synthesis are well described in biomedical translation research ( 7 , 14 , 15 ). LingualAI’s pipeline operates in five real-time stages: (i) voice capture, ii) text conversion via ASR, (iii) refinement using prior dialogue and domain-specific vocabulary, (iv) translation into the target language, and (v) speech synthesis for natural-sounding delivery. Each stage is designed for real-time performance with fallback strategies in case of network interruptions. The most recent update includes support for custom vocabularies, allowing clinicians to register specialized or uncommon medical terms to improve transcription and translation accuracy in clinical contexts. Comparable staged architectures with domain adaptation have been reported for ambient clinical transcription and multilingual MT systems ( 14 ). In the human arm, two certified medical interpreters independently translated the scenarios, producing two human-translated audio files per phrase. Outputs were similarly pooled and randomized to reduce interpreter-specific bias. All translations were recorded under uniform audio conditions. Evaluators Nine fluent bilingual clinicians (English–Spanish) served as evaluators. Participation in the evaluation was voluntary and completed outside protected clinical time. Evaluators submitted ratings anonymously; unique rater identifiers were not collected. Audio clips were randomized and presented blinded to translation source (AI vs. human), but with scenario context provided to support intended meaning. Although blinding was applied, it is possible that evaluators could have inferred AI outputs due to the use of the same synthetic voice and a slightly mechanistic tone compared with human translations. We did not conduct a calibration round, as the study was designed to capture first-impression ratings rather than consensus. All evaluators provided Consent to Participate prior to beginning the study procedures. Blinded bilingual rater designs have been applied in comparable translation and medical communication studies ( 10 , 16 ). Rating instrument Translation quality was assessed across 12 domains: (i) primary: accuracy of meaning, terminology accuracy; (ii) secondary: completeness, cultural appropriateness, grammar, vocabulary; (iii) voice-related: fluency, clarity, prosody, pacing; and (iv) conclusive: overall quality and clinical confidence. Ratings used 5-point Likert scales (1 = poor, 5 = excellent), and evaluators were provided with a guide containing definitions and examples. Evaluators rated each clip independently using Google Forms, a secure, web-based platform containing domain definitions and illustrative examples. Similar multidomain rubrics have been applied in medical machine translation and LLM evaluation studies ( 10 , 11 ). Established frameworks such as the Multidimensional Quality Metrics (MQM) cover accuracy, fluency, terminology, and adequacy, with error taxonomies also informing clarity, grammar, and overall quality ( 17 ). In line with broader recommendations, our rubric emphasized context awareness, distinguishing accuracy from fluency, and incorporating completeness, cultural appropriateness, and clinical relevance into evaluation. ( 18 , 19 ). Additionally, because of this, we did not conduct a calibration round, as the study was designed to capture first-impression ratings rather than consensus. The complete rating instrument and evaluator instructions are provided in Supplementary Material SM2. Statistical analysis Normality of scores was assessed using the Shapiro–Wilk test. For normally distributed outcomes, paired t-tests were applied; for non-normal distributions, Wilcoxon signed-rank tests were used. Effect sizes were reported as Cohen’s d (t-test) or correlation coefficient ( r ) (Wilcoxon). Inter-rater reliability was estimated using Krippendorff’s α, which accommodates missing data and unequal numbers of raters per item, a necessity given that evaluators submitted ratings anonymously without unique identifiers. Mixed-effects linear models with random intercepts for item (line) were applied to account for repeated ratings of the same line across multiple evaluators. A non-inferiority margin of 0.30 points on the 5-point scale was prespecified. This margin was chosen a priori based on three considerations: (i) the half-SD rule commonly applied to Likert ratings, where typical standard deviations of ~ 0.5–0.7 correspond to a minimal meaningful difference of ~ 0.25–0.35; (ii) prior studies that used similar small margins for clinically relevant differences on 5-point rating scales ( 20 ); and (iii) expert consensus among bilingual clinician-evaluators that a 0.30 shift would begin to alter clinical confidence without changing literal meaning. For each domain, one-sided 95% confidence intervals (CIs) were calculated for the mean difference between human and AI ratings. LingualAI was considered non-inferior if the upper bound of the CI was less than the margin. All analyses were conducted in Python (Ver 3.12.5 using Pandas, Numpy, StatsModels and SciPy libraries) and R (Ver 4.4.2 using ggplot2, dplyr, tibble libraries), with statistical significance set at two-sided p < 0.05. This non-inferiority framework is consistent with prior comparative translation studies ( 9 , 20 , 21 ). For a priori power, we planned for 33 lines, each rated by ~ 7 bilingual clinicians (≈ 231 ratings). Because ratings on the same line are correlated, we adjusted for clustering using an assumed intraclass correlation (ICC) of 0.25, giving an effective sample size of \(\:{n}_{eff}=\:\frac{33\times\:7}{1+(7-1)\times\:0.25}\:\approx\:92\) . With a one-sided non-inferiority test (α = 0.025), a prespecified margin of Δ = 0.30, and an expected true mean difference of 0.10, the projected power was ~89% (Supplementary Material SM3). Sensitivity checks varying ICC (0.20–0.30) and score variability (SD 0.60–0.70) yielded power between ~74% and 93%, supporting the adequacy of the planned sample size. Exploratory System Metrics In addition to translation quality, we recorded system-level performance metrics, including end-to-end translation latency and estimated operational cost. Latency was computed from system logs as the total time between audio input and synthesized output, encompassing automatic speech recognition, neural translation, and text-to-speech synthesis. Operating cost was estimated based on API usage and computational overhead for a 10-minute bidirectional conversation, using unit prices of deployed components. These exploratory analyses aimed to contextualize LingualAI’s technical efficiency and cost-effectiveness relative to conventional interpreter services. RESULTS Results are organized in three analytical layers. Differences are reported as Δ = Human − AI (positive values favor human), with a prespecified non-inferiority margin of 0.30 points on the 5-point scale. Analyses include both paired and mixed-effects models to account for evaluator clustering and confirm robustness. Findings are presented as follows: (i) evaluator participation and reliability, (ii) domain-level performance evaluated using non-inferiority testing and direction-specific contrasts, and (iii) secondary analyses, including preference patterns, robustness checks, and representative error types. Evaluator and Data Overview Nine bilingual clinicians contributed ratings across three standardized scenarios comprising 18 clinician and 15 patient lines. Evaluator participation varied by scenario (8, 7, and 4 raters for Scenarios 1–3, respectively), resulting in an unbalanced rater–item panel. Participation declined across scenarios due to evaluator fatigue and the substantial time required, leading to fewer ratings in Scenario 3 (Table 1). Because raters were anonymous, inter-rater reliability was estimated using Krippendorff’s α, which accommodates missing data and unequal numbers of raters per item. Across all domains, α = 0.31, reflecting fair agreement. This modest reliability likely reflects our design choice to capture first-impression ratings rather than consensus. Domain-specific α values are shown in Table 2. Score distributions were approximately normal except for fluency and prosody, which violated Shapiro–Wilk assumptions and were analyzed with non-parametric tests. Primary Analyses (Fixed and Mixed Effects) Results from paired and mixed-effects models were largely consistent (Table 3). Adjusting for evaluator clustering did not alter the direction or significance of findings, confirming the robustness of observed differences between human and AI translations. Primary factors: Adequacy of meaning (human 4.82 vs. AI 4.70; 95% CI, 0.00 to 0.24; p = 0.050) and terminology accuracy (4.81 vs. 4.75; CI, −0.05 to 0.19; p = 0.282) remained statistically comparable across both models, indicating semantic equivalence between human and AI outputs. In contrast, clarity/intelligibility showed a consistent and large human advantage (4.88 vs. 4.36; p < 0.001; r = 0.76). Secondary factors: Human interpreters achieved higher scores in completeness (4.87 vs. 4.73; p = 0.022; 95% CI, 0.02–0.26), grammar/syntax (4.88 vs. 4.67; p = 0.007; CI, 0.09–0.32), vocabulary (4.85 vs. 4.67; p = 0.060; CI, 0.07–0.29), and cultural appropriateness (4.89 vs. 4.49; p < 0.001; CI, 0.27–0.52). Effect sizes ranged from moderate for vocabulary to large for grammar/syntax and cultural appropriateness. Voice-quality factors: The most pronounced differences were observed in fluency (4.86 vs. 3.72; p < 0.001; CI, 0.97–1.31), prosody (4.89 vs. 4.30; p < 0.001; CI, 0.47–0.72), and pacing (4.88 vs. 4.46; p = 0.002; CI, 0.31–0.53), all demonstrating large, statistically robust advantages for human interpreters. Conclusive factors: Overall quality (4.81 vs. 4.23; p < 0.001; CI, 0.44–0.72) and confidence for clinical use (4.82 vs. 4.22; p < 0.001; CI, 0.47–0.75) were also significantly higher for human translations. Direction-specific contrasts: Stratified analyses confirmed these trends (Supplementary Material SM4 ). For clinician statements (English → Spanish), human interpreters outperformed the AI system in clarity, fluency, prosody, pacing, overall quality, and clinical confidence, while terminology and adequacy of meaning were equivalent. For patient statements (Spanish → English), human interpreters again scored higher across delivery-related and linguistic-mechanics domains (grammar, vocabulary, prosody), whereas accuracy and meaning remained non-inferior. Non-Inferiority Testing A prespecified non-inferiority margin of 0.30 points on the 5-point scale was applied (Fig. 3). Only a subset of domains met this threshold, indicating that while LingualAI preserved semantic accuracy, delivery-related aspects remained inferior to human interpreters. Primary factors: Terminology accuracy (mean difference 0.07; 95% CI, 0.18) and adequacy of meaning (0.13; CI, 0.23) satisfied the non-inferiority criterion, confirming comparable semantic performance. Clarity exceeded the margin (0.50; CI, 0.72), reflecting a perceptible decline in intelligibility relative to human translation. Secondary factors: Completeness (0.14; CI, 0.25) met the criterion, but vocabulary (0.18; CI, 0.32), grammar (0.21; CI, 0.34), and cultural appropriateness (0.39; CI, 0.64) exceeded the margin, suggesting subtle but consistent linguistic differences. Voice-quality factors: None met the non-inferiority threshold, differences were largest for fluency (1.13; CI, 1.49), followed by prosody (0.59; CI, 0.81) and pacing (0.40; CI, 0.55), indicating that human interpreters retained a clear advantage in speech naturalness and rhythm. Conclusive factors: Both overall quality (0.58; CI, 0.81) and confidence for clinical use (0.61; CI, 0.84) exceeded the margin, underscoring the evaluators’ preference and higher perceived reliability for human translations. The mean differences and 95% confidence intervals for all domains are visualized in Table 4 and Figure 3, plotted against the prespecified non-inferiority boundary. When a stricter non-inferiority margin of 0.20 was applied, limited to Scenarios 1–2 due to smaller sample size in Scenario 3, terminology accuracy remained non-inferior while all other domains exceeded the margin. Detailed results are provided in Supplementary Material SM5. Preference Analysis Pairwise preference testing compared individual evaluator choices between human and AI audio clips for each domain (Table 5). Overall, preference patterns closely mirrored the quantitative scoring results: evaluators perceived both systems as equivalent for meaning and terminology but favored human interpreters for delivery quality and naturalness. Primary factors: Terminology accuracy (tie rate, 66%) and adequacy of meaning (65%) showed high equivalence, with minimal net bias toward human translations. Clarity revealed a stronger human preference (40% human wins vs. 4% AI wins; net bias of 0.36). Secondary factors: Completeness also demonstrated a high tie rate (73%), whereas grammar (29% vs. 8%; net bias of 0.21) and cultural appropriateness (31% vs. 8%; net bias of 0.23) favored human interpreters. Vocabulary showed a smaller but consistent human advantage (net bias of 0.12). Voice-quality factors: Preferences were most pronounced in these domains. Fluency had the lowest tie rate (25%) and the highest human win rate (67%; net bias of 0.59), followed by pacing and prosody, both exhibiting substantial net biases (>0.38). Conclusive factors: For overall quality and confidence for clinical use, human translations won in roughly half of all comparisons (~49%), with average net biases near 0.39. These findings emphasize that, while evaluators recognized comparable semantic accuracy, they consistently preferred the tone, rhythm, and expressiveness of human interpretations. Detailed tie rates for each domain can be found in Supplementary Material SM6. Error Analysis Qualitative review of evaluator comments and audio outputs revealed that most residual errors arose from delivery rather than meaning . Specifically, deviations in fluency and prosody , such as monotone intonation, abrupt phrasing, and irregular pacing occasionally disrupted conversational flow, even when lexical and semantic accuracy were preserved. These patterns were observed across all scenarios. For instance, certain clinician prompts (e.g., “Are you still doing the Budesonide irrigations?”) exhibited misplaced pauses or unnatural emphasis, while reassurance statements (e.g., “That’s expected. The swelling should go down over the next few weeks.”) were rendered with rigid or mechanical rhythm. Importantly, semantic fidelity remained high, confirming that LingualAI’s primary limitation lies in speech synthesis naturalness rather than translation accuracy. Representative examples with linked audio clips are provided in Supplementary Material SM7 , illustrating how subtle deviations in tone and pacing can alter perceived empathy and conversational realism. Translation Latency and Cost Based on system log analyses, LingualAI achieved an average end-to-end translation latency of 9.7 seconds per message, encompassing automatic speech recognition, neural translation, and speech synthesis. This turnaround time falls within the typical conversational pause window, allowing smooth, turn-based clinical dialogue without noticeable interruption. The estimated operating cost for a 10-minute bilingual conversation was approximately $0.03–$0.04 (USD), compared with $6.90–$10.60 for phone or video interpreter services, representing a cost reduction exceeding 99% per session. These results highlight LingualAI’s potential for scalable, low-cost deployment in time-sensitive or resource-limited clinical settings. Detailed latency distributions and cost-calculation parameters are provided in Supplementary Material SM8. Summary of Findings Overall, LingualAI preserved semantic accuracy at levels comparable to certified interpreters but showed limitations in delivery quality and expressiveness. Among the primary factors, both terminology accuracy and adequacy of meaning met the non-inferiority thresholds, confirming that the AI system conveyed medical content reliably. Within the secondary factors, only completeness met the non-inferiority criterion, while grammar, vocabulary, and cultural appropriateness favored human interpreters. Notably, differences in vocabulary and cultural appropriateness reached significance primarily in patient statements (Spanish → English). All voice-quality factors – fluency, prosody, and pacing showed large and consistent human advantages, underscoring the gap in naturalness and conversational rhythm. Finally, both conclusive factors – overall quality and confidence for clinical use were significantly higher for human translations. Together, these findings suggest that while LingualAI effectively preserves meaning and accuracy, its synthesized speech still lacks the nuance and expressivity essential for fully natural clinical communication. DISCUSSION Key Findings In this prospective, within-subject comparison of an AI-based translation tool (LingualAI) with certified medical interpreters, we found that LingualAI preserved two primary factors, terminology accuracy and adequacy of meaning, at levels that met non-inferiority thresholds, ensuring reliable conveyance of core medical content. LingualAI also achieved non-inferiority for completeness, supporting its ability to maintain the overall flow of dialogue. However, human interpreters consistently outperformed LingualAI in various secondary and voice-related factors (grammar, vocabulary, cultural appropriateness, fluency, prosody, and pacing), underscoring the continued importance of delivery quality on clinical communications. These findings suggest that while semantic integrity was maintained, elements crucial for patient trust and nuanced interaction remain stronger with professional interpreters. Importantly, the preservation of meaning and terminology, the dimensions most directly linked to clinical safety suggests that LingualAI can reliably convey essential medical information, even if its spoken delivery remains less natural than human interpreters. Clinical Implications The findings indicate that AI-based translation tools such as LingualAI are beginning to demonstrate credible performance in the critical clinical safety dimensions, preserving meaning and terminology accuracy. These strengths suggest a role for AI in supporting basic comprehension and instruction delivery, particularly in settings where interpreter services are unavailable, such as urgent encounters, rural clinics, or after-hours consultations. In such scenarios, LingualAI may be an alternative to delaying care or rendering care without translation. From a practical standpoint, accuracy in conveying clinical meaning is often of higher value than perfectly natural prosody, especially in urgent or resource-limited settings. While human interpreters provide richer tone and empathy, LingualAI’s consistent accuracy and ultra-low operational cost may make it a viable adjunct when access or affordability is constrained. Moving forward, it will be essential to work closely with certified medical translation bodies, since the acceptability of AI-supported translation depends on alignment with professional standards. The objective is not to replace certified medical translators but to augment their role and provide coverage in situations where they are absent. In practice, this could also include hybrid workflows where certified translators remotely monitor AI-facilitated conversations, ensuring they remain in the loop and can intervene when needed. AI translation should be viewed as an adjunct that extends access while maintaining the primacy of professional interpreters to ensure safe, nuanced, and culturally appropriate communication. The baseline capabilities of AI translation models are expected to continue improving, which may steadily raise overall performance on core tasks. At the same time, the inherent risk of errors could also increase as these systems are deployed more widely and in higher-stakes settings. The broader deployment of these systems in critical settings highlights the importance of continuously monitoring primary factors such as meaning and terminology accuracy, alongside steady technological improvements in secondary and voice-related factors. To ensure safe and equitable integration, such progress should be documented through well-structured, real-world studies that evaluate not only technical performance but also clinical usability and patient experience. Under Section 1557 of the Affordable Care Act and related regulations and guidance from the Office for Civil Rights (OCR), AI-based translation tools may be used to support language access in healthcare, but they cannot serve as the sole method for providing meaningful access to individuals with limited English proficiency ( 22 , 23 ). OCR’s regulatory framework generally requires that most substantive clinical communications such as informed consent, treatment instructions, and discharge summaries be reviewed or provided by qualified human interpreters or translators. While AI tools may improve efficiency and accuracy in medical terminology, they must be integrated into workflows that include human oversight to ensure compliance with nondiscrimination standards. Comparison to literature Our findings are consistent with prior evaluations of general-purpose MT tools, which often demonstrate strengths in medical terminology and adequacy of meaning, but show persistent weakness in contextual interpretation, cultural appropriateness, fluency, and other delivery-related aspects. For example, studies report that tools such as Google Translate perform well in terms of adequacy of meaning and basic clinical content, but often struggle with cultural context and have dialect sensitivity issues ( 24 , 25 ). Work on LLMs has further highlighted the limitations of literal translation approaches that miss nuances essential for medical communication ( 26 , 27 ). Evidence also suggests that AI-based translations lack the cultural awareness to support trust and rapport in diverse patient populations, potentially leading to misunderstandings ( 28 ). While some studies indicate that AI can generate fluent output, syntactic and semantic inconsistencies remain common, particularly in languages with complex grammatical structures (29). Our results extend this body of evidence by demonstrating that even a purpose-built, domain-specific system like LingualAI, optimized for clinical communication, continues to mirror these patterns. It performed comparably to certified interpreters on meaning and terminology accuracy but lagged in prosody, fluency, and clinical confidence. Unlike prior studies, however, our evaluation used a prospective, within-subject design with blinded bilingual clinician raters, multidomain scoring, and a non-inferiority framework. This methodological rigor strengthens confidence in the findings and provides a more granular understanding of where AI translation tools may safely support clinical communication and where human oversight remains indispensable. Strengths & Limitations This study has several strengths. The prospective, within-subject paired design ensured direct comparability between AI and human translators, while blinded evaluation by bilingual clinicians reduced bias. A structured, multidomain rubric and a pre-specified non-inferiority framework provided a rigorous translation quality assessment across clinically relevant dimensions. In addition, combining paired statistical tests and mixed-effects modeling strengthened the robustness of findings and enabled a nuanced interpretation of where AI tools achieved equivalence and where differences persisted. Several limitations should be acknowledged. The scenarios were scripted and restricted to otorhinolaryngology, which may not capture the variability of free-flowing clinical dialogue, including patient emotions, interruptions, or background noise. The study evaluated audio translations only, without considering LingualAI’s text-based interface, which could also influence comprehension. The evaluation occurred in a non-live setting, outside of real patient encounters, which may not fully represent the pressures and dynamics of clinical care. Evaluator-related factors add further constraints. The clinical evaluators were anonymized, and an unequal number of ratings were available across scenarios, leading to an unbalanced dataset. Inter-rater reliability was modest (Krippendorff’s α = 0.31), and no calibration was performed to align evaluators before the study; this was an intentional decision to capture first impressions rather than consensus, but one that limited agreement. Additionally, the raters were clinicians rather than patients, meaning domains such as fluency, rapport, or cultural appropriateness may not reflect patient experience directly. Finally, the findings were restricted to English–Spanish translation, and results may not generalize to other language pairs or dialects. These limitations should be considered when interpreting the scope of our results. Future Directions Future development of LingualAI should prioritize enhancements in speech synthesis, particularly prosody, pacing, and fluency, which accounted for the most significant performance gaps compared with certified interpreters. Beyond these technical refinements, validation in live, unscripted clinical encounters will be essential to test performance under real-world conditions such as interruptions, emotional tone, and background noise. Incorporating patient perspectives and expanding evaluation to additional languages and dialects will strengthen generalizability. As language models and translation technologies evolve, monitoring new error models and working closely with certified medical translation bodies to ensure alignment with professional standards will be critical. Therefore, well-structured, real-world studies should assess accuracy, delivery quality, patient-centered outcomes, and clinical usability, aiming to develop evidence-based, reliable, and trusted AI translation tools that extend access to multilingual care while preserving the central role of professional interpreters. Another important consideration is consistency. While human interpreter quality can vary substantially across sessions and providers, automated systems like LingualAI offer the potential for stable, repeatable performance once adequately validated. Future improvements in voice synthesis, particularly expressiveness and conversational pacing, will further enhance user experience and make AI-assisted translation increasingly compelling in real-world use. Conclusion LingualAI demonstrated non-inferior performance in preserving meaning and terminology – the core dimensions of translation accuracy while certified interpreters remained superior in delivery-related qualities such as fluency, prosody, and clinical confidence. These findings suggest that AI translation tools can safely support essential communication when interpreters are unavailable, provided their use remains aligned with professional standards. Consistency and scalability may further enhance their utility, particularly as voice synthesis continues to improve. In practice, the most responsible pathway forward is an interpreter-in-the-loop model, in which professional interpreters oversee or supplement AI-mediated communication to ensure contextual accuracy and patient trust. Professional interpreters therefore remain the gold standard, yet AI systems like LingualAI can serve as valuable supplementary tools that extend access to multilingual care. Continued technical refinement and clinical validation will be essential for ensuring safe, effective, and trusted deployment in real-world healthcare settings. Declarations Ethics Approval: This study has been approved by the Committee for the Protection of Human Subjects (the UTHSC-H IRB) under protocol HSC-SBMI- 13-0549. ACKNOWLEDGEMENTS We gratefully acknowledge Drs. German Martinez-Gamba and Amalia Guardiola for their contributions as clinical evaluators. We also thank William B. McElhiney for his valuable guidance on legal and regulatory considerations. FUNDING This work was supported by institutional funds from The University of Texas Health Science Center at Houston (UTHealth Houston) and by discretionary research funds provided by Dr. Xiaoqian Jiang. DATA AVAILABILITY STATEMENT De-identified audio clips and raw evaluator rating data underlying this study are available from the corresponding author upon reasonable request. Access will require completion of a data-use agreement and approval by the appropriate Institutional Review Board. Summary-level data (means, standard deviations, and confidence intervals) are included in the main manuscript and Supplementary Materials. Due to file size, PHI redaction requirements, and copyright considerations, raw audio files and individual rating sheets are not publicly posted. HUMAN ETHICS AND CONSENT TO PARTICIPATE All clinician evaluators participated on a voluntary basis and provided informed consent before initiating any study-related activities. The study was conducted in accordance with the ethical principles outlined in the Declaration of Helsinki. No patients or members of the public were recruited or enrolled as participants in this study. Certified translators provided translation services as compensated professionals and were not considered study participants. AUTHOR CONTRIBUTIONS STATEMENT UPS led the study conceptualization, methodology, formal analysis, data curation, visualization, and drafting of the original manuscript. CJG contributed to software development, methodology, data curation, validation, and visualization. GMA, JBG, DRJ, EW, VSC, AFV, and BRE contributed to investigation, validation, and manuscript review. JHC and CW supported project administration, with JHC additionally contributing to manuscript review and editing. BOF contributed resources, investigation, project administration, and manuscript review. MJC provided supervision, conceptual guidance, and manuscript review. XJ provided overall conceptualization, supervision, methodological guidance, and manuscript review, editing and funding support. All authors reviewed and approved the final manuscript. References Diamond L, Izquierdo K, Canfield D, Matsoukas K, Gany F. A Systematic Review of the Impact of Patient–Physician Non-English Language Concordance on Quality of Care and Outcomes. J Gen Intern Med. 2019 Aug;34(8):1591–606. U.S. Census Bureau. People that speak English less than very well [Internet]. Washington, DC: U.S. Department of Commerce; n.d. Available from: https://www.census.gov/library/visualizations/interactive/people-that-speak-english-less-than-very-well.html Twersky SE, Jefferson R, Garcia-Ortiz L, Williams E, Pina C. The Impact of Limited English Proficiency on Healthcare Access and Outcomes in the U.S.: A Scoping Review. Healthcare. 2024 Jan 31;12(3):364. Williams KM, Dougherty D, Plagens C, Shah NR, Tubbs D, Ehrlich PF. Limited English Proficiency can Negatively Impact Disease/Treatment in Children With Cancer Compared to Those Who are English Proficient-an Institutional Study. J Pediatr Surg. 2024 May;59(5):800–3. Tran AV, Roberts KP. Language Accommodations for Limited English Proficient Patients in Rural Health Care. J Immigr Minor Health. 2023 June;25(3):674–9. Habhab M, Pham MT. Medical Interpretation Services: Challenges for LEP Communities. Am J Bioeth. 2024 Nov;24(11):72–4. Hudelson P, Chappuis F. Using Voice-to-Voice Machine Translation to Overcome Language Barriers in Clinical Communication: An Exploratory Study. J Gen Intern Med. 2024 May;39(7):1095–102. Genovese A, Borna S, Gomez-Cabello CA, Haider SA, Prabha S, Forte AJ, et al. Artificial intelligence in clinical settings: a systematic review of its role in language translation and interpretation. Ann Transl Med. 2024 Dec;12(6):117–117. Lee W, Khoong EC, Zeng B, Rios-Fetchko F, Ma Y, Liu K, et al. Evaluation of Commercially Available Machine Interpretation Applications for Simple Clinical Communication. J Gen Intern Med. 2023 Aug;38(10):2333–9. Kong M, Fernandez A, Bains J, Milisavljevic A, Brooks KC, Shanmugam A, et al. Evaluation of the accuracy and safety of machine translation of patient-specific discharge instructions: a comparative analysis. BMJ Qual Saf. 2025 July 9;bmjqs-2024-018384. Taira BR, Kreger V, Orue A, Diamond LC. A Pragmatic Assessment of Google Translate for Emergency Department Instructions. J Gen Intern Med. 2021 Nov;36(11):3361–5. Herrmann-Werner A, Loda T, Zipfel S, Holderried M, Holderried F, Erschens R. Evaluation of a Language Translation App in an Undergraduate Medical Communication Course: Proof-of-Concept and Usability Study. JMIR MHealth UHealth. 2021 Dec 2;9(12):e31559. Pinto Taylor E, Mulenos A, Chatterjee A, Talwalkar JS. Partnering With Interpreter Services: Standardized Patient Cases to Improve Communication With Limited English Proficiency Patients. MedEdPORTAL. 2019 May 20;10826. Han L, Gladkoff S, Erofeev G, Sorokina I, Galiano B, Nenadic G. Neural machine translation of clinical text: an empirical investigation into multilingual pre-trained language models and transfer-learning. Front Digit Health. 2024 Feb 26;6:1211564. Zolnoori M, Vergez S, Xu Z, Esmaeili E, Zolnour A, Anne Briggs K, et al. Decoding disparities: evaluating automatic speech recognition system performance in transcribing Black and White patient verbal communication with nurses in home healthcare. JAMIA Open. 2024 Oct 8;7(4):ooae130. Chen CL, Dong Y, Castillo-Zambrano C, Bencheqroun H, Barwise A, Hoffman A, et al. A systematic multimodal assessment of AI machine translation tools for enhancing access to critical care education internationally. BMC Med Educ. 2025 July 8;25(1):1022. Freitag M, Foster G, Grangier D, Ratnakar V, Tan Q, Macherey W. Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation. Trans Assoc Comput Linguist. 2021 Dec 17;9:1460–74. Läubli S, Castilho S, Neubig G, Sennrich R, Shen Q, Toral A. A Set of Recommendations for Assessing Human-Machine Parity in Language Translation. J Artif Intell Res [Internet]. 2020 Mar 23 [cited 2025 Aug 29];67. Available from: http://arxiv.org/abs/2004.01694 Liu T, Lo C kiu, Marshman E, Knowles R. Evaluation Briefs: Drawing on Translation Studies for Human Evaluation of MT. 2024; Gerlinger C, Schmelter T. Determining the non‐inferiority margin for patient reported outcomes. Pharm Stat. 2011 Sept;10(5):410–3. Do Campo Bayón M, Sánchez-Gijón P. Evaluating NMT using the non-inferiority principle. Nat Lang Process. 2025 July;31(4):1042–61. U.S. Department of Health & Human Services, Office for Civil Rights. Dear Colleague Letter: Language Access Provisions of the Final Rule Implementing Section 1557 of the Affordable Care Act [Internet]. 2024 Dec. Available from: https://www.hhs.gov/sites/default/files/ocr-dcl-section-1557-language-access.pdf National Health Law Program. Title VI and Section 1557 Language Access Requirements [Internet]. Washington, DC; 2024 May. Available from: https://healthlaw.org/wp-content/uploads/2024/05/T-VI-and-Sec-1557-explainer-2024.pdf Brandenberger J, Stedman I, Stancati N, Sappleton K, Kanathasan S, Fayyaz J, et al. Using artificial intelligence based language interpretation in non-urgent paediatric emergency consultations: a clinical performance test and legal evaluation. BMC Health Serv Res. 2025 Jan 24;25(1):138. Brewster RCL, Gonzalez P, Khazanchi R, Butler A, Selcer R, Chu D, et al. Performance of ChatGPT and Google Translate for Pediatric Discharge Instruction Translation. Pediatrics. 2024 July 1;154(1):e2023065573. Yan J, Yan P, Chen Y, Li J, Zhu X, Zhang Y. Benchmarking GPT-4 against Human Translators: A Comprehensive Evaluation Across Languages, Domains, and Expertise Levels [Internet]. arXiv; 2024 [cited 2025 Sept 16]. Available from: http://arxiv.org/abs/2411.13775 Keles B, Gunay M, Caglar SI. LLMs-in-the-loop Part-1: Expert Small AI Models for Bio-Medical Text Translation [Internet]. arXiv; 2024 [cited 2025 Sept 16]. Available from: http://arxiv.org/abs/2407.12126 Merx R, Phillips C, Suominen H. Machine Translation Technology in Health: A Scoping Review. In: Bichel-Findlay J, editor. Studies in Health Technology and Informatics [Internet]. IOS Press; 2024 [cited 2025 Sept 16]. Available from: https://ebooks.iospress.nl/doi/10.3233/SHTI240895 Slosarek T, Paeschke D, Sivtsev I, Böttinger EP. Comparison of Machine Translation Services in the Biomedical Context. In: 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) [Internet]. Lisbon, Portugal: IEEE; 2024 [cited 2025 Sept 16]. p. 5539–45. Available from: https://ieeexplore.ieee.org/document/10822680/ Tables Tables 1 to 5 are available in the Supplementary Files section Additional Declarations No competing interests reported. Supplementary Files SM1.Scenarios13usedforvalidationofLingualAI.docx SM2.RatingInstrument.docx SM3.Aprioripowercalculation.docx SM4.StratifiedResultsforClinicianandPatientStatements.docx SM5.Noninferioritywithstrictermargin.docx SM6.TieRateforeachdomain.docx SM7.ErrorAnalysis.docx SM8.TranslationLatencyandEstimatedcostper10minuteconversation.docx Tables.docx Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 25 Dec, 2025 Reviews received at journal 25 Dec, 2025 Reviews received at journal 15 Dec, 2025 Reviewers agreed at journal 11 Dec, 2025 Reviewers agreed at journal 02 Dec, 2025 Reviewers invited by journal 02 Dec, 2025 Editor assigned by journal 29 Nov, 2025 Submission checks completed at journal 28 Nov, 2025 First submitted to journal 25 Nov, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8205991","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":554249143,"identity":"71246afc-c5a7-4f73-8013-1457441f7f92","order_by":0,"name":"Uday P. Singh","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABJklEQVRIie3SPWuDQBjA8UdSdJFmfTL5FZSAZGjJV7lDMItNB6FTG4TCZRGy5mM0FMQxIOhi9wOhtBScpJgltFs1xb7QS9ux0Psv5x3+OE4PQCb7k5HPz6QPoHZz81uid2QQ9NRu9ReknZrrH4iZOSUq8a0x1m6S+j6+mA75aXqnPEUGaJcRikhe2qjkvhXqUxdpnvk2dzQTSGEFenomJJw0hBElBM8GylIacUfFhhBAzxaTybYl47BfDeuWXC87YlR7iLfbhYbomUjZOb3Ct110IRnklT+iOXFC/ticha3pMn+wkbiFxXTXHwnIYTZZ8U1MjueLk2TzzGZ0Macl1keF0deSFRd9ZYADpEE77v5M8rpG4MMd+FqvVt7JbP97MplM9m97ATNuaD8LUuRqAAAAAElFTkSuQmCC","orcid":"","institution":"The University of Texas Health Science Center at Houston","correspondingAuthor":true,"prefix":"","firstName":"Uday","middleName":"P.","lastName":"Singh","suffix":""},{"id":554249144,"identity":"fa5900f3-ff89-4056-ab60-45ea3488ff9e","order_by":1,"name":"Carlos A. Jaimes Garcia","email":"","orcid":"","institution":"The University of Texas Health Science Center at Houston","correspondingAuthor":false,"prefix":"","firstName":"Carlos","middleName":"A. Jaimes","lastName":"Garcia","suffix":""},{"id":554249145,"identity":"08b4681b-5436-4ba1-a469-8686ac1d2292","order_by":2,"name":"Gabriel M. Aisenberg","email":"","orcid":"","institution":"The University of Texas Health Science Center at Houston","correspondingAuthor":false,"prefix":"","firstName":"Gabriel","middleName":"M.","lastName":"Aisenberg","suffix":""},{"id":554249146,"identity":"a8a256fb-5adf-42aa-86e4-ebb11bab9c05","order_by":3,"name":"Javier Barreda Garcia","email":"","orcid":"","institution":"The University of Texas Health Science Center at Houston","correspondingAuthor":false,"prefix":"","firstName":"Javier","middleName":"Barreda","lastName":"Garcia","suffix":""},{"id":554249147,"identity":"d8326821-4b69-4950-ac40-519993efcddd","order_by":4,"name":"Jessica A. Hernandez-Chilatra","email":"","orcid":"","institution":"The University of Texas Health Science Center at Houston","correspondingAuthor":false,"prefix":"","firstName":"Jessica","middleName":"A.","lastName":"Hernandez-Chilatra","suffix":""},{"id":554249148,"identity":"19b8c445-369f-4302-ad70-97e0f80f5be8","order_by":5,"name":"Cecilia Wang","email":"","orcid":"","institution":"The University of Texas Health Science Center at Houston","correspondingAuthor":false,"prefix":"","firstName":"Cecilia","middleName":"","lastName":"Wang","suffix":""},{"id":554249149,"identity":"f00c3c32-71c8-4c1f-a5c2-499408ec335a","order_by":6,"name":"Dalilah Reyes Jesus","email":"","orcid":"","institution":"The University of Texas Health Science Center at Houston","correspondingAuthor":false,"prefix":"","firstName":"Dalilah","middleName":"Reyes","lastName":"Jesus","suffix":""},{"id":554249150,"identity":"157c5015-829d-4829-942e-dd9533f6b506","order_by":7,"name":"Eileen Whalen","email":"","orcid":"","institution":"The University of Texas Health Science Center at Houston","correspondingAuthor":false,"prefix":"","firstName":"Eileen","middleName":"","lastName":"Whalen","suffix":""},{"id":554249151,"identity":"45c84767-9c2a-41b6-8113-6f6950f26ac9","order_by":8,"name":"Veronica Santos Canellas","email":"","orcid":"","institution":"The University of Texas Health Science Center at Houston","correspondingAuthor":false,"prefix":"","firstName":"Veronica","middleName":"Santos","lastName":"Canellas","suffix":""},{"id":554249152,"identity":"9e19f430-98a1-4dbc-85f7-dff4930d8670","order_by":9,"name":"Amanda R. Falk Vargas","email":"","orcid":"","institution":"The University of Texas Health Science Center at Houston","correspondingAuthor":false,"prefix":"","firstName":"Amanda","middleName":"R. Falk","lastName":"Vargas","suffix":""},{"id":554249153,"identity":"a9b4be86-45a2-4ce6-a928-1a2532e9bdb2","order_by":10,"name":"Brian O. Rodriguez Echevarria","email":"","orcid":"","institution":"The University of Texas Health Science Center at Houston","correspondingAuthor":false,"prefix":"","firstName":"Brian","middleName":"O. Rodriguez","lastName":"Echevarria","suffix":""},{"id":554249154,"identity":"5b0c9077-63a9-4a3f-a40e-c4e0c203f130","order_by":11,"name":"Martin J. Citardi","email":"","orcid":"","institution":"The University of Texas Health Science Center at Houston","correspondingAuthor":false,"prefix":"","firstName":"Martin","middleName":"J.","lastName":"Citardi","suffix":""},{"id":554249155,"identity":"cd219334-6b0a-4e1a-baf6-15aa4600a470","order_by":12,"name":"Babatope O. Fatuyi","email":"","orcid":"","institution":"The University of Texas Health Science Center at Houston","correspondingAuthor":false,"prefix":"","firstName":"Babatope","middleName":"O.","lastName":"Fatuyi","suffix":""},{"id":554249156,"identity":"334ad77d-a990-4226-8227-3e71903a9902","order_by":13,"name":"Xiaoqian Jiang","email":"","orcid":"","institution":"The University of Texas Health Science Center at Houston","correspondingAuthor":false,"prefix":"","firstName":"Xiaoqian","middleName":"","lastName":"Jiang","suffix":""}],"badges":[],"createdAt":"2025-11-25 18:23:15","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8205991/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8205991/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":97670197,"identity":"aec5bb75-0aed-463a-8b16-dfe8e4c70ea0","added_by":"auto","created_at":"2025-12-08 09:29:51","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":2079780,"visible":true,"origin":"","legend":"","description":"","filename":"LingualAIPaperFinalDraftv1.docx","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/33e5029ce3514b1ae7d69f7e.docx"},{"id":97487755,"identity":"173b2bd1-efe1-4c6a-8a85-0248f9c34822","added_by":"auto","created_at":"2025-12-05 01:52:57","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":16703,"visible":true,"origin":"","legend":"","description":"","filename":"7512df4286474b40b26cdeb152323568.json","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/bef83835ad8b6bce93c4790f.json"},{"id":97669886,"identity":"a97594da-9488-4f02-a726-1c1f12e2539c","added_by":"auto","created_at":"2025-12-08 09:29:16","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":8802,"visible":true,"origin":"","legend":"","description":"","filename":"SM1.Scenarios13usedforvalidationofLingualAI.docx","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/16fb42116e16f5151c53c0fa.docx"},{"id":97487759,"identity":"7b1b1130-fac4-4807-8657-6d55506786e2","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"docx","order_by":3,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":9345,"visible":true,"origin":"","legend":"","description":"","filename":"SM2.RatingInstrument.docx","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/bfc50ec8da1b9f1411f6d62a.docx"},{"id":97487764,"identity":"02f99ac1-d548-47e9-a6de-9c089c522f4c","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"docx","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":10356,"visible":true,"origin":"","legend":"","description":"","filename":"SM3.Aprioripowercalculation.docx","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/f50f6a966fabe468cfd2478d.docx"},{"id":97487769,"identity":"ac085187-5694-47a9-a70c-462b3424e94e","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"docx","order_by":5,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":541071,"visible":true,"origin":"","legend":"","description":"","filename":"SM4.StratifiedResultsforClinicianandPatientStatements.docx","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/cdd1afe9ec2e8a9fb3651b8c.docx"},{"id":97487767,"identity":"0ae04ba7-7c70-4403-8641-e4cf7dcfa0a5","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"docx","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":9361,"visible":true,"origin":"","legend":"","description":"","filename":"SM5.Noninferioritywithstrictermargin.docx","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/f563b97bc53486b119aa35fe.docx"},{"id":97669675,"identity":"a986720e-efcd-4d80-9e5a-b21d99070d0a","added_by":"auto","created_at":"2025-12-08 09:28:41","extension":"docx","order_by":7,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":259039,"visible":true,"origin":"","legend":"","description":"","filename":"SM6.TieRateforeachdomain.docx","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/b2e1e65f3571496553913f3d.docx"},{"id":97487770,"identity":"0116a79f-e511-4ef4-80fc-235584d3f230","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"docx","order_by":8,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":8719,"visible":true,"origin":"","legend":"","description":"","filename":"SM7.ErrorAnalysis.docx","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/f09ab72c1e90d9804ded3605.docx"},{"id":97487794,"identity":"32cfd93d-ba64-462b-a80a-04d832554102","added_by":"auto","created_at":"2025-12-05 01:52:59","extension":"docx","order_by":9,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":565047,"visible":true,"origin":"","legend":"","description":"","filename":"SM8.TranslationLatencyandEstimatedcostper10minuteconversation.docx","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/eaec0f7c7067cf36a169d3f2.docx"},{"id":97670692,"identity":"cd7c6656-38c5-4c98-b0d5-82d19eeb6a50","added_by":"auto","created_at":"2025-12-08 09:31:11","extension":"xml","order_by":10,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":107783,"visible":true,"origin":"","legend":"","description":"","filename":"7512df4286474b40b26cdeb1523235681enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/ebf41c56c372d59ab3494223.xml"},{"id":97487788,"identity":"94e180d1-4795-4670-8b04-95f488cf38e0","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"png","order_by":11,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":654450,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/6e293b6e5dd2835938b0a058.png"},{"id":97668978,"identity":"3fb4ee8e-0879-474b-9ac3-7f961b54f8ce","added_by":"auto","created_at":"2025-12-08 09:26:53","extension":"png","order_by":12,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":339489,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/3b28235d5f4e3cc0ab761a49.png"},{"id":97487776,"identity":"0c668d01-c4fa-49cf-a65d-5a8683637be7","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"png","order_by":13,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":40416,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/2cded00347982d4a943a2887.png"},{"id":97487781,"identity":"5f084605-c6d5-419b-8f55-0d8f1027790f","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"png","order_by":14,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":232857,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/989804e229a679adb380f4eb.png"},{"id":97670179,"identity":"6f5337c0-c0ef-4996-b592-51018085ae69","added_by":"auto","created_at":"2025-12-08 09:29:48","extension":"png","order_by":15,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":252922,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/40e1fb3d85b2cda1a65d7cf9.png"},{"id":97487786,"identity":"e991c4e2-1eb0-4190-ac3c-79ea9ca4dbc1","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"png","order_by":16,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":235094,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/748ce72b4b87d977b08854d7.png"},{"id":97487773,"identity":"f55e6215-b04e-4d58-8680-f327510cf7da","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"png","order_by":17,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":175867,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage7.png","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/342c040ce7932aa4090610cd.png"},{"id":97669690,"identity":"bea07a45-5a1f-40b3-b4d0-4b53883db1cc","added_by":"auto","created_at":"2025-12-08 09:28:41","extension":"png","order_by":18,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":182530,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage8.png","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/208e17fb22e9b29f376dac86.png"},{"id":97487785,"identity":"a334bd00-9a80-4c3a-b6d9-10429c570da7","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"png","order_by":19,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":96841,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/55d12e70c9e5acacb2d6fbde.png"},{"id":97487780,"identity":"c95ce99a-8fe0-4f2b-924a-f3f6398e951b","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"png","order_by":20,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":59936,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/ac5478e7a3a95c358c1a5045.png"},{"id":97487782,"identity":"e61a0360-a199-4c6d-b0d6-85549ab0ca18","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"png","order_by":21,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":17738,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/ec4d49a3fcb0a7e48d840d15.png"},{"id":97487790,"identity":"b6a83871-40cc-470e-9e45-65832b437297","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"png","order_by":22,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":81116,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/5d42c559ab0ce62dc45bf599.png"},{"id":97487783,"identity":"2366215b-d2ba-41cc-badd-93e6096ee1db","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"png","order_by":23,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":94642,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/4dda7a0049eda8cb1da488bb.png"},{"id":97670856,"identity":"c99461fb-387b-4f99-a374-34c415d74035","added_by":"auto","created_at":"2025-12-08 09:31:26","extension":"png","order_by":24,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":86647,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/273c74db143f6e5b2251676b.png"},{"id":97487789,"identity":"fc17cbef-a8af-47f7-8187-33217d98dd52","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"png","order_by":25,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":50120,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage7.png","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/c9e99d9c95c2409995a6a4e7.png"},{"id":97669673,"identity":"23fe7f19-5aab-41ed-b815-3f5089b748c0","added_by":"auto","created_at":"2025-12-08 09:28:41","extension":"png","order_by":26,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":67180,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage8.png","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/866931d2f759bca1c38a6056.png"},{"id":97487791,"identity":"92935436-f50a-4d50-87b2-9a70fea447c5","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"xml","order_by":27,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":101496,"visible":true,"origin":"","legend":"","description":"","filename":"7512df4286474b40b26cdeb1523235681structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/e0ef81b4ecfe43ce4cdbca56.xml"},{"id":97669452,"identity":"c65ebffb-b5db-411f-8c99-b0f1cf73a9ab","added_by":"auto","created_at":"2025-12-08 09:28:01","extension":"html","order_by":28,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":119948,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/fd4bad447e505f02cf41484e.html"},{"id":97487754,"identity":"fa4eb954-ed6c-4c3c-8d5f-cb02f9709b9c","added_by":"auto","created_at":"2025-12-05 01:52:57","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":185149,"visible":true,"origin":"","legend":"\u003cp\u003eVisual Abstract.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/08213f0a217f82b898dbba66.png"},{"id":97487760,"identity":"f5d2a900-4ade-43b4-a0ed-a11a6f3df41d","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":168847,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eStudy design and analytic workflow: The workflow outlines the sequential stages of data collection, translation, evaluation, and analysis. Three standardized otorhinolaryngology scenarios were enacted by native English- and Spanish-speaking pairs and forked into two translation pipelines. The AI pipeline captured voice, converted it to text, refined clinical content, translated, and synthesized speech. The human pipeline involved two certified interpreters (English→Spanish, Spanish→English) producing parallel recordings that were anonymized. All four outputs per line (two AI, two human) were merged, randomized, and presented to blinded bilingual clinicians for multidomain quality rating. Statistical analysis included paired and mixed-effects models with a prespecified non-inferiority margin of 0.30 points on 5-point Likert scales.\u003c/em\u003e\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/c566a6fe945b24ffdac0a49a.png"},{"id":97670238,"identity":"7c3ae910-7f8b-40dc-ad7e-d2060b11e375","added_by":"auto","created_at":"2025-12-08 09:29:59","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":88245,"visible":true,"origin":"","legend":"\u003cp\u003eNon-inferiority analysis of AI versus human translation across quality domains: Mean differences (Human − LingualAI) with 95% confidence intervals are displayed for all 12 translation quality domains, grouped by category. The vertical dashed black line indicates no difference (Δ = 0), and the red dashed line marks the prespecified non-inferiority margin (Δ = 0.30). Points to the left of the red line represent domains in which LingualAI achieved non-inferior performance relative to human interpreters. Terminology accuracy, adequacy of meaning, and completeness met this criterion, while all voice-related and delivery factors (fluency, prosody, pacing) and conclusive factors (overall quality, clinical confidence) exceeded the margin, indicating superior human performance.\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/73e7f0e270607872cbe9d760.png"},{"id":97677718,"identity":"c4a54d0a-47a1-4a4b-9856-696c887a66c7","added_by":"auto","created_at":"2025-12-08 09:54:14","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1175045,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/15ff07dc-5db0-4c9a-98bb-9d228422b042.pdf"},{"id":97487762,"identity":"14ac096b-88c1-490e-9c10-3e1c34871ffa","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":8802,"visible":true,"origin":"","legend":"","description":"","filename":"SM1.Scenarios13usedforvalidationofLingualAI.docx","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/07184a4bc7273f8b361bedb5.docx"},{"id":97669333,"identity":"7f989eea-5d5c-4f47-b8b0-10cb6a090bcb","added_by":"auto","created_at":"2025-12-08 09:27:49","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":9345,"visible":true,"origin":"","legend":"","description":"","filename":"SM2.RatingInstrument.docx","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/95ee1a4192dd3e1c92847d2e.docx"},{"id":97487756,"identity":"cd572cb9-8f8c-43d5-83a7-bfdbd9b8638f","added_by":"auto","created_at":"2025-12-05 01:52:57","extension":"docx","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":10356,"visible":true,"origin":"","legend":"","description":"","filename":"SM3.Aprioripowercalculation.docx","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/381911405006de8f13ad08af.docx"},{"id":97670300,"identity":"bda6089d-1d88-4ddf-bd62-1b5bceedf226","added_by":"auto","created_at":"2025-12-08 09:30:13","extension":"docx","order_by":4,"title":"","display":"","copyAsset":false,"role":"supplement","size":541071,"visible":true,"origin":"","legend":"","description":"","filename":"SM4.StratifiedResultsforClinicianandPatientStatements.docx","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/ec5dc360ba3d131f75d576c6.docx"},{"id":97669344,"identity":"3c043d52-6eaa-4398-b82d-5256554fa789","added_by":"auto","created_at":"2025-12-08 09:27:50","extension":"docx","order_by":5,"title":"","display":"","copyAsset":false,"role":"supplement","size":9361,"visible":true,"origin":"","legend":"","description":"","filename":"SM5.Noninferioritywithstrictermargin.docx","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/5ecee4fb126807986be52cf7.docx"},{"id":97487775,"identity":"2ededd5f-db40-4107-b1a7-b68cfaf496ba","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"docx","order_by":6,"title":"","display":"","copyAsset":false,"role":"supplement","size":259039,"visible":true,"origin":"","legend":"","description":"","filename":"SM6.TieRateforeachdomain.docx","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/a2d2d70feec315c791fa1ac6.docx"},{"id":97487765,"identity":"d0152e4a-9cce-4846-955b-74913e86620a","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"docx","order_by":7,"title":"","display":"","copyAsset":false,"role":"supplement","size":8719,"visible":true,"origin":"","legend":"","description":"","filename":"SM7.ErrorAnalysis.docx","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/57f0e22915cc5453d61d0f00.docx"},{"id":97670840,"identity":"dab2b7c8-71f2-4656-9c89-efdd28d49766","added_by":"auto","created_at":"2025-12-08 09:31:24","extension":"docx","order_by":8,"title":"","display":"","copyAsset":false,"role":"supplement","size":565047,"visible":true,"origin":"","legend":"","description":"","filename":"SM8.TranslationLatencyandEstimatedcostper10minuteconversation.docx","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/17e47e5ed0b475cb22341443.docx"},{"id":97487779,"identity":"d0f83765-c534-4767-ad9b-04add57463bd","added_by":"auto","created_at":"2025-12-05 01:52:58","extension":"docx","order_by":9,"title":"","display":"","copyAsset":false,"role":"supplement","size":960334,"visible":true,"origin":"","legend":"","description":"","filename":"Tables.docx","url":"https://assets-eu.researchsquare.com/files/rs-8205991/v1/8464d79242fb67491539b56d.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Evaluating LingualAI: A Prospective Validation of AI-Based Real- Time Translation Against Certified Human Interpreters","fulltext":[{"header":"INTRODUCTION","content":"\u003cp\u003eEffective communication is fundamental to safe and equitable healthcare. In countries with substantial linguistic diversity, such as the United States, a growing proportion of clinical encounters involve patients with limited English proficiency (LEP), estimated at approximately 25\u0026nbsp;million people, or about 8% of the population aged 5 years and older who report speaking English less than \u0026ldquo;very well\u0026rdquo;. (\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e) LEP patients face barriers to understanding diagnoses, treatment instructions, and follow-up care, contributing to disparities in safety, quality, and health outcomes. (\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e) A recent systematic review found that when LEP patients receive language-concordant care, clinical outcomes are improved in most settings. (\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e) Although professional medical interpreters are the standard of care, access is often limited, particularly in primary care, rural settings, and time-sensitive encounters, leaving many patients without reliable language support. (\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e)\u003c/p\u003e\u003cp\u003eRecent advances in artificial intelligence (AI) have enabled speech- and language-model\u0026ndash;based systems capable of real-time multilingual translation. (\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e) Several mobile or app-based tools are now available at the point of care; yet systematic evidence regarding their performance, accuracy, and clinical appropriateness remains scarce. In particular, little is known about how AI-based translations compare with certified human interpreters when assessed against clinical communication standards such as terminology accuracy, adequacy of meaning, cultural appropriateness, and speech fluency. Notably, in a recent study evaluating three commercially available applications for bidirectional clinician\u0026ndash;patient dialogue, none were found to be suitable for safe two-way clinical communication when compared with professional interpreters. (\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e) This evidence gap complicates safe integration into health systems.\u003c/p\u003e\u003cp\u003eThe LingualAI application was created to reduce language barriers in clinical settings through real-time voice translation. Developed at UTHealth Houston, its purpose it help clinical team members and patients communicate seamlessly through a continuous flow of voice capture, transcription, translation, and synthesis. Its interface is designed to be intuitive: users can select patient and clinician languages, record conversations linked to each patient, and view interactive bubbles on screen with real-time transcription and translation. In addition, the app offers translation quality feedback, conversation history, secure device synchronization, and an offline mode that ensures reliability even in low-connectivity environments. However, its effectiveness relative to certified interpreters has not been formally evaluated.\u003c/p\u003e\u003cp\u003eWe conducted a prospective, within-subject comparative study using scripted bilingual dialogues to evaluate LingualAI\u0026rsquo;s performance in English\u0026ndash;Spanish interactions in an outpatient otorhinolaryngology setting. We assess whether LingualAI\u0026rsquo;s audio translations are non-inferior to certified human interpreters across multiple domains of translation quality, including terminology accuracy, adequacy of meaning, clarity/fluency, cultural appropriateness, and overall confidence for clinical use. By applying a structured validation framework, this work provides early evidence on the role of AI-based translation tools in supporting equitable, multilingual healthcare delivery.\u003c/p\u003e"},{"header":"METHODS","content":"\u003cp\u003e\u003c/p\u003e\u003cp\u003eWe conducted a prospective, within-subject comparison of translations generated by the LingualAI application versus certified human interpreters (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). Two translation directions were evaluated: English \u0026rarr; Spanish (clinician utterances) and Spanish \u0026rarr; English (patient utterances). Each input phrase was translated by both arms, enabling paired comparisons with evaluators blinded to translation source but not scenario context. Similar prospective, paired or randomized designs have been used in prior translation and interpreter studies that have compared Machine or AI Translation with human translators and bilingual clinical evaluators (\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e).\u003c/p\u003e\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e\u003ch2\u003eClinical scenarios and audio collection\u003c/h2\u003e\u003cp\u003eThree standardized otorhinolaryngology scenarios were developed to represent typical care interactions. Together, the scripts included 18 dialogue paragraphs (33 lines) covering history-taking, clinical instructions, and empathic statements. Each scenario was enacted by two pairs of native English and Spanish speakers to reduce bias from voice quality or pronunciation differences. As a result, each phrase was recorded twice, yielding two independent input sets. Use of standardized scripted encounters with native speakers mirrors prior interpreter research and simulation-based translation evaluations (\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e, \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e). The full text of the three standardized scenarios is provided in Supplementary Material SM1.\u003c/p\u003e\u003c/div\u003e\n\u003ch3\u003eTranslation pipelines\u003c/h3\u003e\n\u003cp\u003eIn the AI arm, recordings were processed through LingualAI, which integrates automatic speech recognition (ASR), neural machine translation (NMT), and text-to-speech synthesis.\u003c/p\u003e\u003cp\u003eEach phrase was submitted twice using the two independent source recordings, generating two AI-translated audio files per phrase in each direction. Outputs were pooled and randomized in blocks of 2 using a fixed random seed to ensure reproducibility, and evaluators were blinded to the translation source (AI vs. human). They were, however, provided with scenario context, line order, and speaker role (clinician vs. patient) so that ratings reflected the intended clinical meaning. Modular pipelines combining ASR, neural MT, and synthesis are well described in biomedical translation research (\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e, \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e).\u003c/p\u003e\u003cp\u003eLingualAI\u0026rsquo;s pipeline operates in five real-time stages: (i) voice capture, ii) text conversion via ASR, (iii) refinement using prior dialogue and domain-specific vocabulary, (iv) translation into the target language, and (v) speech synthesis for natural-sounding delivery. Each stage is designed for real-time performance with fallback strategies in case of network interruptions. The most recent update includes support for custom vocabularies, allowing clinicians to register specialized or uncommon medical terms to improve transcription and translation accuracy in clinical contexts. Comparable staged architectures with domain adaptation have been reported for ambient clinical transcription and multilingual MT systems (\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e).\u003c/p\u003e\u003cp\u003eIn the human arm, two certified medical interpreters independently translated the scenarios, producing two human-translated audio files per phrase. Outputs were similarly pooled and randomized to reduce interpreter-specific bias. All translations were recorded under uniform audio conditions.\u003c/p\u003e\n\u003ch3\u003eEvaluators\u003c/h3\u003e\n\u003cp\u003eNine fluent bilingual clinicians (English\u0026ndash;Spanish) served as evaluators. Participation in the evaluation was voluntary and completed outside protected clinical time. Evaluators submitted ratings anonymously; unique rater identifiers were not collected. Audio clips were randomized and presented blinded to translation source (AI vs. human), but with scenario context provided to support intended meaning. Although blinding was applied, it is possible that evaluators could have inferred AI outputs due to the use of the same synthetic voice and a slightly mechanistic tone compared with human translations. We did not conduct a calibration round, as the study was designed to capture first-impression ratings rather than consensus. All evaluators provided Consent to Participate prior to beginning the study procedures. Blinded bilingual rater designs have been applied in comparable translation and medical communication studies (\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e).\u003c/p\u003e\n\u003ch3\u003eRating instrument\u003c/h3\u003e\n\u003cp\u003eTranslation quality was assessed across 12 domains: (i) primary: accuracy of meaning, terminology accuracy; (ii) secondary: completeness, cultural appropriateness, grammar, vocabulary; (iii) voice-related: fluency, clarity, prosody, pacing; and (iv) conclusive: overall quality and clinical confidence. Ratings used 5-point Likert scales (1\u0026thinsp;=\u0026thinsp;poor, 5\u0026thinsp;=\u0026thinsp;excellent), and evaluators were provided with a guide containing definitions and examples. Evaluators rated each clip independently using Google Forms, a secure, web-based platform containing domain definitions and illustrative examples. Similar multidomain rubrics have been applied in medical machine translation and LLM evaluation studies (\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e). Established frameworks such as the Multidimensional Quality Metrics (MQM) cover accuracy, fluency, terminology, and adequacy, with error taxonomies also informing clarity, grammar, and overall quality (\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e). In line with broader recommendations, our rubric emphasized context awareness, distinguishing accuracy from fluency, and incorporating completeness, cultural appropriateness, and clinical relevance into evaluation. (\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e, \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e). Additionally, because of this, we did not conduct a calibration round, as the study was designed to capture first-impression ratings rather than consensus. The complete rating instrument and evaluator instructions are provided in Supplementary Material SM2.\u003c/p\u003e\u003cdiv id=\"Sec7\" class=\"Section2\"\u003e\u003ch2\u003eStatistical analysis\u003c/h2\u003e\u003cp\u003eNormality of scores was assessed using the Shapiro\u0026ndash;Wilk test. For normally distributed outcomes, paired t-tests were applied; for non-normal distributions, Wilcoxon signed-rank tests were used. Effect sizes were reported as Cohen\u0026rsquo;s \u003cem\u003ed\u003c/em\u003e (t-test) or correlation coefficient (\u003cem\u003er\u003c/em\u003e) (Wilcoxon). Inter-rater reliability was estimated using Krippendorff\u0026rsquo;s α, which accommodates missing data and unequal numbers of raters per item, a necessity given that evaluators submitted ratings anonymously without unique identifiers. Mixed-effects linear models with random intercepts for item (line) were applied to account for repeated ratings of the same line across multiple evaluators.\u003c/p\u003e\u003cp\u003eA non-inferiority margin of 0.30 points on the 5-point scale was prespecified. This margin was chosen a priori based on three considerations: (i) the half-SD rule commonly applied to Likert ratings, where typical standard deviations of ~\u0026thinsp;0.5\u0026ndash;0.7 correspond to a minimal meaningful difference of ~\u0026thinsp;0.25\u0026ndash;0.35; (ii) prior studies that used similar small margins for clinically relevant differences on 5-point rating scales (\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e); and (iii) expert consensus among bilingual clinician-evaluators that a 0.30 shift would begin to alter clinical confidence without changing literal meaning. For each domain, one-sided 95% confidence intervals (CIs) were calculated for the mean difference between human and AI ratings. LingualAI was considered non-inferior if the upper bound of the CI was less than the margin. All analyses were conducted in Python (Ver 3.12.5 using Pandas, Numpy, StatsModels and SciPy libraries) and R (Ver 4.4.2 using ggplot2, dplyr, tibble libraries), with statistical significance set at two-sided p\u0026thinsp;\u0026lt;\u0026thinsp;0.05. This non-inferiority framework is consistent with prior comparative translation studies (\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e, \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e).\u003c/p\u003e\u003cp\u003eFor a priori power, we planned for 33 lines, each rated by ~\u0026thinsp;7 bilingual clinicians (\u0026asymp;\u0026thinsp;231 ratings). Because ratings on the same line are correlated, we adjusted for clustering using an assumed intraclass correlation (ICC) of 0.25, giving an effective sample size of \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{n}_{eff}=\\:\\frac{33\\times\\:7}{1+(7-1)\\times\\:0.25}\\:\\approx\\:92\\)\u003c/span\u003e\u003c/span\u003e. With a one-sided non-inferiority test (α\u0026thinsp;=\u0026thinsp;0.025), a prespecified margin of Δ\u0026thinsp;=\u0026thinsp;0.30, and an expected true mean difference of 0.10, the projected power was ~89% (Supplementary Material SM3). Sensitivity checks varying ICC (0.20\u0026ndash;0.30) and score variability (SD 0.60\u0026ndash;0.70) yielded power between ~74% and 93%, supporting the adequacy of the planned sample size.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec8\" class=\"Section2\"\u003e\u003ch2\u003eExploratory System Metrics\u003c/h2\u003e\u003cp\u003eIn addition to translation quality, we recorded system-level performance metrics, including end-to-end translation latency and estimated operational cost. Latency was computed from system logs as the total time between audio input and synthesized output, encompassing automatic speech recognition, neural translation, and text-to-speech synthesis. Operating cost was estimated based on API usage and computational overhead for a 10-minute bidirectional conversation, using unit prices of deployed components. These exploratory analyses aimed to contextualize LingualAI\u0026rsquo;s technical efficiency and cost-effectiveness relative to conventional interpreter services.\u003c/p\u003e\u003c/div\u003e"},{"header":"RESULTS","content":"\u003cp\u003eResults are organized in three analytical layers. Differences are reported as \u0026Delta; = Human \u0026minus; AI (positive values favor human), with a prespecified non-inferiority margin of 0.30 points on the 5-point scale. Analyses include both paired and mixed-effects models to account for evaluator clustering and confirm robustness. Findings are presented as follows: (i) evaluator participation and reliability, (ii) domain-level performance evaluated using non-inferiority testing and direction-specific contrasts, and (iii) secondary analyses, including preference patterns, robustness checks, and representative error types.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEvaluator and Data Overview\u003cbr\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNine bilingual clinicians contributed ratings across three standardized scenarios comprising 18 clinician and 15 patient lines. Evaluator participation varied by scenario (8, 7, and 4 raters for Scenarios 1\u0026ndash;3, respectively), resulting in an unbalanced rater\u0026ndash;item panel. Participation declined across scenarios due to evaluator fatigue and the substantial time required, leading to fewer ratings in Scenario 3 (Table 1). Because raters were anonymous, inter-rater reliability was estimated using Krippendorff\u0026rsquo;s \u0026alpha;, which accommodates missing data and unequal numbers of raters per item. Across all domains, \u0026alpha; = 0.31, reflecting fair agreement. This modest reliability likely reflects our design choice to capture first-impression ratings rather than consensus. Domain-specific \u0026alpha; values are shown in Table 2. Score distributions were approximately normal except for fluency and prosody, which violated Shapiro\u0026ndash;Wilk assumptions and were analyzed with non-parametric tests.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePrimary Analyses (Fixed and Mixed Effects)\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eResults from paired and mixed-effects models were largely consistent (Table 3). Adjusting for evaluator clustering did not alter the direction or significance of findings, confirming the robustness of observed differences between human and AI translations.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePrimary factors:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAdequacy of meaning (human 4.82 vs. AI 4.70; 95% CI, 0.00 to 0.24; p = 0.050) and terminology accuracy (4.81 vs. 4.75; CI, \u0026minus;0.05 to 0.19; p = 0.282) remained statistically comparable across both models, indicating semantic equivalence between human and AI outputs. In contrast, clarity/intelligibility showed a consistent and large human advantage (4.88 vs. 4.36; p \u0026lt; 0.001; r = 0.76).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSecondary factors:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eHuman interpreters achieved higher scores in completeness (4.87 vs. 4.73; p = 0.022; 95% CI, 0.02\u0026ndash;0.26), grammar/syntax (4.88 vs. 4.67; p = 0.007; CI, 0.09\u0026ndash;0.32), vocabulary (4.85 vs. 4.67; p = 0.060; CI, 0.07\u0026ndash;0.29), and cultural appropriateness (4.89 vs. 4.49; p \u0026lt; 0.001; CI, 0.27\u0026ndash;0.52). Effect sizes ranged from moderate for vocabulary to large for grammar/syntax and cultural appropriateness.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eVoice-quality factors:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe most pronounced differences were observed in fluency (4.86 vs. 3.72; p \u0026lt; 0.001; CI, 0.97\u0026ndash;1.31), prosody (4.89 vs. 4.30; p \u0026lt; 0.001; CI, 0.47\u0026ndash;0.72), and pacing (4.88 vs. 4.46; p = 0.002; CI, 0.31\u0026ndash;0.53), all demonstrating large, statistically robust advantages for human interpreters.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusive factors:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eOverall quality (4.81 vs. 4.23; p \u0026lt; 0.001; CI, 0.44\u0026ndash;0.72) and confidence for clinical use (4.82 vs. 4.22; p \u0026lt; 0.001; CI, 0.47\u0026ndash;0.75) were also significantly higher for human translations.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDirection-specific contrasts:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eStratified analyses confirmed these trends (Supplementary Material\u0026nbsp;\u003ca href=\"https://docs.google.com/document/d/1Dx4waGrBqfCM5BsM2pPDKjUzGJ6k99myS3EJhsMmB9U/edit?usp=sharing\"\u003eSM4\u003c/a\u003e). For clinician statements (English \u0026rarr; Spanish), human interpreters outperformed the AI system in clarity, fluency, prosody, pacing, overall quality, and clinical confidence, while terminology and adequacy of meaning were equivalent. For patient statements (Spanish \u0026rarr; English), human interpreters again scored higher across delivery-related and linguistic-mechanics domains (grammar, vocabulary, prosody), whereas accuracy and meaning remained non-inferior.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eNon-Inferiority Testing\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eA prespecified non-inferiority margin of 0.30 points on the 5-point scale was applied (Fig. 3). Only a subset of domains met this threshold, indicating that while LingualAI preserved semantic accuracy, delivery-related aspects remained inferior to human interpreters.\u003c/p\u003e\n\u003cp\u003ePrimary factors: Terminology accuracy (mean difference 0.07; 95% CI, 0.18) and adequacy of meaning (0.13; CI, 0.23) satisfied the non-inferiority criterion, confirming comparable semantic performance. Clarity exceeded the margin (0.50; CI, 0.72), reflecting a perceptible decline in intelligibility relative to human translation.\u003c/p\u003e\n\u003cp\u003eSecondary factors: Completeness (0.14; CI, 0.25) met the criterion, but vocabulary (0.18; CI, 0.32), grammar (0.21; CI, 0.34), and cultural appropriateness (0.39; CI, 0.64) exceeded the margin, suggesting subtle but consistent linguistic differences.\u003c/p\u003e\n\u003cp\u003eVoice-quality factors: None met the non-inferiority threshold, differences were largest for fluency (1.13; CI, 1.49), followed by prosody (0.59; CI, 0.81) and pacing (0.40; CI, 0.55), indicating that human interpreters retained a clear advantage in speech naturalness and rhythm.\u003c/p\u003e\n\u003cp\u003eConclusive factors: Both overall quality (0.58; CI, 0.81) and confidence for clinical use (0.61; CI, 0.84) exceeded the margin, underscoring the evaluators\u0026rsquo; preference and higher perceived reliability for human translations. The mean differences and 95% confidence intervals for all domains are visualized in Table 4 and Figure 3, plotted against the prespecified non-inferiority boundary.\u003c/p\u003e\n\u003cp\u003eWhen a stricter non-inferiority margin of 0.20 was applied, limited to Scenarios 1\u0026ndash;2 due to smaller sample size in Scenario 3, terminology accuracy remained non-inferior while all other domains exceeded the margin. Detailed results are provided in Supplementary Material SM5.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePreference Analysis\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003ePairwise preference testing compared individual evaluator choices between human and AI audio clips for each domain (Table 5). Overall, preference patterns closely mirrored the quantitative scoring results: evaluators perceived both systems as equivalent for meaning and terminology but favored human interpreters for delivery quality and naturalness.\u003c/p\u003e\n\u003cp\u003ePrimary factors: Terminology accuracy (tie rate, 66%) and adequacy of meaning (65%) showed high equivalence, with minimal net bias toward human translations. Clarity revealed a stronger human preference (40% human wins vs. 4% AI wins; net bias of 0.36).\u003c/p\u003e\n\u003cp\u003eSecondary factors: Completeness also demonstrated a high tie rate (73%), whereas grammar (29% vs. 8%; net bias of 0.21) and cultural appropriateness (31% vs. 8%; net bias of 0.23) favored human interpreters. Vocabulary showed a smaller but consistent human advantage (net bias of 0.12).\u003c/p\u003e\n\u003cp\u003eVoice-quality factors: Preferences were most pronounced in these domains. Fluency had the lowest tie rate (25%) and the highest human win rate (67%; net bias of 0.59), followed by pacing and prosody, both exhibiting substantial net biases (\u0026gt;0.38).\u003c/p\u003e\n\u003cp\u003eConclusive factors: For overall quality and confidence for clinical use, human translations won in roughly half of all comparisons (~49%), with average net biases near 0.39. These findings emphasize that, while evaluators recognized comparable semantic accuracy, they consistently preferred the tone, rhythm, and expressiveness of human interpretations. Detailed tie rates for each domain can be found in Supplementary Material SM6.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eError Analysis\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eQualitative review of evaluator comments and audio outputs revealed that most residual errors arose from \u003cem\u003edelivery\u003c/em\u003e rather than \u003cem\u003emeaning\u003c/em\u003e. Specifically, deviations in \u003cem\u003efluency\u003c/em\u003e and \u003cem\u003eprosody\u003c/em\u003e, such as monotone intonation, abrupt phrasing, and irregular pacing occasionally disrupted conversational flow, even when lexical and semantic accuracy were preserved.\u003c/p\u003e\n\u003cp\u003eThese patterns were observed across all scenarios. For instance, certain clinician prompts (e.g., \u0026ldquo;Are you still doing the Budesonide irrigations?\u0026rdquo;) exhibited misplaced pauses or unnatural emphasis, while reassurance statements (e.g., \u0026ldquo;That\u0026rsquo;s expected. The swelling should go down over the next few weeks.\u0026rdquo;) were rendered with rigid or mechanical rhythm. Importantly, semantic fidelity remained high, confirming that LingualAI\u0026rsquo;s primary limitation lies in \u003cem\u003espeech synthesis naturalness\u003c/em\u003e rather than translation accuracy.\u003c/p\u003e\n\u003cp\u003eRepresentative examples with linked audio clips are provided in Supplementary Material \u003ca href=\"https://docs.google.com/document/d/11I1gsV23mSFwNtDJNBUnclyB-IqW0iVHD1e4eEyKmaY/edit?usp=sharing\"\u003eSM7\u003c/a\u003e, illustrating how subtle deviations in tone and pacing can alter perceived empathy and conversational realism.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTranslation Latency and Cost\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eBased on system log analyses, LingualAI achieved an average end-to-end translation latency of 9.7 seconds per message, encompassing automatic speech recognition, neural translation, and speech synthesis. This turnaround time falls within the typical conversational pause window, allowing smooth, turn-based clinical dialogue without noticeable interruption.\u003c/p\u003e\n\u003cp\u003eThe estimated operating cost for a 10-minute bilingual conversation was approximately $0.03\u0026ndash;$0.04 (USD), compared with $6.90\u0026ndash;$10.60 for phone or video interpreter services, representing a cost reduction exceeding 99% per session. These results highlight LingualAI\u0026rsquo;s potential for scalable, low-cost deployment in time-sensitive or resource-limited clinical settings.\u003c/p\u003e\n\u003cp\u003eDetailed latency distributions and cost-calculation parameters are provided in Supplementary Material SM8.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSummary of Findings\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eOverall, LingualAI preserved semantic accuracy at levels comparable to certified interpreters but showed limitations in delivery quality and expressiveness. Among the primary factors, both terminology accuracy and adequacy of meaning met the non-inferiority thresholds, confirming that the AI system conveyed medical content reliably.\u003c/p\u003e\n\u003cp\u003eWithin the secondary factors, only completeness met the non-inferiority criterion, while grammar, vocabulary, and cultural appropriateness favored human interpreters. Notably, differences in vocabulary and cultural appropriateness reached significance primarily in patient statements (Spanish \u0026rarr; English).\u003c/p\u003e\n\u003cp\u003eAll voice-quality factors \u0026ndash; fluency, prosody, and pacing showed large and consistent human advantages, underscoring the gap in naturalness and conversational rhythm.\u003c/p\u003e\n\u003cp\u003eFinally, both conclusive factors \u0026ndash; overall quality and confidence for clinical use were significantly higher for human translations. Together, these findings suggest that while LingualAI effectively preserves meaning and accuracy, its synthesized speech still lacks the nuance and expressivity essential for fully natural clinical communication.\u003c/p\u003e"},{"header":"DISCUSSION","content":"\u003cdiv id=\"Sec23\" class=\"Section2\"\u003e\u003ch2\u003eKey Findings\u003c/h2\u003e\u003cp\u003eIn this prospective, within-subject comparison of an AI-based translation tool (LingualAI) with certified medical interpreters, we found that LingualAI preserved two primary factors, terminology accuracy and adequacy of meaning, at levels that met non-inferiority thresholds, ensuring reliable conveyance of core medical content. LingualAI also achieved non-inferiority for completeness, supporting its ability to maintain the overall flow of dialogue. However, human interpreters consistently outperformed LingualAI in various secondary and voice-related factors (grammar, vocabulary, cultural appropriateness, fluency, prosody, and pacing), underscoring the continued importance of delivery quality on clinical communications. These findings suggest that while semantic integrity was maintained, elements crucial for patient trust and nuanced interaction remain stronger with professional interpreters. Importantly, the preservation of meaning and terminology, the dimensions most directly linked to clinical safety suggests that LingualAI can reliably convey essential medical information, even if its spoken delivery remains less natural than human interpreters.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec24\" class=\"Section2\"\u003e\u003ch2\u003eClinical Implications\u003c/h2\u003e\u003cp\u003eThe findings indicate that AI-based translation tools such as LingualAI are beginning to demonstrate credible performance in the critical clinical safety dimensions, preserving meaning and terminology accuracy. These strengths suggest a role for AI in supporting basic comprehension and instruction delivery, particularly in settings where interpreter services are unavailable, such as urgent encounters, rural clinics, or after-hours consultations. In such scenarios, LingualAI may be an alternative to delaying care or rendering care without translation. From a practical standpoint, accuracy in conveying clinical meaning is often of higher value than perfectly natural prosody, especially in urgent or resource-limited settings. While human interpreters provide richer tone and empathy, LingualAI\u0026rsquo;s consistent accuracy and ultra-low operational cost may make it a viable adjunct when access or affordability is constrained.\u003c/p\u003e\u003cp\u003eMoving forward, it will be essential to work closely with certified medical translation bodies, since the acceptability of AI-supported translation depends on alignment with professional standards. The objective is not to replace certified medical translators but to augment their role and provide coverage in situations where they are absent. In practice, this could also include hybrid workflows where certified translators remotely monitor AI-facilitated conversations, ensuring they remain in the loop and can intervene when needed. AI translation should be viewed as an adjunct that extends access while maintaining the primacy of professional interpreters to ensure safe, nuanced, and culturally appropriate communication.\u003c/p\u003e\u003cp\u003eThe baseline capabilities of AI translation models are expected to continue improving, which may steadily raise overall performance on core tasks. At the same time, the inherent risk of errors could also increase as these systems are deployed more widely and in higher-stakes settings. The broader deployment of these systems in critical settings highlights the importance of continuously monitoring primary factors such as meaning and terminology accuracy, alongside steady technological improvements in secondary and voice-related factors. To ensure safe and equitable integration, such progress should be documented through well-structured, real-world studies that evaluate not only technical performance but also clinical usability and patient experience.\u003c/p\u003e\u003cp\u003eUnder Section 1557 of the Affordable Care Act and related regulations and guidance from the Office for Civil Rights (OCR), AI-based translation tools may be used to support language access in healthcare, but they cannot serve as the sole method for providing meaningful access to individuals with limited English proficiency (\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e, \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e). OCR\u0026rsquo;s regulatory framework generally requires that most substantive clinical communications such as informed consent, treatment instructions, and discharge summaries be reviewed or provided by qualified human interpreters or translators. While AI tools may improve efficiency and accuracy in medical terminology, they must be integrated into workflows that include human oversight to ensure compliance with nondiscrimination standards.\u003c/p\u003e\u003cdiv id=\"Sec25\" class=\"Section3\"\u003e\u003ch2\u003eComparison to literature\u003c/h2\u003e\u003cp\u003eOur findings are consistent with prior evaluations of general-purpose MT tools, which often demonstrate strengths in medical terminology and adequacy of meaning, but show persistent weakness in contextual interpretation, cultural appropriateness, fluency, and other delivery-related aspects. For example, studies report that tools such as Google Translate perform well in terms of adequacy of meaning and basic clinical content, but often struggle with cultural context and have dialect sensitivity issues (\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e, \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e). Work on LLMs has further highlighted the limitations of literal translation approaches that miss nuances essential for medical communication (\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e). Evidence also suggests that AI-based translations lack the cultural awareness to support trust and rapport in diverse patient populations, potentially leading to misunderstandings (\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e). While some studies indicate that AI can generate fluent output, syntactic and semantic inconsistencies remain common, particularly in languages with complex grammatical structures (29).\u003c/p\u003e\u003cp\u003eOur results extend this body of evidence by demonstrating that even a purpose-built, domain-specific system like LingualAI, optimized for clinical communication, continues to mirror these patterns. It performed comparably to certified interpreters on meaning and terminology accuracy but lagged in prosody, fluency, and clinical confidence. Unlike prior studies, however, our evaluation used a prospective, within-subject design with blinded bilingual clinician raters, multidomain scoring, and a non-inferiority framework. This methodological rigor strengthens confidence in the findings and provides a more granular understanding of where AI translation tools may safely support clinical communication and where human oversight remains indispensable.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec26\" class=\"Section3\"\u003e\u003ch2\u003eStrengths \u0026amp; Limitations\u003c/h2\u003e\u003cp\u003eThis study has several strengths. The prospective, within-subject paired design ensured direct comparability between AI and human translators, while blinded evaluation by bilingual clinicians reduced bias. A structured, multidomain rubric and a pre-specified non-inferiority framework provided a rigorous translation quality assessment across clinically relevant dimensions. In addition, combining paired statistical tests and mixed-effects modeling strengthened the robustness of findings and enabled a nuanced interpretation of where AI tools achieved equivalence and where differences persisted.\u003c/p\u003e\u003cp\u003eSeveral limitations should be acknowledged. The scenarios were scripted and restricted to otorhinolaryngology, which may not capture the variability of free-flowing clinical dialogue, including patient emotions, interruptions, or background noise. The study evaluated audio translations only, without considering LingualAI\u0026rsquo;s text-based interface, which could also influence comprehension. The evaluation occurred in a non-live setting, outside of real patient encounters, which may not fully represent the pressures and dynamics of clinical care.\u003c/p\u003e\u003cp\u003eEvaluator-related factors add further constraints. The clinical evaluators were anonymized, and an unequal number of ratings were available across scenarios, leading to an unbalanced dataset. Inter-rater reliability was modest (Krippendorff\u0026rsquo;s α\u0026thinsp;=\u0026thinsp;0.31), and no calibration was performed to align evaluators before the study; this was an intentional decision to capture first impressions rather than consensus, but one that limited agreement. Additionally, the raters were clinicians rather than patients, meaning domains such as fluency, rapport, or cultural appropriateness may not reflect patient experience directly.\u003c/p\u003e\u003cp\u003eFinally, the findings were restricted to English\u0026ndash;Spanish translation, and results may not generalize to other language pairs or dialects. These limitations should be considered when interpreting the scope of our results.\u003c/p\u003e\u003c/div\u003e\u003cdiv id=\"Sec27\" class=\"Section3\"\u003e\u003ch2\u003eFuture Directions\u003c/h2\u003e\u003cp\u003eFuture development of LingualAI should prioritize enhancements in speech synthesis, particularly prosody, pacing, and fluency, which accounted for the most significant performance gaps compared with certified interpreters. Beyond these technical refinements, validation in live, unscripted clinical encounters will be essential to test performance under real-world conditions such as interruptions, emotional tone, and background noise. Incorporating patient perspectives and expanding evaluation to additional languages and dialects will strengthen generalizability. As language models and translation technologies evolve, monitoring new error models and working closely with certified medical translation bodies to ensure alignment with professional standards will be critical. Therefore, well-structured, real-world studies should assess accuracy, delivery quality, patient-centered outcomes, and clinical usability, aiming to develop evidence-based, reliable, and trusted AI translation tools that extend access to multilingual care while preserving the central role of professional interpreters. Another important consideration is consistency. While human interpreter quality can vary substantially across sessions and providers, automated systems like LingualAI offer the potential for stable, repeatable performance once adequately validated. Future improvements in voice synthesis, particularly expressiveness and conversational pacing, will further enhance user experience and make AI-assisted translation increasingly compelling in real-world use.\u003c/p\u003e\u003c/div\u003e\u003c/div\u003e"},{"header":"Conclusion","content":"\u003cp\u003eLingualAI demonstrated non-inferior performance in preserving meaning and terminology \u0026ndash; the core dimensions of translation accuracy while certified interpreters remained superior in delivery-related qualities such as fluency, prosody, and clinical confidence. These findings suggest that AI translation tools can safely support essential communication when interpreters are unavailable, provided their use remains aligned with professional standards. Consistency and scalability may further enhance their utility, particularly as voice synthesis continues to improve. In practice, the most responsible pathway forward is an interpreter-in-the-loop model, in which professional interpreters oversee or supplement AI-mediated communication to ensure contextual accuracy and patient trust. Professional interpreters therefore remain the gold standard, yet AI systems like LingualAI can serve as valuable supplementary tools that extend access to multilingual care. Continued technical refinement and clinical validation will be essential for ensuring safe, effective, and trusted deployment in real-world healthcare settings.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003eEthics Approval: This study has been approved by the Committee for the Protection of Human Subjects (the UTHSC-H IRB) under protocol HSC-SBMI- 13-0549.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eACKNOWLEDGEMENTS\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe gratefully acknowledge Drs. German Martinez-Gamba and Amalia Guardiola for their contributions as clinical evaluators. We also thank William B. McElhiney for his valuable guidance on legal and regulatory considerations.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFUNDING\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis work was supported by institutional funds from The University of Texas Health Science Center at Houston (UTHealth Houston) and by discretionary research funds provided by Dr. Xiaoqian Jiang.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDATA AVAILABILITY STATEMENT\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eDe-identified audio clips and raw evaluator rating data underlying this study are available from the corresponding author upon reasonable request. Access will require completion of a data-use agreement and approval by the appropriate Institutional Review Board. Summary-level data (means, standard deviations, and confidence intervals) are included in the main manuscript and Supplementary Materials. Due to file size, PHI redaction requirements, and copyright considerations, raw audio files and individual rating sheets are not publicly posted.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eHUMAN ETHICS AND CONSENT TO PARTICIPATE\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAll clinician evaluators participated on a voluntary basis and provided informed consent before initiating any study-related activities. The study was conducted in accordance with the ethical principles outlined in the Declaration of Helsinki.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eNo patients or members of the public were recruited or enrolled as participants in this study. Certified translators provided translation services as compensated professionals and were not considered study participants.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAUTHOR CONTRIBUTIONS STATEMENT\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eUPS led the study conceptualization, methodology, formal analysis, data curation, visualization, and drafting of the original manuscript. CJG contributed to software development, methodology, data curation, validation, and visualization. GMA, JBG, DRJ, EW, VSC, AFV, and BRE contributed to investigation, validation, and manuscript review. JHC and CW supported project administration, with JHC additionally contributing to manuscript review and editing. BOF contributed resources, investigation, project administration, and manuscript review. MJC provided supervision, conceptual guidance, and manuscript review. XJ provided overall conceptualization, supervision, methodological guidance, and manuscript review, editing and funding support.\u003c/p\u003e\n\u003cp\u003eAll authors reviewed and approved the final manuscript.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eDiamond L, Izquierdo K, Canfield D, Matsoukas K, Gany F. A Systematic Review of the Impact of Patient\u0026ndash;Physician Non-English Language Concordance on Quality of Care and Outcomes. J Gen Intern Med. 2019 Aug;34(8):1591\u0026ndash;606.\u003c/li\u003e\n\u003cli\u003eU.S. Census Bureau. People that speak English less than very well [Internet]. Washington, DC: U.S. Department of Commerce; n.d. Available from: https://www.census.gov/library/visualizations/interactive/people-that-speak-english-less-than-very-well.html\u003c/li\u003e\n\u003cli\u003eTwersky SE, Jefferson R, Garcia-Ortiz L, Williams E, Pina C. The Impact of Limited English Proficiency on Healthcare Access and Outcomes in the U.S.: A Scoping Review. Healthcare. 2024 Jan 31;12(3):364.\u003c/li\u003e\n\u003cli\u003eWilliams KM, Dougherty D, Plagens C, Shah NR, Tubbs D, Ehrlich PF. Limited English Proficiency can Negatively Impact Disease/Treatment in Children With Cancer Compared to Those Who are English Proficient-an Institutional Study. J Pediatr Surg. 2024 May;59(5):800\u0026ndash;3.\u003c/li\u003e\n\u003cli\u003eTran AV, Roberts KP. Language Accommodations for Limited English Proficient Patients in Rural Health Care. J Immigr Minor Health. 2023 June;25(3):674\u0026ndash;9.\u003c/li\u003e\n\u003cli\u003eHabhab M, Pham MT. Medical Interpretation Services: Challenges for LEP Communities. Am J Bioeth. 2024 Nov;24(11):72\u0026ndash;4.\u003c/li\u003e\n\u003cli\u003eHudelson P, Chappuis F. Using Voice-to-Voice Machine Translation to Overcome Language Barriers in Clinical Communication: An Exploratory Study. J Gen Intern Med. 2024 May;39(7):1095\u0026ndash;102.\u003c/li\u003e\n\u003cli\u003eGenovese A, Borna S, Gomez-Cabello CA, Haider SA, Prabha S, Forte AJ, et al. Artificial intelligence in clinical settings: a systematic review of its role in language translation and interpretation. Ann Transl Med. 2024 Dec;12(6):117\u0026ndash;117.\u003c/li\u003e\n\u003cli\u003eLee W, Khoong EC, Zeng B, Rios-Fetchko F, Ma Y, Liu K, et al. Evaluation of Commercially Available Machine Interpretation Applications for Simple Clinical Communication. J Gen Intern Med. 2023 Aug;38(10):2333\u0026ndash;9.\u003c/li\u003e\n\u003cli\u003eKong M, Fernandez A, Bains J, Milisavljevic A, Brooks KC, Shanmugam A, et al. Evaluation of the accuracy and safety of machine translation of patient-specific discharge instructions: a comparative analysis. BMJ Qual Saf. 2025 July 9;bmjqs-2024-018384.\u003c/li\u003e\n\u003cli\u003eTaira BR, Kreger V, Orue A, Diamond LC. A Pragmatic Assessment of Google Translate for Emergency Department Instructions. J Gen Intern Med. 2021 Nov;36(11):3361\u0026ndash;5.\u003c/li\u003e\n\u003cli\u003eHerrmann-Werner A, Loda T, Zipfel S, Holderried M, Holderried F, Erschens R. Evaluation of a Language Translation App in an Undergraduate Medical Communication Course: Proof-of-Concept and Usability Study. JMIR MHealth UHealth. 2021 Dec 2;9(12):e31559.\u003c/li\u003e\n\u003cli\u003ePinto Taylor E, Mulenos A, Chatterjee A, Talwalkar JS. Partnering With Interpreter Services: Standardized Patient Cases to Improve Communication With Limited English Proficiency Patients. MedEdPORTAL. 2019 May 20;10826.\u003c/li\u003e\n\u003cli\u003eHan L, Gladkoff S, Erofeev G, Sorokina I, Galiano B, Nenadic G. Neural machine translation of clinical text: an empirical investigation into multilingual pre-trained language models and transfer-learning. Front Digit Health. 2024 Feb 26;6:1211564.\u003c/li\u003e\n\u003cli\u003eZolnoori M, Vergez S, Xu Z, Esmaeili E, Zolnour A, Anne Briggs K, et al. Decoding disparities: evaluating automatic speech recognition system performance in transcribing Black and White patient verbal communication with nurses in home healthcare. JAMIA Open. 2024 Oct 8;7(4):ooae130.\u003c/li\u003e\n\u003cli\u003eChen CL, Dong Y, Castillo-Zambrano C, Bencheqroun H, Barwise A, Hoffman A, et al. A systematic multimodal assessment of AI machine translation tools for enhancing access to critical care education internationally. BMC Med Educ. 2025 July 8;25(1):1022.\u003c/li\u003e\n\u003cli\u003eFreitag M, Foster G, Grangier D, Ratnakar V, Tan Q, Macherey W. Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation. Trans Assoc Comput Linguist. 2021 Dec 17;9:1460\u0026ndash;74.\u003c/li\u003e\n\u003cli\u003eL\u0026auml;ubli S, Castilho S, Neubig G, Sennrich R, Shen Q, Toral A. A Set of Recommendations for Assessing Human-Machine Parity in Language Translation. J Artif Intell Res [Internet]. 2020 Mar 23 [cited 2025 Aug 29];67. Available from: http://arxiv.org/abs/2004.01694\u003c/li\u003e\n\u003cli\u003eLiu T, Lo C kiu, Marshman E, Knowles R. Evaluation Briefs: Drawing on Translation Studies for Human Evaluation of MT. 2024;\u003c/li\u003e\n\u003cli\u003eGerlinger C, Schmelter T. Determining the non‐inferiority margin for patient reported outcomes. Pharm Stat. 2011 Sept;10(5):410\u0026ndash;3.\u003c/li\u003e\n\u003cli\u003eDo Campo Bay\u0026oacute;n M, S\u0026aacute;nchez-Gij\u0026oacute;n P. Evaluating NMT using the non-inferiority principle. Nat Lang Process. 2025 July;31(4):1042\u0026ndash;61.\u003c/li\u003e\n\u003cli\u003eU.S. Department of Health \u0026amp; Human Services, Office for Civil Rights. Dear Colleague Letter: Language Access Provisions of the Final Rule Implementing Section 1557 of the Affordable Care Act [Internet]. 2024 Dec. Available from: https://www.hhs.gov/sites/default/files/ocr-dcl-section-1557-language-access.pdf\u003c/li\u003e\n\u003cli\u003eNational Health Law Program. Title VI and Section 1557 Language Access Requirements [Internet]. Washington, DC; 2024 May. Available from: https://healthlaw.org/wp-content/uploads/2024/05/T-VI-and-Sec-1557-explainer-2024.pdf\u003c/li\u003e\n\u003cli\u003eBrandenberger J, Stedman I, Stancati N, Sappleton K, Kanathasan S, Fayyaz J, et al. Using artificial intelligence based language interpretation in non-urgent paediatric emergency consultations: a clinical performance test and legal evaluation. BMC Health Serv Res. 2025 Jan 24;25(1):138.\u003c/li\u003e\n\u003cli\u003eBrewster RCL, Gonzalez P, Khazanchi R, Butler A, Selcer R, Chu D, et al. Performance of ChatGPT and Google Translate for Pediatric Discharge Instruction Translation. Pediatrics. 2024 July 1;154(1):e2023065573.\u003c/li\u003e\n\u003cli\u003eYan J, Yan P, Chen Y, Li J, Zhu X, Zhang Y. Benchmarking GPT-4 against Human Translators: A Comprehensive Evaluation Across Languages, Domains, and Expertise Levels [Internet]. arXiv; 2024 [cited 2025 Sept 16]. Available from: http://arxiv.org/abs/2411.13775\u003c/li\u003e\n\u003cli\u003eKeles B, Gunay M, Caglar SI. LLMs-in-the-loop Part-1: Expert Small AI Models for Bio-Medical Text Translation [Internet]. arXiv; 2024 [cited 2025 Sept 16]. Available from: http://arxiv.org/abs/2407.12126\u003c/li\u003e\n\u003cli\u003eMerx R, Phillips C, Suominen H. Machine Translation Technology in Health: A Scoping Review. In: Bichel-Findlay J, editor. Studies in Health Technology and Informatics [Internet]. IOS Press; 2024 [cited 2025 Sept 16]. Available from: https://ebooks.iospress.nl/doi/10.3233/SHTI240895\u003c/li\u003e\n\u003cli\u003eSlosarek T, Paeschke D, Sivtsev I, B\u0026ouml;ttinger EP. Comparison of Machine Translation Services in the Biomedical Context. In: 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) [Internet]. Lisbon, Portugal: IEEE; 2024 [cited 2025 Sept 16]. p. 5539\u0026ndash;45. Available from: https://ieeexplore.ieee.org/document/10822680/\u003c/li\u003e\n\u003c/ol\u003e"},{"header":"Tables","content":"\u003cp\u003eTables 1 to 5 are available in the Supplementary Files section\u003c/p\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"npj-health-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [npj Health Systems](https://www.nature.com/npjhealthsyst/)","snPcode":"44401","submissionUrl":"https://submission.springernature.com/new-submission/44401/3","title":"npj Health Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Artificial intelligence, Medical translation, Clinical communication, Limited English proficiency, Interpreter services, Otorhinolaryngology","lastPublishedDoi":"10.21203/rs.3.rs-8205991/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8205991/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground\u003c/strong\u003e: Limited English proficiency (LEP) affects \u0026gt;25 million people in the United States and is linked to health disparities in safety, quality, and outcomes. While professional interpreters remain the standard, access is often constrained. Real-time AI translation systems are increasingly available, yet their clinical performance relative to certified interpreters is uncertain.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eObjective\u003c/strong\u003e: To evaluate whether an in-house AI application (LingualAI) achieves non-inferior translation quality compared with certified interpreters in English–Spanish otorhinolaryngology encounters.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDesign, Setting, and Participants\u003c/strong\u003e: Prospective, within-subject comparison using three standardized outpatient scenarios (33 lines: 18 clinicians, 15 patients) enacted by two pairs of native speakers. Each line was translated by LingualAI and by two certified medical interpreters. Nine bilingual clinicians, blinded to source but given scenario context, independently rated anonymized audio clips.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMain Measures\u003c/strong\u003e: Twelve domains on 5-point Likert scales: primary (\u003cem\u003eterminology\u003c/em\u003e \u003cem\u003eaccuracy\u003c/em\u003e, \u003cem\u003eadequacy of meaning\u003c/em\u003e), secondary (\u003cem\u003ecompleteness\u003c/em\u003e, \u003cem\u003egrammar\u003c/em\u003e, \u003cem\u003evocabulary\u003c/em\u003e, \u003cem\u003ecultural appropriateness\u003c/em\u003e), voice-related (\u003cem\u003efluency\u003c/em\u003e, \u003cem\u003eclarity\u003c/em\u003e, \u003cem\u003eprosody\u003c/em\u003e, \u003cem\u003epacing\u003c/em\u003e), and conclusive (\u003cem\u003eoverall quality\u003c/em\u003e, \u003cem\u003eclinical confidence\u003c/em\u003e). Non-inferiority margin prespecified at 0.30 points (Δ = Human − AI). Analyses used paired tests and mixed-effects models with random intercepts for line; inter-rater reliability via Krippendorff’s α.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults\u003c/strong\u003e: Across models, LingualAI was non-inferior for adequacy of meaning and terminology accuracy; completeness also met the criterion. Human interpreters scored higher on delivery-related and linguistic-mechanics domains, including clarity/intelligibility (Δ≈0.50), fluency (Δ≈1.1), prosody (Δ≈0.6), pacing (Δ≈0.4), grammar, vocabulary, and cultural appropriateness. Conclusive ratings favored humans for overall quality (Δ≈0.6) and clinical confidence (Δ≈0.6). Findings were consistent in direction-specific contrasts (English→Spanish clinician lines; Spanish→English patient lines). Inter-rater reliability was modest (α=0.31), reflecting first-impression scoring. In exploratory system metrics, mean end-to-end translation latency was ~9.7s with substantially lower estimated per-session costs than phone/video interpreter services.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusions\u003c/strong\u003e: LingualAI preserves core meaning and terminology at near-interpreter levels but lags in speech naturalness and delivery (\u003cem\u003efluency\u003c/em\u003e, \u003cem\u003eprosody\u003c/em\u003e, \u003cem\u003epacing\u003c/em\u003e), leading to lower overall quality and clinical confidence. AI translation may serve as a useful aid when interpreters are unavailable; however, its use today should remain aligned with professional standards and ideally follow an \u003cem\u003einterpreter-in-the-loop model\u003c/em\u003e rather than replacement. Continued refinement of voice and delivery features potentially will improve perceived speech naturalness and delivery and thus, in the long-run applications such as Lingual AI, will more closely approximate the performance of human interpreters on all measures. Technical work, plus clinical validation is necessary for the safe and effective deployment of applications such as Lingual AI in real-world settings (Fig. 1).\u003c/p\u003e","manuscriptTitle":"Evaluating LingualAI: A Prospective Validation of AI-Based Real- Time Translation Against Certified Human Interpreters","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-12-05 01:52:53","doi":"10.21203/rs.3.rs-8205991/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-12-25T13:00:19+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-12-25T06:47:58+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-12-15T23:15:53+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"304768716125962170564053020550622060095","date":"2025-12-12T02:52:08+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"164726442268066627339626884578251957195","date":"2025-12-02T18:47:56+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-12-02T14:15:06+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-11-29T12:58:37+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-11-28T10:05:52+00:00","index":"","fulltext":""},{"type":"submitted","content":"npj Health Systems","date":"2025-11-25T18:04:32+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"npj-health-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [npj Health Systems](https://www.nature.com/npjhealthsyst/)","snPcode":"44401","submissionUrl":"https://submission.springernature.com/new-submission/44401/3","title":"npj Health Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"848e7dde-2733-4ac8-a91e-0f9011361c69","owner":[],"postedDate":"December 5th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":58982930,"name":"Health sciences/Health care"},{"id":58982931,"name":"Health sciences/Medical research"}],"tags":[],"updatedAt":"2026-02-16T15:24:31+00:00","versionOfRecord":[],"versionCreatedAt":"2025-12-05 01:52:53","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8205991","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8205991","identity":"rs-8205991","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}