Reproducible Generative AI Evaluation for Healthcare: A Clinician-in-the-Loop Approach

Objective To develop and apply a reproducible methodology for evaluating generative artificial intelligence powered systems in healthcare, addressing the gap between theoretical evaluation frameworks and practical implementation guidance.

A five dimension evaluation framework was developed to assess query comprehension and response helpfulness, correctness, completeness, and potential clinical harm. The framework was applied to evaluate ClinicalKey AI using queries drawn from user logs, a benchmark dataset, and subject matter expert curated queries. Forty one board certified physicians and pharmacists were recruited to independently evaluate query–response pairs. An agreement protocol using the mode and modified Delphi method resolved disagreements in evaluation scores.

Of 633 queries, 614 (96.99%) produced evaluable responses, with subject matter experts completing evaluations of 426 query-response pairs. Results demonstrated high rates of response correctness (95.5%) and query comprehension (98.6%), with 94.4% of responses rated as helpful. Two responses (0.47%) received scores indicating potential clinical harm. Pairwise consensus occurred in 60.6% of evaluations, with remaining cases requiring third tie-breaker review.

The framework demonstrated effectiveness in quantifying performance through comprehensive evaluation dimensions and structured scoring resolution methods. Key strengths included representative query sampling, standardized rating scales, and robust subject matter expert agreement protocols. Challenges emerged in managing subjective assessments of open-ended responses and achieving consensus on potential harm classification.

This framework offers a reproducible methodology for evaluating healthcare generative artificial intelligence systems, establishing foundational processes that can inform future efforts while supporting the implementation of generative AI applications in clinical settings. Competing Interest Statement All authors are current or were former employees of Elsevier, which developed and owns the ClinicalKey AI system evaluated in this study. The authors used paid external clinical contractors to perform the study functions described. The authors declare no additional competing interests. Funding Statement The authors have no funding sources to declare. Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes Footnotes This version of the manuscript has been revised to address peer-reviewer feedback and enhance clarity and applicability. The evaluation framework is now explicitly described as applicable to a variety of generative AI systems, including prompt-engineered LLMs, RAG architectures, fine-tuned models, and agentic AI systems. The frameworks flexibility is highlighted, demonstrating its broad relevance beyond Q&A systems to include other text-generation workflows like automated summaries and report writing. In response to concerns about bias, the manuscript acknowledges the importance of bias detection, particularly in healthcare contexts, and proposes adding a sixth dimension to address biases in both inputs and outputs in future work. The manuscript also discusses the potential benefits of using pre-established ground truths in some cases to reduce subjectivity, suggesting a hybrid approach for future evaluations that combines open-ended queries with standardized benchmarks. The rationale for key methodological choices, such as the selection of the five evaluation dimensions, the use of two SMEs for initial evaluations, and the sampling process, is now more thoroughly explained. The manuscript justifies the use of different rating scales for each dimension and clarifies how these choices enhance the evaluation process. A section is added to the manuscript to describe the process for adapting the framework, making it clear that researchers can modify certain components for different contexts while maintaining core principles. Transparency is increased with the inclusion of additional data in the supplementary materials, such as tables showing the raw counts and distributions of SME ratings and a breakdown of disagreements. The manuscript also clarifies the selection process for SMEs, noting that clinicians were chosen based on their qualifications and active practice, with safeguards to minimize potential conflicts of interest. These revisions strengthen the manuscripts methodology, clarify the frameworks broad applicability, and enhance transparency, addressing the reviewers concerns and improving the manuscripts overall clarity and reproducibility.