Safety, accuracy, empathy, reliability, and readability of large language model responses to patient health questions about adenomyosis: A comparative study

Lingwei Zhang, Wu F, Fen Wu, Duanlian Ye, Ye D, Zhishui Wu, Wu Z, Huan Ma, Ma H, Liu L, Li Liu
International Journal of Gynecology & Obstetrics · 2026 · doi:10.1002/ijgo.71446 · PMID:43001641 · W7225646137
Other OA: bronze public-domain-us
🔓 Open OA copy View on OpenAlex View on PubMed View at publisher

Abstract

BACKGROUND: Patients increasingly use large language model (LLM)-based chatbots to seek health information. Adenomyosis is a common gynecological condition associated with heavy menstrual bleeding, dysmenorrhea, pelvic pain, infertility, and pregnancy-related concerns. The quality and safety of chatbot-generated patient information is clinically relevant. OBJECTIVE: Our objective was to compare the safety, accuracy, empathy, reliability, information quality, and readability of LLM responses to patient health questions about adenomyosis. METHODS: This comparative cross-sectional evaluation was designed and reported according to the CHART checklist for chatbot health advice studies. Standardized chatbot queries were posed from June 13 to June 15, 2026, in Wuhan, China. A 40-item adenomyosis patient question set was submitted to four LLM-based chatbots: ChatGPT, Gemini, DeepSeek, and Doubao. Five trained gynecology experts evaluated the safety, accuracy, empathy, DISCERN score, EQIP (Ensuring Quality Information for Patients) score, Journal of the American Medical Association (JAMA) benchmark criteria, and Global Quality Scale score using prespecified criteria; readability was calculated algorithmically using automated text-processing procedures. Safety was coded as 0 for safe and 1 for potentially unsafe. Inter-rater agreement was assessed using Fleiss' kappa for safety and intraclass correlation coefficient (2,1) for the remaining expert-rated metrics. Between-model comparisons used paired nonparametric methods, with Kendall's W and 95% confidence intervals reported as the omnibus effect-size measure for Friedman tests. RESULTS: A total of 160 model-question responses were evaluated. Inter-rater agreement was substantial to excellent across metrics, with Fleiss' kappa of 0.892 for safety and intraclass correlation coefficient values ranging from 0.768 to 0.886 for the remaining metrics. Most responses were classified as safe; observed safe-response proportions differed descriptively across models, and no pairwise safety comparison remained statistically significant after Benjamini-Hochberg adjustment. Accuracy and empathy differed significantly among models (both P < 0.001): ChatGPT achieved the highest median accuracy score. DISCERN, EQIP, and JAMA scores differed significantly among models, whereas Global Quality Scale scores did not. All readability indices differed significantly among models: Doubao had the highest grade-level readability scores and the lowest Flesch Reading Ease score, whereas Gemini had comparatively easier readability. CONCLUSIONS: LLMs produced generally safe and moderately high-quality responses to adenomyosis-related patient questions, but performance varied by model and by evaluation domain. These findings support the need for transparent, multidimensional assessment of LLM-generated health advice before patient-facing use.

My notes (saved in your browser only)

Citation neighborhood (2-hop)

Papers in the corpus that this work cites (lower rings, blue) and that cite this one (upper rings, green). Dot size scales with the paper's in-corpus citation count — bigger dot = more influential within the endo/adeno field. Click a dot to open that paper. Outer rings show 2-hop neighbours — papers reached through the immediate citers/citees. [ collapse to 1-hop ]

References (32)

Source provenance

europepmc
last seen: 2026-11-13T07:00:47.251693+00:00
openalex
last seen: 2026-11-13T06:53:04.716285+00:00
pubmed
last seen: 2026-11-13T06:54:48.021411+00:00
unpaywall
last seen: 2026-11-12T06:32:18.685444+00:00
License: public-domain-us · commercial use OK · attribution required
Courtesy of the U.S. National Library of Medicine