Quality, Reliability, and Readability of AI-Generated Breastfeeding Information: A Comparative Evaluation of Four Large Language Models

Quality, Reliability, and Readability of AI-Generated Breastfeeding Information: A Comparative Evaluation of Four Large Language Models | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Quality, Reliability, and Readability of AI-Generated Breastfeeding Information: A Comparative Evaluation of Four Large Language Models Sibel Ejder Tekgunduz, Ayse Gurol, Serap Ejder Apay This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8558387/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 7 You are reading this latest preprint version Abstract Background Large language models (LLMs) are increasingly used to provide breastfeeding information, yet concerns remain regarding the quality, reliability, and readability of AI-generated health content. Objective To comparatively evaluate the information quality, scientific reliability, and readability of breastfeeding-related responses generated by four widely used LLMs. Methods This descriptive cross-sectional study (September 2025) assessed responses from ChatGPT-5, Google Gemini, DeepSeek, and Claude to 10 expert-validated, clinically critical breastfeeding FAQs derived from an initial pool of 100 questions (LLM-generated and Google “People also ask”). Prompts were submitted in newly initiated chat sessions on the same day. A blinded panel of three independent experts (pediatrician, obstetrician–gynecologist, senior midwife) rated each response using DISCERN (16–80) for information quality and a 5-point Likert scale for scientific reliability; readability was assessed with Flesch Reading Ease Score (FRES) and Flesch–Kincaid Grade Level (FKGL). Differences across models were tested using Friedman/Dunn (DISCERN, Likert) and one-way ANOVA (readability). Ethical approval was obtained from the Atatürk University Faculty of Health Sciences Ethics Committee. Results DISCERN scores differed significantly across models (χ²(3) = 76.50, p < .001). DeepSeek (37.20 ± 7.17) and Claude (34.27 ± 4.93) scored higher than ChatGPT (19.93 ± 2.86) and Gemini (22.40 ± 2.19) (p < .05); no model reached “excellent” quality (≥ 63). Likert reliability also varied (χ²(3) = 62.50, p < .001), highest for DeepSeek (3.47 ± 0.63) and Claude (3.17 ± 0.38) versus ChatGPT (2.03 ± 0.18) and Gemini (2.07 ± 0.37). Readability differed (FRES: F(3,36) = 3.54, p = .024; FKGL: F(3,36) = 3.57, p = .023); all models exceeded the ideal ≤ 6 FKGL benchmark. Conclusions LLMs show a clear trade-off between informational quality and readability. DeepSeek and Claude produced more comprehensive, guideline-consistent content, but it was less readable. In contrast, ChatGPT and Gemini were more readable, albeit with lower quality and reliability. Expert oversight remains essential before integrating LLM outputs into breastfeeding education. Figures Figure 1 Introduction The exponential growth of artificial intelligence (AI)– based large language models (LLMs) has created unprecedented opportunities, alongside significant uncertainties, in health communication and patient education. Owing to their advanced natural language processing capabilities, LLMs can generate rapid, accessible, and user-tailored responses to frequently encountered health-related questions. These models are increasingly regarded as promising tools for democratizing access to health information, reducing disparities associated with low health literacy, and supporting patient decision-making—particularly among individuals who struggle to interpret complex medical terminology. Recent studies suggest that LLM-based platforms are capable of providing generally consistent and scientifically grounded responses across various health domains (Aslan & Aslan, 2025 ; Stephenson-Moe et al., 2025 ; Tepe & Emekli, 2024 ). Despite these potential benefits, growing evidence indicates that the quality, reliability, and comprehensiveness of AI-generated health information remain inconsistent. Prior research has demonstrated that clinical content produced by LLMs may vary substantially in accuracy, depth, and internal consistency; in some cases, responses fail to align with current clinical guidelines or contain hallucinations—that is, fabricated or inaccurate information presented with apparent confidence (Gökalp et al., 2025 ; Shao et al., 2025 ). These concerns underscore the crucial need for a systematic evaluation of LLM-generated health information using validated assessment frameworks, such as the DISCERN instrument for information quality and the Journal of the American Medical Association (JAMA) benchmark criteria for transparency and credibility (Aslan & Aslan, 2025 ; Davey et al., 2025 ). Such variability is particularly concerning in the context of breastfeeding, a sensitive and time-critical period in which accurate guidance is essential for maternal and infant health. Breastfeeding support is often sought outside formal clinical encounters, with parents frequently relying on online and AI-generated sources to address practical challenges. Consequently, misinformation or incomplete guidance may lead to early cessation of breastfeeding or inappropriate practices with direct consequences for neonatal outcomes. In this context, the readability of health information plays a pivotal role. Evidence consistently shows that AI-generated health content frequently exceeds the recommended sixth-grade reading level, thereby limiting accessibility for mothers with low health literacy (Behers et al., 2024 ; Kacer, 2025 lıcoglu et al., 2025 ). In addition to readability concerns, the high compliance tendency of LLMs—particularly when responding to vague or ambiguously framed prompts—may increase the risk of producing misleading or overly generalized recommendations. Within neonatal and maternal health contexts, such responses pose significant safety risks, as users may interpret them as authoritative and act upon them without clinical verification (Chen et al., 2025a ; Pierri, 2025 ). However, mitigation strategies such as retrieval-augmented generation have been proposed to enhance factual grounding, unresolved technical limitations related to model safety, consistency, and information validity persist (Alber et al., 2025 ; Bunnell et al., 2025 ). Taken together, existing evidence suggests that LLM outputs must be evaluated not only for factual accuracy but also for readability, reliability, and susceptibility to misinformation. Despite this growing body of research, studies that systematically compare AI-generated responses to frequently asked questions about breastfeeding remain limited. Addressing this gap, the present study conducts a structured evaluation of breastfeeding-related responses generated by four state-of-the-art LLMs. Specifically, we examined the quality of information (DISCERN), reliability (using a 5-point Likert scale), and readability (using the Flesch Reading Ease Score and Flesch-Kincaid Grade Level). We hypothesized that while the LLMs would generally provide accurate clinical information, the readability of the responses would be too complex for the general public. By identifying the strengths and limitations of these models, this study aims to provide clinicians, health educators, and policymakers with an evidence-based framework for the safe integration of LLMs into breastfeeding education. Methods Study Design This descriptive cross-sectional study aimed to evaluate the information quality, reliability, and readability of responses generated by four Large Language Models (LLMs) to frequently asked questions (FAQs) about breastfeeding. Although the study analyzed publicly available AI-generated content and did not involve human participants or patient data, ethical approval was obtained from the Atatürk University Faculty of Health Sciences Ethics Committee. The study was conducted in accordance with ethical standards and principles of responsible research. Data collection was carried out in September 2025. Question Generation and Data Collection An initial pool of 100 breastfeeding-related questions was compiled using a dual-source approach. First, four LLMs (ChatGPT-5, Google Gemini, DeepSeek, and Claude) were individually queried with the prompt: “Most frequently asked 20 breastfeeding questions by mothers.” This yielded 80 questions (20 from each model). Second, the term “breastfeeding” was searched on Google, and the top 20 questions from the “People also ask” section were recorded. The Google search was conducted in a non-logged-in browser session to minimize personalization bias. All questions were compiled into a Microsoft Excel file (Microsoft Corp., Redmond, WA, USA). To ensure clinical relevance and eliminate redundancy, the initial pool was independently reviewed by a pediatric nursing specialist and a midwifery specialist. Duplicate, highly similar, or clinically ambiguous items were excluded based on expert consensus. This rigorous filtration process yielded a final set of 10 distinct and clinically critical breastfeeding questions, representing the most common concerns encountered during routine breastfeeding counseling (Table 1 , Fig. 1 ). Each of the 10 final questions was entered individually into all four LLMs on the same day to minimize temporal variability. All prompts were submitted in newly initiated chat sessions to prevent context retention from prior interactions. The responses generated by each model were recorded verbatim without modification and subsequently evaluated for information quality, reliability, and readability. Table 1 Final Set of 10 Breastfeeding Questions Evaluated Across AI Models No Breastfeeding Question 1 How can I tell if my breast milk is enough for my baby? 2 What can I do to increase my milk supply? 3 How often should I breastfeed my baby? 4 How can I prevent or treat nipple cracks? 5 My baby is refusing to breastfeed—what should I do? 6 My baby is crying constantly—could it mean my milk is not enough? 7 My baby spits up/vomits after breastfeeding—what should I do? 8 How should I store and give expressed breast milk when I return to work? 9 How can a baby latch and breastfeed if the mother has flat or inverted nipples? 10 Is breastfeeding while lying down safe for my baby? Assessments A panel of three independent experts, comprising one pediatrician, one obstetrician-gynecologist, and one senior midwife, individually evaluated the responses generated by the AI LLMs. The evaluators were blinded to the specific AI model associated with each response to prevent assessment bias. The evaluation process focused on three dimensions: information quality, reliability, and readability. To capture the full range of clinical perspectives and maximize the granularity of the assessment, individual ratings from all three experts were included in the final dataset. This approach resulted in a total sample size of 30 evaluations per AI model (10 questions × 3 independent raters). Information Quality and Reliability Information quality was assessed using the DISCERN instrument, a validated tool for evaluating written consumer health information (Charnock et al., 1999 ). The DISCERN tool consists of 16 items scored from 1 (“criterion not fulfilled”) to 5 (“criterion fully fulfilled”), yielding total scores ranging from 16 to 80. Information quality was classified as very poor (≤ 27), poor (28–38), average (39–50), good (51–62), or excellent (63–80). Scientific reliability was evaluated separately using a 5-point Likert scale, reflecting the perceived trustworthiness, internal consistency, and clinical appropriateness of each response. Ratings ranged from 1 (“strongly disagree; the answer is not reliable”) to 5 (“strongly agree; the answer is scientifically trustworthy”). Responses achieving a DISCERN score of ≥ 39 (average or higher) and a Likert reliability score of ≥ 3 were considered acceptable for patient education. Readability Readability was assessed using two widely accepted metrics: the Flesch Reading Ease Score (FRES) and the Flesch–Kincaid Grade Level (FKGL) formula. The FRES assigns scores from 0 to 100, with higher scores indicating easier readability (Kincaid et al., 1975 ). Scores between 70 and 80 correspond to a reading level suitable for students in grades 7 to 8. The FKGL estimates the U.S. school grade level required to comprehend the text (e.g., a score of 8 indicates eighth-grade readability or higher). Both FRES and FKGL were calculated using an online readability calculator ( https://goodcalculators.com/flesch-kincaid-calculator/ ). In line with previous studies (Walters & Hamrell, 2008 ), responses with an FKGL score of ≤ 8 were considered acceptable for patient education. Together, these validated instruments enabled a comprehensive evaluation of the quality, reliability, and readability of AI-generated information on breastfeeding. Statistical Analysis All statistical analyses were conducted using IBM SPSS Statistics version 30.0 (IBM Corp., Armonk, NY, USA). Descriptive statistics were reported as mean ± standard deviation (SD) for all variables to ensure consistency in reporting. Data normality was assessed using the Shapiro–Wilk test. Since DISCERN and Likert scores did not meet the assumption of normality, differences among the four AI models were evaluated using the non-parametric Friedman test, followed by Dunn pairwise comparisons with adjusted p-values for post-hoc analysis. Readability measures (FRES and FKGL) demonstrated normal distribution; therefore, differences among AI models were examined using one-way analysis of variance (ANOVA). Inter-rater reliability between expert evaluators was assessed using the intraclass correlation coefficient (ICC), calculated with a two-way mixed-effects model based on absolute agreement. A p-value < 0.05 was considered statistically significant. Results A total of 40 unique AI-generated breastfeeding responses (10 questions × 4 models) were analyzed based on the consensus scores of two independent experts. The findings are presented below regarding information quality, reliability, and readability. 1. Information Quality (DISCERN) Information quality scores differed significantly across the four AI models (χ²(3) = 76.50, p < .001). As shown in Table 1 , DeepSeek achieved the highest mean DISCERN score (37.20 ± 7.17), followed by Claude (34.27 ± 4.93). Post-hoc pairwise comparisons indicated that DeepSeek and Claude scored significantly higher than ChatGPT and Gemini (p < .05). In contrast, ChatGPT and Gemini demonstrated the lowest information quality, with mean scores of 19.93 ± 2.86 and 22.40 ± 2.19, respectively. Their responses were generally brief, less structured, and often lacked adequate discussion of management options or potential risks. Notably, no AI model reached the DISCERN threshold for excellent quality (score ≥ 63). Inter-rater reliability for DISCERN assessments was excellent (ICC = 0.95, p < .001), indicating a high level of agreement between expert raters regarding content depth and completeness. Table 1 Comparison of DISCERN Scores Among AI Models Model Mean ± SD Median (IQR) ChatGPT 19.93 ± 2.86 19.0 (2.0) Gemini 22.40 ± 2.19 23.0 (2.0) Claude 34.27 ± 4.93 33.0 (7.75) DeepSeek 37.20 ± 7.17 39.0 (13.75) Data are presented as mean ± SD and median (IQR). P-values derived from Friedman test. 2. Reliability (Likert Scale) Likert reliability scores varied significantly across models (χ²(3) = 62.50, p < .001) (Table 2 ). DeepSeek received the highest mean reliability score (3.47 ± 0.63), suggesting greater clinical appropriateness and alignment with breastfeeding guidelines. ChatGPT (2.03 ± 0.18) and Gemini (2.07 ± 0.37) demonstrated significantly lower reliability scores (p < .05), with many responses receiving ratings of ≤ 2, indicating substantial limitations in completeness and clinical utility. Inter-rater agreement for Likert reliability scores was good to excellent (ICC = 0.89, p < .001), reflecting consistent expert evaluations of factual accuracy. Table 2 Comparison of Likert Reliability Scores Among AI Models Model Mean ± SD Median (IQR) ChatGPT 2.03 ± 0.18 2.0 (0.0) Gemini 2.07 ± 0.37 2.0 (0.0) Claude 3.17 ± 0.38 3.0 (0.0) DeepSeek 3.47 ± 0.63 3.0 (1.0) 3. Readability (FRES and FKGL) Readability metrics showed statistically significant variation among the AI models (Table 3 ). There was a significant difference in FRES scores (F(3,36) = 3.54, p = .024). ChatGPT (70.54 ± 7.17) and Gemini (69.41 ± 5.17) produced texts categorized as “fairly easy,” whereas Claude and DeepSeek generated comparatively more complex content. Similarly, FKGL scores differed significantly (F(3,36) = 3.57, p = .023). While ChatGPT demonstrated relatively high reading ease, its grade-level requirement (7.18 ± 1.53) exceeded that of Claude (6.24 ± 1.36) and Gemini (6.57 ± 1.18). DeepSeek generated responses requiring the highest literacy level (7.92 ± 0.73). Overall, although some individual responses from Claude and Gemini fell within the recommended range, the mean FKGL scores for all models exceeded the ideal benchmark of ≤ 6, which is recommended for patient-facing educational materials. Table 3 Comparison of Readability Scores (FRES and FKGL) Model FRES FKGL Mean ± SD Mean ± SD ChatGPT 70.54 ± 7.17 7.18 ± 1.53 Gemini 69.41 ± 5.17 6.57 ± 1.18 Claude 63.83 ± 10.65 6.24 ± 1.36 DeepSeek 61.60 ± 4.34 7.92 ± 0.73 Test and p* 3.539; 0.024 3.572; 0.023 *Derived from One-way ANOVA. Discussion The present study provides one of the first systematic evaluations of breastfeeding-related information generated by four widely used large language models (LLMs), examining their performance in terms of information quality (DISCERN), reliability, and readability. Overall, the findings demonstrate substantial variability across models, with significant implications for clinical practice, patient education, and the integration of AI systems into maternal–child health communication. These results align with a growing body of literature documenting heterogeneity in the accuracy, comprehensiveness, and linguistic accessibility of AI-generated health information (Aslan & Aslan, 2025 ; Gökalp et al., 2025 ; Shao et al., 2025 ). Information Quality (DISCERN) and Expert Consensus Our findings revealed that Claude and DeepSeek produced markedly higher DISCERN scores compared to ChatGPT and Gemini. This performance gap is consistent with prior evaluations in clinical domains, where models optimized for reasoning (such as Claude and DeepSeek) often demonstrate superior adherence to guidelines and greater depth of content. For instance, while earlier studies found ChatGPT-4 to be exceptional in terms of guideline alignment, our results suggest that newer competitors, such as DeepSeek, may offer enhanced comprehensiveness for complex queries. Notably, widely used models such as ChatGPT and Gemini achieved significantly lower quality scores in our dataset. This may reflect the “fluency hallucination” phenomenon (i.e., fluent but shallow content), whereby models prioritize conversational smoothness over the granular, structured detail required for high DISCERN scores. A critical finding of our study was the excellent inter-rater reliability (ICC = 0.95) observed for DISCERN assessments. While previous studies have often reported moderate agreement due to the subjective nature of the tool, the high consensus in our research suggests that the disparity in model performance was distinct and consistently identifiable. Experts repeatedly recognized the superior structural quality and comprehensiveness of DeepSeek and Claude compared with the more superficial responses generated by ChatGPT and Gemini. Reliability and Clinical Safety Likert-based reliability scores similarly revealed significant differences, with Claude and DeepSeek outperforming ChatGPT and Gemini. This finding aligns with systematic evaluations indicating that some LLMs exhibit stronger consistency with established clinical guidelines, while others produce more variable or incomplete responses (Beheshti et al., 2025; Johnson et al., 2023 ). Studies in maternal and neonatal health have documented that ChatGPT—despite providing linguistically fluent answers—may omit crucial risk–benefit explanations or fail to contextualize recommendations for specific patient subgroups (Amin et al., 2024 ; Temsah et al., 2025 ). The high inter-rater agreement for Likert scores (ICC = 0.89) further reinforces that factual accuracy is reliably distinguishable among models. Consistent with Shiferaw et al. ( 2024 ), this finding suggests that experts can distinguish between evidence-based guidance and generic advice, even when stylistic or linguistic variations exist. However, the results also highlight that producing factually correct statements does not necessarily translate into well-structured, patient-centered educational content—an aspect more effectively captured by DISCERN than by Likert-based reliability ratings. Readability: The Quality–Accessibility Trade-off A key finding of this study was the significant variation in readability across the four models. ChatGPT generated the most readable output, followed by Gemini and Claude, whereas DeepSeek produced the most linguistically complex responses. These results are consistent with prior research demonstrating variability in readability across AI systems (Kirchner et al., 2023 ; Patel et al., 2024 ; Rouhi et al., 2024 ). Critically, none of the models in our study met the readability standards recommended by the AMA and NIH for patient-facing educational materials. This finding aligns with previous evidence suggesting that AI-generated health texts often exceed recommended readability thresholds (Jindal & MacDermid, 2017 ; Shafau & Wahl, 2025 ). Given the well-established association between low maternal health literacy and early cessation of breastfeeding (Gaupšienė et al., 2023 ; Valero-Chillerón et al., 2021 , 2022 ), the production of overly complex breastfeeding content by LLMs may disproportionately disadvantage vulnerable populations. Mothers with lower health literacy are also more likely to supplement unnecessarily or discontinue breastfeeding due to misinterpretation of infant behaviors such as crying or perceived insufficient milk supply (Kilfoyle et al., 2016 ). Accordingly, the readability limitations observed across all AI systems represent a clinically significant barrier to equitable lactation support. Safety Risks and Misinformation: Clinical Relevance of Model Variability Beyond quality and readability, LLMs pose clinically significant safety concerns, particularly in neonatal and perinatal contexts. Our findings of inconsistent quality across models align with recent analyses demonstrating that LLMs can confidently generate clinically incorrect or misleading breastfeeding advice (Blease & Torous, 2023 ; Chen et al., 2025a ). When prompts contain false premises, models may reinforce misinformation—a phenomenon with important implications for lactation counseling (Chen et al., 2025b ). Moreover, as Hatem et al. ( 2023 ) and Pierri ( 2025 ) have noted, AI-generated misinformation can spread rapidly, amplifying anxiety, fueling misconceptions, and undermining maternal confidence. Considering that perceived insufficient milk supply is one of the leading modifiable causes of undesired breastfeeding cessation (Cheng et al., 2023 ), erroneous or ambiguous AI-generated information may have substantial downstream consequences for maternal and infant health. Limitations This study has several limitations that should be considered when interpreting the findings. First, the evaluation was based on a limited set of ten breastfeeding questions, which, although clinically relevant and expert-validated, may not fully represent the breadth of real-world breastfeeding information needs. Second, AI responses were generated at a single time point, yet large language models evolve rapidly; thus, the results may not reflect future model updates or performance variations across different versions. Third, the DISCERN and Likert assessments relied on subjective expert judgments, and although inter-rater reliability was examined, the inherent subjectivity of narrative content evaluation may have influenced the scoring. Additionally, readability metrics such as FRES and FKGL, initially developed for English texts, were applied to Turkish-language outputs and may not capture all linguistic nuances or structural complexities of Turkish grammar. The study also did not evaluate other vital dimensions such as cultural appropriateness, actionability, or alignment with national breastfeeding guidelines. Finally, the analysis was limited to textual responses, excluding the examination of multimodal capabilities (e.g., images, audio explanations), which are becoming increasingly relevant in patient education. These limitations suggest that while the findings provide an essential comparative snapshot of current AI performance, they should be interpreted with caution and complemented by future research employing broader datasets, longitudinal designs, and additional evaluation tools. Implications for Clinical Practice and Future Research Taken together, these findings underscore the need for structured oversight, expert review, and the design of tailored prompts before AI systems can be responsibly integrated into breastfeeding education. Although LLMs offer considerable promise in expanding access to breastfeeding information, the combined challenges of variable information quality, limited readability, and potential for misinformation necessitate cautious implementation. Emerging solutions, such as retrieval-augmented generation (Bunnell et al., 2025 ) and domain-specific fine-tuning (Kalai et al., 2025 ), may mitigate these risks; however, further empirical testing is required. Conclusion This study presents a comprehensive comparison of four large language models in generating breastfeeding-related health information, with a focus on information quality, reliability, and readability. The findings demonstrate that while all models could produce clinically relevant responses, the quality and accessibility of these outputs varied substantially. DeepSeek and Claude generated higher-quality, more guideline-consistent information, but the resulting text required higher literacy levels. In contrast, ChatGPT and Gemini delivered more readable content at the expense of information quality and reliability. Notably, none of the models met the ideal readability thresholds recommended for patient education materials. These findings reveal a clear trade-off between informational depth and linguistic accessibility across LLMs. Despite strong inter-rater agreement for both DISCERN and reliability assessments, the results highlight the complexity of evaluating narrative health information and underscore the need for multidimensional assessment frameworks. Overall, the study highlights the importance of cautious, expert-guided integration of AI systems into breastfeeding education to ensure safety, accuracy, and equity. Declarations Ethical approval was obtained from the Atatürk University Faculty of Health Sciences Ethics Committee. (Date: 05.12.2025, Decision No: 2025/12/17) Consent for publication: Not applicable Competing interests: The authors declare no competing interests Funding: No funding. Availability of data and materials: The datasets used and analyzed in this study are available from the corresponding author upon reasonable request. Authors' contributions: Study design: SEA, AG, SET; Data collection: SEA, AG, SET; Data analysis: SEA, AG, SET, Draft preparation: SEA, AG, SET; Critical review for content: SEA, AG, Final approval of the version to be published: SEA, AG, SET References Alber D, Yang Z, Alyakin A, Yang E, Shesh N, Valliani A, Oermann E. Medical large language models are vulnerable to data-poisoning attacks. Nat Med. 2025;31(2):618–26. https://doi.org/10.1038/s41591-024-03445-1 . Amin KS, Mayes LC, Khosla P, Doshi RH. Assessing the Efficacy of Large Language Models in Health Literacy: A Comprehensive Cross-Sectional Study. Yale J Biol Med. 2024;97(1):17–27. https://doi.org/10.59249/ZTOZ1966 . Aslan İÖ, Aslan MT. Benchmarking ai chatbots for maternal lactation support: a cross-platform evaluation of quality, readability, and clinical accuracy. Healthcare. 2025;13(14):1756. https://doi.org/10.3390/healthcare13141756 . Behers B, Vargas I, Behers B, Rosario M, Wojtas C, Deevers A, Hamad K. Assessing the readability of patient education materials on cardiac catheterization from artificial intelligence chatbots: an observational cross-sectional study. Cureus. 2024;16(7):e63865. https://doi.org/10.7759/cureus.63865 . Beheshti M, Toubal IE, Alaboud K, Almalaysha M, Ogundele OB, Turabieh H, Dahu BM. Evaluating the reliability of chatgpt for health-related questions: A systematic review. Informatics. 2025, January;12(1):9. https://doi.org/10.3390/informatics12010009 . Blease C, Torous J. ChatGPT and mental healthcare: balancing benefits with risks of harms. BMJ Ment Health. 2023;26(1):e300884. https://doi.org/10.1136/bmjment-2023-300884 . Bunnell D, Bondy M, Fromtling L, Ludeman E, Gourab K. (2025). Bridging ai and healthcare: a scoping review of retrieval-augmented generation-ethics, bias, transparency, improvements, and applications. medRxiv 2025.04.01.25325033. https://doi.org/10.1101/2025.04.01.25325033 Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health. 1999;53(2):105–11. Chen S, Gao M, Sasse K, Hartvigsen T, Anthony B, Fan L, Bitterman D. Abstract ia04: evaluating and mitigating medical misinformation risk in large language models. Clin Cancer Res. 2025a;31(2Supplement):IA04. https://doi.org/10.1158/1557-3265.targetedtherap-ia04 . Chen S, Gao M, Sasse K, Hartvigsen T, Anthony B, Fan L, Bitterman DS. When helpfulness backfires: LLMs and the risk of false medical information due to sycophantic behavior. npj Digit Med. 2025b;8(1):605. https://doi.org/10.1038/s41746-025-02008-z . Cheng GZ, Chen A, Xin Y, Ni QQ. Using the teach-back method to improve postpartum maternal-infant health among women with limited maternal health literacy: a randomized controlled study. BMC Pregnancy Childbirth. 2023;23(1):13. https://doi.org/10.1186/s12884-022-05302-w . Davey M, Hayes J, Hurley E, Cassidy J. Evaluation and comparison of the quality and readability of online information for patients using the chat gpt (artificial intelligence) platform versus other online sources regarding hip fractures management. Trauma. 2025;27(4):301–6. https://doi.org/10.1177/14604086251345001 . Gaupšienė A, Vainauskaitė A, Baglajeva J, Stukas R, Ramašauskaitė D, Paliulytė V, Istomina N. Associations between maternal health literacy, neonatal health and breastfeeding outcomes in the early postpartum period. Eur J midwifery. 2023;7:25. 10.18332/ejm/170161 . Gökalp M, Yücel Ş, Çakır Z. (2025). Readability, reliability, and quality of nursing care plan texts generated by chatgpt. BMC Nursing, under review https://doi.org/10.21203/rs.3.rs-7237457/v1 Hatem R, Simmons B, Thornton JE. (2023). A call to address AI hallucinations and how healthcare professionals can mitigate their risks. Cureus, 15 (9). https://goodcalculators.com/flesch-kincaid-calculator/ Jindal P, MacDermid JC. (2017). Assessing reading levels of health information: uses and limitations of flesch formula. Education for Health , 30 (1), 84–88. DOI: 10.4103/1357-6283.210517 . Johnson D, Goodman R, Patrinely J, Stone C, Zimmerman E, Donald R, Wheless L. Assessing the accuracy and reliability of AI-generated medical responses: an evaluation of the Chat-GPT model. Res square. 2023;rs–3. 10.21203/rs.3.rs-2566942/v1 . Kacer EO. Evaluating AI-based breastfeeding chatbots: quality, readability, and reliability analysis. PLoS ONE. 2025;20(3):e0319782. https://doi.org/10.1371/journal.pone.0319782 . Kalai AT, Nachum O, Vempala SS, Zhang E. (2025). Why language models hallucinate. arXiv preprint arXiv:2509.04664 . https://doi.org/10.48550/arXiv.2509.04664 Kilfoyle KA, Vitko M, O'Conor R, Bailey SC. Health literacy and women's reproductive health: a systematic review. J women's health. 2016;25(12):1237–55. Kılıcoglu MS, Yurdakul OV, Aydın T. Evaluation of Chat Generative Pre-trained Transformer’s responses to frequently asked questions about psoriatic arthritis: A study on quality and readability. Ann Med Res. 2025;32(2):79–84. https://doi.org/10.5455/annalsmedres.2024.10.231 . Kincaid JP, Fishburne RP, Rogers RL, Chissom BS. (1975). Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy enlisted personnel. Research Branch Report 8–75. Naval Technical Training Command, Millington, TN. Kirchner GJ, Kim RY, Weddle JB, Bible JE. (2023). Can artificial intelligence improve the readability of patient education materials? Clinical Orthopaedics and Related Research , 481 (11), 2260–2267. DOI: 10.1097/CORR.0000000000002668 . Patel EA, Fleischer L, Filip P, Eggerstedt M, Hutz M, Michaelides E, Tajudeen BA. The use of artificial intelligence to improve readability of otolaryngology patient education materials. Otolaryngology–Head Neck Surg. 2024;171(2):603–8. Pierri F. The role of ai agents for online information disorders. XRDS Crossroads Acm Magazine Students. 2025;31(3):24–7. https://doi.org/10.1145/3729511 . Rouhi AD, Ghanem YK, Yolchieva L, Saleh Z, Joshi H, Moccia MC, Han JJ. Can artificial intelligence improve the readability of patient education materials on aortic stenosis? A pilot study. Cardiol therapy. 2024;13(1):137–47. https://doi.org/10.1007/s40119-023-00347-0 . Shafau F, Wahl C. (2025). Evaluating the Readability of AI-Generated Patient Information on Chronic Diseases. Chronic Diseases and Translational Medicine . Shao X, Ruan T, Ju X, Sun Y, Cui J. Evaluating artificial intelligence chatbots’ responses to gynecomastia inquiries: comparative study of information quality, readability, and guideline consistency. Digit Health. 2025;11. https://doi.org/10.1177/20552076251367645 . Shiferaw MW, Zheng T, Winter A, Mike LA, Chan LN. Assessing the accuracy and quality of artificial intelligence (AI) chatbot-generated responses in making patient-specific drug-therapy and healthcare-related decisions. BMC Med Inf Decis Mak. 2024;24(1):404. https://doi.org/10.1186/s12911-024-02824-5 . Stephenson-Moe C, Behers B, Gibons R, Behers B, Herrera L, Anneaud D, Hamad K. Assessing the quality and readability of patient education materials on chemotherapy cardiotoxicity from artificial intelligence chatbots: an observational cross-sectional study. Medicine. 2025;104(15):e42135. https://doi.org/10.1097/md.0000000000042135 . Temsah A, Alhasan K, Altamimi I, Jamal A, Al-Eyadhy A, Malki K, Temsah M. DeepSeek in Healthcare: Revealing Opportunities and Steering Challenges of a New Open-Source Artificial Intelligence Frontier. Cureus. 2025;17(2):e79221. 10.7759/cureus.79221 . Tepe M, Emekli E. Assessing the responses of large language models (chatgpt-4, gemini, and microsoft copilot) to frequently asked questions in breast imaging: a study on readability and accuracy. Cureus. 2024;16(5):e59960. https://doi.org/10.7759/cureus.59960 . Valero-Chillerón MJ, González‐Chordà VM, Cervera‐Gasch Á, Vila‐Candel R, Soriano‐Vidal FJ, Mena‐Tudela D. Health literacy and its relation to continuing with breastfeeding at six months post‐partum in a sample of Spanish women. Nurs Open. 2021;8(6):3394–402. https://doi.org/10.1002/nop2.885 . Valero-Chillerón MJ, Mena-Tudela D, Cervera-Gasch Á, González-Chordá VM, Soriano-Vidal FJ, Quesada JA, Vila-Candel R. Influence of health literacy on maintenance of exclusive breastfeeding at 6 months postpartum: a multicentre study. Int J Environ Res Public Health. 2022;19(9):5411. https://doi.org/10.3390/ijerph19095411 . Walters KA, Hamrell MR. Consent forms, lower reading levels, and using Flesch–Kincaid readability software. Drug Inform J. 2008;42(5):385–94. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Reviews received at journal 13 Feb, 2026 Reviewers agreed at journal 09 Feb, 2026 Reviewers invited by journal 09 Feb, 2026 Editor invited by journal 15 Jan, 2026 Editor assigned by journal 13 Jan, 2026 Submission checks completed at journal 13 Jan, 2026 First submitted to journal 09 Jan, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8558387","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":589078893,"identity":"3a7a7005-0510-4c82-ae5f-d9e7f339df94","order_by":0,"name":"Sibel Ejder Tekgunduz","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Sibel","middleName":"Ejder","lastName":"Tekgunduz","suffix":""},{"id":589078894,"identity":"0f54f58d-ac6d-4e28-bade-5282dcbe33dd","order_by":1,"name":"Ayse Gurol","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Ayse","middleName":"","lastName":"Gurol","suffix":""},{"id":589078895,"identity":"da6a0753-27a3-437f-b5d6-22fd9d4cca20","order_by":2,"name":"Serap Ejder Apay","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAz0lEQVRIiWNgGAWjYHCCBAkGBpsECLOAeC1pCQxsIKYBkdYAtRyGaGEgRot8e8PDGx/3nM/jl+9O/PDAgEGeX+wAfi0GZw4kW854drtYso13swTQYYYzZycQ0CKRkCbNc+B24oZjvBtAWhIMbhPQIj//QZr0nwPnQFo2/yBKC8MNhjRphgMHQFq2EWeLwZmEZMueA8mJM9tyt1kkGEgQ9ot8+5nEGz8O2CX2M5/dfPNHhY08vzQhhzHwoKiQIKQcBNgPEKNqFIyCUTAKRjIAABSQRs1Lz3NJAAAAAElFTkSuQmCC","orcid":"","institution":"","correspondingAuthor":true,"prefix":"","firstName":"Serap","middleName":"Ejder","lastName":"Apay","suffix":""}],"badges":[],"createdAt":"2026-01-09 08:08:58","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8558387/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8558387/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":102403160,"identity":"515d0087-191f-4651-9dfb-9eb5fc843388","added_by":"auto","created_at":"2026-02-11 10:46:06","extension":"jpeg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":209625,"visible":true,"origin":"","legend":"\u003cp\u003eEvaluation process of breastfeeding-related questions\u003c/p\u003e","description":"","filename":"floatimage1.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-8558387/v1/93788e94ab94296403b0546f.jpeg"},{"id":102403637,"identity":"a60f1a9c-087e-4536-9466-a8cd82d010c7","added_by":"auto","created_at":"2026-02-11 10:47:21","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":952564,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8558387/v1/67a1f2cb-a20d-4137-aa33-86664a808671.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Quality, Reliability, and Readability of AI-Generated Breastfeeding Information: A Comparative Evaluation of Four Large Language Models","fulltext":[{"header":"Introduction","content":"\u003cp\u003eThe exponential growth of artificial intelligence (AI)\u0026ndash; based large language models (LLMs) has created unprecedented opportunities, alongside significant uncertainties, in health communication and patient education. Owing to their advanced natural language processing capabilities, LLMs can generate rapid, accessible, and user-tailored responses to frequently encountered health-related questions. These models are increasingly regarded as promising tools for democratizing access to health information, reducing disparities associated with low health literacy, and supporting patient decision-making\u0026mdash;particularly among individuals who struggle to interpret complex medical terminology. Recent studies suggest that LLM-based platforms are capable of providing generally consistent and scientifically grounded responses across various health domains (Aslan \u0026amp; Aslan, \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Stephenson-Moe et al., \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Tepe \u0026amp; Emekli, \u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e2024\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eDespite these potential benefits, growing evidence indicates that the quality, reliability, and comprehensiveness of AI-generated health information remain inconsistent. Prior research has demonstrated that clinical content produced by LLMs may vary substantially in accuracy, depth, and internal consistency; in some cases, responses fail to align with current clinical guidelines or contain hallucinations\u0026mdash;that is, fabricated or inaccurate information presented with apparent confidence (G\u0026ouml;kalp et al., \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Shao et al., \u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). These concerns underscore the crucial need for a systematic evaluation of LLM-generated health information using validated assessment frameworks, such as the DISCERN instrument for information quality and the Journal of the American Medical Association (JAMA) benchmark criteria for transparency and credibility (Aslan \u0026amp; Aslan, \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Davey et al., \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e2025\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eSuch variability is particularly concerning in the context of breastfeeding, a sensitive and time-critical period in which accurate guidance is essential for maternal and infant health. Breastfeeding support is often sought outside formal clinical encounters, with parents frequently relying on online and AI-generated sources to address practical challenges. Consequently, misinformation or incomplete guidance may lead to early cessation of breastfeeding or inappropriate practices with direct consequences for neonatal outcomes. In this context, the readability of health information plays a pivotal role. Evidence consistently shows that AI-generated health content frequently exceeds the recommended sixth-grade reading level, thereby limiting accessibility for mothers with low health literacy (Behers et al., \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Kacer, \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e2025\u003c/span\u003elıcoglu et al., \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2025\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eIn addition to readability concerns, the high compliance tendency of LLMs\u0026mdash;particularly when responding to vague or ambiguously framed prompts\u0026mdash;may increase the risk of producing misleading or overly generalized recommendations. Within neonatal and maternal health contexts, such responses pose significant safety risks, as users may interpret them as authoritative and act upon them without clinical verification (Chen et al., \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2025a\u003c/span\u003e; Pierri, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). However, mitigation strategies such as retrieval-augmented generation have been proposed to enhance factual grounding, unresolved technical limitations related to model safety, consistency, and information validity persist (Alber et al., \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Bunnell et al., \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2025\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eTaken together, existing evidence suggests that LLM outputs must be evaluated not only for factual accuracy but also for readability, reliability, and susceptibility to misinformation. Despite this growing body of research, studies that systematically compare AI-generated responses to frequently asked questions about breastfeeding remain limited. Addressing this gap, the present study conducts a structured evaluation of breastfeeding-related responses generated by four state-of-the-art LLMs. Specifically, we examined the quality of information (DISCERN), reliability (using a 5-point Likert scale), and readability (using the Flesch Reading Ease Score and Flesch-Kincaid Grade Level). We hypothesized that while the LLMs would generally provide accurate clinical information, the readability of the responses would be too complex for the general public. By identifying the strengths and limitations of these models, this study aims to provide clinicians, health educators, and policymakers with an evidence-based framework for the safe integration of LLMs into breastfeeding education.\u003c/p\u003e"},{"header":"Methods","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eStudy Design\u003c/h2\u003e \u003cp\u003eThis descriptive cross-sectional study aimed to evaluate the information quality, reliability, and readability of responses generated by four Large Language Models (LLMs) to frequently asked questions (FAQs) about breastfeeding. Although the study analyzed publicly available AI-generated content and did not involve human participants or patient data, ethical approval was obtained from the Atat\u0026uuml;rk University Faculty of Health Sciences Ethics Committee. The study was conducted in accordance with ethical standards and principles of responsible research. Data collection was carried out in September 2025.\u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003eQuestion Generation and Data Collection\u003c/h3\u003e\n\u003cp\u003eAn initial pool of 100 breastfeeding-related questions was compiled using a dual-source approach. First, four LLMs (ChatGPT-5, Google Gemini, DeepSeek, and Claude) were individually queried with the prompt: \u0026ldquo;Most frequently asked 20 breastfeeding questions by mothers.\u0026rdquo; This yielded 80 questions (20 from each model). Second, the term \u0026ldquo;breastfeeding\u0026rdquo; was searched on Google, and the top 20 questions from the \u0026ldquo;People also ask\u0026rdquo; section were recorded. The Google search was conducted in a non-logged-in browser session to minimize personalization bias. All questions were compiled into a Microsoft Excel file (Microsoft Corp., Redmond, WA, USA).\u003c/p\u003e \u003cp\u003eTo ensure clinical relevance and eliminate redundancy, the initial pool was independently reviewed by a pediatric nursing specialist and a midwifery specialist. Duplicate, highly similar, or clinically ambiguous items were excluded based on expert consensus. This rigorous filtration process yielded a final set of 10 distinct and clinically critical breastfeeding questions, representing the most common concerns encountered during routine breastfeeding counseling (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e1\u003c/span\u003e, Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eEach of the 10 final questions was entered individually into all four LLMs on the same day to minimize temporal variability. All prompts were submitted in newly initiated chat sessions to prevent context retention from prior interactions. The responses generated by each model were recorded verbatim without modification and subsequently evaluated for information quality, reliability, and readability.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eFinal Set of 10 Breastfeeding Questions Evaluated Across AI Models\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNo\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eBreastfeeding Question\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHow can I tell if my breast milk is enough for my baby?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eWhat can I do to increase my milk supply?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHow often should I breastfeed my baby?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHow can I prevent or treat nipple cracks?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMy baby is refusing to breastfeed\u0026mdash;what should I do?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMy baby is crying constantly\u0026mdash;could it mean my milk is not enough?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMy baby spits up/vomits after breastfeeding\u0026mdash;what should I do?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHow should I store and give expressed breast milk when I return to work?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHow can a baby latch and breastfeed if the mother has flat or inverted nipples?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e10\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eIs breastfeeding while lying down safe for my baby?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e\n\u003ch3\u003eAssessments\u003c/h3\u003e\n\u003cp\u003eA panel of three independent experts, comprising one pediatrician, one obstetrician-gynecologist, and one senior midwife, individually evaluated the responses generated by the AI LLMs. The evaluators were blinded to the specific AI model associated with each response to prevent assessment bias.\u003c/p\u003e \u003cp\u003eThe evaluation process focused on three dimensions: information quality, reliability, and readability. To capture the full range of clinical perspectives and maximize the granularity of the assessment, individual ratings from all three experts were included in the final dataset. This approach resulted in a total sample size of 30 evaluations per AI model (10 questions \u0026times; 3 independent raters).\u003c/p\u003e\n\u003ch3\u003eInformation Quality and Reliability\u003c/h3\u003e\n\u003cp\u003eInformation quality was assessed using the DISCERN instrument, a validated tool for evaluating written consumer health information (Charnock et al., \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e1999\u003c/span\u003e). The DISCERN tool consists of 16 items scored from 1 (\u0026ldquo;criterion not fulfilled\u0026rdquo;) to 5 (\u0026ldquo;criterion fully fulfilled\u0026rdquo;), yielding total scores ranging from 16 to 80. Information quality was classified as very poor (\u0026le;\u0026thinsp;27), poor (28\u0026ndash;38), average (39\u0026ndash;50), good (51\u0026ndash;62), or excellent (63\u0026ndash;80).\u003c/p\u003e \u003cp\u003eScientific reliability was evaluated separately using a 5-point Likert scale, reflecting the perceived trustworthiness, internal consistency, and clinical appropriateness of each response. Ratings ranged from 1 (\u0026ldquo;strongly disagree; the answer is not reliable\u0026rdquo;) to 5 (\u0026ldquo;strongly agree; the answer is scientifically trustworthy\u0026rdquo;). Responses achieving a DISCERN score of \u0026ge;\u0026thinsp;39 (average or higher) and a Likert reliability score of \u0026ge;\u0026thinsp;3 were considered acceptable for patient education.\u003c/p\u003e\n\u003ch3\u003eReadability\u003c/h3\u003e\n\u003cp\u003eReadability was assessed using two widely accepted metrics: the Flesch Reading Ease Score (FRES) and the Flesch\u0026ndash;Kincaid Grade Level (FKGL) formula. The FRES assigns scores from 0 to 100, with higher scores indicating easier readability (Kincaid et al., \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e1975\u003c/span\u003e). Scores between 70 and 80 correspond to a reading level suitable for students in grades 7 to 8. The FKGL estimates the U.S. school grade level required to comprehend the text (e.g., a score of 8 indicates eighth-grade readability or higher). Both FRES and FKGL were calculated using an online readability calculator (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://goodcalculators.com/flesch-kincaid-calculator/\u003c/span\u003e\u003cspan address=\"https://goodcalculators.com/flesch-kincaid-calculator/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003cp\u003e). In line with previous studies (Walters \u0026amp; Hamrell, \u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e2008\u003c/span\u003e), responses with an FKGL score of \u0026le;\u0026thinsp;8 were considered acceptable for patient education.\u003c/p\u003e \u003cp\u003eTogether, these validated instruments enabled a comprehensive evaluation of the quality, reliability, and readability of AI-generated information on breastfeeding.\u003c/p\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eStatistical Analysis\u003c/h2\u003e \u003cp\u003eAll statistical analyses were conducted using IBM SPSS Statistics version 30.0 (IBM Corp., Armonk, NY, USA). Descriptive statistics were reported as mean\u0026thinsp;\u0026plusmn;\u0026thinsp;standard deviation (SD) for all variables to ensure consistency in reporting.\u003c/p\u003e \u003cp\u003eData normality was assessed using the Shapiro\u0026ndash;Wilk test. Since DISCERN and Likert scores did not meet the assumption of normality, differences among the four AI models were evaluated using the non-parametric Friedman test, followed by Dunn pairwise comparisons with adjusted p-values for post-hoc analysis.\u003c/p\u003e \u003cp\u003eReadability measures (FRES and FKGL) demonstrated normal distribution; therefore, differences among AI models were examined using one-way analysis of variance (ANOVA).\u003c/p\u003e \u003cp\u003eInter-rater reliability between expert evaluators was assessed using the intraclass correlation coefficient (ICC), calculated with a two-way mixed-effects model based on absolute agreement. A p-value\u0026thinsp;\u0026lt;\u0026thinsp;0.05 was considered statistically significant.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"Results","content":"\u003cp\u003eA total of 40 unique AI-generated breastfeeding responses (10 questions \u0026times; 4 models) were analyzed based on the consensus scores of two independent experts. The findings are presented below regarding information quality, reliability, and readability.\u003c/p\u003e \u003cp\u003e \u003cb\u003e1. Information Quality (DISCERN)\u003c/b\u003e \u003c/p\u003e \u003cp\u003eInformation quality scores differed significantly across the four AI models (χ\u0026sup2;(3)\u0026thinsp;=\u0026thinsp;76.50, p\u0026thinsp;\u0026lt;\u0026thinsp;.001). As shown in Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e1\u003c/span\u003e, DeepSeek achieved the highest mean DISCERN score (37.20\u0026thinsp;\u0026plusmn;\u0026thinsp;7.17), followed by Claude (34.27\u0026thinsp;\u0026plusmn;\u0026thinsp;4.93). Post-hoc pairwise comparisons indicated that DeepSeek and Claude scored significantly higher than ChatGPT and Gemini (p\u0026thinsp;\u0026lt;\u0026thinsp;.05).\u003c/p\u003e \u003cp\u003eIn contrast, ChatGPT and Gemini demonstrated the lowest information quality, with mean scores of 19.93\u0026thinsp;\u0026plusmn;\u0026thinsp;2.86 and 22.40\u0026thinsp;\u0026plusmn;\u0026thinsp;2.19, respectively. Their responses were generally brief, less structured, and often lacked adequate discussion of management options or potential risks. Notably, no AI model reached the DISCERN threshold for excellent quality (score\u0026thinsp;\u0026ge;\u0026thinsp;63).\u003c/p\u003e \u003cp\u003eInter-rater reliability for DISCERN assessments was excellent (ICC\u0026thinsp;=\u0026thinsp;0.95, p\u0026thinsp;\u0026lt;\u0026thinsp;.001), indicating a high level of agreement between expert raters regarding content depth and completeness.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eComparison of DISCERN Scores Among AI Models\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\"\u0026plusmn;\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMean\u0026thinsp;\u0026plusmn;\u0026thinsp;SD\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMedian (IQR)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eChatGPT\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e19.93\u0026thinsp;\u0026plusmn;\u0026thinsp;2.86\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e19.0 (2.0)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eGemini\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e22.40\u0026thinsp;\u0026plusmn;\u0026thinsp;2.19\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e23.0 (2.0)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eClaude\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e34.27\u0026thinsp;\u0026plusmn;\u0026thinsp;4.93\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e33.0 (7.75)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eDeepSeek\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e37.20\u0026thinsp;\u0026plusmn;\u0026thinsp;7.17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e39.0 (13.75)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003eData are presented as mean\u0026thinsp;\u0026plusmn;\u0026thinsp;SD and median (IQR).\u003c/em\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003eP-values derived from Friedman test.\u003c/em\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003e2. Reliability (Likert Scale)\u003c/b\u003e \u003c/p\u003e \u003cp\u003eLikert reliability scores varied significantly across models (χ\u0026sup2;(3)\u0026thinsp;=\u0026thinsp;62.50, p\u0026thinsp;\u0026lt;\u0026thinsp;.001) (Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e2\u003c/span\u003e). DeepSeek received the highest mean reliability score (3.47\u0026thinsp;\u0026plusmn;\u0026thinsp;0.63), suggesting greater clinical appropriateness and alignment with breastfeeding guidelines.\u003c/p\u003e \u003cp\u003eChatGPT (2.03\u0026thinsp;\u0026plusmn;\u0026thinsp;0.18) and Gemini (2.07\u0026thinsp;\u0026plusmn;\u0026thinsp;0.37) demonstrated significantly lower reliability scores (p\u0026thinsp;\u0026lt;\u0026thinsp;.05), with many responses receiving ratings of \u0026le;\u0026thinsp;2, indicating substantial limitations in completeness and clinical utility.\u003c/p\u003e \u003cp\u003eInter-rater agreement for Likert reliability scores was good to excellent (ICC\u0026thinsp;=\u0026thinsp;0.89, p\u0026thinsp;\u0026lt;\u0026thinsp;.001), reflecting consistent expert evaluations of factual accuracy.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eComparison of Likert Reliability Scores Among AI Models\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\"\u0026plusmn;\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMean\u0026thinsp;\u0026plusmn;\u0026thinsp;SD\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMedian (IQR)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eChatGPT\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e2.03\u0026thinsp;\u0026plusmn;\u0026thinsp;0.18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e2.0 (0.0)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eGemini\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e2.07\u0026thinsp;\u0026plusmn;\u0026thinsp;0.37\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e2.0 (0.0)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eClaude\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e3.17\u0026thinsp;\u0026plusmn;\u0026thinsp;0.38\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e3.0 (0.0)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eDeepSeek\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e3.47\u0026thinsp;\u0026plusmn;\u0026thinsp;0.63\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e3.0 (1.0)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003e3. Readability (FRES and FKGL)\u003c/b\u003e \u003c/p\u003e \u003cp\u003eReadability metrics showed statistically significant variation among the AI models (Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e3\u003c/span\u003e). There was a significant difference in FRES scores (F(3,36)\u0026thinsp;=\u0026thinsp;3.54, p\u0026thinsp;=\u0026thinsp;.024). ChatGPT (70.54\u0026thinsp;\u0026plusmn;\u0026thinsp;7.17) and Gemini (69.41\u0026thinsp;\u0026plusmn;\u0026thinsp;5.17) produced texts categorized as \u0026ldquo;fairly easy,\u0026rdquo; whereas Claude and DeepSeek generated comparatively more complex content.\u003c/p\u003e \u003cp\u003eSimilarly, FKGL scores differed significantly (F(3,36)\u0026thinsp;=\u0026thinsp;3.57, p\u0026thinsp;=\u0026thinsp;.023). While ChatGPT demonstrated relatively high reading ease, its grade-level requirement (7.18\u0026thinsp;\u0026plusmn;\u0026thinsp;1.53) exceeded that of Claude (6.24\u0026thinsp;\u0026plusmn;\u0026thinsp;1.36) and Gemini (6.57\u0026thinsp;\u0026plusmn;\u0026thinsp;1.18). DeepSeek generated responses requiring the highest literacy level (7.92\u0026thinsp;\u0026plusmn;\u0026thinsp;0.73).\u003c/p\u003e \u003cp\u003eOverall, although some individual responses from Claude and Gemini fell within the recommended range, the mean FKGL scores for all models exceeded the ideal benchmark of \u0026le;\u0026thinsp;6, which is recommended for patient-facing educational materials.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eComparison of Readability Scores (FRES and FKGL)\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eModel\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFRES\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eFKGL\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMean\u0026thinsp;\u0026plusmn;\u0026thinsp;SD\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMean\u0026thinsp;\u0026plusmn;\u0026thinsp;SD\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eChatGPT\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e70.54\u0026thinsp;\u0026plusmn;\u0026thinsp;7.17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e7.18\u0026thinsp;\u0026plusmn;\u0026thinsp;1.53\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eGemini\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e69.41\u0026thinsp;\u0026plusmn;\u0026thinsp;5.17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e6.57\u0026thinsp;\u0026plusmn;\u0026thinsp;1.18\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eClaude\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e63.83\u0026thinsp;\u0026plusmn;\u0026thinsp;10.65\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e6.24\u0026thinsp;\u0026plusmn;\u0026thinsp;1.36\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eDeepSeek\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e61.60\u0026thinsp;\u0026plusmn;\u0026thinsp;4.34\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e7.92\u0026thinsp;\u0026plusmn;\u0026thinsp;0.73\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eTest and p*\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e3.539; 0.024\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e3.572; 0.023\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003e*Derived from One-way ANOVA.\u003c/em\u003e \u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eThe present study provides one of the first systematic evaluations of breastfeeding-related information generated by four widely used large language models (LLMs), examining their performance in terms of information quality (DISCERN), reliability, and readability. Overall, the findings demonstrate substantial variability across models, with significant implications for clinical practice, patient education, and the integration of AI systems into maternal\u0026ndash;child health communication. These results align with a growing body of literature documenting heterogeneity in the accuracy, comprehensiveness, and linguistic accessibility of AI-generated health information (Aslan \u0026amp; Aslan, \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; G\u0026ouml;kalp et al., \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Shao et al., \u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e2025\u003c/span\u003e).\u003c/p\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003eInformation Quality (DISCERN) and Expert Consensus\u003c/h2\u003e \u003cp\u003eOur findings revealed that Claude and DeepSeek produced markedly higher DISCERN scores compared to ChatGPT and Gemini. This performance gap is consistent with prior evaluations in clinical domains, where models optimized for reasoning (such as Claude and DeepSeek) often demonstrate superior adherence to guidelines and greater depth of content. For instance, while earlier studies found ChatGPT-4 to be exceptional in terms of guideline alignment, our results suggest that newer competitors, such as DeepSeek, may offer enhanced comprehensiveness for complex queries.\u003c/p\u003e \u003cp\u003eNotably, widely used models such as ChatGPT and Gemini achieved significantly lower quality scores in our dataset. This may reflect the \u0026ldquo;fluency hallucination\u0026rdquo; phenomenon (i.e., fluent but shallow content), whereby models prioritize conversational smoothness over the granular, structured detail required for high DISCERN scores.\u003c/p\u003e \u003cp\u003eA critical finding of our study was the excellent inter-rater reliability (ICC\u0026thinsp;=\u0026thinsp;0.95) observed for DISCERN assessments. While previous studies have often reported moderate agreement due to the subjective nature of the tool, the high consensus in our research suggests that the disparity in model performance was distinct and consistently identifiable. Experts repeatedly recognized the superior structural quality and comprehensiveness of DeepSeek and Claude compared with the more superficial responses generated by ChatGPT and Gemini.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003eReliability and Clinical Safety\u003c/h2\u003e \u003cp\u003eLikert-based reliability scores similarly revealed significant differences, with Claude and DeepSeek outperforming ChatGPT and Gemini. This finding aligns with systematic evaluations indicating that some LLMs exhibit stronger consistency with established clinical guidelines, while others produce more variable or incomplete responses (Beheshti et al., 2025; Johnson et al., \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). Studies in maternal and neonatal health have documented that ChatGPT\u0026mdash;despite providing linguistically fluent answers\u0026mdash;may omit crucial risk\u0026ndash;benefit explanations or fail to contextualize recommendations for specific patient subgroups (Amin et al., \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Temsah et al., \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e2025\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThe high inter-rater agreement for Likert scores (ICC\u0026thinsp;=\u0026thinsp;0.89) further reinforces that factual accuracy is reliably distinguishable among models. Consistent with Shiferaw et al. (\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e2024\u003c/span\u003e), this finding suggests that experts can distinguish between evidence-based guidance and generic advice, even when stylistic or linguistic variations exist. However, the results also highlight that producing factually correct statements does not necessarily translate into well-structured, patient-centered educational content\u0026mdash;an aspect more effectively captured by DISCERN than by Likert-based reliability ratings.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003eReadability: The Quality\u0026ndash;Accessibility Trade-off\u003c/h2\u003e \u003cp\u003eA key finding of this study was the significant variation in readability across the four models. ChatGPT generated the most readable output, followed by Gemini and Claude, whereas DeepSeek produced the most linguistically complex responses. These results are consistent with prior research demonstrating variability in readability across AI systems (Kirchner et al., \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Patel et al., \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Rouhi et al., \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e2024\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eCritically, none of the models in our study met the readability standards recommended by the AMA and NIH for patient-facing educational materials. This finding aligns with previous evidence suggesting that AI-generated health texts often exceed recommended readability thresholds (Jindal \u0026amp; MacDermid, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2017\u003c/span\u003e; Shafau \u0026amp; Wahl, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). Given the well-established association between low maternal health literacy and early cessation of breastfeeding (Gaupšienė et al., \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Valero-Chiller\u0026oacute;n et al., \u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e2021\u003c/span\u003e, \u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e2022\u003c/span\u003e), the production of overly complex breastfeeding content by LLMs may disproportionately disadvantage vulnerable populations. Mothers with lower health literacy are also more likely to supplement unnecessarily or discontinue breastfeeding due to misinterpretation of infant behaviors such as crying or perceived insufficient milk supply (Kilfoyle et al., \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2016\u003c/span\u003e). Accordingly, the readability limitations observed across all AI systems represent a clinically significant barrier to equitable lactation support.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003eSafety Risks and Misinformation: Clinical Relevance of Model Variability\u003c/h2\u003e \u003cp\u003eBeyond quality and readability, LLMs pose clinically significant safety concerns, particularly in neonatal and perinatal contexts. Our findings of inconsistent quality across models align with recent analyses demonstrating that LLMs can confidently generate clinically incorrect or misleading breastfeeding advice (Blease \u0026amp; Torous, \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Chen et al., \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2025a\u003c/span\u003e). When prompts contain false premises, models may reinforce misinformation\u0026mdash;a phenomenon with important implications for lactation counseling (Chen et al., \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2025b\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eMoreover, as Hatem et al. (\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2023\u003c/span\u003e) and Pierri (\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) have noted, AI-generated misinformation can spread rapidly, amplifying anxiety, fueling misconceptions, and undermining maternal confidence. Considering that perceived insufficient milk supply is one of the leading modifiable causes of undesired breastfeeding cessation (Cheng et al., \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2023\u003c/span\u003e), erroneous or ambiguous AI-generated information may have substantial downstream consequences for maternal and infant health.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003eLimitations\u003c/h2\u003e \u003cp\u003eThis study has several limitations that should be considered when interpreting the findings. First, the evaluation was based on a limited set of ten breastfeeding questions, which, although clinically relevant and expert-validated, may not fully represent the breadth of real-world breastfeeding information needs. Second, AI responses were generated at a single time point, yet large language models evolve rapidly; thus, the results may not reflect future model updates or performance variations across different versions. Third, the DISCERN and Likert assessments relied on subjective expert judgments, and although inter-rater reliability was examined, the inherent subjectivity of narrative content evaluation may have influenced the scoring. Additionally, readability metrics such as FRES and FKGL, initially developed for English texts, were applied to Turkish-language outputs and may not capture all linguistic nuances or structural complexities of Turkish grammar. The study also did not evaluate other vital dimensions such as cultural appropriateness, actionability, or alignment with national breastfeeding guidelines. Finally, the analysis was limited to textual responses, excluding the examination of multimodal capabilities (e.g., images, audio explanations), which are becoming increasingly relevant in patient education. These limitations suggest that while the findings provide an essential comparative snapshot of current AI performance, they should be interpreted with caution and complemented by future research employing broader datasets, longitudinal designs, and additional evaluation tools.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003eImplications for Clinical Practice and Future Research\u003c/h2\u003e \u003cp\u003eTaken together, these findings underscore the need for structured oversight, expert review, and the design of tailored prompts before AI systems can be responsibly integrated into breastfeeding education. Although LLMs offer considerable promise in expanding access to breastfeeding information, the combined challenges of variable information quality, limited readability, and potential for misinformation necessitate cautious implementation. Emerging solutions, such as retrieval-augmented generation (Bunnell et al., \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) and domain-specific fine-tuning (Kalai et al., \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e2025\u003c/span\u003e), may mitigate these risks; however, further empirical testing is required.\u003c/p\u003e \u003c/div\u003e"},{"header":"Conclusion","content":"\u003cp\u003eThis study presents a comprehensive comparison of four large language models in generating breastfeeding-related health information, with a focus on information quality, reliability, and readability. The findings demonstrate that while all models could produce clinically relevant responses, the quality and accessibility of these outputs varied substantially.\u003c/p\u003e \u003cp\u003e DeepSeek and Claude generated higher-quality, more guideline-consistent information, but the resulting text required higher literacy levels. In contrast, ChatGPT and Gemini delivered more readable content at the expense of information quality and reliability. Notably, none of the models met the ideal readability thresholds recommended for patient education materials.\u003c/p\u003e \u003cp\u003eThese findings reveal a clear trade-off between informational depth and linguistic accessibility across LLMs. Despite strong inter-rater agreement for both DISCERN and reliability assessments, the results highlight the complexity of evaluating narrative health information and underscore the need for multidimensional assessment frameworks. Overall, the study highlights the importance of cautious, expert-guided integration of AI systems into breastfeeding education to ensure safety, accuracy, and equity.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003eEthical approval was obtained from the Atat\u0026uuml;rk University Faculty of Health Sciences Ethics Committee.\u0026nbsp;(Date: 05.12.2025, Decision No: 2025/12/17)\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eConsent for publication: Not applicable\u003c/p\u003e\n\u003cp\u003eCompeting interests: The authors declare no competing interests\u003c/p\u003e\n\u003cp\u003eFunding: No funding.\u003c/p\u003e\n\u003cp\u003eAvailability of data and materials: The datasets used and analyzed in this study are available from the corresponding author upon reasonable request.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthors\u0026apos; contributions:\u003c/strong\u003e Study design: SEA, AG, SET; Data collection: SEA, AG, SET; Data analysis: SEA, AG, SET, Draft preparation: SEA, AG, SET; Critical review for content: SEA, AG, Final approval of the version to be published: SEA, AG, SET\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n \u003cli\u003e\u003cspan\u003eAlber D, Yang Z, Alyakin A, Yang E, Shesh N, Valliani A, Oermann E. Medical large language models are vulnerable to data-poisoning attacks. Nat Med. 2025;31(2):618\u0026ndash;26. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41591-024-03445-1\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eAmin KS, Mayes LC, Khosla P, Doshi RH. Assessing the Efficacy of Large Language Models in Health Literacy: A Comprehensive Cross-Sectional Study. Yale J Biol Med. 2024;97(1):17\u0026ndash;27. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.59249/ZTOZ1966\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eAslan İ\u0026Ouml;, Aslan MT. Benchmarking ai chatbots for maternal lactation support: a cross-platform evaluation of quality, readability, and clinical accuracy. Healthcare. 2025;13(14):1756. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3390/healthcare13141756\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eBehers B, Vargas I, Behers B, Rosario M, Wojtas C, Deevers A, Hamad K. Assessing the readability of patient education materials on cardiac catheterization from artificial intelligence chatbots: an observational cross-sectional study. Cureus. 2024;16(7):e63865. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.7759/cureus.63865\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eBeheshti M, Toubal IE, Alaboud K, Almalaysha M, Ogundele OB, Turabieh H, Dahu BM. Evaluating the reliability of chatgpt for health-related questions: A systematic review. Informatics. 2025, January;12(1):9. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3390/informatics12010009\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eBlease C, Torous J. ChatGPT and mental healthcare: balancing benefits with risks of harms. BMJ Ment Health. 2023;26(1):e300884. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1136/bmjment-2023-300884\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eBunnell D, Bondy M, Fromtling L, Ludeman E, Gourab K. (2025). Bridging ai and healthcare: a scoping review of retrieval-augmented generation-ethics, bias, transparency, improvements, and applications. \u003cem\u003emedRxiv 2025.04.01.25325033.\u003c/em\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1101/2025.04.01.25325033\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eCharnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health. 1999;53(2):105\u0026ndash;11.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eChen S, Gao M, Sasse K, Hartvigsen T, Anthony B, Fan L, Bitterman D. Abstract ia04: evaluating and mitigating medical misinformation risk in large language models. Clin Cancer Res. 2025a;31(2Supplement):IA04. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1158/1557-3265.targetedtherap-ia04\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eChen S, Gao M, Sasse K, Hartvigsen T, Anthony B, Fan L, Bitterman DS. When helpfulness backfires: LLMs and the risk of false medical information due to sycophantic behavior. npj Digit Med. 2025b;8(1):605. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41746-025-02008-z\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eCheng GZ, Chen A, Xin Y, Ni QQ. Using the teach-back method to improve postpartum maternal-infant health among women with limited maternal health literacy: a randomized controlled study. BMC Pregnancy Childbirth. 2023;23(1):13. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1186/s12884-022-05302-w\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eDavey M, Hayes J, Hurley E, Cassidy J. Evaluation and comparison of the quality and readability of online information for patients using the chat gpt (artificial intelligence) platform versus other online sources regarding hip fractures management. Trauma. 2025;27(4):301\u0026ndash;6. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1177/14604086251345001\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eGaup\u0026scaron;ienė A, Vainauskaitė A, Baglajeva J, Stukas R, Rama\u0026scaron;auskaitė D, Paliulytė V, Istomina N. Associations between maternal health literacy, neonatal health and breastfeeding outcomes in the early postpartum period. Eur J midwifery. 2023;7:25. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.18332/ejm/170161\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eG\u0026ouml;kalp M, Y\u0026uuml;cel Ş, \u0026Ccedil;akır Z. (2025). Readability, reliability, and quality of nursing care plan texts generated by chatgpt. \u003cem\u003eBMC Nursing, under review\u003c/em\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.21203/rs.3.rs-7237457/v1\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eHatem R, Simmons B, Thornton JE. (2023). A call to address AI hallucinations and how healthcare professionals can mitigate their risks. Cureus, \u003cem\u003e15\u003c/em\u003e(9).\u0026nbsp;\u003cspan\u003e\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://goodcalculators.com/flesch-kincaid-calculator/\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eJindal P, MacDermid JC. (2017). Assessing reading levels of health information: uses and limitations of flesch formula. \u003cem\u003eEducation for Health\u003c/em\u003e, \u003cem\u003e30\u003c/em\u003e(1), 84\u0026ndash;88. \u003cem\u003eDOI: 10.4103/1357-6283.210517\u003c/em\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eJohnson D, Goodman R, Patrinely J, Stone C, Zimmerman E, Donald R, Wheless L. Assessing the accuracy and reliability of AI-generated medical responses: an evaluation of the Chat-GPT model. Res square. 2023;rs\u0026ndash;3. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.21203/rs.3.rs-2566942/v1\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eKacer EO. Evaluating AI-based breastfeeding chatbots: quality, readability, and reliability analysis. PLoS ONE. 2025;20(3):e0319782. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1371/journal.pone.0319782\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eKalai AT, Nachum O, Vempala SS, Zhang E. (2025). Why language models hallucinate. \u003cem\u003earXiv preprint arXiv:2509.04664\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.2509.04664\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eKilfoyle KA, Vitko M, O\u0026apos;Conor R, Bailey SC. Health literacy and women\u0026apos;s reproductive health: a systematic review. J women\u0026apos;s health. 2016;25(12):1237\u0026ndash;55.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eKılıcoglu MS, Yurdakul OV, Aydın T. Evaluation of Chat Generative Pre-trained Transformer\u0026rsquo;s responses to frequently asked questions about psoriatic arthritis: A study on quality and readability. Ann Med Res. 2025;32(2):79\u0026ndash;84. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.5455/annalsmedres.2024.10.231\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eKincaid JP, Fishburne RP, Rogers RL, Chissom BS. (1975). Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy enlisted personnel. Research Branch Report 8\u0026ndash;75. Naval Technical Training Command, Millington, TN.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eKirchner GJ, Kim RY, Weddle JB, Bible JE. (2023). Can artificial intelligence improve the readability of patient education materials? \u003cem\u003eClinical Orthopaedics and Related Research\u003c/em\u003e, \u003cem\u003e481\u003c/em\u003e(11), 2260\u0026ndash;2267. \u003cem\u003eDOI: 10.1097/CORR.0000000000002668\u003c/em\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003ePatel EA, Fleischer L, Filip P, Eggerstedt M, Hutz M, Michaelides E, Tajudeen BA. The use of artificial intelligence to improve readability of otolaryngology patient education materials. Otolaryngology\u0026ndash;Head Neck Surg. 2024;171(2):603\u0026ndash;8.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003ePierri F. The role of ai agents for online information disorders. XRDS Crossroads Acm Magazine Students. 2025;31(3):24\u0026ndash;7. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/3729511\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eRouhi AD, Ghanem YK, Yolchieva L, Saleh Z, Joshi H, Moccia MC, Han JJ. Can artificial intelligence improve the readability of patient education materials on aortic stenosis? A pilot study. Cardiol therapy. 2024;13(1):137\u0026ndash;47. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s40119-023-00347-0\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eShafau F, Wahl C. (2025). Evaluating the Readability of AI-Generated Patient Information on Chronic Diseases. \u003cem\u003eChronic Diseases and Translational Medicine\u003c/em\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eShao X, Ruan T, Ju X, Sun Y, Cui J. Evaluating artificial intelligence chatbots\u0026rsquo; responses to gynecomastia inquiries: comparative study of information quality, readability, and guideline consistency. Digit Health. 2025;11. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1177/20552076251367645\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eShiferaw MW, Zheng T, Winter A, Mike LA, Chan LN. Assessing the accuracy and quality of artificial intelligence (AI) chatbot-generated responses in making patient-specific drug-therapy and healthcare-related decisions. BMC Med Inf Decis Mak. 2024;24(1):404. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1186/s12911-024-02824-5\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eStephenson-Moe C, Behers B, Gibons R, Behers B, Herrera L, Anneaud D, Hamad K. Assessing the quality and readability of patient education materials on chemotherapy cardiotoxicity from artificial intelligence chatbots: an observational cross-sectional study. Medicine. 2025;104(15):e42135. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1097/md.0000000000042135\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eTemsah A, Alhasan K, Altamimi I, Jamal A, Al-Eyadhy A, Malki K, Temsah M. DeepSeek in Healthcare: Revealing Opportunities and Steering Challenges of a New Open-Source Artificial Intelligence Frontier. Cureus. 2025;17(2):e79221. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.7759/cureus.79221\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eTepe M, Emekli E. Assessing the responses of large language models (chatgpt-4, gemini, and microsoft copilot) to frequently asked questions in breast imaging: a study on readability and accuracy. Cureus. 2024;16(5):e59960. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.7759/cureus.59960\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eValero-Chiller\u0026oacute;n MJ, Gonz\u0026aacute;lez‐Chord\u0026agrave; VM, Cervera‐Gasch \u0026Aacute;, Vila‐Candel R, Soriano‐Vidal FJ, Mena‐Tudela D. Health literacy and its relation to continuing with breastfeeding at six months post‐partum in a sample of Spanish women. Nurs Open. 2021;8(6):3394\u0026ndash;402. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/nop2.885\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eValero-Chiller\u0026oacute;n MJ, Mena-Tudela D, Cervera-Gasch \u0026Aacute;, Gonz\u0026aacute;lez-Chord\u0026aacute; VM, Soriano-Vidal FJ, Quesada JA, Vila-Candel R. Influence of health literacy on maintenance of exclusive breastfeeding at 6 months postpartum: a multicentre study. Int J Environ Res Public Health. 2022;19(9):5411. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3390/ijerph19095411\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\n \u003cli\u003e\u003cspan\u003eWalters KA, Hamrell MR. Consent forms, lower reading levels, and using Flesch\u0026ndash;Kincaid readability software. Drug Inform J. 2008;42(5):385\u0026ndash;94.\u003c/span\u003e\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-pregnancy-and-childbirth","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"prch","sideBox":"Learn more about [BMC Pregnancy and Childbirth](http://bmcpregnancychildbirth.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/prch/default.aspx","title":"BMC Pregnancy and Childbirth","twitterHandle":"@BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-8558387/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8558387/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e \u003cp\u003eLarge language models (LLMs) are increasingly used to provide breastfeeding information, yet concerns remain regarding the quality, reliability, and readability of AI-generated health content.\u003c/p\u003e\u003ch2\u003eObjective\u003c/h2\u003e \u003cp\u003eTo comparatively evaluate the information quality, scientific reliability, and readability of breastfeeding-related responses generated by four widely used LLMs.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eThis descriptive cross-sectional study (September 2025) assessed responses from ChatGPT-5, Google Gemini, DeepSeek, and Claude to 10 expert-validated, clinically critical breastfeeding FAQs derived from an initial pool of 100 questions (LLM-generated and Google \u0026ldquo;People also ask\u0026rdquo;). Prompts were submitted in newly initiated chat sessions on the same day. A blinded panel of three independent experts (pediatrician, obstetrician\u0026ndash;gynecologist, senior midwife) rated each response using DISCERN (16\u0026ndash;80) for information quality and a 5-point Likert scale for scientific reliability; readability was assessed with Flesch Reading Ease Score (FRES) and Flesch\u0026ndash;Kincaid Grade Level (FKGL). Differences across models were tested using Friedman/Dunn (DISCERN, Likert) and one-way ANOVA (readability). Ethical approval was obtained from the Atat\u0026uuml;rk University Faculty of Health Sciences Ethics Committee.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eDISCERN scores differed significantly across models (χ\u0026sup2;(3)\u0026thinsp;=\u0026thinsp;76.50, p\u0026thinsp;\u0026lt;\u0026thinsp;.001). DeepSeek (37.20\u0026thinsp;\u0026plusmn;\u0026thinsp;7.17) and Claude (34.27\u0026thinsp;\u0026plusmn;\u0026thinsp;4.93) scored higher than ChatGPT (19.93\u0026thinsp;\u0026plusmn;\u0026thinsp;2.86) and Gemini (22.40\u0026thinsp;\u0026plusmn;\u0026thinsp;2.19) (p\u0026thinsp;\u0026lt;\u0026thinsp;.05); no model reached \u0026ldquo;excellent\u0026rdquo; quality (\u0026ge;\u0026thinsp;63). Likert reliability also varied (χ\u0026sup2;(3)\u0026thinsp;=\u0026thinsp;62.50, p\u0026thinsp;\u0026lt;\u0026thinsp;.001), highest for DeepSeek (3.47\u0026thinsp;\u0026plusmn;\u0026thinsp;0.63) and Claude (3.17\u0026thinsp;\u0026plusmn;\u0026thinsp;0.38) versus ChatGPT (2.03\u0026thinsp;\u0026plusmn;\u0026thinsp;0.18) and Gemini (2.07\u0026thinsp;\u0026plusmn;\u0026thinsp;0.37). Readability differed (FRES: F(3,36)\u0026thinsp;=\u0026thinsp;3.54, p\u0026thinsp;=\u0026thinsp;.024; FKGL: F(3,36)\u0026thinsp;=\u0026thinsp;3.57, p\u0026thinsp;=\u0026thinsp;.023); all models exceeded the ideal\u0026thinsp;\u0026le;\u0026thinsp;6 FKGL benchmark.\u003c/p\u003e\u003ch2\u003eConclusions\u003c/h2\u003e \u003cp\u003eLLMs show a clear trade-off between informational quality and readability. DeepSeek and Claude produced more comprehensive, guideline-consistent content, but it was less readable. In contrast, ChatGPT and Gemini were more readable, albeit with lower quality and reliability. Expert oversight remains essential before integrating LLM outputs into breastfeeding education.\u003c/p\u003e","manuscriptTitle":"Quality, Reliability, and Readability of AI-Generated Breastfeeding Information: A Comparative Evaluation of Four Large Language Models","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-02-11 10:36:44","doi":"10.21203/rs.3.rs-8558387/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"editorInvitedReview","content":"","date":"2026-02-13T20:01:50+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"72088729539009633312827813887353296504","date":"2026-02-09T14:19:20+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-02-09T09:47:39+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2026-01-15T17:38:35+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-01-13T10:59:10+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-01-13T10:53:51+00:00","index":"","fulltext":""},{"type":"submitted","content":"BMC Pregnancy and Childbirth","date":"2026-01-09T07:50:53+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-pregnancy-and-childbirth","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"prch","sideBox":"Learn more about [BMC Pregnancy and Childbirth](http://bmcpregnancychildbirth.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/prch/default.aspx","title":"BMC Pregnancy and Childbirth","twitterHandle":"@BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"54c7fde2-c7bb-40cf-ba68-4f79185c0536","owner":[],"postedDate":"February 11th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-02-11T10:36:44+00:00","versionOfRecord":[],"versionCreatedAt":"2026-02-11 10:36:44","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8558387","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8558387","identity":"rs-8558387","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}