Artificial Intelligence in Obstetrics: Evaluating ChatGPT and Google Gemini in Answering Patient Questions

Objective: To evaluate the accuracy and completeness among different Large Language Models (LLMs), such as ChatGPT and Google Gemini, in responding to patients’ frequently asked questions regarding obstetric topics. Design: Comparative analysis of responses generated by LLMs to obstetric-related patient questions. Setting: Online platforms utilizing ChatGPT and Google Gemini. Population or Sample: Responses from ChatGPT and Google Gemini to patient-generated questions on five obstetrics topics, evaluated by board-certified Obstetrics and Gynecology physicians. Methods: : Five common obstetric topics were identified: prenatal labs, extended carrier screen, treatments for nausea and vomiting in pregnancy, gestational diabetes, and trial of labor after cesarean section. ChatGPT was used to generate the five most frequently asked patient questions for each topic. These questions were posed to ChatGPT and Google Gemini. Physician evaluators used Likert scales to rate response accuracy (1–6) and completeness (1–3). Main Outcome Measures: Accuracy and completeness ratings, inter-rater agreement scores, and statistical comparison between the models using a Wilcoxon signed-rank test. Results: : Most responses from both models met the criteria for acceptable accuracy (≥5) and completeness (≥2). A Wilcoxon Signed-Rank Test revealed a statistically significant difference in accuracy and completeness between models (p < 0.05). Inter-rater agreement scores for ChatGPT were -0.047 (completeness) and 0.112 (accuracy), and for Google Gemini were 0.367 (completeness) and 0.205 (accuracy). Conclusions: : ChatGPT and Google Gemini provided accurate and complete responses to common obstetric questions, suggesting potential utility in patient medical education. However, patients should always confirm online information with their physician.