Enhancing Patient Education With AI:A Readability Analysis of AI-Generated Versus American Academy of Ophthalmology Online Patient Education Materials

Abstract Purpose: This study aims to compare the readability of patient education materials (PEMs) written by the American Academy of Ophthalmology (AAO) with those generated by large language models (LLMs), including ChatGPT-4o, Microsoft Copilot, and Meta-Llama-3.1-70B-Instruct. Methods: LLMs were prompted to generate PEMs for 15 common diagnoses relating to cornea and anterior chamber, which was followed by a prompt to reword the content at a 6th-grade reading level. The readability of these materials was evaluated using nine different readability analysis python libraries and compared to existing PEMs found on the AAO website. Results: For all 15 topics, ChatGPT, Copilot, and Llama successfully generated PEMs, though all exceeded the recommended 6th-grade reading level. While unprompted ChatGPT, Copilot, and Llama outputs were 10.8, 12.2, and 13.2, respectively, prompting significantly improving readability to 8.3 for ChatGPT, 11.2 for Copilot, and 9.3 for Llama (p < 0.001). While readability improved, AI-generated PEMs were on average, not statistically easier to read than AAO PEMs, which averaged an 8.0 Flesch-Kincaid grade level. Conclusions: Prompted AI chatbots can generate PEMs with improved readability, nearing the level of AAO materials. However, most outputs remain above the recommended 6th-grade reading level, and the brevity of Copilot's responses raises concerns about content quality. By creating a blueprint, AI chatbots show promise as tools for ophthalmologists to increase the availability of accessible PEMs in ophthalmology.