Closing the Gap: Using Chatbots to Improve the Readability of Patient Education Materials in Urogynecology

Abstract

INTRODUCTION: Patient education materials created by professional societies are valuable clinical resources, being freely accessible online, frequently distributed by physician offices, and covering a variety of topics in urogynecology. Although a 6th-grade reading level is recommended for patient materials by the American Medical Association and Agency for Healthcare Research and Quality, many available resources are written at a higher level, posing a challenge to meaningful patient education with implications for disparities related to social determinants of health. OBJECTIVE: We aim to determine if 1) current patient education materials created by urogynecology professional societies are written at the recommended reading level, 2) artificial intelligence (AI) chatbots can rewrite these materials at a 6 th-grade reading level, and 3) the writing quality differs between original and chatbot-generated versions. METHODS: All 25 American Urogynecologic Society (AUGS) fact sheets and 45 International Urogynecological Association (IUGA) leaflets were selected. Publicly available AI chatbots, ChatGPT free, ChatGPT 4o, Claude, and Gemini, were provided with the original text and prompted, “Rewrite the entire following text at a 6th grade reading level.” All original and chatbot-generated versions were analyzed for reading level using validated readability formulas, Flesch-Kincaid Grade Level, Simple Measures of Gobbledygook, and Gunning Fog Index. They were also examined for writing quality using the AI writing assistant Grammarly. Readability and grammar measures of the original and chatbot-generated documents were compared using analysis of variance techniques employing a Dunnett adjustment for multiple comparisons. Post hoc Tukey's studentized range for pairwise comparisons of means compared differences among the chatbots. Statistical significance was defined as p<.05. RESULTS: The average reading levels of AUGS and IUGA materials were grades 10.67 and 11.72, respectively. All chatbot-generated versions achieved a significantly lower average reading level than the original AUGS and IUGA materials, except for those generated by ChatGPT 4o based on original AUGS documents (Table 1). Chatbots Gemini and Claude performed significantly better than both versions of ChatGPT across all materials on pairwise comparison. The only paid chatbot, ChatGPT 4o, underperformed compared to the 3 free chatbots. Claude was the only chatbot to consistently achieve the goal reading level for most documents (80%) (Figures 1 and 2). All chatbot-generated versions of the patient education materials had significantly fewer average total writing issues than the original versions by AUGS and IUGA (Table 1). CONCLUSIONS: Existing patient education materials available through AUGS and IUGA are written at reading levels higher than the recommended 6th-grade level. All AI chatbots were able to rewrite the currently available materials at a lower reading level. However, most chatbots still fell short of achieving the goal reading level. Of the four chatbots, Claude performed the best, most consistently achieving a 6th-grade reading level. Future studies will assess the medical accuracy and patient understanding of the chatbot-generated materials.

Ähnliche Arbeiten