Assessing the Accuracy, Readability, and Clinical Applicability of Artificial Intelligence Chatbots in Primary Bladder Pain Syndrome Management: A Cross-Sectional Methodological Study

Abstract

Objective: This study aimed to evaluate the quality, readability, actionability, and guideline adherence of medical information provided by artificial intelligence chatbots (AICs) regarding treatment options for primary bladder pain syndrome (PBPS). Material and Methods: Four AICs were queried with the question: ''What treatments are available for bladder pain syndrome?''. Their responses were evaluated by 5 expert urologists using DISCERN Patient Education Materials Assessment Tool for Print Materials (PEMAT-P), the Web Resource Rating (WRR), the Coleman-Liau Index, and a guideline adherence Likert scale based on the European Association of Urology (EAU) guidelines. Data were analysed and reported using median (minimum-maximum) values for subjective scores and mean values for word count and readability. Results: Perplexity and Gemini achieved the highest median DISCERN scores (52), followed by Copilot and Chat Generative Pretrained Transformer (ChatGPT). Understandability was highest for Perplexity (75%), while actionability remained low across all platforms. Perplexity achieved the best WRR score (44.2), while ChatGPT scored the lowest (14.3). Readability analysis showed that AIC responses required a university-level education for comprehension, with Coleman-Liau Index scores ranging from 16.02 to 19.35. Guideline adherence according to EAU was moderate, with ChatGPT and Perplexity scoring highest. Conclusion: Although AICs demonstrated moderate to good reliability and understandability in providing information about PBPS treatment, concerns regarding high reading complexity and low actionability remain. AICs offer promising supplementary tools for patient education, but significant improvements in readability, actionable guidance, and clinical accuracy are needed before broader implementation in urological practice.

Ähnliche Arbeiten