EP63 CHATGPT Generates Responses Accurate, Yet Insufficient and Accurately Tailors Responses Based on Prompted Reading Levels When Responding to Frequently Asked Patient Questions Regarding Hamstring Injuries

Abstract

Abstract Purpose This study aimed to evaluate the accuracy of ChatGPT’s responses to frequently asked questions (FAQs) about hamstring injuries and to determine if prompted, if ChatGPT could appropriately tailor the reading level to that suggested. Methods A preliminary list of 15 questions was developed, from which the authors selected the 10 most encountered patient questions. Three queries were performed, inputting the questions into ChatGPT-4.0: 1) unprompted, naïve, 2) additional prompt specifying the response being tailored to a seventh grade reading level, 3) additional prompt specifying the response being tailored to a college graduate reading level. The responses from the unprompted query were independently evaluated by two of the authors. To assess the quality of the answers, a grading system was applied: (A) correct and sufficient response; (B) correct but insufficient response; (C) response containing both correct and incorrect information; and (D) incorrect response. Additionally, the readability of each response was measured using the Flesch-Kinkaid Reading Ease Score (FRES) and Grade Level (FKGL) scales. Results Ten responses were evaluated. Inter-rater reliability was 0.6 regarding grading. Of the initial query, two out of ten responses received a grade of A, seven were graded as B, and one were graded as C. The average cumulative FRES and FKGL scores of the initial query was 61.64 and 10.28, respectively. The average cumulative FRES and FKGL scores of the secondary query were 75.2 and 6.1, respectively. Finally, the average FRES and FKGL scores of the third query were 12.08 and 17.23. Conclusion ChatGPT demonstrated generally satisfactory accuracy in responding to questions regarding hamstring injuries, although certain responses lacked completeness or specificity. On initial, unprompted queries, the readability of responses aligned with a tenth-grade level. However, when explicitly prompted, ChatGPT reliably adjusted the complexity of its responses to both a seventh-grade and a graduate-level reading standard. These findings suggest that while ChatGPT may not consistently deliver fully comprehensive medical information, it possesses the capacity to adapt its output to meet specific readability targets.

Ähnliche Arbeiten