Quality Assessment of Pathology Board-Exam-Style MCQs Produced by ChatGPT3.5: A Comparative Study

Abstract

Residents preparing for pathology board exams frequently use multiple-choice questions (MCQs) from question banks (QBs) like PathDojo and PathPrimer, which can be costly. ChatGPT, a free tool, has been used to generate MCQs for other tests like the SAT. This study compared the quality of pathology MCQs created by ChatGPT versus commercially available study questions for the American Board of Pathology’s (ABPath) certifying exams. A rubric adapted from the National Board of Medical Examiners’ (NBME) question writing guide was validated by two pathologists using commercially available pathology board exam questions. This rubric was then used to evaluate MCQs from commercially available pathology board study books as well as MCQs created by ChatGPT. The results compared the percentage of criteria met between ChatGPT and control MCQs using chi-square analysis with significance set at <0.05. While ChatGPT MCQs were less likely to be accurate compared to commercially available MCQs in four criteria (the best answer choice (82.5% vs. 100%), reflection of current practice (84.6% vs. 100%), error-free explanation (87.9% vs. 100%), and explanation reflecting current practice (87.9% vs. 100%)), the complexity of the ChatGPT-generated questions was higher (78.5% vs. 47.2%). At this time, ChatGPT-generated MCQs should not be used in the same way as commercially available study guides, however there is potential for learned language models (LLM)s to create quality study materials and exam questions with careful monitoring

Ähnliche Arbeiten