Evaluating the Accuracy of ChatGPT in the Japanese Board-Certified Physiatrist Examination

Abstract

Background Generative artificial intelligence (AI), such as Chat Generative Pre-trained Transformer (ChatGPT), has shown potential in various medical applications, including answering licensing examination questions. However, its performance in rehabilitation medicine remains underexplored. This study aimed to evaluate the accuracy of ChatGPT4o in answering questions from the Japanese Board-Certified Physiatrist Examination and assess its potential as an educational and clinical support tool. Methods This study assessed the performance of ChatGPT4o on questions from the 2021-2023 Japanese Board-Certified Physiatrist Examinations. Questions were categorized into text- and image-based types and correct response rates were calculated. Errors were classified into informational, logical, or statistical. Results ChatGPT4o achieved correct response rates of 79.1% in 2021, 80.0% in 2022, and 86.3% in 2023, with an overall accuracy of 81.8%. The AI performed better on text-based (83.0%) than on image-based (70.0%) questions. Most errors (92.8%) were related to information. Conclusions ChatGPT4o demonstrated high accuracy in the Japanese Board-Certified Physiatrist Examination, particularly for text-based questions, demonstrating its potential as an educational tool. However, limitations in image interpretation and specialized topics indicate the need for further improvements for clinical application.

Ähnliche Arbeiten