Artificial intelligence versus physical medicine and rehabilitation residents: Can <scp>ChatGPT</scp> compete in clinical exam performance?

Abstract

BACKGROUND: Artificial intelligence has begun to replace human power in many areas today. OBJECTIVE: To assess the performance of Chat Generative Pretrained Transformer (ChatGPT) on examinations administered to physical medicine and rehabilitation (PM&R) residents. DESIGN: Cross-sectional study. SETTING: Tertiary-care training and research hospital, department of physical medicine and rehabilitation. PARTICIPANTS: ChatGPT-4o and PM&R residents. INTERVENTION: ChatGPT was presented with questions from the annual nationwide in-training exams administered to PM&R residents at different postgraduate years. The exam is a national requirement for the majority of PM&R residents in Turkey and is administered annually. MAIN OUTCOME MEASURES: The responses to these multiple-choice questions were evaluated as correct or incorrect, and ChatGPT's performance was then compared to that of the residents of each postgraduate year (PGY) term. The time taken by ChatGPT to answer each question was also recorded. Additionally, its learning ability was assessed by reasking the questions it initially answered incorrectly, this time providing the correct answers to evaluate improvement. RESULTS: ChatGPT received a score of 88 out of 100 points in the PGY1 exam, 84 points in the PGY2 exam, 78 points in the PGY3 exam, and 80 points in the PGY4 exam. When compared with the performance distribution of residents, ChatGPT ranked in the 40th-50th percentile for PGY1, 70th-80th percentile for PGY2, 30th-40th percentile for PGY3, and 40th-50th percentile for PGY4. It has been demonstrated that ChatGPT has achieved a learning rate of 65%. CONCLUSION: Despite the potential of ChatGPT to surpass PM&R physicians in terms of learning capabilities and extensive knowledge network, several functional limitations remain. In its current form, it is not capable of replacing a physician, especially in the field of PM&R, where clinical examination and patient interaction play a critical role.

Ähnliche Arbeiten