Human vs. AI: Can You ChatGPT Your Way Into Residency?

Abstract

Background: Chat Generative Pretrained Transformer (ChatGPT) is a popular and easily accessible artificial intelligence (AI) program often used for creation of human- like text. It was released in November 2022 to the public. It has the capability to provide long format text with minimal prompting, including residency applicant personal statements. Previous studies have shown AI-generated anesthesia residency application statements are acceptable1 and AI-generated plastic surgery statements are indistinguishable from human. Objective: The purpose of this study is to determine if Ob-Gyn physician faculty were able to differentiate between ChatGPT-generated and human-generated statements. This study further analyzed 3 different AI detection softwares’ abilities to detect AI-generated personal statements in comparison to human-generated statements. Methods: Six human statements were voluntarily submitted, each created prior to release of ChatGPT. For AI statement generation, investigators summarized the topics of the human statements to create outlines matching the content of the original statements. This outline was then fed into ChatGPT-3.5 with the instruction to generate an obstetrics and gynecology residency application personal statement. No editing was performed after AI statement generation. A total of 12 statements were included: 6 human-generated statements and 6 AI-generated equivalents. Two packets containing 6 randomized statements were created, each containing 3 human and 3 AI statements. Eleven faculty each reviewed one packet and were asked to determine which were ChatGPT- versus human- generated. Statement content did not cross over among the packets. Each statement was fed into 3 different free AI detection programs. Each program provided an overall assessment of statement originality. Two of three programs provided a more in-depth analysis including a percentage of content suspicious for AI-generation. Results: Statement reviewers correctly determined statement origin 57% of the time. AI statements were correctly identified 52% of the time and human-generated statements were correctly identified 64% of the time. The combined accuracy of AI detection software totaled 86%. AI detection software detected 89% of AI-generated statements and 83% of human-generated statements. Conclusions: Physician faculty were not able to determine AI-generated statements notably more than chance (50%). Three AI detection software programs performed more accurately but also incorrectly flagged human generated statements as possible AI. This could have significant negative impacts on applicants. It is likely residency programs will see increased reliance on AI language generators for preparation of residency applications. Programs may need to evaluate their perception of the personal statement as the primary metric of written communication ability, as well as confront the ethical question of statement originality and requirements thereof.

Ähnliche Arbeiten