The AI challenge: A turing test pilot study of attendings and residents in identifying AI-Generated content

Abstract

Generative Artificial Intelligence (AI) models have demonstrated strong potential in radiology report generation, but their clinical adoption depends on physician trust. In this pilot study, we conducted a radiology-focused Turing test to evaluate how well attendings and residents distinguish AI-generated reports from those written by radiologists, and how their confidence and decision time reflect trust. We developed an integrated web-based platform for report evaluation. Using the web-based platform, eight participants (4 attendings and 4 residents) evaluated 48 anonymized X-ray cases, each paired with two reports from three comparison groups: radiologist vs. AI model 1, radiologist vs. AI model 2, and AI model 1 vs. AI model 2. Participants were asked to select the AI-generated report, rate their confidence, and indicate report preference. Results show that attendings outperformed residents in identifying AI-generated reports (49.9 % vs. 41.1 %) and exhibited longer decision times, suggesting more deliberate judgment. Both groups took more time when both reports were AI-generated. Our findings highlight the role of clinical experience in AI acceptance and the need for design strategies that foster trust in clinical applications.

Ähnliche Arbeiten