Comparative Evaluation of GPT Models in FHIR Proficiency

Abstract

Ensuring interoperability in healthcare data exchange is vital for advancing patient care, and Fast Healthcare Interoperability Resources (FHIR®) has emerged as a cornerstone standard in this effort. As healthcare increasingly integrates AI for managing and interpreting complex data, proficiency in FHIR is essential to ensure seamless and reliable interactions with healthcare systems. This study evaluates the FHIR proficiency of Generative Pre-trained Transformer (GPT) models, which serves as a critical benchmark for applying artificial intelligence (AI) in healthcare. The performance of GPT-3.5, GPT-4.0, and two custom models was assessed in two FHIR examination scenarios using novel metrics, including Token Processing Cost (TPC), Accuracy-Adjusted Token Processing Cost (ATPC), Comprehensive Performance Index (CPI), and Quality-Adjusted Performance Score (QAPS). GPT-4.0 demonstrated superior accuracy and robustness, while custom models such as the “FHIR Interop Expert” showed strengths in domain-specific tasks through effective prompt engineering. Despite these capabilities, none of the models consistently achieved the \(\geq 99\) % accuracy required for high-stakes healthcare applications. The findings underscore the importance of refining domain-specific training and evaluation methods. The proposed metrics provide a replicable framework for assessing AI readiness, offering a foundation for the responsible and effective integration of AI into healthcare workflows.

Ähnliche Arbeiten