Validation of the QAMAI tool to assess the quality of health information provided by AI

Abstract

Abstract Objective To propose and validate the Quality Assessment of Medical Artificial Intelligence (QAMAI), a tool specifically designed to assess the quality of health information provided by AI platforms. Study design observational and valuative study Setting 27 surgeons from 25 academic centers worldwide. Methods The QAMAI tool has been developed by a panel of experts following guidelines for the development of new questionnaires. A total of 30 responses from ChatGPT4, addressing patient queries, theoretical questions, and clinical head and neck surgery scenarios were assessed. Construct validity, internal consistency, inter-rater and test-retest reliability were assessed to validate the tool. Results The validation was conducted on the basis of 792 assessments for the 30 responses given by ChatGPT4. The results of the exploratory factor analysis revealed a unidimensional structure of the QAMAI with a single factor comprising all the items that explained 51.1% of the variance with factor loadings ranging from 0.449 to 0.856. Overall internal consistency was high (Cronbach’s alpha=0.837). The Interclass Correlation Coefficient was 0.983 (95%CI 0.973-0.991; F(29,542)=68.3; p <0.001), indicating excellent reliability. Test-retest reliability analysis revealed a moderate-to-strong correlation with a Pearson’s coefficient of 0.876 (95%CI 0.859-0.891; p <0.001) Conclusions The QAMAI tool demonstrated significant reliability and validity in assessing the quality of health information provided by AI platforms. Such a tool might become particularly important/useful for physicians as patients increasingly seek medical information on AI platforms.

Ähnliche Arbeiten