MEDAI-LLM-SUMM: a reporting checklist for medical text summarization studies using large language models

Abstract

The final MEDAI-LLM-SUMM checklist comprises 24 items organized into six sections: (A) Clinical validity (4 items addressing clinical task definition, expert involvement, hypothesis formulation, and medical expertise requirements); (B) Model Selection (5 items covering model justification, system requirements, deployment environment, LLM-as-judge approach, and prompt documentation); (C) Data (3 items on datasets, reference summaries with expert consensus, and data stratification); (D) Quality Assessment (8 items including evaluation metrics, clinical metrics, expert evaluation, hallucination detection, LLM-judge assessment, sample size justification, pilot testing, and limitations documentation); (E) Safety (2 items on ethical approval and data anonymization); and (F) Data Availability (2 items on code and dataset accessibility). Comparative analysis with six existing reporting standards demonstrated that MEDAI-LLM-SUMM uniquely addresses hallucination assessment requirements, reference summary creation methodology, LLM-as-judge validation protocols, and detailed pilot testing specifications.

Ähnliche Arbeiten