Autonomous Analysis of Curated Patient Data Using a Large Language Model–Based Multiagent Framework

Abstract

PURPOSE Analyzing complex medical data sets is specialized and time-consuming. This study aimed to develop and evaluate a novel multiagent artificial intelligence (AI) framework for automating medical data analysis workflows and to compare its performance against nonagent-based approaches using large language models (LLMs). METHODS A six-party AI agent system was developed using the AutoGen platform, with specialized agents for planning, data retrieval, cleaning, statistical analysis, and review, powered by OpenAI gpt-4o. This framework was applied to deidentified single patient-level data sets from 20 recent studies in the field of bone marrow transplantation (2021-2023). The primary objective was to evaluate its accuracy in replicating published primary outcomes, benchmarked against direct use of the Web site–based ChatGPT 4o. RESULTS The multiagent framework successfully replicated 53.3% (95% CI, 40.7 to 66.0) of primary outcomes, significantly outperforming ChatGPT 4o (35.0% [95% CI, 22.9 to 47.1]; P = .04). The agent framework's failures were predominantly due to data transformation issues (46.4%) and analysis code errors (21.4%). In contrast, ChatGPT 4o failures largely stemmed from incorrect statistical method application (38.4%) and data transformation (45.6%), often attempting to resolve code errors by switching to alternative, incorrect statistical methods. Hallucinations of variables or results were not observed in the multiagent approach. CONCLUSION Our multiagent AI framework demonstrated superior accuracy and robustness in automating biomedical data analysis compared with a generalized LLM.

Ähnliche Arbeiten