1631: ENHANCING PHYSICIAN SEP1 COMPLIANCE WITH AUTOMATED FEEDBACK FROM A LARGE LANGUAGE MODEL

Abstract

Introduction: the Centers for Medicare and Medicaid Services (CMS) to encourage rapid recognition and treatment of sepsis. Compliance with SEP-1 may reduce mortality, rates of organ failure and decrease length of hospital stay. We previously demonstrated that large language models (LLM) determine SEP-1 compliance as accurately as expert human reviewers. We sought to test the hypothesis that automated feedback to physicians using an LLM increases SEP-1 compliance compared to usual care. Methods: A prospective randomized study was conducted between December 2024 to July 2025 and included 62 attending physicians at the University of California, San Diego Health (UCSDH). Using a baseline 65% SEP-1 compliance, we calculated that 300 cases would be needed to demonstrate a 15% increase in SEP-1 compliance in the intervention group. Physicians were randomized to either receive timely feedback on quality of care using the LLM (intervention group) or standard care. If intervention physicians met all elements of SEP-1, they received a standardized email congratulating them and a brief reminder of UCSD’s sepsis policy. Physicians whose patient care did not meet the SEP-1 metric were sent an email describing the criteria that failed to be met. Physicians in the usual care group received feedback only if a sepsis case was sent to CMS for public reporting. Comparisons between the intervention and control groups were completed with Fisher’s exact tests in Python version 3.9.21 using Scipy version 1.13.1. An alpha > 0.05 was considered significant. Results: From December 2024 to July 2025, there were 180 cases in the intervention arm and 121 cases in the control arm. Patients whose cases fell in the intervention group had physician compliance rates of 81.1% and patients whose cases fell in the control group had physician compliance rates of 69.2% (p=0.024). This finding was largely driven by an increased compliance with the 30 mL/kg fluid bolus component. Conclusions: This study suggests that automated feedback significantly improves physician compliance with the SEP-1 metric. Future studies should explore physician perceptions toward automated feedback based on LLM abstraction and its impact on clinical decision making.

Ähnliche Arbeiten