System Architecture and Prospective Evaluation Protocol of AskDoc: A Secure Multimodal mHealth Triage Assistant — Pilot Safety and Feasibility Study

Abstract

Aim: This study is explicitly designed as a feasibility and safety pilot, not a powered efficacy trial. Its dual purpose is: (i) to establish whether an AI-augmented, rule-constrained triage strategy can reduce clinically unsafe under-triage compared with a rule-based baseline while maintaining acceptable accuracy across a range of adult symptom presentations; and (ii) to pre-register a comprehensive evaluation protocol for future adequately powered studies. Materials and Methods: In a prospective observational study, adult symptom presentations were triaged using a baseline rules engine and an AI-augmented system (rules + LLM). The final analytic sample consisted of 20 index presentations with clinician reference labels serving as the ground truth. The primary endpoint was exact-match triage accuracy; secondary endpoints were under-triage, over-triage, and specialist-routing accuracy. Results: The baseline system correctly classified 12 of 20 presentations (60%); the AI-augmented system correctly classified 13 of 20 (65%; paired risk difference = 0.05; 95% CI −0.15 to +0.25; p = 0.727, McNemar’s exact test). Critically, zero red-flag misses were recorded under either condition, and the AI-augmented system achieved 100% query coverage with no uninformative deferrals. Under-triage decreased from 2 to 1 of 20 presentations; over-triage from 3 to 2 of 20; specialist-routing accuracy improved from 11 to 12 of 20. Conclusions: Within the scope of a feasibility and safety pilot, the AI-augmented system demonstrated perfect safety preservation (zero red-flag misses, 100% coverage) alongside a small, non-significant improvement in overall triage accuracy. These findings constitute a preliminary safety signal warranting a larger, adequately powered trial.

Ähnliche Arbeiten