Large language models and conditional rules in clinical decision support systems

Abstract

Background: Clinical Decision Support Systems (CDSS) improve patient outcomes and support sustainable health services by enhancing medical decisions. Developing rules for a CDSS is expensive due to delays in capturing and defining the rules through multiple iterations between clinicians and developers as the role of a clinician is patient care. Objective: We investigate the effectiveness of large language models (LLMs) and large reasoning models (LRMs) in generating a triaging rule set for a CDSS. Methods: We prompt various LLMs (GPT-3.5, GPT-4, GPT-4o, Gemini, Claude 3.5 Sonnet) and various LRMs (GPT-o1-mini, Grok-4, Claude 4 Sonnet) using alternative prompting techniques. We compare the LLM generated rule sets against the clinical rule set from our Pandemic Intervention Monitoring System (PiMS); a triaging CDSS built in collaboration with clinicians to monitor COVID-19 positive patients. Effectiveness is evaluated based on the accuracy, interpretability, and rule complexity. Results: We identified that LLMs generated COVID-19 screening rule sets compared to triaging rule sets when not specifying the variables from our PiMS rule set. By including PiMS variables in our prompts, we discovered LLMs 1) had lower interpretability and rule complexity compared to the PiMS rule set, and 2) resulted in an average accuracy between 31.62% ± 0.19% and 70.71% ± 0.02%. While for LRMs, we identified that 1) interpretability varied between 3 and 94 compared to 41 identified in our PiMS rule set and 2) resulted in an average accuracy between 31.62% ± 0.19% and 81.70 ± 0.05%. Conclusions: LLMs are limited in emulating clinical rule sets due to their simplicity and lack of complex reasoning. Despite LRMs improving effectiveness, they are still limited. LLMs and LRMs can generate a feasible initial rule set for CDSS. This can reduce time invested by clinicians and developers by minimising the number of iterations for refinement. Future work can explore integrating LLMs and LRMs with decision trees to improve effectiveness. Supplementary Information: The online version contains supplementary material available at 10.1007/s13755-026-00428-z.

Ähnliche Arbeiten