Cross-Modal Retrieval and Retrieval-Augmented Inference for IoT-Enabled Clinical Decision Support

Abstract

Early clinical decisions increasingly rely on heterogeneous IoT streams (wearables, bedside waveforms, point-of-care imaging), yet current multimodal systems remain vulnerable to retrieval errors and unsupported LLM assertions. This paper proposes a topology-aware cross-modal retrieval and retrieval-augmented inference pipeline that fuses image and time-series embeddings via a domain-adapted contrastive dual encoder, regularizes embedding neighbourhood geometry with persistent homology, and conditions a frozen LLM through a BLIP-2-style Q-former for evidence-grounded generation. Evaluation on MIMIC-CXR and ROCO-style radiology corpora demonstrates substantial gains: Recall@1 <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\cong 0.78$</tex> (<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\approx+9$</tex> percentage points vs. a BiomedCLIP baseline) and mAP @ 10 <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\approx$</tex> 0.72, together with an absolute +0.15 increase in evidencecoverage (0.81 vs. 0.66), corresponding to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\approx 23 \%$</tex> relative reduction in unsupported LLM assertions. Key contributions are: (i) a persistent-homology embedding regularizer for modality-invariant geometric alignment; (ii) an IoT-aware hybrid retrieval index and pruning strategy for bounded latency; and (iii) comprehensive empirical validation including ablations and clinician-adjudicated concordance analyses on MIMIC-derived datasets.

Ähnliche Arbeiten