A Modular Multimodal Language Model for Radiation Oncology

Abstract

Multimodal language models (MLMs) have the potential to integrate and interpret heterogeneous data sources, such as clinical text and medical imaging (e.g., CT, PET), to support diagnosis and treatment planning in radiation oncology. Despite their promise, existing general-purpose MLMs often fail to consistently provide accurate and clinically relevant information. Even if domainspecific MLMs are developed at select research institutions, they are likely inaccessible to the broader clinical community for use in practice, due to barriers such as the need for programming expertise, lack of access to high-performance computers, and the impracticality of keyboard-based interaction within hands-occupied clinical workflows. To address this gap, we propose a modular MLM framework designed to improve accessibility, usability, and clinical relevance of MLMs for practitioners in radiation oncology. Our framework offers three key features: (1) a web-based deployment, making it broadly accessible via any internet-connected computer; (2) Integrated voice input and text-to-speech output modules enable clinicians to interact with the AI system hands-free, accommodating scenarios in which manual keyboard use is impractical; (3) our framework currently supports several pre-trained general-purpose MLM modules, with the capacity to incorporate additional domain-specific MLMs in future iterations. By lowering the barrier to experiencing advanced MLMs, our modular framework aims to increase clinician familiarity with emerging AI tools and accelerate their translation into routine clinical practice.

Ähnliche Arbeiten