Bias-Aware Multimodal Medical Diagnosis with Large Language Models for Underrepresented Conditions

Abstract

We introduce a real-time multimodal AI assistant for medical diagnosis that combines large language models (LLMs) with structured patient data, medical imaging, and spoken physician inputs. The system aims to enhance early and accurate diagnosis by integrating diverse clinical information within a unified workflow. It employs a CNN for interpreting medical images, leverages speech-to-text to transcribe verbal notes, and transforms structured patient data into natural language prompts for LLM-driven reasoning.To mitigate diagnostic bias from class-imbalanced training data, we propose a bias-aware approach that integrates class-weighted loss for image models, input prompt rebalancing for LLMs, and human-in-the-loop feedback. Tests on a synthetic yet clinically relevant dataset show improved performance on underrepresented classes versus a baseline multimodal LLM system. We also evaluate GPT-4 (OpenAI), LLaMA-3 (Meta), and DeepSeek V3, comparing diagnostic accuracy, latency, and bias sensitivity. Results underscore the value of combining multimodal AI with imbalance-aware techniques to enhance fairness and reliability in AI-driven healthcare.

Ähnliche Arbeiten