Eye Guided Multimodal Fusion: Toward an Adaptive Learning Framework Using Explainable Artificial Intelligence

Interpreting diagnostic imaging and identifying relevant features in healthcare present significant challenges. For novices, the risk of misdiagnosis can be overwhelming, particularly in the absence of structured guidance and supervision. Furthermore, radiologists' expertise is not always accessible to trainees when needed. Consequently, explicit, structured guidance is essential to help novices interpret complex imaging data accurately and enhance their learning process. Therefore, developing an approach to transfer expert knowledge to novices would be invaluable, bridging the gap between theoretical understanding and practical skills in medical imaging. Eye-tracking has surged in popularity in recent years for analyzing medical images. Incorporating experts' eye-gaze patterns in an artificial intelligence (AI)- driven web tool offers an intuitive learning experience. Highlighting the regions of interest (ROI) can facilitate feedback and accelerate students' learning and clinical decision-making. Our multimodal approach integrates chest X-ray (CXR) images with expert eye-tracking fixation maps as auxiliary data, explicitly highlighting radiologists' visual attention during medical image assessment. We employ a unified core architecture to minimize the influence of noisy fixation data and avoid treating the imaging and eye-tracking modalities as independent contributors, thereby enhancing abnormality detection in CXRs. Gradient-weighted Class Activation Mapping (Grad-CAM) validates our model interpretability and influences radiologist decision-making, underscoring the framework's practical application in clinical contexts. Finally, we conducted a comprehensive evaluation of our model using both qualitative and quantitative analyses.