Visualize what you learn: a well-explainable joint-learning framework based on multi-view mammograms and associated reports

Abstract Radiological image and report joint pre-training learning lays the foundation for label-limited medical image analysis, especially when training of effective neural networks is hindered by insufficient manually labeled samples. It can extract abundant medical-related knowledge from paired images and reports to learn the well-transferable features, thus alleviating the need for labor-intensive and time-consuming labeling in downstream applications. However, existing methods prioritize exploring transferable representations rather than explaining how these representations are learned. In this study, we propose a well-explained radiograph-report joint pre-training framework, applied to label-efficient abnormality recognition in mammograms. We address the challenges of learning and visualizing the alignment between the exam-level report and the corresponding mammographic examination consisting of bilateral views in several directions. Qualitatively, our model provides researchers with a better understanding of representation learning for further improvement of pre-training and downstream tasks. We evaluate the label-efficiency ability of our method by comparing it to several state-of-the-art uni-modal and multi-modal methods. The results on four datasets show that our approach brings significant improvements under limited supervision for various downstream tasks. It, therefore, may serve as a novel practical framework exploiting co-learning between multi-view radiological images and radiologists' reports. Furthermore, it has the potential to offer valuable insights into interpretable and comparable tools in the domain of pre-training.