Distilling Missing Modality Knowledge from Ultrasound for Endometriosis Diagnosis with Magnetic Resonance Images

DISTILLING MISSING MODALITY KNOWLEDGE FROM ULTRASOUND FOR ENDOMETRIOSIS DIAGNOSIS WITH MAGNETIC RESONANCE IMAGES Yuan Zhang1, Hu Wang1, David Butler 1, Minh-Son To2, Jodie Avery3, M Louise Hull 3, Gustavo Carneiro4 1 Australian Institute for Machine Learning, University of Adelaide, Australia 2 Flinders Health and Medical Research Institute, Flinders University, Australia 3 Robinson Research Institute, University of Adelaide, Australia 4 Centre for Vision, Speech and Signal Processing, University of Surrey, UK ABSTRACT Endometriosis is a common chronic gynecological disorder that has many characteristics, including the pouch of Douglas (POD) obliteration, which can be diagnosed using Transvagi- nal gynecological ultrasound (TVUS) scans and magnetic res- onance imaging (MRI). TVUS and MRI are complementary non-invasive endometriosis diagnosis imaging techniques, but patients are usually not scanned using both modalities and, it is generally more challenging to detect POD obliter- ation from MRI than TVUS. To mitigate this classification imbalance, we propose in this paper a knowledge distillation training algorithm to improve the POD obliteration detection from MRI by leveraging the detection results from unpaired TVUS data. More specifically, our algorithm pre-trains a teacher model to detect POD obliteration from TVUS data, and it also pre-trains a student model with 3D masked auto- encoder using a large amount of unlabelled pelvic 3D MRI volumes. Next, we distill the knowledge from the teacher TVUS POD obliteration detector to train the student MRI model by minimizing a regression loss that approximates the output of the student to the teacher using unpaired TVUS and MRI data. Experimental results on our endometriosis dataset containing TVUS and MRI data demonstrate the effective- ness of our method to improve the POD detection accuracy from MRI. Index Terms— Endometriosis, Knowledge Distillation, Masked Auto-Encoder, Pouch of Douglas Obliteration 1. INTRODUCTION Endometriosis is a gynecological disorder associated with the growth of endometrial glands and stroma outside the uter- ine cavity [1, 2]. The clinical manifestations of endometrio- sis include infertility and endometriosis-related pain [3]. As a common chronic gynecological disease, it affects approxi- mately 1.5 million women worldwide [4]. There is currently no known way to prevent or cure endometriosis, but early di- agnosis, intervention and management may slow or stop the natural disease progression. However, the diagnosis of en- dometriosis can take about 7 years on average after the ap- pearance of initial symptoms [5]. Laparoscopy used to be the diagnostic gold standard [6], but with the improvement in the quality and availability of imaging modalities for endometrio- sis diagnosis, there has been evidence suggesting that accu- rate endometriosis diagnosis can be achieved with the analy- sis of TVUS sequences and MRI volumes [7, 8]. Many of the symptomatic endometriosis cases can be as- sociated with the pouch of Douglas (POD) obliteration, which can be diagnosed from complementary TVUS and MRI data, as shown in Fig. 1. However, in clinical practice, it is difficult to access clinicians who can diagnose endometriosis with one of these modalities, not to mention those who are proficient in both modalities. For TVUS, POD obliteration can be ac- curately detected manually [9] and automatically [10] via the ultrasound ‘sliding sign’ [11], which is classified as positive (i.e. normal) or negative (i.e. abnormal), where a negative sliding sign is considered when the anterior rectum or bowel glides cannot freely slide over the posterior upper uterine fundus. For MRI, POD can be classified as obliterated or normal, where the POD obliteration can be characterized by endometrial plaques and dense adhesions between uterosacral ligaments, uterine serosa, ovaries, rectum and vagina on T2- weighted and T1-weighted images [12]. However, differently from TVUS, the manual POD obliteration detection from MRI can only reach 61.4-71.9% accuracy [13]. There has been some effort to propose methods that can automatically diagnose deep pelvic endometriosis classification from MRI1, but we are not aware of methods that can detect POD obliter- ation from MRI. Leveraging the TVUS POD obliteration detection to im- prove the automated detection accuracy from MRI using an unpaired training set containing scans from both modalities is the main goal of this paper. We achieve this goal by proposing a new knowledge distillation algorithm based on two stages: 1) pre-training a teacher model to detect POD obliteration from TVUS data, and pre-training a student model with 3D masked auto-encoder (MAE) using a large amount of unla- belled pelvic 3D MRI volumes; and 2) knowledge distilla- tion from the teacher TVUS detector to train the student MRI model by minimizing a regression loss that approximates the output of the student to the teacher using unpaired TVUS and 1Deep infiltrating endometriosis (DIE) can lead to a partial or complete obliteration of the pouch of Douglas (POD) [14]. arXiv:2307.02000v1 [eess.IV] 5 Jul 2023 Fig. 1 . Examples of POD obliteration on MRI and sliding sign on TVUS. (a) and (b) represent negative and positive POD obliteration sign on sagittal plane MRI, respectively. (c) and (d) represent positive and negative sliding sign on TVUS, respectively. MRI data. The testing is realised using only MRI data. The main innovations of this work are: • To the best of our knowledge, this is the first POD oblitera- tion detection method that distills knowledge from TVUS to MRI using unpaired data, with the objective of improv- ing the accuracy of diagnosing endometriosis from MRI; • It is also the first machine learning method that can auto- matically detect POD obliteration from MRI data with the goal of diagnosing endometriosis. Experimental results on a private endometriosis training set containing unpaired TVUS and MRI data show the effective- ness of our method to increase the POD detection accuracy from testing MRI volumes. 2. RELA TED WORK The automated detection of endometriosis from medical imaging has received some attention lately. Using ultrasound images, Guerriero et al. [15] compared the ability of six ma- chine learning algorithms and neural networks for the diag- nosis of endometriosis in the rectosigmoid, where the neural network achieved the highest classification accuracy of 0.73. Maicas et al. [10] constructed a deep learning model based on a temporal residual network to classify POD obliteration from TVUS videos, achieving an AUC of 96.5% and an accu- racy of 88.8%. The methods above use TVUS data alone, but recently, Yang et al. [16] built a bi-model method with one model for MRI and another for TVUS, but they do not ex- plore the relationship between MRI and TVUS, like we pro- pose in this paper. Furthermore, as mentioned above, it is unlikely that patients will have access to both modalities in clinical practice, which justifies the need for single-modality approaches that have highly accurate endometriosis detection. Knowledge distillation is a general framework to extract the knowledge learned from a teacher model to a student model by soft-label supervision. The original purpose of knowledge distillation is to compress deep learning mod- els, so they can run on resource-constrained devices [17], but in this paper we focus on the transfer of knowledge from a teacher network trained on source modalities to a student network that is trained on different target modali- ties [18]. In medical image analysis, data from different modalities can provide rich and complementary information about diseases, so multimodal knowledge distillation is suit- able for scenarios where data or labels for some modalities are missing during training or testing. Inspired by knowl- edge distillation, Dou et al. [19] tackled the task of unpaired multi-modal image segmentation. Guan et al. [20] leverage additional supervision distilled from clinical data to improve MRI-based Alzheimer’s disease prediction models. Cheng et al. [21] utilise both a pixel-level and an image-level distilla- tion scheme to transfer knowledge from a multimodal-MRI teacher network to a unimodal segmentation student network. However, most unpaired multi-modal studies above focus on MRI and CT scans, which is arguably easier than focusing on MRI and TVUS, which is the case considered in this paper. 3. METHOD An overview of our proposed TVUS to MRI knowledge distil- lation model is shown in Figure 2, which consists of two mod- els: a teacher model pre-trained with a TVUS dataset and a student model pre-trained on an MRI dataset, and then trained on an unpaired dataset of TVUS and MRI data by distilling the knowledge in the representation learned by the teacher model to the student model. Formally, letDM = {(xi, yi)}N i=1 denote the endometrio- sis MRI dataset, with N T2 SPC images x ∈ X ⊂ RH×W ×D and corresponding POD obliteration one-hot label y ∈ {0, 1}2, where H, W and D are height, width and depth of the MRI, respectively. For TVUS dataset, let Ds U = {(xs i , ys i )}M i=1 be the video clips dataset, where xs ∈ X s ⊂ RH×W ×T (H, W and T are height, width and number of frames), and ys ∈ {0, 1}2 denotes the POD obliteration one- hot label. For the self-supervised pre-training of the MRI POD obliteration detector, we have DP = {xp i }K i=1, which contains K unlabeled MRI volumes, where the number of unlabeled images is much larger than the labeled images (i.e., K >> N and K >> M ). The teacher model fθU : X s → ∆ (with ∆ ⊂ [0, 1]2 being the probability simplex) is trained with dataset Ds U , the student model fθM : X → ∆ is pre- trained with dataset DP and fine-tuned using DM . The final knowledge distillation model is initialised by the pre-trained student model fθM (.), which is then trained from DM and Ds U . The testing to classify POD obliteration uses the trained fθM (.) on the MRI testing images from DM . 3.1. Pre-training For pre-training the MRI encoder, we explore the self- supervised masked auto-encoder (MAE) method [22] us- ing the large dataset of un-annotated T2 MRI images DP . Fig. 2 . Proposed POD obliteration detector trained by dis- tilling knowledge to MRI from unpaired TVUS. (a) MRI pre- training with 3D masked auto-encoder, (b) TVUS pre-training with ResNet(2+1)D, (c) MRI Knowledge Distillation from the frozen teacher model pretrained on TVUS. 3D-MAE has an asymmetric encoder-decoder architecture based on 3D Vision Transformer (3D ViT) [23], as shown in Figure 2. During pre-training, each volume is cropped into 8 × 8 × 8 patches, then 50% of the patches are randomly masked. The encoder, denoted by fθE : X → E , takes visible patches embedded by a linear projection (of dimension 768) with additional positional embeddings, and processes via the 3D ViT with 12 Transformer blocks whose output is then fed into the decoder, denoted by fθD : D → X , together with the masked volume tokens, to reconstruct the original volume at the voxel level. After pre-training, the decoder is discarded and the pre-trained encoder is applied to extract MRI fea- tures for the downstream classification task. During MRI pre-training, we minimize the mean squared error (MSE) on the masked patches between the reconstructed and original volumes in the voxel space, defined as follows: ℓM AE(DP ; θE, θD) = 1 L × K KX i=1 LX l=1 ∥xi(l) − ˆxi(l)∥2 2, (1) where K is the size of the unlabelled MRI dataset,L indicates the number of masked patches, xi(l) and ˆxi(l) represent the voxel values of thelth masked patch in the original and recon- structed volumes, respectively, with the reconstructed volume obtained from ˆxi = fθD(fθE(xi)). After this self-supervised pre-training, we take the en- coder fθE(.), add a linear layer to change the size from 768 (output size of the MRI encoder) to 512 (output size of the TVUS encoder), and add a final classification layer with a 2- dimensional output activated by softmax to form the student network, denoted by fθM (.). We fine-tune all transformer blocks in the encoder and fully connected layer to classify POD obliteration by minimizing the cross-entropy (CE) loss: ℓP T M(DM; θM) = − X (xi,yi)∈DM ℓCE(yi, fθM (xi)). (2) For the TVUS pre-training, we adopted the ResNet (2+1)D model proposed in [10]. This model contains 18 modules of R(2+1) convolutional layers, with each convo- lution being followed by batch normalization. During this TVUS pre-training, we also minimize the CE loss, as fol- lows: ℓP T U(Ds U; θU) = − X (xs i ,ys i )∈DM ℓCE(ys i , fθU (xs i )). (3) 3.2. Knowledge Distillation In the knowledge distillation (KD) stage, we consider the pre- trained TVUS model fθU (.) as the teacher model, and the pre- trained MRI model fθM (.) as the student model. Given that the pre-trained TVUS model tends to produce superior classi- fication accuracy than the pre-trained MRI model, we aim to fine-tune the MRI model to match the predictions produced by the TVUS model. The goal is to use this knowledge distil- lation procedure to improve the classification accuracy of the MRI model, which uses only the MRI volume during testing. However, recall that we do not have matched TVUS and MRI data for this knowledge distillation procedure, so we match the data based only on their classification labels, i.e., an MRI sample with positive POD obliteration is matched with a ran- dom TVUS sample with positive POD obliteration, and simi- larly for the negative POD obliteration. Then, the KD training minimises the following loss ℓKD(DM , Ds U; θM) = − X (xi,yi)∈DM (xs j ,ys j )∈Ds U yi=ys j ∥fθU (xs j) − fθM (xi))∥1. (4) The loss in (4) is added to the ℓP T M(.) loss from (2) to form the final loss that encourages the model to pull the TVUS and the MRI outputs closer to distill the TVUS classification in- formation to the student network, as follows ℓ(DM , Ds U; θM) =αepoch × ℓKD(DM , Ds U; θM)+ (1 − αepoch) × ℓP T M(DM; θM), (5) which is used to estimate the optimal value of θM , where αepoch is a parameter to dynamically balance the contribu- tions of the two loss terms during training. 4. EXPERIMENTS 4.1. Dataset Our private dataset contains: an MRI endometriosis dataset, a TVUS endometriosis dataset, and a female pelvic MRI dataset for self-supervised pre-training. The MRI endometriosis dataset contains 88 T2 SPACE MRI scans from women aged 18-45, including 19 POD oblit- eration cases. These examinations were performed in several clinical sites in Australia. The scans contain the whole pelvic area, but we focus on the region around the uterus that can display the POD obliteration. We performed 3D contrast lim- ited adaptive histogram equalization (CLAHE) to improve the local contrast and enhance the definitions of edges in each re- gion of a sequence. For the experiments, we use stratified random sampling to split the dataset into 5 folds, each with 17-18 subjects, with 4 folds used for training and 1 for test- ing, which are rotated to produce a cross-validated result. The TVUS endometriosis dataset has 749 video clips of the ’sliding sign’, including 103 negative (POD obliterated) cases. We follow the data preparation and pre-processing pro- tocol as well as model parameter settings proposed in [10] to pre-train the TVUS model. For the knowledge distillation phase, we divide the dataset into negative and positive groups, then use stratified random sampling to split each group into 5 folds and use 4 folds in each group as the training set. The female pelvic MRI dataset contains 8,984 volumes. In the context of endometriosis research, we accept all scans when the patient is aged between 18 to 45, the physical ex- amination site is pelvis, and the sequence description con- tains ’T2’. It is worth noting that most of these volumes were scanned in different settings, so they may contain signs of other diseases and the scanned area may or may not overlap with the diagnostic area of endometriosis. We re-sampled all scans by SimpleITK with an output spacing of 1 × 1 × 1mm, and filtered out volumes if the number of slices in any dimen- sion is less than 65. 4.2. Training and Evaluation In the pre-training phase, the 3D MAE network is trained for 200 epochs on the female pelvic MRI dataset using a batch size of 3 with AdamW optimizer and a learning rate of 1e- 3. Then we fine-tuned the pre-trained checkpoint saved from epoch 195 for 25 epochs on the endometriosis MRI dataset with a “5-fold cross validation” strategy. The ResNet(2+1)D network was pre-trained on the Kinetics-400 dataset then fine- tuned for 30 epochs on the TVUS endometriosis dataset using a batch size of 30 with Adam optimizer and a learning rate of 1e-5. For knowledge distillation, we loaded the weights of the 10th checkpoint fine-tuned from each fold as the MRI feature extractor of the MRI model and the pre-trained TVUS feature extractor. The knowledge distillation network is trained for 10 epochs on the endometriosis MRI dataset using a batch size of 7 with AdamW optimizer and a learning rate of 1e-3 with the same “5-fold cross validation” strategy. We set the hyper- parameter α = 0.85 based on the cross-validation results. We evaluate our method by computing Area Under the Receiver Operating Characteristic Curve (ROC AUC) for five folds and calculate the mean and standard deviation of them. 4.3. Results The classification results with our proposed method is shown in Table 1. The ResNet(2+1)D model shows an outstanding classification AUC of 96.9% from TVUS data. The small amount of training samples limited the generalisation of 3D ViT to classify POD obliteration from MRI volumes with an AUC of 65.0%, but the MAE pre-training (PT) partially mit- igates the issue, improving the AUC to 87.2%. With knowl- edge distillation (KD), we observe that training a 3D ViT from scratch on such a small dataset is still challenging, with an AUC of 66.7%. Also, the KD performance of 3D ViT with MAE pre-training (PT) reaches AUC=77.2%), which is worse than without KD, with AUC=87.2%, which may be due to the excessive domain shift between the pre-training dataset and TVUS dataset. By fine-tuning (FT) the model from MAE pre-training, the model improves accuracy from AUC=87.2% (without KD and FT) to AUC=90.6% (with KD and FT). Table 1. POD obliteration classification results.Method Training Testing AUC Modality Modality mean±stddev ResNet(2+1)D TVUS TVUS 0.969±0.012 3D ViT MRI MRI 0.650±0.102 3D ViT: MAE PT MRI MRI 0.872±0.094 3D ViT: KD MRI,TVUSMRI 0.667±0.107 3D ViT: MAE PT + KD MRI,TVUSMRI 0.772±0.087 3D ViT: MAE PT + KD + FTMRI,TVUSMRI 0.906±0.099 5. CONCLUSION In this paper, we proposed a two-stage algorithm to distill the knowledge from a TVUS to an MRI classifier, thereby improving the POD obliteration classification accuracy of the MRI classifier. Through the MAE pre-training, knowl- edge distillation and fine-tuning, we are able to significantly reduce the distance between the two domains and accom- plish a promising knowledge distillation from TVUS to MRI. The efficacy and superiority of our proposed approach are demonstrated by experimental results on our endometriosis datasets. In the future, we will introduce a missing modality deep learning approach and expand our proposed method to perform weakly-supervised lesion segmentation, thereby im- proving the interpretability of the model, so it can be widely applied in future clinical trials. 6. COMPLIANCE WITH ETHICAL STANDARDS This study was performed in line with the principles of the Declaration of Helsinki. Approval was granted by Human Research Ethics Committee (HREC) of University of Ade- laide(Date 01-03-2020/No. H-2020-051) and the Southern Adelaide Clinical Human Research Ethics Committee (SAC HREC) (Date 01-11-2021/No. 111.20). 7. ACKNOWLEDGMENTS This work received funding from the Australian Government through the Medical Research Futures Fund: Primary Health Care Research Data Infrastructure Grant 2020 and from En- dometriosis Australia. 8. REFERENCES [1] Antonio Simone Lagana et al., “Evaluation of m1 and m2 macrophages in ovarian endometriomas from women affected by endometriosis at different stages of the disease,” Gynecological Endocrinology, vol. 36, no. 5, pp. 441–444, 2020. [2] KM Moss, J Doust, H Homer, IJ Rowlands, R Hockey, and GD Mishra, “Delayed diagnosis of endometriosis disadvantages women in art: a retrospective population linked data study,” Human Reproduction, vol. 36, no. 12, pp. 3074–3082, 2021. [3] Paolo Vercellini et al., “Endometriosis: pathogenesis and treatment,” Nature Reviews Endocrinology, vol. 10, no. 5, pp. 261–275, 2014. [4] Krina T. Zondervan et al., “Endometriosis,” New Eng- land Journal of Medicine , vol. 382, no. 13, pp. 1244– 1256, 2020, PMID: 32212520. [5] Australian Institute of Health and Welfare, “En- dometriosis in australia: prevalence and hospitalisa- tions,” 2019, [Online; accessed 18-02-2022]. [6] Christian M Becker et al., “Eshre guideline: en- dometriosis,” Human reproduction open, vol. 2022, no. 2, pp. hoac009, 2022. [7] Alison Deslandes et al., “Current status of transvaginal ultrasound accuracy in the diagnosis of deep infiltrating endometriosis before surgery: a systematic review of the literature,” Journal of Ultrasound in Medicine , vol. 39, no. 8, pp. 1477–1490, 2020. [8] T Indrielle-Kelly et al., “Diagnostic accuracy of ul- trasound and mri in the mapping of deep pelvic en- dometriosis using the international deep endometriosis analysis (idea) consensus,” BioMed research interna- tional, vol. 2020, 2020. [9] Le Chi Chiu et al., “Predicting pouch of douglas oblit- eration using ultrasound and laparoscopic video sets: an interobserver and diagnostic accuracy study,”Journal of Ultrasound in Medicine, vol. 38, no. 12, pp. 3155–3161, 2019. [10] Gabriel Maicas et al., “Deep learning to diagnose pouch of douglas obliteration with ultrasound sliding sign,” Reproduction and Fertility , vol. 2, no. 4, pp. 236–243, 2021. [11] Stefano Guerriero et al., “Systematic approach to sono- graphic evaluation of the pelvis in women with sus- pected endometriosis, including terms, definitions and measurements: a consensus opinion from the interna- tional deep endometriosis analysis (idea) group,” Ultra- sound in Obstetrics & Gynecology , vol. 48, no. 3, pp. 318–332, 2016. [12] Anitha L Thalluri et al., “Mri findings in deep infiltrat- ing endometriosis: a pictorial essay,” Journal of Medi- cal Imaging and Radiation Oncology , vol. 61, no. 6, pp. 767–773, 2017. [13] Milliam L Kataoka et al., “Posterior cul-de-sac obliter- ation associated with endometriosis: Mr imaging evalu- ation,” Radiology, vol. 234, no. 3, pp. 815–823, 2005. [14] Kristina Arion et al., “Prediction of pouch of douglas obliteration: point-of-care ultrasound versus pelvic ex- amination,” Journal of Minimally Invasive Gynecology , vol. 26, no. 5, pp. 928–934, 2019. [15] Stefano Guerriero et al., “Artificial intelligence (ai) in the detection of rectosigmoid deep endometriosis,” Eu- ropean Journal of Obstetrics & Gynecology and Repro- ductive Biology, vol. 261, pp. 29–33, 2021. [16] Minmin Yang et al., “Diagnostic efficacy of ultrasound combined with magnetic resonance imaging in diag- nosis of deep pelvic endometriosis under deep learn- ing,” The Journal of Supercomputing , vol. 77, no. 7, pp. 7598–7619, 2021. [17] Geoffrey Hinton et al., “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, vol. 2, no. 7, 2015. [18] Nikolaos Passalis et al., “Learning deep representa- tions with probabilistic knowledge transfer,” in Pro- ceedings of the European Conference on Computer Vi- sion (ECCV), 2018, pp. 268–284. [19] Qi Dou et al., “Unpaired multi-modal segmentation via knowledge distillation,” IEEE transactions on medical imaging, vol. 39, no. 7, pp. 2415–2425, 2020. [20] Hao Guan et al., “Mri-based alzheimer’s disease predic- tion via distilling the knowledge in multi-modal data,” NeuroImage, vol. 244, pp. 118586, 2021. [21] Cheng Chen et al., “Learning with privileged multi- modal knowledge for unimodal segmentation,” IEEE Transactions on Medical Imaging , vol. 41, no. 3, pp. 621–632, 2021. [22] Kaiming He et al., “Masked autoencoders are scal- able vision learners,” arXiv preprint arXiv:2111.06377, 2021. [23] Alexey Dosovitskiy et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.