{"paper_id":"b4ab562d-c925-4090-8b30-792779bd8080","body_text":"Augmenting endometriosis analysis from ultrasound data with deep \nlearning  \nAdrian Balica*,a and Jennifer Dai*,b and Kayla Piiwaab and Xiao Qic and Ashlee N. Greena and \nNancy Phillipsa and Susan Egana and Ilker Hacihaliloglud \na Department of Obstetrics and Gynecology, Robert Wood Johnson University Hospital, USA \nb Robert Wood Johnson Medical School, Rutgers University, USA \nc Department of Electrical and Computer Engineering, Rutgers University, USA \nd Department of Radiology, Department of Medicine, University of British Columbia, Canada \nABSTRACT  \nEndometriosis is a non-malignant disorder that affects 176 million women globally. Diagnostic delays result in severe \ndysmenorrhea, dyspareunia, chronic pelvic pain, and infertility. Therefore, there is a significant need to diagnose patients \nat an early stage. Our objective in this work is to investigate the potential of deep learning methods to classify \nendometriosis from ultrasound data. Retrospective data from 100 subjects were collected at the Rutgers Robert Wood \nJohnson University Hospital (New Brunswick, NJ, USA). Endometriosis was diagnosed via laparoscopy or laparotomy. \nWe designed and trained five different deep learning methods (Xception, Inception-V4, ResNet50, DenseNet, and \nEfficientNetB2) for the classification of endometriosis from ultrasound data. Using 5-fold cross-validation study we \nachieved an average area under the receiver operator curve (AUC) of 0.85 and 0.90 respectively for the two evaluation \nstudies.  \nKeywords: Artificial intelligence (AI), deep learning, convolutional neural networks, endometriosis \n \n1. INTRODUCTION  \nEndometriosis is a non-malignant disorder that affects 10-15% of American women and 176 million women globally1,2. \nSymptoms of endometriosis include severe dysmenorrhea, dyspareunia, and chronic pelvic pain2. Without treatment, \nendometriosis can lead to infertility; thus, there is a significant need to diagnose patients early3,4. Despite this, the mean \ntime to diagnosis is seven years5. While the current gold standard for diagnosis is laparoscopy with tissue diagnosis, there \nis not yet a reliable, nonsurgical, diagnostic method5. Ultrasound could aid in clinical diagnosis and provide a safer, non-\ninvasive, and more cost-effective diagnostic option6. However, ultrasound data is characterized by high levels of noise, \nimaging artifacts, and variability during data collection (manual data collection). Furthermore, identification of early \nlesions in qualitative ultrasound is challenging5,6.  \nDeep learning, a subfield of machine learning, has recently been successfully deployed for processing various \nmedical data to support clinical decision-making for a range of diseases7,8, including endometriosis9-12. In Hu et. al. (2019)9, \na deep learning method (based on previously proposed two well-known neural network architectures VGG13 U-net14) was \nproposed for segmentation and thickness measurement of the endometrium. In Guerroro et. al. (2021)10, k-nearest \nneighbors algorithm (k-NN), Naive Bayes, Neural Networks (NNET-neuralnet), Support Vector Machine (SVM), \nDecision Tree, Random Forest, and Logistic Regression models were investigated. The highest reported area under the \nreceiver operating curve (AUC) was 81.73% for NNET-neuralnet using single-fold validation. In Yang M et. al. (2021) 11, \na traditional deep learning method13 was extended for classifying deep pelvic endometriosis (DPE). The reported single-\nfold classification accuracy was 96.5%, however, quantitative results for AUC were not reported. The authors also did not \nmention the size of the training and validations set nor provided information if the same patient scans were used for training \nand testing. In Maicas et. al. (2021)12, ResNet with spatiotemporal information15, for the classification of the sliding sign \n  \n*These authors contributed equally to this work. kap421@rwjms.rutgers.edu, ilker.hacihaliloglu@ubc.ca \n\n \n \n \n \nfor the diagnosis of the pouch of Douglas (POD) obliteration, was investigated. Single-fold validation achieved an average \naccuracy of 88.8% and AUC 96.5%. The evaluation dataset included 86.2% positive and 13.8% negative POD group. Data \naugmentation to balance the class distribution was not performed.  \nThe primary objective of this pilot study was to evaluate the feasibility of using deep learning methods to diagnose \nthe presence of endometriosis from ultrasound data. Specifically, we optimize the design of the five most widely used deep \nlearning methods for processing medical image data, Xception26, Inception-V427, ResNet5016, DenseNet18 and Efficient-\nNetB217, for improved classification endometriosis from clinical ultrasound data.  \n2. METHODS AND MATERIALS \n2.1 Data collection \nThe data was collected as part of an IRB-approved retrospective chart review, so consent was waived. In total 100 subjects \nwere included in the study: 50 subjects with no endometrial findings on ultrasound, and 50 with diagnosed endometriosis \nwho were seen at Rutgers University Robert Wood Johnson University Hospital (New Brunswick, NJ, USA) between \n2017-2020. Subjects in the endometriosis group were identified with a positive diagnosis of endometriosis from the \nmedical record. All the subjects 21 years of age or older and diagnosed with endometriosis via laparoscopy or laparotomy \nand assessed via transabdominal and transvaginal ultrasounds were included in the data collection. Out of the 50 \nendometriosis subjects, 34 subjects had laparoscopy or laparotomy procedures at Rutgers, while 16 had a documented \nprocedure in the electronic medical records (EMR) at another site. A total of 174 subjects > 21 years old from 2018-2020 \nwith normal ultrasound findings evaluated at Rutgers were identified for inclusion in the normal group. The normal group \n(no endometriosis group) was composed of patients with at least one symptom suggestive of endometriosis but without \nprevious treatment for endometriosis and ultrasound findings not suggestive of endometriosis. From the normal group, 2 \nsubjects were removed because they were later diagnosed with endometriosis. Subjects who had undergone \nhysterectomies, oophorectomies, or had ultrasounds outside of IRB-approved dates were also excluded from both groups.   \nUltrasound protocol at Rutgers included the following. Transabdominal ultrasound was performed with a Voluson \nE8 scanner equipped with an abdominal 4C-D 2–5-MHz transducer (GE Healthcare, Milwaukee, WI). The ultrasound data \nwere collected by an AIUM-certified ultrasound technician. For each subject, all ultrasound images within the IRB-\napproved time frame were downloaded. These ultrasound images were stratified by subject for analysis. In total, we had \n1,788 ultrasound scans from the endometriosis group and 812 scans from the normal group. All the subjects also had video \nultrasound recordings available as part of their scanning protocol, and these were included in the evaluation study as well. \n2.2 Data Pre-processing \nUltrasound images were processed to standardize input into the deep learning algorithm. The ultrasound image intensities \nwere rescaled to a certain size (0-255). To normalize the image size, we cropped a centered square region of interest, with \nthe size of the smallest image axis. Images were resized to 512×512 pixels to be processed by the CNN architecture. image \nzooming by a factor uniformly sampled from [0.95, 1.05], translation in 4 directions by a factor uniformly sampled from \n+- 5% of height and width, and rotation by a factor uniformly sampled from [-15, 15] degrees. After data augmentation, \nthe training dataset consisted of 3,412 images (1,788 images from endometriosis patients, and 1,624 images from normal \npatients). We referred to the test data used in this analysis as ‘Test 1’ data. Five-fold cross-validation was used during \nevaluation. The data partition for each fold was divided into 60% of images used for training, 20% for validation, and 20% \nfor testing. Test data did not include augmented data to ensure accuracy metrics were calculated on previously unseen and \nunprocessed images. During data partitioning, all images from the same patient appeared in the same partition.  At the end \nof this process, ‘Test 1’ data had 520 images (357 images from the endometriosis group and 163 images from the normal \ngroup). An additional 41,186 images including 26,250 positive images (diagnosed with endometriosis) and 14,936 \nnegative images (no endometriosis diagnosis) were extracted from video data and were referred to as ‘Test 2’ data. We \nalso ensured that the ‘Test 2’ data did not have scans used for training the deep learning methods. \n \n2.3 Deep Learning Methods \nAs part of this work, we have evaluated three different convolutional neural network (CNN) architectures: Xception26, \nInception-V427, ResNet5016 DenseNet18 and EfficientNetB217. These are the most widely investigated CNN models for \nclassifying medical image data7, 8. Each of these architectures was extended by using one fully connected layer with half \nthe length of the input features with rectified linear unit (ReLU) activation23 and a final classification layer of length one \n\n \n \n \n \nwith sigmoid activation. Dropout layers with a dropout coefficient of 0.2 were interleaved between the fully connected \nlayers. The model weights were pre-trained using the ImageNet10 database. The loss function employed was binary cross-\nentropy loss7,8. We balanced the binary cross-entropy loss of positive and negative samples within each batch following19. \nThe experiments were implemented in Python using the Pytorch framework22. The networks were trained using an AdamW \noptimizer21 with a learning rate schedule that decreased the learning rate between 10−4 for the first epochs to 10−6 for the \nlast ones. The mini-batch size was set to 16. The learning rate was reduced by a factor of 10 if the validation area under \nthe receiver operating curve (AUC) did not improve for 10 epochs. The training was performed for 60 epochs.  \n3. RESULTS \nThe Densenet CNN algorithm successfully detected the presence of endometriosis on ultrasound with an AUC of 90% and \nan accuracy of 80% using heterogeneous patterns of endometriosis. This algorithm has the potential to be extremely useful \nin the clinic, as it can take as input a patient’s ultrasound, and suggest with 90% probability and 80% accuracy whether \nthe patient has endometriosis. This tool, combined with the patient’s clinical presentation, has the potential to advance our \ncurrent capability to diagnose endometriosis. \nOther algorithms were also evaluated. Table 1 shows the average accuracy of the 5-fold cross-validation on the \n'Test 1’ and ‘Test 2’ datasets. All investigated network designs achieved an AUC above 80% and an accuracy above 70%. \nFor ‘Test 1’ data, the original dataset with data augmentation, Densenet outperformed the other networks (p<0.05 for \npaired t-test, Figure 1, Table 1). For ‘Test 2’ data, the AUC and accuracy results improved to 90% and 84% respectively \n(Figure 2, Table 1). The highest AUC and accuracy were again obtained using the Densenet network architecture. \nTable 1. Precision, Recall, F1-Score,  Accuracy, and AUC values for the designed and optimized deep learning methods. Average \nAUC and ACC values across five folds. Results show the average values across five folds.  \n  \n \nFig. 1. AUC plots and confusion matrix results for all the investigated network architectures. Quantitative results were obtained using \n‘Test 1’ data, which had 2,600 ultrasound images in total after data augmentation for 5-fold cross-validation.  \n \n\n\n \n \n \n \n \nFig. 2. AUC plots and confusion matrix results for all the investigated network architectures. Quantitative results were obtained using \n‘Test 2’ data which had 205,930 ultrasound images in total for 5-fold cross validation. Data augmentation was not performed for this \ndata set.   \n4. DISCUSSIONS AND CONCLUSIONS \nEarly detection of endometriosis is crucial to prevent adverse outcomes such as infertility. Ultrasound, due to being non-\ninvasive, safe, and cost-effective (compared to MRI), is a promising imaging modality providing a safer alternative to gold \nstandard invasive laparoscopy-based diagnosis. However, high levels of noise, imaging, artifacts, and being a user-\noperated imaging modality makes the diagnosis of endometriosis from ultrasound data challenging. In this work, we have \ninvestigated the potential of artificial intelligence, based on deep learning, methods to diagnose endometriosis from \nultrasound data. Five different deep-learning methods were designed and optimized to classify endometriosis from \nultrasound data. We have achieved close to 0.9 average AUC and 80% accuracy in a 5-fold cross-validation study meaning \nthat given an ultrasound, we can predict with 90% probability and 80% accuracy that the patient has endometriosis. To \nbalance the dataset distribution we have performed data augmentation. Our test data did not include any augmented data \nto ensure that the quantitative evaluation was not influenced by the augmented data. Our study is an initial successful pilot \nstudy to determine the presence of endometriosis using deep learning. Strengths include being the first study to evaluate \nthe classification of heterogeneous patterns of endometriosis on patients with a surgical diagnosis of endometriosis using \nCNN architectures, making our study most applicable to the clinic. We also designed and optimized the most widely used \nCNN architectures for medical image classification as a proof-of-concept study. The present study is only a preliminary \nattempt to integrate deep learning into the clinical decision-making process for improved management of endometriosis. \nSeveral challenges remain to be addressed which are discussed in the next section.  \n \nOur initial pilot study included scans collected only from 100 patients retrospectively. This is not a significant amount of \ndata to harness the real power of deep learning methods. Clinical data collection is currently ongoing in our institute. \nDuring the data collection, patients with fibroids and other reproductive abnormalities were included, which may have \naffected the regions the algorithm uses to classify endometriosis. However, women with endometriosis commonly present \nwith these comorbidities, making our analysis applicable in the clinic. In Hacihaliloglu and Balica (2020)24, we have shown \nthat ultrasound tissue representation of endometriosis can be improved by using local phase-based image enhancement \nmethods. Local phase tissue signatures are intensity invariant and are not affected by the ultrasound transducer operating \nfrequency, ultrasound machine settings, and the body mass index of the patient. Figure 3 shows the enhancement results \nobtained from our local phase-based image processing method. In Che et al (2021)25, we have shown that by using the \nlocal phase image features the success of deep learning methods for diagnosing liver disease from ultrasound data can be \nimproved compared to the most widely used deep learning methods such as the ones investigated in this work. We did not \ninclude such an analysis in this work. Our future work will involve the incorporation of the local phase images for the \ndesign of multi-feature deep learning methods for improving classification performance.  \n \n\n\n \n \n \n \n \n Fig. 3. Local phase-based image enhancement results. (a) B-mode ultrasound image with laparoscopy confirmed endometriosis. (b-d) \nthree different local phase images. (e) Multi-feature image obtained using the combination of the local phase images shown in (b-d). \nMany clinical applications have successfully analyzed ultrasound images using convolutional neural networks (CNN). \nWith this pilot study, we have shown that traditional CNN architectures can classify the presence of endometriosis from \nultrasound data. Future work will include increasing our patient population and evaluating multi-feature CNN \narchitectures25 for improved diagnostic accuracy.    \n5. ACKNOWLEDGEMENTS \nThe authors thank Allison Cabinian (Rutgers Women Health Institute, New Brunswick, New Jersey, USA) for her \ncontribution.  \nREFERENCES \n[1] Louis GM, Hediger ML, Peterson CM, Croughan M, Sundaram R, Stanford J, Chen Z, Fujimoto VY, Varner \nMW, Trumble A, Giudice LC. Incidence of endometriosis by study population and diagnostic method: the ENDO \nstudy. Fertility and sterility. 2011 Aug 1;96(2):360-5. \n[2] Bulletti C, Coccia ME, Battistoni S, Borini A. Endometriosis and infertility. Journal of assisted reproduction and \ngenetics. 2010 Aug;27(8):441-7. 2017 Sep 8. PMID: 28893546.  \n[3] Tsamantioti ES, Mahdy H. Endometriosis. [Updated 2021 Aug 25]. In: StatPearls [Internet]. Treasure Island (FL): \nStatPearls Publishing; 2022 Jan-. Available from: https://www.ncbi.nlm.nih.gov/books/NBK567777/ \n[4] Arruda MS, Petta CA, Abrão MS, et al. Time elapsed from onset of symptoms to diagnosis of endometriosis in a \ncohort study of Brazilian women. Hum Reprod. 2003 Apr;18(4):756-9. doi: 10.1093/humrep/deg136. PMID: \n12660267. \n[5] Berker B, Seval M. Problems with the diagnosis of endometriosis. Womens Health (Lond). 2015 Aug;11(5):597-\n601. doi: 10.2217/whe.15.44. Epub 2015 Sep 21. PMID: 26389666. \n[6] Scioscia M, Virgilio BA, Laganà AS, Bernardini T, Fattizzi N, Neri M, Guerriero S. Differential diagnosis of \nendometriosis by ultrasound: a rising challenge. Diagnostics. 2020 Oct;10(10):848. \n[7] Zhou SK, Greenspan H, Davatzikos C, Duncan JS, Van Ginneken B, Madabhushi A, Prince JL, Rueckert D, \nSummers RM. A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with \nprogress highlights, and future promises. Proceedings of the IEEE. 2021 Feb 26. \n[8] Shen D, Wu G, Suk HI. Deep learning in medical image analysis. Annual review of biomedical engineering. 2017 \nJun 21;19:221-48. \n[9] Hu SY, Xu H, Li Q, Telfer BA, Brattain LJ, Samir AE. Deep Learning-Based Automatic Endometrium \nSegmentation and Thickness Measurement for 2D Transvaginal Ultrasound. In2019 41st Annual International \nConference of the IEEE Engineering in Medicine and Biology Society (EMBC) 2019 Jul 23 (pp. 993-997). IEEE. \n[10] Guerriero S, Pascual M, Ajossa S, Neri M, Musa E, Graupera B, Rodriguez I, Alcazar JL. Artificial intelligence \n(AI) in the detection of rectosigmoid deep endometriosis. European Journal of Obstetrics & Gynecology and \nReproductive Biology. 2021 Jun 1;261:29-33. \n[11] Yang M, Liu M, Chen Y, He S, Lin Y. Diagnostic efficacy of ultrasound combined with magnetic resonance \nimaging in diagnosis of deep pelvic endometriosis under deep learning. The Journal of Supercomputing. 2021 \nJul;77(7):7598-619. \n[12] Maicas G, Leonardi M, Avery J, Panuccio C, Carneiro G, Hull ML, Condous G. Deep learning to diagnose pouch \nof Douglas obliteration with ultrasound sliding sign. Reproduction and Fertility. 2021 Oct 19;2(4):236-43. \n\n\n \n \n \n \n[13] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv preprint \narXiv:1409.1556. 2014 Sep 4. \n[14] Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. \nInInternational Conference on Medical image computing and computer-assisted intervention 2015 Oct 5 (pp. 234-\n241). Springer, Cham. \n[15] Tran D, Wang H, Torresani L, Ray J, LeCun Y, Paluri M. A closer look at spatiotemporal convolutions for action \nrecognition. InProceedings of the IEEE conference on Computer Vision and Pattern Recognition 2018 (pp. 6450-\n6459). \n[16] He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. InProceedings of the IEEE \nconference on computer vision and pattern recognition 2016 (pp. 770-778). \n[17] Tan M, Le Q. Efficientnet: Rethinking model scaling for convolutional neural networks. InInternational \nconference on machine learning 2019 May 24 (pp. 6105-6114). PMLR. \n[18] Huang G, Liu Z, Van Der Maaten L, Weinberger KQ. Densely connected convolutional networks. InProceedings \nof the IEEE conference on computer vision and pattern recognition 2017 (pp. 4700-4708). \n[19] Loshchilov I, Hutter F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. 2017 Nov 14. \n[20] Russakovsky O, Deng J, Su H, et al. Imagenet large scale visual recognition challenge. International journal of \ncomputer vision. 2015 Dec;115(3):211-52. \n[21] Kingma DP, Ba J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014 Dec 22. \n[22] Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, \nDesmaison A. Pytorch: An imperative style, high-performance deep learning library. Advances in neural \ninformation processing systems. 2019;32. \n[23] Agarap AF. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375. 2018 Mar 22. \n[24] Hacihaliloglu I, Balica AC. A Method of Improving Representation of Endometriosis Ultrasound Data By Using \nLocal Phase Tissue Signatures. Journal of Minimally Invasive Gynecology. 2020 Nov 1;27(7):S112. \n[25] Che H, Brown LG, Foran DJ, et al. Liver disease classification from ultrasound using multi-scale CNN. \nInternational Journal of Computer Assisted Radiology and Surgery. 2021 Jun 7:1-2.  \n[26] Chollet F. Xception: Deep learning with depthwise separable convolutions. InProceedings of the IEEE conference \non computer vision and pattern recognition 2017 (pp. 1251-1258). \n[27] Szegedy C, Ioffe S, Vanhoucke V, Alemi AA. Inception-v4, inception-resnet and the impact of residual \nconnections on learning. InThirty-first AAAI conference on artificial intelligence 2017 Feb 12.","source_license":"CC0","license_restricted":false}