Endometriosis detection and localization in laparoscopic gynecology

Endometriosis is a common gynecologic condition typically treated via laparoscopic sur - gery. Its visual versatility makes it hard to identify for non-specialized physicians and chal- lenging to classify or localize via computer-aided analysis. In this work, we take a first step in the direction of localized endometriosis recognition in laparoscopic gynecology videos using region-based deep neural networks Faster R-CNN and Mask R-CNN. We in particu - lar use and further develop publicly available data for transfer learning deep detection mod- els according to distinctive visual lesion characteristics. Subsequently, we evaluate the per- formance impact of different data augmentation techniques, including selected geometrical and visual transformations, specular reflection removal as well as region tracking across video frames. Finally, particular attention is given to creating reasonable data segmentation for training, validation and testing. The best performing result surprisingly is achieved by randomly applying simple cropping combined with rotation, resulting in a mean average segmentation precision of 32.4% at 50-95% intersection over union overlap (64.2% for 50% overlap).

Endometriosis segmentation · Lesion detection · Deep learning * Andreas Leibetseder [email protected] Klaus Schoeffmann [email protected] Jörg Keckstein [email protected] Simon Keckstein [email protected] 1 Institute of Information Technology, Klagenfurt University, Klagenfurt, Austria 2 Medical Faculty, Ulm University, Ulm, Germany 3 University Hospital, Ludwig-Maximilians-University Munich, Munich, Germany Published online: 7 January 2022 Multimedia Tools and Applications (2022) 81:6191–6215 / 1 3 1 Introduction Minimally invasive surgery (MIS) in gynecology focuses on the endoscopic treatment of conditions and diseases of the female inner genital tract. It is conducted by using special camera systems – endoscopes – as well as surgical instruments introduced through small incisions into the abdomen of a patient. In this way, the physician is able to perform treat- ments via observing the camera feed, while greatly reducing the physical injury imposed on patients as well as facilitating intervention monitoring, computer-aided live assistance and even the creation of medical video archives for post-surgical analysis. In particular, the latter benefit enables physicians to revisit interventions at any time for gaining useful insights or improving treatment planning and medical education. Revisiting intervention archives can especially be important for the treatment of patients with endometriosis and infertility. These patients suffer from endometrial-like tissue in a variety of abnormal locations external to the uterus. In over 60% of diagnosed cases, endo- metriosis treatment requires multiple surgeries [34] due to unidentified (missed) lesions, which can be mitigated by consulting archived intervention videos. Depending on the prac- tice of an individual physician, it certainly is possible to record specific short clips of rel- evant content during surgery, however, typically surgeries are archived in full meaning that the resulting videos generally contain a considerable amount of irrelevant content such as out-of-patient views, visually inexpressive or static scenes where treatment is interrupted and even common surgery phases that are of little importance for post-surgical analysis. Such properties combined with the typical routine of using conventional video players and file browsers for manually examining surgical video archives greatly impair revisiting past interventions, while also making this procedure error-prone. To improve this situation, computer-aided automatic content analysis can be employed for creating systems that high- light potentially relevant content to physicians during patient case inspections. However, although obvious irrelevant video segments such as overly blurry frames or camera testing screens can reasonably be identified via video analysis [24], more sophisticated approaches are required for identifying very specific content such as scenes showing endometriosis lesions. The main symptoms of endometriosis, a benign inflammatory disease, are menstrual- related pelvic pain and unexplainable infertility. It is found in a large variety of locations and treatment strategies depend on the symptoms, the extent and localization of the lesions. The best known classification systems for endometriosis are the revised Ameri- can Society for Reproductive Medicine (rASRM) score [1 ] and the Enzian classification scheme [14, 15]. The rASRM score describes superficial lesions of the peritoneum and ovarian endometriosis in four stages, whereas the Enzian classification categorizes deep endometriosis. Alongside these different possible locations, endometriotic lesions also strongly vary in their visual appearance, both intra- and interpersonal. This is illustrated by Fig.  1 showing a set of positive and negative endometriosis examples 1 at three com- mon locations: ovary, uterus and peritoneum. In direct comparison and without a spe- cific medical background, the differences between normal and pathological tissue are very difficult to discern, which evidently holds true for laymen but even inexperienced medical practitioners. Consequently, with the current successful application of deep 1 Please no te that Fig.  1b, d and f merely depict a subset of all the conditions’ visual appearances – refer to Section 3.1 for more examples. 6192 Multimedia Tools and Applications (2022) 81:6191–6215 1 3 learning in many medical fields, attempting to solve this problem via computer-aided analysis seems reasonable. Moreover, being able to classify and potentially locate endo- metriosis can not only be helpful during interventions but also in treatment planning and particularly in teaching/training. In this work we create, utilize and evaluate different datasets for a binary endometriosis detection and localization task using state-of-the-art region-based convolutional neural net- works (R-CNNs). Since as of yet to the best of our knowledge no comparable work exists, this paper’s findings can be considered benchmarks for future research on the topic. We define our contributions as follows: Dataset development, Section  3.1. By revising the labeling strategy of publicly avail- able endometriosis dataset GLENDA  [20] towards visual similarity, we discover a large improvement in lesion segmentation performance. Further refinement of a newly (a) (b) (c) (d) (e) (f) Fig. 1 Endome triosis examples – negative (left column) and positive (right column) examples at locations ovary, uterus and peritoneum 6193Multimedia Tools and Applications (2022) 81:6191–6215 1 3 defined specifically well performing class – endometrial implants – leads to the release of a novel dataset named ENdometrial Implants Dataset2 (ENID). Data augmentation, Section  3.2. Utilizing ENID, we investigate the segmentation performance impact of a variety of data augmentation techniques including rotation, blurring, cropping, perspective transformation, reflection removal and frame-by-frame object tracking. Performance evaluation, Sections  3.3&  3.4. For training deep models, we employ state-of-the-art object detection and segmentation models Faster R-CNN [33] and Mask R-CNN [9] using ResNet [8] variants as backbone networks. For creating training, vali- dation and testing splits we divide the data twofold and pay specific attention to avoid- ing similar images across splits, thus risking overfitting our models. This work is mainly partitioned into three sections: related work in Section  2, methodology description in Section 3 and discussion in Section 4. 2 Related work The concept of learning features rather than defining them by hand had a tremendous impact on image classification over the last years. Deep convolutional networks such as AlexNet [17], GoogLeNet [35] or ResNet [8] have been successfully applied to countless domains, even non-image-related ones. Such networks not only are utilized on their own, they also represent valuable backbones to many deep architectures for image as well as video analysis. One such architecture family heavily using CNNs as backbones for region of interest (ROI) prediction and labeling are named region based convolutional neural net- works or R-CNNs. These R-CNN networks after sufficient training are capable of locating, classifying and even segmenting objects in images through smart arrangement and utiliza- tion of CNN components. When regarding the evolution of R-CNNs from Fast R-CNN [6] over Faster R-CNN [33] up to Mask R-CNN [9], large improvements in runtime perfor - mance can be observed, as architecture bottlenecks were gradually resolved. Albeit said R-CNNs are developed for the common objects in context (COCO) challenge [21], which addresses the problem of real-world object detection, their successful application in many domains including medical imaging encourages us to as well employ variants of Faster R-CNN and Mask R-CNN architectures for this study’s challenging lesion detection tasks. The rising performance improvements of deep learning in medicine entails an ever- expanding range of applications targeted at providing medical staff with digital assistance for patient treatment. While similarly to the general multimedia domain, a large variety of data is analyzed ranging from electronic health records (E.H.R) over biosignals up to images produced by manifold clinical imaging technologies [30]. Apart from being used across many special fields, medical imaging also varies strongly depending on its purpose: monochrome images taken from computer tomography (CT) scans or ultrasound are very different compared to open surgery or endoscopic recordings. This in general renders sci- entific research on medical image classification more difficult to compare than in traditional multimedia analysis. Surprisingly, endoscopic images as of yet are not nearly as much digi- tally analyzed as many other technologies such as magnetic resonance imaging (MRI) [31, 2 http:// ftp. itec. aau. at/ datas ets/ ENID or https:// doi. org/ 10. 5281/ zenodo. 45709 69 6194 Multimedia Tools and Applications (2022) 81:6191–6215 1 3 32] or even the very similar field of microscopy [22]. Nevertheless, when regarding exist- ing research on endoscopic video or image analysis three application areas can be defined: pre-processing methods, real-time support at procedure time and post-procedural applica- tions [24]. Lesion detection in the laparoscopic field, as targeted in this study, can, depend- ing on the use case, be placed into both of the latter categories. For instance, it can be help- ful for tasks such as pointing out pathology during surgery as well as having post-surgical functions like automatically annotating medical video archives. Generally, endoscopic surgery videos offer a great variety of research topics for apply - ing deep learning. For instance, many studies are found on classifying or detecting content such as anatomy [19, 43], surgical tasks [29], instruments [12, 16, 26] or even the occur - rence of smoke [18]. Furthermore, attempts have been made to recognize surgical phases based on that content [37, 42]. Finally, laparoscopic lesion segmentation using deep neural networks is a comparably scarcely researched field, which for a large part can be attributed to the limited availability of expert-annotated medical datasets. While lesion detection is much addressed in the more restrictive field of gastrointes- tinal endoscopy  [2], e.g. for polyp detection  [11, 27, 28], to best of our knowledge, no directly comparable work has been conducted regarding image or video-based lapa- roscopic endometriosis segmentation. When, however, targeting classification we find almost all image-based works are on the different subject of endometrial cancer detection in MRI [38]. Nevertheless, we identify two approaches targeting endometriosis classifica- tion, albeit one does not contain any results besides extracting visual features  [39]. The other approach uses the previously published GLENDA [20] dataset, which is as well used and further developed in this study, for binary classifying endometriotic regions using sev- eral CNNs [40]. They obtain classification accuracies between 80-90%. In the absence of any other related work to the best of our knowledge, we further attempt to give the reader an overview and list a small number of notable studies in the field of laparoscopic object detection and segmentation together with their findings. Predominantly, studies approach- ing object detection in laparoscopy attempt to localize instruments as is highlighted by the existence of literature surveys such as the one written by Yang et  al.  [41]. In fact, sev - eral studies achieve good results for this task: Jin et al. [12] use Faster R-CNN reporting a bounding box mean average precision (mAP) in terms of intersection over union (IoU) of 63.1%, while Kletz et al. [16] report a similar mAP of 64.5% for bounding boxes and in addition, since they are using the segmentation-capable Mask R-CNN, a segmentation mAP of 54.3%. Furthermore, in addition to surgical instruments, Zadeh et al. [43] attempt to localize anatomical regions, also using Mask R-CNN and achieving high segmentation mAPs for uterus (84.5%), ovary (29.6%) and tools (54.5%, similar to Kletz et  al.  [16]). Finally, Gibson et al. [5] define a deep residual network for supervised liver segmentation yielding a median Dice similarity score of 95%. Fu et  al.  [4] even show that this result score can be improved to 96.4% by following a different model training strategy, i.e. using a semi-supervised mean teacher method. 3 Methodology Our approach targeting endometriosis detection and segmentation is summarized in Fig.  2. We start by gradually developing ENID by revising publicly released dataset GLENDA  [20] and creating intermediate dataset GLENDA-VIS in Section  3.1. Sub- sequently, we thoroughly describe all employed data augmentation techniques such as 6195Multimedia Tools and Applications (2022) 81:6191–6215 1 3 blurring, cropping and desaturation but also reflection removal as well as tracking in Sec- tion  3.2. Finally, after outlining our data partitioning approaches, Section  3.3 details our model training strategies and Section 3.4 lists our evaluation results. 3.1 Datasets When considering all potential endometriosis locations and severities using both, the rASRM as well as the Enzian system, a complete dataset would have to include well over 50 classes, which would be challenging to collect in large enough quantities. Therefore, the publicly available dataset GLENDA [20] follows the strategy of disregarding lesion size altogether and focusing on three common locations as well as one specific type of endo- metriosis. Figure  3 depicts example images of the dataset’s classes: peritoneum, ovary, uterus and deep infiltrating endometriosis 3 (DIE). As indicated with a light green color, in every image at least one region is annotated using rectangular bounding boxes or polygon shapes. All of these annotations fully enclose the lesions, albeit mostly do not mark their exact boundaries. This is due to the difficulty of reasonably identifying such boundaries: often, pathologic areas contain multiple very small lesions that are infeasible to annotate individually (c.f. Fig.  3d, second image) or the borders of affected tissue blend in with adjacent healthy tissue (c.f. Fig.  3a, second image). Apart from this annotation strategy likely already entailing consequences for model training, an arguably much larger impact factor can be identified when regarding the visual dissimilarity of annotated regions within classes. Although partially showing similar patterns across classes as well as within them, the lesions nevertheless can also vary strongly, which is a highly detrimental character - istic for convolutional deep networks attempting to learn visually distinct features across classes. Fig. 2 Me thodology overview – dataset creation, augmentations and deep model training 3 Deep infiltr ating endometriosis is found several millimeters into the lining of the abdmonial cavity, i.e. the peritoneal tissue. 6196 Multimedia Tools and Applications (2022) 81:6191–6215 1 3 Addressing these problems, we perform two steps for improvement. First, we reorgan- ize the dataset’s classes by disregarding lesion location and grouping pathologic areas by their visual similarity. This is accomplished by using most of GLENDA’s images and altering their annotations accordingly. Examples of the resulting visual GLENDA dataset (GLENDA-VIS) are illustrated in Fig. 4 . GLENDA-VIS contains four classes: Mucus white-toned mucous areas, often part of adhesions or sclerotic areas Vesicles clear blister-like areas, often resembling water drops Implants dark-toned growths, often are surrounded by white sclerotic areas Abnormal Tissue irritated/bleeding tissue, e.g. increased/deformed blood vessels or bleeding endometrial cells GLENDA-VIS is a preliminary dataset with annotations created exclusively from rec- tangles that can overlap to form simple polygonal areas. It is intentionally created in this rudimentary manner, as for the purposes of this work it is merely used to determine the most promising visual endometriosis characteristic for the second improvement step performed: creating a larger binary dataset using the best performing GLENDA-VIS class. After evaluating the instance segmentation performance of both GLENDA and GLENDA-VIS (c.f. Tables  9 and  10), we identify Implants as the most promising can- didate for our final ENdometrial Implants Dataset (ENID), which is shown exemplary in Fig.  5. We base most of our evaluations in this work on ENID, in which we finally (a) (b) (c) (d) Fig. 3 GLEND A dataset class overview 6197Multimedia Tools and Applications (2022) 81:6191–6215 1 3 address the above issue of annotation imprecision – lesions are enclosed by carefully freehand-drawn polygons created using the Endoscopic Concept Annotation Tool [25] and according to following criteria: color Implants generally appear as dark, sometimes dotted areas with varying color tones. Annotators predominantly focus on clearly visible implants. size Annotated implants are required to be of a certain, well visible size – very small lesions are very likely not have a significant impact. boundary Annotations are created precisely, carefully identifying lesion boundaries. Here annotators focus on the transition from darker to lighter tissue. (a) (b) (c) (d) Fig. 4 GLEND A-VIS dataset class overview Fig. 5 ENID dat aset examples 6198 Multimedia Tools and Applications (2022) 81:6191–6215 1 3 Similar to GLENDA/GLENDA-VIS, ENID is collected from a selection of frames belonging to a laparoscopic video archive of over 500 individual surgery recordings. Table  1 presents a comparison of statistics for all three datasets: number of patient cases, images and annotations. 4 As indicated by the cases column, all datasets contain frames taken from close to 100 surgeries, which on one hand ensures data variety for increased generalizability and on the other hand lowers the risk of overfitting due to potential simi- larities in training and validation data. This specifically is deemed important for creating the single-class ENID dataset, which is collected from altogether 108 surgeries – more than the number of cases for for the combination of all classes in GLENDA or GLENDA-VIS. 3.2 Augmentation Apart from comparing the instance segmentation performance of the raw datasets, we also investigate the performance impact of applying several augmentations to ENID. We alto- gether explore seven augmentation techniques, five of which are commonly used in the field of image retrieval. All augmentations are exclusively applied to the training portion of the dataset only, while the validation and test sets are left unchanged. Figure  6 shows six applied augmentations, where Fig.  6a-e are applied in the following manner: – F or every image i of training split St , one or multiple augment ations are applied n times increasing St by a factor of n + 1 , i.e. Sa t = St ∪{ ia 1, ..., ia n} . In all our e xperiments we use n = 2 effectively tripling the training datasets. – A ugmentations are applied in random intensities on a set interval of potential values differing for every augmentation type. Table 1 Dat aset Statistics Comparison – number of patient cases, images and annotations Dataset Cases Images Annotations GLENDA peritoneum 84 257 489 ovary 27 53 54 uterus 11 17 25 die 29 55 59 total 102 373 628 GLENDA-VIS mucus 77 246 442 vesicles 59 153 250 implants 86 281 652 abnormal tissue 41 66 85 total 96 342 1485 ENID implants 108 160 358 4 Please note that individual frames are allowed to contain annotations for multiple classes, hence, the cases/images totals for GLENDA/GLENDA-VIS in fact are lower than they would be when simply sum- ming up all class statistics. 6199Multimedia Tools and Applications (2022) 81:6191–6215 1 3 In the following we describe corresponding image augmentations 5 and their intensity scales: rotating rotate images on a scale between [−45, 45] degrees blurring apply motion blur using random angles and directions with kernel sizes between [3, 11] pixels cropping crop each image side by a percentage between [0.0,  0.25], input size is retained perspective transform transform a frame’s perspective by placing 4 points relative to its corners in a distance (scale) between [0.01, 0.15], input size is retained desaturation reduce image saturation by a percentage between [0.15, 0.40] The remaining augmentation shown in Fig.  6f represents a more special case, hence, it is applied differently. The introduction of bright light sources for illumination during laparo- scopic surgery frequently creates specular reflections, especially on wet surfaces or bulging tissue as is formed in the case of shallow endometrial implants. This can have detrimental effects on the learning performance (cf. Fig.  8d), since it may lead to associating reflec- tions with lesions. Therefore, we explore the effects of automatically removing very bright reflections, i.e. regions above a certain brightness threshold, and correct the images using inpainting as proposed by Telea et al. [36]. As we want to remove reflections altogether from St , when appl ying this augmentation we simply replace these training images with the processed ones, i.e. altering instead of enlarging the corresponding data. Although remov - ing potentially important information using such an approach, we, nevertheless, expect it to mitigate the negative performance impact of specular reflections. (a) (c) (e) (f) (d) (b) Fig. 6 Summar y of applied augmentations – original (left), augmented (right) 5 F or more detailed information about these standard augmentations please refer to the utilized library imgaug [13]. 6200 Multimedia Tools and Applications (2022) 81:6191–6215 1 3 For our final augmentation, we perform object tracking using the annotated frames together with their source videos. We accomplish this task by using kernelized correla- tion filters [10], a simple and fast bounding box tracking methodology. We process each annotation on a frame-to-frame basis, i.e. every successfully tracked region is used for tracking the next region. Albeit simple bounding boxes can not capture the exact ground truth region or its deformation, we nevertheless achieve good results by simply relocat- ing and resizing the original annotations according to the tracked bounding boxes – Fig.  7 shows some of the results. Subsequently, we manually review all of the tracked frames and remove evidently incorrect results. Following this approach, we are able to augment the full dataset approximately by a factor of 81 (160 vs. 12981 frames). The final augmenta- tion factor, however, depends on how the dataset is split into training, validation and test set, since similar to all other augmentations this one is as well only applied to the training portion. Also, in order to avoid feeding the network with almost identical images, as a last step we uniformly sample tracked frames with three different frame intervals: v=10, v=20 and v=30. For each interval, starting with the ground truth frame, we pick every v-th frame from a reviewed tracking sequence and repeating this process for the entire training set, yielding three augmented datasets: tracked 10 , tracked20 and tracked30. Finally, in order to compromise between swift model training and result expressiveness, we use two dataset splits throughout all our experiments (training/validation/test): 60/20/20 and 80/10/10. For splitting, special attention is paid to avoid similar frames across splits and risk overfitting: patient cases, i.e. full surgery recordings, can only be part of a single split portion. For brevity, we only include result listings for the 60/20/20 split throughout the paper, while listing everything related to 80/10/10 in the Appendix, as to provide the interested reader with further insights. Accordingly, Table  2 lists augmentation statistics related to the 60/20/20 split and Table  8 represents its Appendix-counterpart for 80/10/10. When regarding the number of images in the augmented training splits we can conclude that, while tripling our data with basic augmentations, we achieve an augmentation fac- tor of over 7 with tracking. Additionally, convincing findings in our evaluations (cf. Sec- tion  3.4) also encourage us to combine augmentations, which are listed as the last four entries in the TRAINING section of Table  2. First, we investigate the effects of combin- ing all simple augmentations for non-tracked (all w/o tracking) as well as tracked (all w/ tracked 30 ) dat a and second, discuss other promising augmentation combinations per split Fig. 7 T racked ENID examples, one tracked annotation per row (GT marks original ground truth annota- tion) 6201Multimedia Tools and Applications (2022) 81:6191–6215 1 3 (blurring/cropping/rotating). Analogously to the above descriptions, we keep using n = 2 for these additional approaches and apply all involved augmentations in a sequence to both images. Lastly, it is worth mentioning that, albeit the number of images stays consist- ent across augmentations, the number of annotations varies because they potentially are removed by geometrical image transformations. Additionally, if they are located near an image border a transformation can as well possibly dissect them. As a consequence during the augmentation process, we verify that every augmented image at least contains one valid annotation. 3.3 Model training Having developed the datasets and prepared ENID in the above described way, we at this point are ready for model training. As mentioned in the introductory section, we use both Faster R-CNN (Faster) as well as Mask R-CNN (Mask) as architectures, each trained with ResNet-50 (R-50) and ResNet-101 (R-101) as backbones. We conduct transfer learn- ing, i.e. the models are initialized with pre-trained weights from the COCO dataset [21], while adjusting the last layer for our detection and segmentation purposes: since we want to compare the performance of single lesion classes, we exclusively train our models to give binary output. To achieve this, we correspondingly split up all multi-class datasets GLENDA and GLENDA-VIS into binary datasets of four classes each. The utilized loss functions incorporate class (log loss), bounding box (smooth L1 loss) and for Mask R-CNN pixel-based segmentation (binary cross entropy loss) predictions as described in  [6, 9]. Additionally, for optimization we choose stochastic gradient descent (SGD) and conduct Table 2 ENID (60/20/20) – augment ation statistics, cases, images and annotations Augmentation Cases Images Annotations TRAINING None 64 96 240 Rotating 64 288 713 Blurring, desaturation 64 288 720 Cropping 64 288 672 Perspective transform 64 288 703 Reflection removal 64 96 240 Tracked10 64 653 1110 Tracked20 64 361 649 Tracked30 64 265 486 All w/o tracking 64 288 637 All w/ tracked30 64 786 1292 Blurring & rotating 64 288 713 Cropping & rotating 64 288 658 VALIDATION All 24 32 61 TESTING All 20 32 57 6202 Multimedia Tools and Applications (2022) 81:6191–6215 1 3 initial experiments for finding the optimal learning rate /u1D702∈{ 0.01, 0.001, 0.0001} with the

of choosing /u1D702= 0.001 due to its overall superior performance. We also conduct pre- liminary experiments using Adam, yet, find that this optimization strategy tends to perform better for classifying rather than segmenting endometriosis. Training is conducted for a fixed number of 50 epochs (approx. 25.5K iterations) and we create a model checkpoint approximately every 5 epochs. The utilized networks are implemented in Python using Detectron2 6 – an object detec- tion framework powered by the PyTorch 7 deep learning framework. Regarding hardware, a workstation running Linux Ubuntu 18.x with the following specifications was used: Intel Core i7-5820K CPU @ 3.30GHz x 6, 32 GiB DDR3 @ 1333 MHz, Nvidia GeForce GTX 1080. Depending on the amount of input data, training a single network for 50 epochs requires approximately 2-4h to complete. 3.4 Evaluation This section contains all the results achieved for evaluating our trained models (Sec- tion  3.4.1) as well as qualitative prediction insights (Section  3.4.2). Similarly to Sec- tion  3.2, we only report results for the 60/20/20 split, while listing the ones for 80/10/10 in the Appendix. Additionally, in order to further condense table sizes, for every potentially augmented dataset, we merely report the best performing network combination (R-CNN + backbone) evaluated at the epoch checkpoint that achieves the highest mean average preci- sion (mAP) value. 3.4.1 Quantitative results Using the COCO-detection [21] metrics, we evaluate a number of mAP values for different intersection over union (IoU) thresholds – the higher the threshold, the more a predicted area is required to overlap with a given ground truth to be considered correct. Since we are using Faster R-CNN and Mask R-CNN architectures, we report bounding box 8 mAPs (mAPbb ) as w ell as pixel mask segmentation mAPs (mAP). IoU thresholds are identified by subscript numbers: mAP 50−95 indicates the mAP calculated over 10 thresholds starting from 50% percent up to 95% IoU overlap using intervals of 5%, while mAP 50 and mAP 75 describe the mAP at single specific thresholds 50% as well as 75%. First, we evaluate the performance of the raw (unaugmented) datasets GLENDA, GLENDA-VIS and ENID listing mAP comparisons for bounding box prediction in Table  3. Throughout GLENDA as well as GLENDA-VIS, we discover very low precisions of at most close or equal to 3% (mAP bb 50−95=0.030). For many classes (ovary, uterus, die, abnormal tissue), result precisions for any trained models even are almost 0% (mAPbb 50−95 < 0.000). This is as well reflected in the 80/10/10 split (cf. Table  9) with the exception of ovary scoring around 10% (mAPbb 50−95=0.107). When specifically observing GLENDA-VIS, we identify vesicles (mAP bb 50−95=0.030) and implants (mAP bb 50−95=0.029) as the best per - forming classes and their similar performance is confirmed by regarding the evaluations for 6 https:// github. com/ faceb ookre search/ detec tron2 7 https:// pytor c h. org/ 8 Please no te that for Mask vs. Faster R-CNN bounding boxes we essentially are comparing RoI Pooling vs. RoI Align, the latter of which is not suffering from quantization loss, hence, is expected to yield more precise results. 6203Multimedia Tools and Applications (2022) 81:6191–6215 1 3 the 80/10/10 split (cf. Table  9). ENID being the more refined endometrial implants dataset outperforms both other datasets significantly with a precision over 30% (mAPbb 50−95=0.308), which is even increased to over 50% (mAPbb 50 =0.561) when merely considering a 50% IoU overlap threshold. When regarding the mask segmentation results listed in Table  4, we observe similar results: GLENDA as well as GLENDA-VIS classes yield equally low or even lower segmentation scores, while ENID on the other hand scores significantly higher with over 30% for all thresholds (mAP 50−95=0.309) or above 50% for a 50% IoU thresh- old (mAP 50−95=0.581). Albeit performing slightly worse (mAP bb 50−95=0.288, mAP bb 50=0.5, mAP50−95=0.250, mAP50=0.522), the same trend can be observed for split 80/10/10 (cf. Tables 9 and 10). Having observed superior results for ENID in all mAP categories, we finally compare the raw dataset’s performance against the effects of applying different augmentations. Table  5 lists all precision values for bounding box predictions and only few of them seem to to have a slight impact on the results. While cropping, perspective transform and desatu- ration yield very similar results to the raw dataset (0-0.5% improvement in mAP bb 50−95 ), we disco ver that the best performing augmentations for bounding box prediction are blurring and rotating with 0.7-2.5% improvements in mAPbb 50−95 . Albeit t hese improvements can not be regarded as significant, in particular since they can not be observed for the 80/10/10 split Table 3 GLEND A, GLENDA-VIS, ENID (60/20/20) – bounding box prediction precision, raw; best (bold) and worst (italic) Dataset Arch. Backb. Ep. mAPbb 50−95 mAPbb 50 mAPbb 75 GLENDA peritoneum Mask R-101 24 0.030 0.068 0.010 ovary all all − <0.000 <0.000 <0.000 uterus all all − <0.000 <0.000 <0.000 die all all − <0.000 <0.000 <0.000 GLENDA-VIS mucus Mask R-101 35 0.015 0.026 0.020 vesicles Mask R-101 30 0.030 0.068 0.004 implants Mask R-101 35 0.029 0.069 0.020 abn. tissue all all − <0.000 <0.000 <0.000 ENID implants Mask R-50 34 0.308 0.561 0.276 Table 4 GLEND A, GLENDA-VIS, ENID (60/20/20) – mask segmentation prediction precision, raw; best (bold) and worst (italic) Dataset Arch. Backb. Ep. mAP 50−95 mAP50 mAP75 GLENDA peritoneum Mask R-101 24 0.025 0.074 0.004 ovary all all − <0.000 <0.000 <0.000 uterus all all − <0.000 <0.000 <0.000 die all all − <0.000 <0.000 <0.000 GLENDA-VIS mucus Mask R-101 35 0.015 0.026 0.020 vesicles Mask R-101 30 0.029 0.068 0.027 implants Mask R-101 35 0.028 0.069 0.019 abn. tissue all all − <0.000 <0.000 <0.000 ENID implants Mask R-50 39 0.309 0.581 0.325 6204 Multimedia Tools and Applications (2022) 81:6191–6215 1 3 – here all augmentations even perform worse than the raw dataset (cf. Table  11). Looking at the mask segmentation results, however, we discover a slight impact of rotating for both, split 60/20/20 (0.320 vs. 0.309, cf. Table  6) as well as split 80/10/10 (0.253 vs. 0.250, cf. Table  12). The latter split even indicates a performance boost of 2.3-3.85% for cropping in all listed mAP values compared to the raw dataset performance. Further, for the com- bination of all augmentations applied to the raw and tracking 30 data, we observe similar performance to most individually augmented data – for 80/10/10 we even discover many of the worst-performing mAPs for both bounding box and mask predictions. Lastly, it is worth noting that although the tracking approach consistently decreases the performance, Table 5 ENID (60/20/20) – bounding bo x prediction precision, raw vs. augmented; best (bold) and worst (italic) Augmentation Arch. Backb. Ep. mAPbb 50−95 mAPbb 50 mAPbb 75 None Mask R-50 34 0.308 0.561 0.276 Rotating Mask R-101 24 0.333 0.633 0.322 Blurring Mask R-50 44 0.315 0.582 0.309 Cropping Mask R-50 44 0.308 0.544 0.269 Perspective transform Mask R-101 44 0.311 0.591 0.292 Desaturation Mask R-50 49 0.313 0.566 0.303 Reflection removal Mask R-101 44 0.281 0.544 0.274 Tracking10 Faster R-101 15 0.234 0.460 0.183 Tracking20 Mask R-50 40 0.268 0.516 0.183 Tracking30 Mask R-50 50 0.270 0.515 0.232 All w/o tracking Mask R-101 19 0.299 0.590 0.231 all w/ tracked30 Faster R-101 19 0.276 0.526 0.216 Blurring & rotating Mask R-101 24 0.307 0.638 0.250 Cropping & rotating Mask R-101 24 0.344 0.627 0.278 Table 6 ENID (60/20/20) – mask segment ation prediction precision, raw vs. augmented; best (bold) and worst (italic) Augmentation Arch. Backb. Ep. mAP 50−95 mAP50 mAP75 None Mask R-50 39 0.309 0.581 0.325 Rotating Mask R-50 24 0.320 0.640 0.255 Blurring Mask R-50 44 0.290 0.582 0.289 Cropping Mask R-101 24 0.298 0.580 0.250 Perspective transform Mask R-101 14 0.283 0.573 0.267 Desaturation Mask R-50 19 0.286 0.575 0.229 Reflection removal Mask R-101 24 0.286 0.547 0.244 Tracking10 Mask R-101 30 0.197 0.489 0.102 Tracking20 Mask R-101 10 0.228 0.465 0.175 Tracking30 Mask R-101 15 0.217 0.473 0.119 All w/o tracking Mask R-101 19 0.277 0.572 0.229 All w/ tracked30 Faster R-101 14 0.250 0.487 0.214 Blurring & rotating Mask R-101 19 0.294 0.608 0.265 Cropping & rotating Mask R-101 29 0.324 0.642 0.255 6205Multimedia Tools and Applications (2022) 81:6191–6215 1 3 the results indicate that more samples generally perform worse, i.e. tracked 20 and tracked30 mostly show higher mAP values than tracked 10 with the exception of mAP bb 50−95 for split 80/10/10 (cf. Tables 5 and 11) Based on above observations, we identify three augmentations that generally seem to have an impact on the resulting precision values, whether for bounding box prediction or mask segmentation on either of the splits: rotating, cropping and blurring. Therefore, we choose to combine these particular augmentations in order to determine, if the outcomes can further be improved. After training additional models on ENID that is augmented by jointly applying above techniques, we subsequently decide to add the following two aug- mentation combinations to our evaluations: blurring & rotating as well as cropping & rotat- ing. Unfortunately, for both splits, the best models for augmenting ENID through blur - ring- and rotating are not improving prediction precision and, in fact, even show slightly detrimental effects on mask segmentation results for split 60/20/20 (0.294 vs. 0.309 mAP 50−95 , cf. T able  6) and split 80/10/10 (0.238 vs. 0.250 mAP 50−95 , cf. T able  12). On the other hand, models for cropping- & rotating-augmented ENID mostly outperform raw ENID models for bounding box prediction and segmentation in both splits, albeit merely by up to 3.6% precision. Nevertheless, this ultimately seems to be the most robust perfor - mance improvement indicator. Thus, we finally summarize the overall best augmentation techniques for any of the splits to to be cropping, rotating and their combination. Subse- quently, we put a stronger focus on the segmentation results of the 60/20/20 split and com- pare the best models trained on raw ENID to those trained on rotating- as well as crop- ping- & rotating-augmented ENID: although the performance improvement of applying augmentation in this case merely amounts to 1.1-1.5% mAP 50−95 , t he models’ differences, nevertheless, become apparent when qualitatively inspecting their prediction outputs. 3.4.2 Qualitative results A qualitative inspection of results taken from the 60/20/20 testing set reveals advantages as well as disadvantages in our evaluation’s best performing models, which are chosen accord- ing to segmentation precision (mAP 50−95 ). Figure  8 compares selected prediction results of the best model for raw ENID (M M-R-50 raw ) with the best models trained on ENID with augmentations, i.e. rotation (M M-R-101 rot. ) and cr opping combined with rotation (M M-R-101 crop+rot. ). Ev ery row in the figure compares a test split ground truth annotation with predictions of all chosen models in the same order as mentioned above and using a confidence threshold of 50%. In contrast to the green outlined ground truth annotations, the model predictions are randomly colored and include pixel segmentation masks as well as bounding boxes for increased visibility. Starting with Fig.  8a, M M-R-101 rot. fails to predict some lesion areas, yet, according to the ground truth, correctly is not mispredicting the small dark spot on the right side of the image, in contrast to the other models. This region, however, is not as easily distinguish- able from a pathologic area: in fact, ground truth annotations are only made for obvious lesions, therefore, the prediction in question could be considered suspicious and in need of further examination by an expert. Thus, arguably in this case, an expert might even prefer such suspicious areas to be included in the results, despite their potential of being false alarms. Subsequently, in all of the remaining figures, we examine the most common mis- predictions discovered throughout all trained models. Figure  8b demonstrates how M M-R-50 raw and M M-R-101 rot. confuse a darker background area for a lesion, while M M-R-101 crop+rot. manages to avoid this mistake. In fact, this is discovered most frequently with differing severity, i.e. 6206 Multimedia Tools and Applications (2022) 81:6191–6215 1 3 multi-shaded background areas potentially are mispredicted partially up to fully. Moreo- ver, a further cause of mispredictions are prominent color transitions, as can be discovered in Fig.  8c, where M M-R-101 rot. confuses a blood vessel for a lesion. Blood or blood vessels are contrasting in color with their surrounding tissue, which makes them easy targets to mispredict, especially when embedded in mucus-like surrounding tissue as is the case in the figure. Another less obvious but prevalent misprediction type are reflections as shown in Fig.  8d, which demonstrates M M-R-50 raw falsely predicting a large specular reflection to be a lesion, while the other models behave correctly in this situation. This presumably hap- pens because reflections are part of many implant annotations (cf. Fig.  5), hence, they are learned to be part of the lesion regions. Although mitigating this particular problem, reflec- tion removal evidently increases the amount of other mispredictions, thus, altogether hav - ing a detrimental rather than beneficial effect (cf. Table  6). Surprisingly, at least in the depicted case, cropping seems to as well have a positive effect on this problem. Lastly. (a) (b) (c) (d) (e) Fig. 8 Selected q ualitative result comparison of best performing models using a 50% confidence threshold – ground truth (left), Mask R-CNN with ResNet-50 on raw ENID (MM-R-50 raw , middle lef t), Mask R-CNN with ResNet-101 on rotating-augmented ENID (MM-R-101 rot. , middle r ight), Mask R-CNN with ResNet-101 on crop- ping- & rotating-augmented ENID (MM-R-101 crop+rot. , right) 6207Multimedia Tools and Applications (2022) 81:6191–6215 1 3 instruments also exhibit attributes that can lead to confusion: many possess black shafts that under certain lighting conditions are easily confused for implants and their often silvery top parts frequently reflect light, which results in reflection mispredictions. For instance, Fig.  8e shows M M-R-50 raw struggling with instrument shafts, while the other models appear to less susceptible to this kind of misprediction. Overall, judging by the examples shown in Fig.  8, it seems apparent to deem M M-R-101 rot. and M M-R-101 crop+rot. as more robust than M M-R-50 raw . Ho wever, it is worth again referring to their mere slight overall performance improvement of at most 1.5% mAP 50−95 . Ther efore, they ultimately many times are as well suffering from above mispredictions and lesion localiza- tion failures, yet, on different test images. Nevertheless, every above model seems to have its merits in certain situations, which may allow for increasing performance by applying them in sequence and taking decisions using majority voting. 4 Discussion Reviewing the entirety of results, we observe several expected, yet, also unexpected out- comes. First, looking at the prediction performance of GLENDA, GLENDA-VIS and ENID we can confirm our conjecture that categorizing endometriosis by visual appear - ance rather than location is more suitable for state-of-the-art R-CNN networks. This particularly seems reasonable when focusing on a lesion manifestation like endometrial implants, which potentially can occur at any bodily region susceptible to endometrio- sis. Moreover, we ultimately base choosing to further investigate this particular class on observing its overall best performance as part of GLENDA-VIS, while taking into account two different dataset splits. This decision can arguably even stronger be justified by observing the somewhat exceptional performance of models trained on GLENDA’s ovary class, which appears to be influenced by its inclusion of many implants-like anno- tations (cf. Figs.  3b or 7, first row). Nevertheless, ovary-trained models in direct compar - ison to ENID-trained models show significantly lower performance, which is attributable to the difference in annotation strategies used when creating the classes: apart from the necessity of consulting medical experts, precisely enclosing lesion regions as well as set- ting clear annotation guidelines are key components for largely increasing segmentation performance. On the downside, we discover a strong decrease in precision when requiring pre- dictions to have a higher lesion area overlap. While this is expected considering the difficulty of the problem, we on average identify a considerable decrease of close to 40% between results calculated with a 50% and 75% overlap requirement, which holds true for models trained on raw as well as augmented ENID. Surprisingly, most pro- posed augmentations do not yield any significant performance improvements, which, although standing in contrast to results achieved in traditional image segmentation tasks [23], we find to be in line with the discoveries in other medical fields, such as the findings of Fox et al., who apply Mask R-CNN for tool segmentation in cataract surgery using microscopy videos [3 ]. Nevertheless, when striving to identify the best performing augmentations by regard- ing quantitative as well as qualitative evaluations we identify simple rotation, the

of which can even further be improved in combination with cropping. Interest- ingly, this combination for various examples counteracts some of the major causes of mispredictions: dark background areas, blood vessels, specular reflections and surgical 6208 Multimedia Tools and Applications (2022) 81:6191–6215 1 3 instruments. Nevertheless, the best performing augmented models still merely show slight performance improvements over the non-augmented models. Interestingly, some mispredictions could be regarded as questionable and in need of further expert inspec- tion, since it is possible that pathological regions are omitted during ground truth anno- tation, either because of not conforming to the annotation policy or being too incon- clusive. Since larger numbers of such unannotated lesions could influence training, this particularly could explain the comparably poor performance of models trained on tracked data: single annotated frames do not necessarily contain all lesion regions of a whole scene, however, during tracking they can easily be brought into view by camera movements or surgery actions like moving an organ to reveal occluded implants. Finally, it is also possible that, despite uniform sampling, tracked images still are too similar, causing a disproportionate amount of weight to be put on easily trackable lesion scenes while decreasing the impact of others. 5 Contributions As outlined in Section  2 at the time of writing, we are unable to retrieve any directly com- parable work on endometriosis segmentation. Nevertheless, we list some loosely related work in Table  7, where we set our work in contrast to classification, object detection and segmentation tasks on endoscopy [12, 16, 40, 43] as well as microscopy [3] datasets. For classification, we list Visalaxi et al. [40] who achieve a very high binary endome- triosis classification accuracy of 90.0% using ResNet on the GLENDA [20] dataset. When compared to our work, besides classification being a different task, with defining endome- trial implants we additionally are targeting a more confined type of lesion than the various ones included in GLENDA. Additionally, it remains unclear how the dataset was used or Table 7 Com parison of loosely related work targeting Instruments (Instr.), Anatomy (Anat.) and Endome- triosis (Endom.). Results are reported for classification (accuracy), object detection  (mAPbb 50−95, mAPbb 50) and segmentation  (mAP50−95, mAP50)

Topic Data Method Results CLASSIFICATION accuracy Visalaxi et al. [40] Endom. [20] ResNet 90.0 OBJECT DETECTION mAPbb 50−95 mAPbb 50 Jin et al. [12] Instr. [12] Faster R-CNN − 63.1 Fox et al. [3] Instr. [7] Mask R-CNN 59.2 83.9 Kletz et al. [16] Instr. custom Mask R-CNN 64.5 83.1 Ours (cropping & rotating) Endom. [20]/ENID Mask R-CNN 34.4 62.7 SEGMENTATION mAP50−95 mAP50 Zadeh et al. [43] Anat. custom Mask R-CNN − 84.5/29.6 Zadeh et al. [43] Instr. custom Mask R-CNN − 54.5 Kletz et al. [16] Instr. custom Mask R-CNN 54.3 81.4 Fox et al. [3] Instr. [7] Mask R-CNN 40.9 71.0 Ours (cropping & rotating) Endom. [20]/ENID Mask R-CNN 32.4 64.2 6209Multimedia Tools and Applications (2022) 81:6191–6215 1 3 processed. Specifically, the authors used 6000/25682 images for their evaluations but omit how many of them show pathology as well as which lesion types are included. For object detection, we contrast our work against the task of localizing instru- ments. Albeit we achieve comparable results to Jin et al. [12], we generally discover that other approaches [3 , 16] outperform ours by approximately doubling mAP bb 50−95 . The f act that both of these studies also used Mask R-CNN confirms our assumptions that it is much easier to detect and localize instruments than endometrial implants. This becomes even more apparent when visually comparing the prominent appear - ance of surgical instruments against a much more subtle endometriosis region, e.g. by observing Fig. 8 e. For segmentation, we similarly include studies targeting instruments, yet, also another study that additionally targets anatomy. Here we discover a lower performance gap to both above-addressed studies and even reach almost comparable performance to Fox et al. [3 ]. However, since this work is using microscopy images, we argue that it is the overall least comparable study besides Visalaxi et  al.  [40]. Although Zadeh et al. [43] are missing mAP 50−95 values, they achieve good performance for uterus seg- mentation (84.5% mAP 50 ), y et, much lower results for uterus (29.6% mAP 50 ) and ins tru- ment segmentation (54.0% mAP 50 ). This is sur prising, given that instruments should be more distinguishable from any anatomical structure. Nevertheless, it underlines our main conception that segmentation strongly depends on the distinct visual appearance of a given target object. Finally, given the overall absence of directly comparable work, we believe that this study makes an important step in the direction of automatic endometriosis segmentation in the domain of laparoscopic gynecology. Specifically, reorganizing medically accepted endometriosis classes according to their visual similarity shows a large performance improvement. Therefore, we consider this study as well as the accompanying publicly released endometrial implant dataset (ENID) to be of particular importance for advancing computer-aided endometriosis analysis. 6 Conclusion We target a visually-oriented approach for endometriosis segmentation in laparoscopic surgery videos utilizing a publicly available dataset to develop a novel endometrial implants dataset including precise hand-drawn annotations. Further, we evaluate this dataset using Faster as well as Mask R-CNN combined with different ResNet back - bones. Results show a large performance increase compared to the public dataset, yet merely slight improvements when applying geometrical data augmentations. The best

are achieved using simple random image rotation combined with cropping, which surprisingly counteracts some of the major causes of mispredictions. Establish- ing a baseline for future work in this domain, we publish the novel dataset as well as selected deep models with this study. For ensuing research, we plan on further inves- tigating data augmentations and their limited applicability to laparoscopic videos as well as adding additional classes to the currently binary dataset. Finally, we believe this study lays important groundwork for the topic of computer-aided endometriosis lesion detection and we hope it inspires other researchers to contribute to this yet scarcely researched topic. 6210 Multimedia Tools and Applications (2022) 81:6191–6215 1 3 Appendix This Appendix contains all statistics and evaluations for ENID split according to percent- ages 80/10/10 (training/validation/test). Table 8 ENID, 80/10/10 – augment ation statistics, cases, images and annotations Augmentation Cases Images Annotations TRAINING None 83 128 279 Rotating 83 384 832 Blurring, desaturation 83 384 837 Cropping 83 384 809 Perspective transform 83 384 826 Reflection removal 83 128 279 Tracked10 83 1095 1654 Tracked20 83 595 937 Tracked30 83 419 673 Blurring & rotating 83 384 834 Cropping & rotating 83 384 788 All w/o tracking 64 384 751 All w/ tracked30 64 1245 1841 VALIDATION All 12 16 32 TESTING All 13 16 47 Table 9 GLEND A, GLENDA-VIS, ENID (80/10/10) – bounding box prediction precision, raw; best (bold) and worst (italic) Dataset Arch. Backb. Ep. mAPbb 50−95 mAPbb 50 mAPbb 75 GLENDA peritoneum Faster R-101 45 0.028 0.078 0.003 ovary Mask R-101 45 0.107 0.119 0.119 uterus all all − <0.000 <0.000 <0.000 die Mask R-50 50 0.036 0.119 <0.000 GLENDA-VIS mucus Faster R-101 35 0.029 0.053 0.04 vesicles Mask R-101 39 0.046 0.105 0.03 implants Faster R-50 45 0.054 0.127 0.028 abn. tissue Mask R-101 49 0.008 0.016 <0.000 ENID implants Mask R-101 39 0.288 0.5 0.332 6211Multimedia Tools and Applications (2022) 81:6191–6215 1 3 Table 10 GLEND A, GLENDA-VIS, ENID (80/10/10) – mask segmentation prediction precision, raw; best (bold) and worst (italic) Dataset Arch. Backb. Ep. mAP 50−95 mAP50 mAP75 GLENDA peritoneum Mask R-101 15 0.015 0.037 <0.000 ovary Mask R-101 45 0.048 0.119 <0.000 uterus Mask all - <0.000 <0.000 <0.000 die Mask R-101 50 0.036 0.119 <0.000 GLENDA-VIS mucus Mask R-101 35 0.023 0.062 0.021 vesicles Mask R-101 44 0.042 0.100 0.033 implants Mask R-101 20 0.048 0.137 0.033 abn. tissue Mask R-101 49 0.010 0.016 0.016 ENID implants Mask R-50 39 0.250 0.522 0.195 Table 11 ENID (80/10/10) – bounding bo x prediction precision, raw vs. augmented; best (bold) and worst (italic) Augmentation Arch. Backb. Ep. mAPbb 50−95 mAPbb 50 mAPbb 75 None Mask R-101 39 0.288 0.500 0.332 Rotating Mask R-101 44 0.287 0.440 0.395 Blurring Mask R-101 24 0.249 0.459 0.271 Cropping Mask R-50 44 0.266 0.499 0.216 Perspective transform Faster R-101 24 0.249 0.497 0.207 Desaturation Mask R-101 29 0.252 0.397 0.306 Reflection removal Mask R-101 49 0.253 0.417 0.317 Tracking10 Mask R-101 20 0.269 0.432 0.321 Tracking20 Mask R-101 10 0.240 0.459 0.206 Tracking30 Mask R-50 35 0.248 0.497 0.227 All w/o tracking Mask R-101 44 0.231 0.388 0.234 All w/ tracked30 Mask R-101 30 0.236 0.472 0.159 Blurring & rotating Mask R-50 29 0.257 0.479 0.260 Cropping & rotating Mask R-101 49 0.261 0.472 0.278 6212 Multimedia Tools and Applications (2022) 81:6191–6215 1 3

This w ork was funded by the FWF Austrian Science Fund under grant P 32010-N38. Funding Open access funding pr ovided by University of Klagenfurt. Open Access This ar ticle is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Com- mons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/.

1. Canis M, Donnez J, Guzic k D, Halme J, Rock J, Schenken R, Vernon M (1997) Revised american soci- ety for reproductive medicine classification of endometriosis: 1996. Fertility and Sterility 67(5):817– 821. https:// doi. org/ 10. 1016/ S0015- 0282(97) 81391-X 2. Du W , Rao N, Liu D, Jiang H, Luo C, Li Z, Gan T, Zeng B (2019) Review on the applications of deep learning in the analysis of gastrointestinal endoscopy images. IEEE Access 7:142053–142069 3. F ox M, Taschwer M, Schoeffmann K (2020) Pixel-based tool segmentation in cataract surgery videos with mask r-cnn. In: 2020 IEEE 33rd international symposium on computer-based medical systems (CBMS), IEEE, pp 565–568 4. F u Y, Robu MR, Koo B, Schneider C, van Laarhoven S, Stoyanov D, Davidson B, Clarkson MJ, Hu Y (2019) More unlabelled data or label more data? a study on semi-supervised laparoscopic image segmentation. In: Domain adaptation and representation transfer and medical image learning with less labels and imperfect data, Springer, pp 173–180 5. Gibson E, R obu MR, Thompson S, Edwards PE, Schneider C, Gurusamy K, Davidson B, Hawkes DJ, Barratt DC, Clarkson MJ (2017) Deep residual networks for automatic segmentation of laparoscopic videos of the liver. In: Medical imaging 2017: Image-guided procedures, robotic interventions, and modeling, international society for optics and photonics, vol 10135, pp 101351M Table 12 ENID (80/10/10) – mask segment ation prediction precision, raw vs. augmented; best (bold) and worst (italic) Augmentation Arch. Backb. Ep. mAP 50−95 mAP50 mAP75 None Mask R-50 39 0.250 0.522 0.195 Rotating Mask R-101 44 0.253 0.440 0.231 Blurring Mask R-101 24 0.242 0.434 0.253 Cropping Mask R-50 44 0.277 0.545 0.258 Perspective transform Mask R-50 49 0.252 0.472 0.183 Desaturation Mask R-50 34 0.241 0.492 0.196 Reflection removal Mask R-101 29 0.228 0.421 0.215 Tracking10 Mask R-101 25 0.226 0.429 0.195 Tracking20 Mask R-50 25 0.236 0.466 0.157 Tracking30 Mask R-50 35 0.246 0.457 0.185 All w/o tracking Mask R-101 44 0.200 0.374 0.213 All w/ tracked30 Mask R-101 30 0.217 0.472 0.099 Blurring & rotating Mask R-50 29 0.238 0.479 0.179 Cropping & rotating Mask R-101 49 0.253 0.472 0.228 6213Multimedia Tools and Applications (2022) 81:6191–6215 1 3 6. Girshic k R (2015) Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp 1440–1448 7. Gr ammatikopoulou M, Flouty E, Kadkhodamohammadi A, Quellec G, Chow A, Nehme J, Luengo I, Stoyanov D (2019) Cadis: Cataract dataset for image segmentation. arXiv: 19061 1586 8. He K, Zhang X, R en S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE con- ference on computer vision and pattern recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, IEEE Computer Society, pp 770–778. https:// doi. org/ 10. 1109/ CVPR. 2016. 90 9. He K, Gkio xari G, Dollár P, Girshick RB (2020) Mask R-CNN. IEEE Trans Pattern Anal Mach Intell 42(2):386–397. https:// doi. org/ 10. 1109/ TPAMI. 2018. 28441 75 10. Henr iques JF, Caseiro R, Martins P, Batista J (2014) High-speed tracking with kernelized correla- tion filters. IEEE Transactions on Pattern Analysis and Machine Intelligence 37(3):583–596 11. Jha D, Ali S, Johansen HD, Johansen DD, Rittsc her J, Riegler MA, Halvorsen P (2020) Real-time polyp detection, localisation and segmentation in colonoscopy using deep learning. arXiv: 20110 7631 12. Jin A , Yeung S, Jopling J, Krause J, Azagury D, Milstein A, Fei-Fei L (2018) Tool detection and operative skill assessment in surgical videos using region-based convolutional neural networks. In: 2018 IEEE winter conference on applications of computer vision (WACV), IEEE, pp 691–699 13. Jung AB, Wada K, Crall J, Tanaka S, Graving J, Reinders C, Yadav S, Banerjee J, Vecsei G, Kraft A, Rui Z, Borovec J, Vallentin C, Zhydenko S, Pfeiffer K, Cook B, Fernández I, De Rainville FM, Weng CH, Ayala-Acevedo A, Meudec R, Laporte M, et al. (2020) Imgaug. https:// github. com/ aleju/ img aug, Accessed 01 Feb 2020 14. K eckstein J, Hudelist G (2020) Classification of die including bowel endometriosis: from r-asrm to #enzian-classification. Best Pract Res Clin Obstet Gynaecol 15. K eckstein J, Ulrich U, Possover M, Schweppe K et al (2003) Enzian-klassifikation der tief infiltrier - enden endometriose. Zentralblatt für Gynäkologie 125:291 16. Kle tz S, Schoeffmann K, Benois-Pineau J, Husslein H (2019) Identifying surgical instruments in laparoscopy using deep learning instance segmentation. In: 2019 International conference on con- tent-based multimedia indexing (CBMI), IEEE, pp 1–6 17. Kr izhevsky A, Sutskever I, Hinton GE (2012) ImageNet Classification with Deep Convolutional Neural Networks. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ (eds) Advances in neural information processing systems 25, Curran Associates, Inc., pp 1097–1105. http:// papers. nips. cc/ paper/ 4824- image net- class ifica tion- with- deep- convo lutio nal- neural- netwo rks. pdf 18. Leibe tseder A, Primus MJ, Petscharnig S, Schoeffmann K (2017) Real-time image-based smoke detection in endoscopic videos. In: Wu W, Yang J, Tian Q, Zimmermann R (eds) Proceedings of the on thematic workshops of ACM multimedia 2017, October 23 - 27, 2017. ACM, Mountain View, CA, USA, pp 296–304. https:// doi. org/ 10. 1145/ 31266 86. 31266 90 19. Leibe tseder A, Petscharnig S, Primus MJ, Kletz S, Münzer B, Schoeffmann K, Keckstein J (2018) Lapgyn4: a dataset for 4 automatic content analysis problems in the domain of laparoscopic gyne- cology. In: César P, Zink M, Murray N (eds) Proceedings of the 9th ACM multimedia systems con- ference, MMSys 2018, June 12-15, 2018. ACM, Amsterdam, The Netherlands, pp 357–362. https:// doi. org/ 10. 1145/ 32049 49. 32081 27 20. Leibe tseder A, Kletz S, Schoeffmann K, Keckstein S, Keckstein J (2020) GLENDA: gynecologic laparoscopy endometriosis dataset. In: Ro YM, Cheng W, Kim J, Chu W, Cui P, Choi J, Hu M, Neve WD (eds) MultiMedia Modeling - 26th International Conference, MMM 2020, Daejeon, South Korea, January 5-8, 2020, Proceedings, Part II, Springer, Lecture Notes in Computer Sci- ence, vol 11962, pp 439–450. https:// doi. org/ 10. 1007/ 978-3- 030- 37734-2_ 36, 21. Lin T Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Micro- soft coco: Common objects in context. In: European conference on computer vision, Springer, pp 740–755 22. Litjens G, K ooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, Van Der Laak JA, Van Gin- neken B, Sánchez CI (2017) A survey on deep learning in medical image analysis. Medical Image Analysis 42:60–88 23. Minaee S, Bo ykov Y, Porikli F, Plaza A, Kehtarnavaz N, Terzopoulos D (2020) Image segmenta- tion using deep learning: A survey. arXiv: 20010 5566 24. Münzer B, Sc hoeffmann K, Böszörmenyi L (2017) Content-based processing and analysis of endoscopic images and videos: A survey. Multimed Tools Appl. https:// doi. org/ 10. 1007/ s11042- 016- 4219-z 25. Münzer B, Leibe tseder A, Kletz S, Schoeffmann K (2019) ECAT - endoscopic concept annotation tool. In: Kompatsiaris I, Huet B, Mezaris V, Gurrin C, Cheng W, Vrochidis S (eds) MultiMedia Modeling - 25th International Conference, MMM 2019, Thessaloniki, Greece, January 8-11, 2019, 6214 Multimedia Tools and Applications (2022) 81:6191–6215 1 3 Proceedings, Part II, Springer, Lecture Notes in Computer Science, vol 11296, pp 571–576. https:// doi. org/ 10. 1007/ 978-3- 030- 05716-9_ 48, 26. Nw oye CI, Mutter D, Marescaux J, Padoy N (2019) Weakly supervised convolutional lstm approach for tool tracking in laparoscopic videos. InternationAl Journal of Computer Assisted Radiology and Surgery 14(6):1059–1067 27. Oza wa T, Ishihara S, Fujishiro M, Kumagai Y, Shichijo S, Tada T (2020) Automated endoscopic detection and classification of colorectal polyps using convolutional neural networks. TherapeutiC Advances in Gastroenterology 13:1756284820910659 28. P ark SY, Sargent D (2016) Colonoscopic polyp detection using convolutional neural networks. In: Tourassi GD, Armato SG (eds) Medical Imaging 2016: Computer-Aided Diagnosis, San Diego, California, United States, 27 February - 3 March 2016, International Society for Optics and Pho- tonics, pp 978528. https:// doi. org/ 10. 1117/ 12. 22171 48 29. P etscharnig S, Schöffmann K (2018) Learning laparoscopic video shot classification for gyneco- logical surgery. Multimedia Tools and Applications 77(7):8061–8079 30. Piccialli F , Di Somma V, Giampaolo F, Cuomo S, Fortino G (2021) A survey on deep learning in medicine: Why, how and when? Information Fusion 66:111–137 31. R ai HM, Chatterjee K, Gupta A, Dubey A (2020) A novel deep cnn model for classification of brain tumor from mr images. In: 2020 IEEE 1st international conference for convergence in engi- neering (ICCE), pp 134–138.https:// doi. org/ 10. 1109/ ICCE5 0343. 2020. 92907 40 32. R ai HM, Chatterjee K, Dashkevich S (2021) Automatic and accurate abnormality detection from brain mr images using a novel hybrid unetresnext-50 deep cnn model. Biomedical Signal Process- ing and Control 66:102477 33. R en S, He K, Girshick R, Sun J (2017) Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 39(6):1137– 1149. https:// doi. org/ 10. 1109/ TPAMI. 2016. 25770 31 34. Sar aswat L, Ayansina D, Cooper K, Bhattacharya S, Horne A, Bhattacharya S (2018) Impact of endometriosis on risk of further gynaecological surgery and cancer: a national cohort study. BJOG : An International Journal of Obstetrics & Gynaecology 125(1):64–72 35. Szeg edy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9 36. T elea A (2004) An image inpainting technique based on the fast marching method. Journal of Graphics Tools 9(1):23–34 37. T winanda AP, Shehata S, Mutter D, Marescaux J, de Mathelin M, Padoy N (2017) EndoNet: A deep architecture for recognition tasks on laparoscopic videos. IEEE Transactions on Medical Imaging 36(1):86–97. https:// doi. org/ 10. 1109/ TMI. 2016. 25939 57 38. V isalaxi S, Punnoose D, Muthu TS (2021a) An analogy of endometriosis recognition using machine learning techniques. In: 2021 third international conference on intelligent communication technologies and virtual mobile networks (ICICV), IEEE, pp 739–746 39. V isalaxi S, Punnoose D, Muthu TS (2021b) Lesion extraction of endometriotic images using open computer vision. In: 2021 international conference on artificial intelligence and smart systems (ICAIS), IEEE, pp 747–751 40. V isalaxia S, Muthua TS (2021) Automated prediction of endometriosis using deep learning. Int J Nonlinear Anal Appl 12(2):2403–2416 41. Y ang C, Zhao Z, Hu S (2020) Image-based laparoscopic tool detection and tracking using convolu- tional neural networks: a review of the literature. Computer Assisted Surgery 25(1):15–28 42. Y engera G, Mutter D, Marescaux J, Padoy N (2018) Less is more: Surgical phase recognition with less annotations through self-supervised pre-training of cnn-lstm networks. arXiv: 18050 8569 43. Zadeh SM, F rancois T, Calvet L, Chauvet P, Canis M, Bartoli A, Bourdel N (2020) Surgai: deep learning for computerized laparoscopic image understanding in gynaecology. Surgical Endoscopy 34(12):5377–5383 6215Multimedia Tools and Applications (2022) 81:6191–6215