{"paper_id":"cc085ba5-524e-490c-88c2-2e96bb1bc2c6","body_text":"Vol.:(0123456789)\nhttps://doi.org/10.1007/s11042-021-11730-1\n1 3\nEndometriosis detection and localization in laparoscopic \ngynecology\nAndreas Leibetseder1  · Klaus Schoeffmann1 · Jörg Keckstein2 · Simon Keckstein3\nReceived: 17 March 2021 / Revised: 30 September 2021 / Accepted: 2 November 2021 \n© The Author(s) 2021\nAbstract\nEndometriosis is a common gynecologic condition typically treated via laparoscopic sur -\ngery. Its visual versatility makes it hard to identify for non-specialized physicians and chal-\nlenging to classify or localize via computer-aided analysis. In this work, we take a first step \nin the direction of localized endometriosis recognition in laparoscopic gynecology videos \nusing region-based deep neural networks Faster R-CNN and Mask R-CNN. We in particu -\nlar use and further develop publicly available data for transfer learning deep detection mod-\nels according to distinctive visual lesion characteristics. Subsequently, we evaluate the per-\nformance impact of different data augmentation techniques, including selected geometrical \nand visual transformations, specular reflection removal as well as region tracking across \nvideo frames. Finally, particular attention is given to creating reasonable data segmentation \nfor training, validation and testing. The best performing result surprisingly is achieved by \nrandomly applying simple cropping combined with rotation, resulting in a mean average \nsegmentation precision of 32.4% at 50-95% intersection over union overlap (64.2% for 50% \noverlap).\nKeywords Endometriosis segmentation · Lesion detection · Deep learning\n * Andreas Leibetseder \n aleibets@itec.aau.at\n Klaus Schoeffmann \n ks@itec.aau.at\n Jörg Keckstein \n joerg@keckstein.at\n Simon Keckstein \n simon.keckstein@med.uni-muenchen.de\n1 Institute of Information Technology, Klagenfurt University, Klagenfurt, Austria \n2 Medical Faculty, Ulm University, Ulm, Germany \n3 University Hospital, Ludwig-Maximilians-University Munich, Munich, Germany\nPublished online: 7 January 2022\nMultimedia Tools and Applications (2022) 81:6191–6215\n/\n\n1 3\n1 Introduction\nMinimally invasive surgery (MIS) in gynecology focuses on the endoscopic treatment of \nconditions and diseases of the female inner genital tract. It is conducted by using special \ncamera systems – endoscopes – as well as surgical instruments introduced through small \nincisions into the abdomen of a patient. In this way, the physician is able to perform treat-\nments via observing the camera feed, while greatly reducing the physical injury imposed \non patients as well as facilitating intervention monitoring, computer-aided live assistance \nand even the creation of medical video archives for post-surgical analysis. In particular, \nthe latter benefit enables physicians to revisit interventions at any time for gaining useful \ninsights or improving treatment planning and medical education.\nRevisiting intervention archives can especially be important for the treatment of patients \nwith endometriosis and infertility. These patients suffer from endometrial-like tissue in a \nvariety of abnormal locations external to the uterus. In over 60% of diagnosed cases, endo-\nmetriosis treatment requires multiple surgeries [34] due to unidentified (missed) lesions, \nwhich can be mitigated by consulting archived intervention videos. Depending on the prac-\ntice of an individual physician, it certainly is possible to record specific short clips of rel-\nevant content during surgery, however, typically surgeries are archived in full meaning that \nthe resulting videos generally contain a considerable amount of irrelevant content such as \nout-of-patient views, visually inexpressive or static scenes where treatment is interrupted \nand even common surgery phases that are of little importance for post-surgical analysis. \nSuch properties combined with the typical routine of using conventional video players and \nfile browsers for manually examining surgical video archives greatly impair revisiting past \ninterventions, while also making this procedure error-prone. To improve this situation, \ncomputer-aided automatic content analysis can be employed for creating systems that high-\nlight potentially relevant content to physicians during patient case inspections. However, \nalthough obvious irrelevant video segments such as overly blurry frames or camera testing \nscreens can reasonably be identified via video analysis [24], more sophisticated approaches \nare required for identifying very specific content such as scenes showing endometriosis \nlesions.\nThe main symptoms of endometriosis, a benign inflammatory disease, are menstrual-\nrelated pelvic pain and unexplainable infertility. It is found in a large variety of locations \nand treatment strategies depend on the symptoms, the extent and localization of the \nlesions. The best known classification systems for endometriosis are the revised Ameri-\ncan Society for Reproductive Medicine (rASRM) score [1 ] and the Enzian classification \nscheme [14, 15]. The rASRM score describes superficial lesions of the peritoneum and \novarian endometriosis in four stages, whereas the Enzian classification categorizes deep \nendometriosis. Alongside these different possible locations, endometriotic lesions also \nstrongly vary in their visual appearance, both intra- and interpersonal. This is illustrated \nby Fig.  1 showing a set of positive and negative endometriosis examples\n1 at three com-\nmon locations: ovary, uterus and peritoneum. In direct comparison and without a spe-\ncific medical background, the differences between normal and pathological tissue are \nvery difficult to discern, which evidently holds true for laymen but even inexperienced \nmedical practitioners. Consequently, with the current successful application of deep \n1 Please no te that Fig.  1b, d and f merely depict a subset of all the conditions’ visual appearances – refer to \nSection 3.1 for more examples.\n6192 Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\nlearning in many medical fields, attempting to solve this problem via computer-aided \nanalysis seems reasonable. Moreover, being able to classify and potentially locate endo-\nmetriosis can not only be helpful during interventions but also in treatment planning \nand particularly in teaching/training.\nIn this work we create, utilize and evaluate different datasets for a binary endometriosis \ndetection and localization task using state-of-the-art region-based convolutional neural net-\nworks (R-CNNs). Since as of yet to the best of our knowledge no comparable work exists, \nthis paper’s findings can be considered benchmarks for future research on the topic. We \ndefine our contributions as follows:\nDataset development, Section  3.1. By revising the labeling strategy of publicly avail-\nable endometriosis dataset GLENDA  [20] towards visual similarity, we discover a \nlarge improvement in lesion segmentation performance. Further refinement of a newly \n(a) (b)\n(c) (d)\n(e) (f)\nFig. 1  Endome triosis examples – negative (left column) and positive (right column) examples at locations \novary, uterus and peritoneum\n6193Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\ndefined specifically well performing class – endometrial implants – leads to the release \nof a novel dataset named ENdometrial Implants Dataset2 (ENID).\nData augmentation, Section  3.2. Utilizing ENID, we investigate the segmentation \nperformance impact of a variety of data augmentation techniques including rotation, \nblurring, cropping, perspective transformation, reflection removal and frame-by-frame \nobject tracking.\nPerformance evaluation, Sections  3.3&  3.4. For training deep models, we employ \nstate-of-the-art object detection and segmentation models Faster R-CNN [33] and Mask \nR-CNN [9] using ResNet [8] variants as backbone networks. For creating training, vali-\ndation and testing splits we divide the data twofold and pay specific attention to avoid-\ning similar images across splits, thus risking overfitting our models.\nThis work is mainly partitioned into three sections: related work in Section  2, methodology \ndescription in Section 3 and discussion in Section 4.\n2  Related work\nThe concept of learning features rather than defining them by hand had a tremendous \nimpact on image classification over the last years. Deep convolutional networks such as \nAlexNet [17], GoogLeNet [35] or ResNet [8] have been successfully applied to countless \ndomains, even non-image-related ones. Such networks not only are utilized on their own, \nthey also represent valuable backbones to many deep architectures for image as well as \nvideo analysis. One such architecture family heavily using CNNs as backbones for region \nof interest (ROI) prediction and labeling are named region based convolutional neural net-\nworks or R-CNNs. These R-CNN networks after sufficient training are capable of locating, \nclassifying and even segmenting objects in images through smart arrangement and utiliza-\ntion of CNN components. When regarding the evolution of R-CNNs from Fast R-CNN [6] \nover Faster R-CNN [33] up to Mask R-CNN [9], large improvements in runtime perfor -\nmance can be observed, as architecture bottlenecks were gradually resolved. Albeit said \nR-CNNs are developed for the common objects in context (COCO) challenge [21], which \naddresses the problem of real-world object detection, their successful application in many \ndomains including medical imaging encourages us to as well employ variants of Faster \nR-CNN and Mask R-CNN architectures for this study’s challenging lesion detection tasks.\nThe rising performance improvements of deep learning in medicine entails an ever-\nexpanding range of applications targeted at providing medical staff with digital assistance \nfor patient treatment. While similarly to the general multimedia domain, a large variety \nof data is analyzed ranging from electronic health records (E.H.R) over biosignals up to \nimages produced by manifold clinical imaging technologies [30]. Apart from being used \nacross many special fields, medical imaging also varies strongly depending on its purpose: \nmonochrome images taken from computer tomography (CT) scans or ultrasound are very \ndifferent compared to open surgery or endoscopic recordings. This in general renders sci-\nentific research on medical image classification more difficult to compare than in traditional \nmultimedia analysis. Surprisingly, endoscopic images as of yet are not nearly as much digi-\ntally analyzed as many other technologies such as magnetic resonance imaging (MRI) [31, \n2 http://  ftp.  itec.  aau.  at/  datas  ets/  ENID or https://  doi.  org/  10.  5281/  zenodo.  45709  69\n6194 Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\n32] or even the very similar field of microscopy [22]. Nevertheless, when regarding exist-\ning research on endoscopic video or image analysis three application areas can be defined: \npre-processing methods, real-time support at procedure time and post-procedural applica-\ntions [24]. Lesion detection in the laparoscopic field, as targeted in this study, can, depend-\ning on the use case, be placed into both of the latter categories. For instance, it can be help-\nful for tasks such as pointing out pathology during surgery as well as having post-surgical \nfunctions like automatically annotating medical video archives.\nGenerally, endoscopic surgery videos offer a great variety of research topics for apply -\ning deep learning. For instance, many studies are found on classifying or detecting content \nsuch as anatomy [19, 43], surgical tasks [29], instruments [12, 16, 26] or even the occur -\nrence of smoke [18]. Furthermore, attempts have been made to recognize surgical phases \nbased on that content [37, 42]. Finally, laparoscopic lesion segmentation using deep neural \nnetworks is a comparably scarcely researched field, which for a large part can be attributed \nto the limited availability of expert-annotated medical datasets.\nWhile lesion detection is much addressed in the more restrictive field of gastrointes-\ntinal endoscopy  [2], e.g. for polyp detection  [11, 27, 28], to best of our knowledge, no \ndirectly comparable work has been conducted regarding image or video-based lapa-\nroscopic endometriosis segmentation. When, however, targeting classification we find \nalmost all image-based works are on the different subject of endometrial cancer detection \nin MRI [38]. Nevertheless, we identify two approaches targeting endometriosis classifica-\ntion, albeit one does not contain any results besides extracting visual features  [39]. The \nother approach uses the previously published GLENDA [20] dataset, which is as well used \nand further developed in this study, for binary classifying endometriotic regions using sev-\neral CNNs [40]. They obtain classification accuracies between 80-90%. In the absence of \nany other related work to the best of our knowledge, we further attempt to give the reader \nan overview and list a small number of notable studies in the field of laparoscopic object \ndetection and segmentation together with their findings. Predominantly, studies approach-\ning object detection in laparoscopy attempt to localize instruments as is highlighted by the \nexistence of literature surveys such as the one written by Yang et  al.  [41]. In fact, sev -\neral studies achieve good results for this task: Jin et al. [12] use Faster R-CNN reporting \na bounding box mean average precision (mAP) in terms of intersection over union (IoU) \nof 63.1%, while Kletz et al. [16] report a similar mAP of 64.5% for bounding boxes and \nin addition, since they are using the segmentation-capable Mask R-CNN, a segmentation \nmAP of 54.3%. Furthermore, in addition to surgical instruments, Zadeh et al. [43] attempt \nto localize anatomical regions, also using Mask R-CNN and achieving high segmentation \nmAPs for uterus (84.5%), ovary (29.6%) and tools (54.5%, similar to Kletz et  al.  [16]). \nFinally, Gibson et al. [5] define a deep residual network for supervised liver segmentation \nyielding a median Dice similarity score of 95%. Fu et  al.  [4] even show that this result \nscore can be improved to 96.4% by following a different model training strategy, i.e. using \na semi-supervised mean teacher method.\n3  Methodology\nOur approach targeting endometriosis detection and segmentation is summarized in \nFig.  2. We start by gradually developing ENID by revising publicly released dataset \nGLENDA  [20] and creating intermediate dataset GLENDA-VIS in Section  3.1. Sub-\nsequently, we thoroughly describe all employed data augmentation techniques such as \n6195Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\nblurring, cropping and desaturation but also reflection removal as well as tracking in Sec-\ntion  3.2. Finally, after outlining our data partitioning approaches, Section  3.3 details our \nmodel training strategies and Section 3.4 lists our evaluation results.\n3.1  Datasets\nWhen considering all potential endometriosis locations and severities using both, the \nrASRM as well as the Enzian system, a complete dataset would have to include well over \n50 classes, which would be challenging to collect in large enough quantities. Therefore, the \npublicly available dataset GLENDA [20] follows the strategy of disregarding lesion size \naltogether and focusing on three common locations as well as one specific type of endo-\nmetriosis. Figure  3 depicts example images of the dataset’s classes: peritoneum, ovary, \nuterus and deep infiltrating endometriosis\n3 (DIE). As indicated with a light green color, in \nevery image at least one region is annotated using rectangular bounding boxes or polygon \nshapes. All of these annotations fully enclose the lesions, albeit mostly do not mark their \nexact boundaries. This is due to the difficulty of reasonably identifying such boundaries: \noften, pathologic areas contain multiple very small lesions that are infeasible to annotate \nindividually (c.f. Fig.  3d, second image) or the borders of affected tissue blend in with \nadjacent healthy tissue (c.f. Fig.  3a, second image). Apart from this annotation strategy \nlikely already entailing consequences for model training, an arguably much larger impact \nfactor can be identified when regarding the visual dissimilarity of annotated regions within \nclasses. Although partially showing similar patterns across classes as well as within them, \nthe lesions nevertheless can also vary strongly, which is a highly detrimental character -\nistic for convolutional deep networks attempting to learn visually distinct features across \nclasses.\nFig. 2  Me thodology overview – dataset creation, augmentations and deep model training\n3 Deep infiltr ating endometriosis is found several millimeters into the lining of the abdmonial cavity, i.e. \nthe peritoneal tissue.\n6196 Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\nAddressing these problems, we perform two steps for improvement. First, we reorgan-\nize the dataset’s classes by disregarding lesion location and grouping pathologic areas \nby their visual similarity. This is accomplished by using most of GLENDA’s images and \naltering their annotations accordingly. Examples of the resulting visual GLENDA dataset \n(GLENDA-VIS) are illustrated in Fig. 4 . GLENDA-VIS contains four classes:\nMucus white-toned mucous areas, often part of adhesions or sclerotic areas\nVesicles clear blister-like areas, often resembling water drops\nImplants dark-toned growths, often are surrounded by white sclerotic areas\nAbnormal Tissue irritated/bleeding tissue, e.g. increased/deformed blood vessels or \nbleeding endometrial cells\nGLENDA-VIS is a preliminary dataset with annotations created exclusively from rec-\ntangles that can overlap to form simple polygonal areas. It is intentionally created in \nthis rudimentary manner, as for the purposes of this work it is merely used to determine \nthe most promising visual endometriosis characteristic for the second improvement step \nperformed: creating a larger binary dataset using the best performing GLENDA-VIS \nclass.\nAfter evaluating the instance segmentation performance of both GLENDA and \nGLENDA-VIS (c.f. Tables  9 and  10), we identify Implants as the most promising can-\ndidate for our final ENdometrial Implants Dataset (ENID), which is shown exemplary \nin Fig.  5. We base most of our evaluations in this work on ENID, in which we finally \n(a)\n(b)\n(c)\n(d)\nFig. 3  GLEND A dataset class overview\n6197Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\naddress the above issue of annotation imprecision – lesions are enclosed by carefully \nfreehand-drawn polygons created using the Endoscopic Concept Annotation Tool [25] \nand according to following criteria:\ncolor Implants generally appear as dark, sometimes dotted areas with varying color \ntones. Annotators predominantly focus on clearly visible implants.\nsize Annotated implants are required to be of a certain, well visible size – very small \nlesions are very likely not have a significant impact.\nboundary Annotations are created precisely, carefully identifying lesion boundaries. \nHere annotators focus on the transition from darker to lighter tissue.\n(a)\n(b)\n(c)\n(d)\nFig. 4  GLEND A-VIS dataset class overview\nFig. 5  ENID dat aset examples\n6198 Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\nSimilar to GLENDA/GLENDA-VIS, ENID is collected from a selection of frames \nbelonging to a laparoscopic video archive of over 500 individual surgery recordings. \nTable  1 presents a comparison of statistics for all three datasets: number of patient cases, \nimages and annotations. 4 As indicated by the cases column, all datasets contain frames \ntaken from close to 100 surgeries, which on one hand ensures data variety for increased \ngeneralizability and on the other hand lowers the risk of overfitting due to potential simi-\nlarities in training and validation data. This specifically is deemed important for creating \nthe single-class ENID dataset, which is collected from altogether 108 surgeries – more than \nthe number of cases for for the combination of all classes in GLENDA or GLENDA-VIS.\n3.2  Augmentation\nApart from comparing the instance segmentation performance of the raw datasets, we also \ninvestigate the performance impact of applying several augmentations to ENID. We alto-\ngether explore seven augmentation techniques, five of which are commonly used in the \nfield of image retrieval. All augmentations are exclusively applied to the training portion of \nthe dataset only, while the validation and test sets are left unchanged.\nFigure  6 shows six applied augmentations, where Fig.  6a-e are applied in the following \nmanner:\n–\n F\nor every image i of training split St , one or multiple augment ations are applied n times \nincreasing St by a factor of n + 1 , i.e. Sa\nt = St ∪{ ia\n1, ..., ia\nn} . In all our e xperiments we use \nn = 2 effectively tripling the training datasets.\n–\n A\nugmentations are applied in random intensities on a set interval of potential values \ndiffering for every augmentation type.\nTable 1  Dat aset Statistics Comparison – number of patient cases, images and annotations\nDataset Cases Images Annotations\nGLENDA peritoneum 84 257 489\novary 27 53 54\nuterus 11 17 25\ndie 29 55 59\ntotal 102 373 628\nGLENDA-VIS mucus 77 246 442\nvesicles 59 153 250\nimplants 86 281 652\nabnormal tissue 41 66 85\ntotal 96 342 1485\nENID implants 108 160 358\n4 Please note that individual frames are allowed to contain annotations for multiple classes, hence, the \ncases/images totals for GLENDA/GLENDA-VIS in fact are lower than they would be when simply sum-\nming up all class statistics.\n6199Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\nIn the following we describe corresponding image augmentations 5 and their intensity \nscales:\nrotating rotate images on a scale between [−45, 45] degrees\nblurring apply motion blur using random angles and directions with kernel sizes \nbetween [3, 11] pixels\ncropping crop each image side by a percentage between [0.0,  0.25], input size is \nretained\nperspective transform transform a frame’s perspective by placing 4 points relative to \nits corners in a distance (scale) between [0.01, 0.15], input size is retained\ndesaturation reduce image saturation by a percentage between [0.15, 0.40]\nThe remaining augmentation shown in Fig.  6f represents a more special case, hence, it is \napplied differently. The introduction of bright light sources for illumination during laparo-\nscopic surgery frequently creates specular reflections, especially on wet surfaces or bulging \ntissue as is formed in the case of shallow endometrial implants. This can have detrimental \neffects on the learning performance (cf. Fig.  8d), since it may lead to associating reflec-\ntions with lesions. Therefore, we explore the effects of automatically removing very bright \nreflections, i.e. regions above a certain brightness threshold, and correct the images using \ninpainting as proposed by Telea et al. [36]. As we want to remove reflections altogether \nfrom \nSt , when appl ying this augmentation we simply replace these training images with the \nprocessed ones, i.e. altering instead of enlarging the corresponding data. Although remov -\ning potentially important information using such an approach, we, nevertheless, expect it to \nmitigate the negative performance impact of specular reflections.\n(a)\n(c)\n(e) (f)\n(d)\n(b)\nFig. 6  Summar y of applied augmentations – original (left), augmented (right)\n5 F or more detailed information about these standard augmentations please refer to the utilized library \nimgaug [13].\n6200 Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\nFor our final augmentation, we perform object tracking using the annotated frames \ntogether with their source videos. We accomplish this task by using kernelized correla-\ntion filters [10], a simple and fast bounding box tracking methodology. We process each \nannotation on a frame-to-frame basis, i.e. every successfully tracked region is used for \ntracking the next region. Albeit simple bounding boxes can not capture the exact ground \ntruth region or its deformation, we nevertheless achieve good results by simply relocat-\ning and resizing the original annotations according to the tracked bounding boxes – Fig.  7 \nshows some of the results. Subsequently, we manually review all of the tracked frames and \nremove evidently incorrect results. Following this approach, we are able to augment the \nfull dataset approximately by a factor of 81 (160 vs. 12981 frames). The final augmenta-\ntion factor, however, depends on how the dataset is split into training, validation and test \nset, since similar to all other augmentations this one is as well only applied to the training \nportion. Also, in order to avoid feeding the network with almost identical images, as a last \nstep we uniformly sample tracked frames with three different frame intervals: v=10, v=20 \nand v=30. For each interval, starting with the ground truth frame, we pick every v-th frame \nfrom a reviewed tracking sequence and repeating this process for the entire training set, \nyielding three augmented datasets: tracked\n10 , tracked20 and tracked30.\nFinally, in order to compromise between swift model training and result expressiveness, \nwe use two dataset splits throughout all our experiments (training/validation/test): 60/20/20 \nand 80/10/10. For splitting, special attention is paid to avoid similar frames across splits \nand risk overfitting: patient cases, i.e. full surgery recordings, can only be part of a single \nsplit portion. For brevity, we only include result listings for the 60/20/20 split throughout \nthe paper, while listing everything related to 80/10/10 in the Appendix, as to provide the \ninterested reader with further insights. Accordingly, Table  2 lists augmentation statistics \nrelated to the 60/20/20 split and Table  8 represents its Appendix-counterpart for 80/10/10. \nWhen regarding the number of images in the augmented training splits we can conclude \nthat, while tripling our data with basic augmentations, we achieve an augmentation fac-\ntor of over 7 with tracking. Additionally, convincing findings in our evaluations (cf. Sec-\ntion  3.4) also encourage us to combine augmentations, which are listed as the last four \nentries in the TRAINING section of Table  2. First, we investigate the effects of combin-\ning all simple augmentations for non-tracked (all w/o tracking) as well as tracked (all w/ \ntracked\n30 ) dat a and second, discuss other promising augmentation combinations per split \nFig. 7  T racked ENID examples, one tracked annotation per row (GT marks original ground truth annota-\ntion)\n6201Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\n(blurring/cropping/rotating). Analogously to the above descriptions, we keep using n = 2 \nfor these additional approaches and apply all involved augmentations in a sequence to \nboth images. Lastly, it is worth mentioning that, albeit the number of images stays consist-\nent across augmentations, the number of annotations varies because they potentially are \nremoved by geometrical image transformations. Additionally, if they are located near an \nimage border a transformation can as well possibly dissect them. As a consequence during \nthe augmentation process, we verify that every augmented image at least contains one valid \nannotation.\n3.3  Model training\nHaving developed the datasets and prepared ENID in the above described way, we at this \npoint are ready for model training. As mentioned in the introductory section, we use both \nFaster R-CNN (Faster) as well as Mask R-CNN (Mask) as architectures, each trained \nwith ResNet-50 (R-50) and ResNet-101 (R-101) as backbones. We conduct transfer learn-\ning, i.e. the models are initialized with pre-trained weights from the COCO dataset [21], \nwhile adjusting the last layer for our detection and segmentation purposes: since we want \nto compare the performance of single lesion classes, we exclusively train our models to \ngive binary output. To achieve this, we correspondingly split up all multi-class datasets \nGLENDA and GLENDA-VIS into binary datasets of four classes each. The utilized loss \nfunctions incorporate class (log loss), bounding box (smooth \nL1 loss) and for Mask R-CNN \npixel-based segmentation (binary cross entropy loss) predictions as described in  [6, 9]. \nAdditionally, for optimization we choose stochastic gradient descent (SGD) and conduct \nTable 2  ENID (60/20/20) – augment ation statistics, cases, images and annotations\nAugmentation Cases Images Annotations\nTRAINING\nNone 64 96 240\nRotating 64 288 713\nBlurring, desaturation 64 288 720\nCropping 64 288 672\nPerspective transform 64 288 703\nReflection removal 64 96 240\nTracked10 64 653 1110\nTracked20 64 361 649\nTracked30 64 265 486\nAll w/o tracking 64 288 637\nAll w/ tracked30 64 786 1292\nBlurring & rotating 64 288 713\nCropping & rotating 64 288 658\nVALIDATION\nAll 24 32 61\nTESTING\nAll 20 32 57\n6202 Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\ninitial experiments for finding the optimal learning rate /u1D702∈{ 0.01, 0.001, 0.0001} with the \nresult of choosing /u1D702= 0.001 due to its overall superior performance. We also conduct pre-\nliminary experiments using Adam, yet, find that this optimization strategy tends to perform \nbetter for classifying rather than segmenting endometriosis. Training is conducted for a \nfixed number of 50 epochs (approx. 25.5K iterations) and we create a model checkpoint \napproximately every 5 epochs.\nThe utilized networks are implemented in Python using Detectron2\n6 – an object detec-\ntion framework powered by the PyTorch 7 deep learning framework. Regarding hardware, \na workstation running Linux Ubuntu 18.x with the following specifications was used: Intel \nCore i7-5820K CPU @ 3.30GHz x 6, 32 GiB DDR3 @ 1333 MHz, Nvidia GeForce GTX \n1080. Depending on the amount of input data, training a single network for 50 epochs \nrequires approximately 2-4h to complete.\n3.4  Evaluation\nThis section contains all the results achieved for evaluating our trained models (Sec-\ntion  3.4.1) as well as qualitative prediction insights (Section  3.4.2). Similarly to Sec-\ntion  3.2, we only report results for the 60/20/20 split, while listing the ones for 80/10/10 in \nthe Appendix. Additionally, in order to further condense table sizes, for every potentially \naugmented dataset, we merely report the best performing network combination (R-CNN + \nbackbone) evaluated at the epoch checkpoint that achieves the highest mean average preci-\nsion (mAP) value.\n3.4.1  Quantitative results\nUsing the COCO-detection [21] metrics, we evaluate a number of mAP values for different \nintersection over union (IoU) thresholds – the higher the threshold, the more a predicted \narea is required to overlap with a given ground truth to be considered correct. Since we \nare using Faster R-CNN and Mask R-CNN architectures, we report bounding box\n8 mAPs \n(mAPbb ) as w ell as pixel mask segmentation mAPs (mAP). IoU thresholds are identified \nby subscript numbers: mAP 50−95 indicates the mAP calculated over 10 thresholds starting \nfrom 50% percent up to 95% IoU overlap using intervals of 5%, while mAP 50 and mAP 75 \ndescribe the mAP at single specific thresholds 50% as well as 75%.\nFirst, we evaluate the performance of the raw (unaugmented) datasets GLENDA, \nGLENDA-VIS and ENID listing mAP comparisons for bounding box prediction in \nTable  3. Throughout GLENDA as well as GLENDA-VIS, we discover very low precisions \nof at most close or equal to 3% (mAP\nbb\n50−95=0.030). For many classes (ovary, uterus, die, \nabnormal tissue), result precisions for any trained models even are almost 0% (mAPbb\n50−95 < \n0.000). This is as well reflected in the 80/10/10 split (cf. Table  9) with the exception of \novary scoring around 10% (mAPbb\n50−95=0.107). When specifically observing GLENDA-VIS, \nwe identify vesicles (mAP bb\n50−95=0.030) and implants (mAP bb\n50−95=0.029) as the best per -\nforming classes and their similar performance is confirmed by regarding the evaluations for \n6 https://  github.  com/  faceb  ookre  search/  detec  tron2\n7 https://  pytor c h.  org/\n8 Please no te that for Mask vs. Faster R-CNN bounding boxes we essentially are comparing RoI Pooling \nvs. RoI Align, the latter of which is not suffering from quantization loss, hence, is expected to yield more \nprecise results.\n6203Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\nthe 80/10/10 split (cf. Table  9). ENID being the more refined endometrial implants dataset \noutperforms both other datasets significantly with a precision over 30% (mAPbb\n50−95=0.308), \nwhich is even increased to over 50% (mAPbb\n50 =0.561) when merely considering a 50% IoU \noverlap threshold. When regarding the mask segmentation results listed in Table  4, we \nobserve similar results: GLENDA as well as GLENDA-VIS classes yield equally low or \neven lower segmentation scores, while ENID on the other hand scores significantly higher \nwith over 30% for all thresholds (mAP\n50−95=0.309) or above 50% for a 50% IoU thresh-\nold (mAP 50−95=0.581). Albeit performing slightly worse (mAP bb\n50−95=0.288, mAP bb\n50=0.5, \nmAP50−95=0.250, mAP50=0.522), the same trend can be observed for split 80/10/10 (cf. \nTables 9 and 10).\nHaving observed superior results for ENID in all mAP categories, we finally compare \nthe raw dataset’s performance against the effects of applying different augmentations. \nTable  5 lists all precision values for bounding box predictions and only few of them seem \nto to have a slight impact on the results. While cropping, perspective transform and desatu-\nration yield very similar results to the raw dataset (0-0.5% improvement in mAP\nbb\n50−95 ), we \ndisco\nver that the best performing augmentations for bounding box prediction are blurring \nand rotating with 0.7-2.5% improvements in mAPbb\n50−95 . Albeit t hese improvements can not \nbe regarded as significant, in particular since they can not be observed for the 80/10/10 split \nTable 3  GLEND A, GLENDA-VIS, ENID (60/20/20) – bounding box prediction precision, raw; best (bold) \nand worst (italic)\nDataset Arch. Backb. Ep. mAPbb\n50−95 mAPbb\n50 mAPbb\n75\nGLENDA peritoneum Mask R-101 24 0.030 0.068 0.010\novary all all − <0.000 <0.000 <0.000\nuterus all all − <0.000 <0.000 <0.000\ndie all all − <0.000 <0.000 <0.000\nGLENDA-VIS mucus Mask R-101 35 0.015 0.026 0.020\nvesicles Mask R-101 30 0.030 0.068 0.004\nimplants Mask R-101 35 0.029 0.069 0.020\nabn. tissue all all − <0.000 <0.000 <0.000\nENID implants Mask R-50 34 0.308 0.561 0.276\nTable\n \n4\n  GLEND\nA, GLENDA-VIS, ENID (60/20/20) – mask segmentation prediction precision, raw; best \n(bold) and worst (italic)\nDataset Arch. Backb. Ep. mAP 50−95 mAP50 mAP75\nGLENDA peritoneum Mask R-101 24 0.025 0.074 0.004\novary all all − <0.000 <0.000 <0.000\nuterus all all − <0.000 <0.000 <0.000\ndie all all − <0.000 <0.000 <0.000\nGLENDA-VIS mucus Mask R-101 35 0.015 0.026 0.020\nvesicles Mask R-101 30 0.029 0.068 0.027\nimplants Mask R-101 35 0.028 0.069 0.019\nabn. tissue all all − <0.000 <0.000 <0.000\nENID implants Mask R-50 39 0.309 0.581 0.325\n6204 Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\n– here all augmentations even perform worse than the raw dataset (cf. Table  11). Looking \nat the mask segmentation results, however, we discover a slight impact of rotating for both, \nsplit 60/20/20 (0.320 vs. 0.309, cf. Table  6) as well as split 80/10/10 (0.253 vs. 0.250, cf. \nTable  12). The latter split even indicates a performance boost of 2.3-3.85% for cropping \nin all listed mAP values compared to the raw dataset performance. Further, for the com-\nbination of all augmentations applied to the raw and tracking\n30 data, we observe similar \nperformance to most individually augmented data – for 80/10/10 we even discover many \nof the worst-performing mAPs for both bounding box and mask predictions. Lastly, it is \nworth noting that although the tracking approach consistently decreases the performance, \nTable 5  ENID (60/20/20) – bounding bo x prediction precision, raw vs. augmented; best (bold) and worst (italic)\nAugmentation Arch. Backb. Ep. mAPbb\n50−95 mAPbb\n50 mAPbb\n75\nNone Mask R-50 34 0.308 0.561 0.276\nRotating Mask R-101 24 0.333 0.633 0.322\nBlurring Mask R-50 44 0.315 0.582 0.309\nCropping Mask R-50 44 0.308 0.544 0.269\nPerspective transform Mask R-101 44 0.311 0.591 0.292\nDesaturation Mask R-50 49 0.313 0.566 0.303\nReflection removal Mask R-101 44 0.281 0.544 0.274\nTracking10 Faster R-101 15 0.234 0.460 0.183\nTracking20 Mask R-50 40 0.268 0.516 0.183\nTracking30 Mask R-50 50 0.270 0.515 0.232\nAll w/o tracking Mask R-101 19 0.299 0.590 0.231\nall w/ tracked30 Faster R-101 19 0.276 0.526 0.216\nBlurring & rotating Mask R-101 24 0.307 0.638 0.250\nCropping & rotating Mask R-101 24 0.344 0.627 0.278\nTable\n \n6\n  ENID (60/20/20) – mask segment\nation prediction precision, raw vs. augmented; best (bold) and \nworst (italic)\nAugmentation Arch. Backb. Ep. mAP 50−95 mAP50 mAP75\nNone Mask R-50 39 0.309 0.581 0.325\nRotating Mask R-50 24 0.320 0.640 0.255\nBlurring Mask R-50 44 0.290 0.582 0.289\nCropping Mask R-101 24 0.298 0.580 0.250\nPerspective transform Mask R-101 14 0.283 0.573 0.267\nDesaturation Mask R-50 19 0.286 0.575 0.229\nReflection removal Mask R-101 24 0.286 0.547 0.244\nTracking10 Mask R-101 30 0.197 0.489 0.102\nTracking20 Mask R-101 10 0.228 0.465 0.175\nTracking30 Mask R-101 15 0.217 0.473 0.119\nAll w/o tracking Mask R-101 19 0.277 0.572 0.229\nAll w/ tracked30 Faster R-101 14 0.250 0.487 0.214\nBlurring & rotating Mask R-101 19 0.294 0.608 0.265\nCropping & rotating Mask R-101 29 0.324 0.642 0.255\n6205Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\nthe results indicate that more samples generally perform worse, i.e. tracked 20 and tracked30 \nmostly show higher mAP values than tracked 10 with the exception of mAP bb\n50−95 for split \n80/10/10 (cf. Tables 5 and 11)\nBased on above observations, we identify three augmentations that generally seem to \nhave an impact on the resulting precision values, whether for bounding box prediction or \nmask segmentation on either of the splits: rotating, cropping and blurring. Therefore, we \nchoose to combine these particular augmentations in order to determine, if the outcomes \ncan further be improved. After training additional models on ENID that is augmented by \njointly applying above techniques, we subsequently decide to add the following two aug-\nmentation combinations to our evaluations: blurring & rotating as well as cropping & rotat-\ning. Unfortunately, for both splits, the best models for augmenting ENID through blur -\nring- and rotating are not improving prediction precision and, in fact, even show slightly \ndetrimental effects on mask segmentation results for split 60/20/20 (0.294 vs. 0.309 \nmAP\n50−95 , cf. T able  6) and split 80/10/10 (0.238 vs. 0.250 mAP 50−95 , cf. T able  12). On \nthe other hand, models for cropping- & rotating-augmented ENID mostly outperform raw \nENID models for bounding box prediction and segmentation in both splits, albeit merely \nby up to 3.6% precision. Nevertheless, this ultimately seems to be the most robust perfor -\nmance improvement indicator. Thus, we finally summarize the overall best augmentation \ntechniques for any of the splits to to be cropping, rotating and their combination. Subse-\nquently, we put a stronger focus on the segmentation results of the 60/20/20 split and com-\npare the best models trained on raw ENID to those trained on rotating- as well as crop-\nping- & rotating-augmented ENID: although the performance improvement of applying \naugmentation in this case merely amounts to 1.1-1.5% mAP\n50−95 , t he models’ differences, \nnevertheless, become apparent when qualitatively inspecting their prediction outputs.\n3.4.2  Qualitative results\nA qualitative inspection of results taken from the 60/20/20 testing set reveals advantages as \nwell as disadvantages in our evaluation’s best performing models, which are chosen accord-\ning to segmentation precision (mAP\n50−95 ). Figure  8  compares selected prediction results \nof the best model for raw ENID (M M-R-50\nraw  ) with the best models trained on ENID with \naugmentations, i.e. rotation (M M-R-101\nrot.  ) and cr opping combined with rotation (M M-R-101\ncrop+rot. ). \nEv\nery row in the figure compares a test split ground truth annotation with predictions of \nall chosen models in the same order as mentioned above and using a confidence threshold \nof 50%. In contrast to the green outlined ground truth annotations, the model predictions \nare randomly colored and include pixel segmentation masks as well as bounding boxes for \nincreased visibility.\nStarting with Fig.  8a, M\n M-R-101\nrot.  fails to predict some lesion areas, yet, according to the \nground truth, correctly is not mispredicting the small dark spot on the right side of the \nimage, in contrast to the other models. This region, however, is not as easily distinguish-\nable from a pathologic area: in fact, ground truth annotations are only made for obvious \nlesions, therefore, the prediction in question could be considered suspicious and in need of \nfurther examination by an expert. Thus, arguably in this case, an expert might even prefer \nsuch suspicious areas to be included in the results, despite their potential of being false \nalarms. Subsequently, in all of the remaining figures, we examine the most common mis-\npredictions discovered throughout all trained models. Figure  8b demonstrates how M\n M-R-50\nraw  \nand M\n M-R-101\nrot.  confuse a darker background area for a lesion, while M  M-R-101\ncrop+rot. manages to \navoid this mistake. In fact, this is discovered most frequently with differing severity, i.e. \n6206 Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\nmulti-shaded background areas potentially are mispredicted partially up to fully. Moreo-\nver, a further cause of mispredictions are prominent color transitions, as can be discovered \nin Fig.  8c, where M\n M-R-101\nrot.  confuses a blood vessel for a lesion. Blood or blood vessels \nare contrasting in color with their surrounding tissue, which makes them easy targets to \nmispredict, especially when embedded in mucus-like surrounding tissue as is the case in \nthe figure. Another less obvious but prevalent misprediction type are reflections as shown \nin Fig.  8d, which demonstrates M\n M-R-50\nraw  falsely predicting a large specular reflection to be \na lesion, while the other models behave correctly in this situation. This presumably hap-\npens because reflections are part of many implant annotations (cf. Fig.  5), hence, they are \nlearned to be part of the lesion regions. Although mitigating this particular problem, reflec-\ntion removal evidently increases the amount of other mispredictions, thus, altogether hav -\ning a detrimental rather than beneficial effect (cf. Table  6). Surprisingly, at least in the \ndepicted case, cropping seems to as well have a positive effect on this problem. Lastly. \n(a)\n(b)\n(c)\n(d)\n(e)\nFig. 8  Selected q ualitative result comparison of best performing models using a 50% confidence threshold \n– ground truth (left), Mask R-CNN with ResNet-50 on raw ENID (MM-R-50\nraw  , middle lef t), Mask R-CNN with \nResNet-101 on rotating-augmented ENID (MM-R-101\nrot.  , middle r ight), Mask R-CNN with ResNet-101 on crop-\nping- & rotating-augmented ENID (MM-R-101\ncrop+rot. , right)\n6207Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\ninstruments also exhibit attributes that can lead to confusion: many possess black shafts \nthat under certain lighting conditions are easily confused for implants and their often \nsilvery top parts frequently reflect light, which results in reflection mispredictions. For \ninstance, Fig.  8e shows M\n M-R-50\nraw  struggling with instrument shafts, while the other models \nappear to less susceptible to this kind of misprediction.\nOverall, judging by the examples shown in Fig.  8, it seems apparent to deem M\n M-R-101\nrot.  \nand M\n M-R-101\ncrop+rot. as more robust than M  M-R-50\nraw  . Ho wever, it is worth again referring to their \nmere slight overall performance improvement of at most 1.5% mAP 50−95 . Ther efore, they \nultimately many times are as well suffering from above mispredictions and lesion localiza-\ntion failures, yet, on different test images. Nevertheless, every above model seems to have \nits merits in certain situations, which may allow for increasing performance by applying \nthem in sequence and taking decisions using majority voting.\n4  Discussion\nReviewing the entirety of results, we observe several expected, yet, also unexpected out-\ncomes. First, looking at the prediction performance of GLENDA, GLENDA-VIS and \nENID we can confirm our conjecture that categorizing endometriosis by visual appear -\nance rather than location is more suitable for state-of-the-art R-CNN networks. This \nparticularly seems reasonable when focusing on a lesion manifestation like endometrial \nimplants, which potentially can occur at any bodily region susceptible to endometrio-\nsis. Moreover, we ultimately base choosing to further investigate this particular class \non observing its overall best performance as part of GLENDA-VIS, while taking into \naccount two different dataset splits. This decision can arguably even stronger be justified \nby observing the somewhat exceptional performance of models trained on GLENDA’s \novary class, which appears to be influenced by its inclusion of many implants-like anno-\ntations (cf. Figs.  3b or 7, first row). Nevertheless, ovary-trained models in direct compar -\nison to ENID-trained models show significantly lower performance, which is attributable \nto the difference in annotation strategies used when creating the classes: apart from the \nnecessity of consulting medical experts, precisely enclosing lesion regions as well as set-\nting clear annotation guidelines are key components for largely increasing segmentation \nperformance.\nOn the downside, we discover a strong decrease in precision when requiring pre-\ndictions to have a higher lesion area overlap. While this is expected considering the \ndifficulty of the problem, we on average identify a considerable decrease of close to \n40% between results calculated with a 50% and 75% overlap requirement, which holds \ntrue for models trained on raw as well as augmented ENID. Surprisingly, most pro-\nposed augmentations do not yield any significant performance improvements, which, \nalthough standing in contrast to results achieved in traditional image segmentation \ntasks [23], we find to be in line with the discoveries in other medical fields, such as \nthe findings of Fox et al., who apply Mask R-CNN for tool segmentation in cataract \nsurgery using microscopy videos [3 ].\nNevertheless, when striving to identify the best performing augmentations by regard-\ning quantitative as well as qualitative evaluations we identify simple rotation, the \nresults of which can even further be improved in combination with cropping. Interest-\ningly, this combination for various examples counteracts some of the major causes of \nmispredictions: dark background areas, blood vessels, specular reflections and surgical \n6208 Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\ninstruments. Nevertheless, the best performing augmented models still merely show \nslight performance improvements over the non-augmented models. Interestingly, some \nmispredictions could be regarded as questionable and in need of further expert inspec-\ntion, since it is possible that pathological regions are omitted during ground truth anno-\ntation, either because of not conforming to the annotation policy or being too incon-\nclusive. Since larger numbers of such unannotated lesions could influence training, \nthis particularly could explain the comparably poor performance of models trained on \ntracked data: single annotated frames do not necessarily contain all lesion regions of a \nwhole scene, however, during tracking they can easily be brought into view by camera \nmovements or surgery actions like moving an organ to reveal occluded implants. Finally, \nit is also possible that, despite uniform sampling, tracked images still are too similar, \ncausing a disproportionate amount of weight to be put on easily trackable lesion scenes \nwhile decreasing the impact of others.\n5  Contributions\nAs outlined in Section  2 at the time of writing, we are unable to retrieve any directly com-\nparable work on endometriosis segmentation. Nevertheless, we list some loosely related \nwork in Table  7, where we set our work in contrast to classification, object detection and \nsegmentation tasks on endoscopy [12, 16, 40, 43] as well as microscopy [3] datasets.\nFor classification, we list Visalaxi et al. [40] who achieve a very high binary endome-\ntriosis classification accuracy of 90.0% using ResNet on the GLENDA [20] dataset. When \ncompared to our work, besides classification being a different task, with defining endome-\ntrial implants we additionally are targeting a more confined type of lesion than the various \nones included in GLENDA. Additionally, it remains unclear how the dataset was used or \nTable 7  Com parison of loosely related work targeting Instruments (Instr.), Anatomy (Anat.) and Endome-\ntriosis (Endom.). Results are reported for classification (accuracy), object detection  (mAPbb\n50−95, mAPbb\n50) and \nsegmentation  (mAP50−95, mAP50) \nReference Topic Data Method Results\nCLASSIFICATION accuracy\nVisalaxi et al. [40] Endom. [20] ResNet 90.0\nOBJECT DETECTION mAPbb\n50−95 mAPbb\n50\nJin et al. [12] Instr. [12] Faster R-CNN − 63.1\nFox et al. [3] Instr. [7] Mask R-CNN 59.2 83.9\nKletz et al. [16] Instr. custom Mask R-CNN 64.5 83.1\nOurs (cropping & rotating) Endom. [20]/ENID Mask R-CNN 34.4 62.7\nSEGMENTATION mAP50−95 mAP50\nZadeh et al. [43] Anat. custom Mask R-CNN − 84.5/29.6\nZadeh et al. [43] Instr. custom Mask R-CNN − 54.5\nKletz et al. [16] Instr. custom Mask R-CNN 54.3 81.4\nFox et al. [3] Instr. [7] Mask R-CNN 40.9 71.0\nOurs (cropping & rotating) Endom. [20]/ENID Mask R-CNN 32.4 64.2\n6209Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\nprocessed. Specifically, the authors used 6000/25682 images for their evaluations but omit \nhow many of them show pathology as well as which lesion types are included.\nFor object detection, we contrast our work against the task of localizing instru-\nments. Albeit we achieve comparable results to Jin et al. [12], we generally discover \nthat other approaches [3 , 16] outperform ours by approximately doubling mAP\nbb\n50−95 . \nThe f\nact that both of these studies also used Mask R-CNN confirms our assumptions \nthat it is much easier to detect and localize instruments than endometrial implants. \nThis becomes even more apparent when visually comparing the prominent appear -\nance of surgical instruments against a much more subtle endometriosis region, e.g. by \nobserving Fig. 8 e.\nFor segmentation, we similarly include studies targeting instruments, yet, also \nanother study that additionally targets anatomy. Here we discover a lower performance \ngap to both above-addressed studies and even reach almost comparable performance \nto Fox et al. [3 ]. However, since this work is using microscopy images, we argue that \nit is the overall least comparable study besides Visalaxi et  al.  [40]. Although Zadeh \net al. [43] are missing mAP\n50−95 values, they achieve good performance for uterus seg-\nmentation (84.5% mAP 50 ), y et, much lower results for uterus (29.6% mAP 50 ) and ins tru-\nment segmentation (54.0% mAP 50 ). This is sur prising, given that instruments should \nbe more distinguishable from any anatomical structure. Nevertheless, it underlines our \nmain conception that segmentation strongly depends on the distinct visual appearance \nof a given target object.\nFinally, given the overall absence of directly comparable work, we believe that this \nstudy makes an important step in the direction of automatic endometriosis segmentation \nin the domain of laparoscopic gynecology. Specifically, reorganizing medically accepted \nendometriosis classes according to their visual similarity shows a large performance \nimprovement. Therefore, we consider this study as well as the accompanying publicly \nreleased endometrial implant dataset (ENID) to be of particular importance for advancing \ncomputer-aided endometriosis analysis.\n6  Conclusion\nWe target a visually-oriented approach for endometriosis segmentation in laparoscopic \nsurgery videos utilizing a publicly available dataset to develop a novel endometrial \nimplants dataset including precise hand-drawn annotations. Further, we evaluate this \ndataset using Faster as well as Mask R-CNN combined with different ResNet back -\nbones. Results show a large performance increase compared to the public dataset, yet \nmerely slight improvements when applying geometrical data augmentations. The best \nresults are achieved using simple random image rotation combined with cropping, \nwhich surprisingly counteracts some of the major causes of mispredictions. Establish-\ning a baseline for future work in this domain, we publish the novel dataset as well as \nselected deep models with this study. For ensuing research, we plan on further inves-\ntigating data augmentations and their limited applicability to laparoscopic videos as \nwell as adding additional classes to the currently binary dataset. Finally, we believe this \nstudy lays important groundwork for the topic of computer-aided endometriosis lesion \ndetection and we hope it inspires other researchers to contribute to this yet scarcely \nresearched topic.\n6210 Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\nAppendix\nThis Appendix contains all statistics and evaluations for ENID split according to percent-\nages 80/10/10 (training/validation/test).\nTable 8  ENID, 80/10/10 – augment ation statistics, cases, images and annotations\nAugmentation Cases Images Annotations\nTRAINING\nNone 83 128 279\nRotating 83 384 832\nBlurring, desaturation 83 384 837\nCropping 83 384 809\nPerspective transform 83 384 826\nReflection removal 83 128 279\nTracked10 83 1095 1654\nTracked20 83 595 937\nTracked30 83 419 673\nBlurring & rotating 83 384 834\nCropping & rotating 83 384 788\nAll w/o tracking 64 384 751\nAll w/ tracked30 64 1245 1841\nVALIDATION\nAll 12 16 32\nTESTING\nAll 13 16 47\nTable\n \n9\n  GLEND\nA, GLENDA-VIS, ENID (80/10/10) – bounding box prediction precision, raw; best (bold) \nand worst (italic)\nDataset Arch. Backb. Ep. mAPbb\n50−95 mAPbb\n50 mAPbb\n75\nGLENDA peritoneum Faster R-101 45 0.028 0.078 0.003\novary Mask R-101 45 0.107 0.119 0.119\nuterus all all − <0.000 <0.000 <0.000\ndie Mask R-50 50 0.036 0.119 <0.000\nGLENDA-VIS mucus Faster R-101 35 0.029 0.053 0.04\nvesicles Mask R-101 39 0.046 0.105 0.03\nimplants Faster R-50 45 0.054 0.127 0.028\nabn. tissue Mask R-101 49 0.008 0.016 <0.000\nENID implants Mask R-101 39 0.288 0.5 0.332\n6211Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\nTable 10  GLEND A, GLENDA-VIS, ENID (80/10/10) – mask segmentation prediction precision, raw; best \n(bold) and worst (italic)\nDataset Arch. Backb. Ep. mAP 50−95 mAP50 mAP75\nGLENDA peritoneum Mask R-101 15 0.015 0.037 <0.000\novary Mask R-101 45 0.048 0.119 <0.000\nuterus Mask all - <0.000 <0.000 <0.000\ndie Mask R-101 50 0.036 0.119 <0.000\nGLENDA-VIS mucus Mask R-101 35 0.023 0.062 0.021\nvesicles Mask R-101 44 0.042 0.100 0.033\nimplants Mask R-101 20 0.048 0.137 0.033\nabn. tissue Mask R-101 49 0.010 0.016 0.016\nENID implants Mask R-50 39 0.250 0.522 0.195\nTable\n \n11\n  ENID (80/10/10) – bounding bo\nx prediction precision, raw vs. augmented; best (bold) and worst (italic)\nAugmentation Arch. Backb. Ep. mAPbb\n50−95 mAPbb\n50 mAPbb\n75\nNone Mask R-101 39 0.288 0.500 0.332\nRotating Mask R-101 44 0.287 0.440 0.395\nBlurring Mask R-101 24 0.249 0.459 0.271\nCropping Mask R-50 44 0.266 0.499 0.216\nPerspective transform Faster R-101 24 0.249 0.497 0.207\nDesaturation Mask R-101 29 0.252 0.397 0.306\nReflection removal Mask R-101 49 0.253 0.417 0.317\nTracking10 Mask R-101 20 0.269 0.432 0.321\nTracking20 Mask R-101 10 0.240 0.459 0.206\nTracking30 Mask R-50 35 0.248 0.497 0.227\nAll w/o tracking Mask R-101 44 0.231 0.388 0.234\nAll w/ tracked30 Mask R-101 30 0.236 0.472 0.159\nBlurring & rotating Mask R-50 29 0.257 0.479 0.260\nCropping & rotating Mask R-101 49 0.261 0.472 0.278\n6212 Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\nAcknowledgements This w ork was funded by the FWF Austrian Science Fund under grant P 32010-N38.\nFunding\n Open access funding pr\novided by University of Klagenfurt.\nOpen Access\n This ar\nticle is licensed under a Creative Commons Attribution 4.0 International License, \nwhich permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long \nas you give appropriate credit to the original author(s) and the source, provide a link to the Creative Com-\nmons licence, and indicate if changes were made. The images or other third party material in this article \nare included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the \nmaterial. If material is not included in the article’s Creative Commons licence and your intended use is not \npermitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly \nfrom the copyright holder. To view a copy of this licence, visit http://\n \ncreat\n \niveco\n \nmmons.\n \norg/\n \nlicen\n \nses/\n \nby/4.\n \n0/.\nReferences\n 1.  Canis M, Donnez J, Guzic k D, Halme J, Rock J, Schenken R, Vernon M (1997) Revised american soci-\nety for reproductive medicine classification of endometriosis: 1996. Fertility and Sterility 67(5):817–\n821. https://\n \ndoi.\n \norg/\n \n10.\n \n1016/\n \nS0015-\n \n0282(97)\n \n81391-X\n 2.\n Du W\n, Rao N, Liu D, Jiang H, Luo C, Li Z, Gan T, Zeng B (2019) Review on the applications of deep \nlearning in the analysis of gastrointestinal endoscopy images. IEEE Access 7:142053–142069\n 3.\n F\nox M, Taschwer M, Schoeffmann K (2020) Pixel-based tool segmentation in cataract surgery videos \nwith mask r-cnn. In: 2020 IEEE 33rd international symposium on computer-based medical systems \n(CBMS), IEEE, pp 565–568\n 4.\n F\nu Y, Robu MR, Koo B, Schneider C, van Laarhoven S, Stoyanov D, Davidson B, Clarkson MJ, Hu \nY (2019) More unlabelled data or label more data? a study on semi-supervised laparoscopic image \nsegmentation. In: Domain adaptation and representation transfer and medical image learning with less \nlabels and imperfect data, Springer, pp 173–180\n 5.\n Gibson E, R\nobu MR, Thompson S, Edwards PE, Schneider C, Gurusamy K, Davidson B, Hawkes DJ, \nBarratt DC, Clarkson MJ (2017) Deep residual networks for automatic segmentation of laparoscopic \nvideos of the liver. In: Medical imaging 2017: Image-guided procedures, robotic interventions, and \nmodeling, international society for optics and photonics, vol 10135, pp 101351M\nTable\n \n12\n  ENID (80/10/10) – mask segment\nation prediction precision, raw vs. augmented; best (bold) and \nworst (italic)\nAugmentation Arch. Backb. Ep. mAP 50−95 mAP50 mAP75\nNone Mask R-50 39 0.250 0.522 0.195\nRotating Mask R-101 44 0.253 0.440 0.231\nBlurring Mask R-101 24 0.242 0.434 0.253\nCropping Mask R-50 44 0.277 0.545 0.258\nPerspective transform Mask R-50 49 0.252 0.472 0.183\nDesaturation Mask R-50 34 0.241 0.492 0.196\nReflection removal Mask R-101 29 0.228 0.421 0.215\nTracking10 Mask R-101 25 0.226 0.429 0.195\nTracking20 Mask R-50 25 0.236 0.466 0.157\nTracking30 Mask R-50 35 0.246 0.457 0.185\nAll w/o tracking Mask R-101 44 0.200 0.374 0.213\nAll w/ tracked30 Mask R-101 30 0.217 0.472 0.099\nBlurring & rotating Mask R-50 29 0.238 0.479 0.179\nCropping & rotating Mask R-101 49 0.253 0.472 0.228\n6213Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\n 6.  Girshic k R (2015) Fast r-cnn. In: Proceedings of the IEEE international conference on computer \nvision, pp 1440–1448\n 7.\n Gr\nammatikopoulou M, Flouty E, Kadkhodamohammadi A, Quellec G, Chow A, Nehme J, Luengo I, \nStoyanov D (2019) Cadis: Cataract dataset for image segmentation. arXiv:\n \n19061\n \n1586\n 8.\n He K, Zhang X, R\nen S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE con-\nference on computer vision and pattern recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, \n2016, IEEE Computer Society, pp 770–778. https://\n \ndoi.\n \norg/\n \n10.\n \n1109/\n \nCVPR.\n \n2016.\n \n90\n 9.\n He K, Gkio\nxari G, Dollár P, Girshick RB (2020) Mask R-CNN. IEEE Trans Pattern Anal Mach \nIntell 42(2):386–397. https://\n \ndoi.\n \norg/\n \n10.\n \n1109/\n \nTPAMI.\n \n2018.\n \n28441\n \n75\n 10.\n Henr\niques JF, Caseiro R, Martins P, Batista J (2014) High-speed tracking with kernelized correla-\ntion filters. IEEE Transactions on Pattern Analysis and Machine Intelligence 37(3):583–596\n 11.\n Jha D, Ali S, Johansen HD, Johansen DD, Rittsc\nher J, Riegler MA, Halvorsen P (2020) Real-time \npolyp detection, localisation and segmentation in colonoscopy using deep learning. arXiv:\n \n20110 \n7631\n 12.\n Jin A\n, Yeung S, Jopling J, Krause J, Azagury D, Milstein A, Fei-Fei L (2018) Tool detection and \noperative skill assessment in surgical videos using region-based convolutional neural networks. In: \n2018 IEEE winter conference on applications of computer vision (WACV), IEEE, pp 691–699\n 13.\n Jung \nAB, Wada K, Crall J, Tanaka S, Graving J, Reinders C, Yadav S, Banerjee J, Vecsei G, Kraft \nA, Rui Z, Borovec J, Vallentin C, Zhydenko S, Pfeiffer K, Cook B, Fernández I, De Rainville FM, \nWeng CH, Ayala-Acevedo A, Meudec R, Laporte M, et al. (2020) Imgaug. https://\n \ngithub.\n \ncom/\n \naleju/ \nimg\naug, Accessed 01 Feb 2020\n 14.\n K\neckstein J, Hudelist G (2020) Classification of die including bowel endometriosis: from r-asrm to \n#enzian-classification. Best Pract Res Clin Obstet Gynaecol\n 15.\n K\neckstein J, Ulrich U, Possover M, Schweppe K et al (2003) Enzian-klassifikation der tief infiltrier -\nenden endometriose. Zentralblatt für Gynäkologie 125:291\n 16.\n Kle\ntz S, Schoeffmann K, Benois-Pineau J, Husslein H (2019) Identifying surgical instruments in \nlaparoscopy using deep learning instance segmentation. In: 2019 International conference on con-\ntent-based multimedia indexing (CBMI), IEEE, pp 1–6\n 17.\n Kr\nizhevsky A, Sutskever I, Hinton GE (2012) ImageNet Classification with Deep Convolutional \nNeural Networks. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ (eds) Advances in neural \ninformation processing systems 25, Curran Associates, Inc., pp 1097–1105. http://\n \npapers.\n \nnips.\n \ncc/ \npaper/\n \n4824-\n \nimage\n \nnet-\n \nclass\n \nifica\n \ntion-\n \nwith-\n \ndeep-\n \nconvo\n \nlutio\n \nnal-\n \nneural-\n \nnetwo\n \nrks.\n \npdf\n 18.\n Leibe\ntseder A, Primus MJ, Petscharnig S, Schoeffmann K (2017) Real-time image-based smoke \ndetection in endoscopic videos. In: Wu W, Yang J, Tian Q, Zimmermann R (eds) Proceedings of \nthe on thematic workshops of ACM multimedia 2017, October 23 - 27, 2017. ACM, Mountain \nView, CA, USA, pp 296–304. https://\n \ndoi.\n \norg/\n \n10.\n \n1145/\n \n31266\n \n86.\n \n31266\n \n90\n 19.\n Leibe\ntseder A, Petscharnig S, Primus MJ, Kletz S, Münzer B, Schoeffmann K, Keckstein J (2018) \nLapgyn4: a dataset for 4 automatic content analysis problems in the domain of laparoscopic gyne-\ncology. In: César P, Zink M, Murray N (eds) Proceedings of the 9th ACM multimedia systems con-\nference, MMSys 2018, June 12-15, 2018. ACM, Amsterdam, The Netherlands, pp 357–362. https://\n \ndoi.\n \norg/\n \n10.\n \n1145/\n \n32049\n \n49.\n \n32081\n \n27\n 20.\n Leibe\ntseder A, Kletz S, Schoeffmann K, Keckstein S, Keckstein J (2020) GLENDA: gynecologic \nlaparoscopy endometriosis dataset. In: Ro YM, Cheng W, Kim J, Chu W, Cui P, Choi J, Hu M, \nNeve WD (eds) MultiMedia Modeling - 26th International Conference, MMM 2020, Daejeon, \nSouth Korea, January 5-8, 2020, Proceedings, Part II, Springer, Lecture Notes in Computer Sci-\nence, vol 11962, pp 439–450. https://\n \ndoi.\n \norg/\n \n10.\n \n1007/\n \n978-3-\n \n030-\n \n37734-2_\n \n36,\n 21.\n Lin T\nY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Micro-\nsoft coco: Common objects in context. In: European conference on computer vision, Springer, pp \n740–755\n 22.\n Litjens G, K\nooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, Van Der Laak JA, Van Gin-\nneken B, Sánchez CI (2017) A survey on deep learning in medical image analysis. Medical Image \nAnalysis 42:60–88\n 23.\n Minaee S, Bo\nykov Y, Porikli F, Plaza A, Kehtarnavaz N, Terzopoulos D (2020) Image segmenta-\ntion using deep learning: A survey. arXiv:\n \n20010\n \n5566\n 24.\n Münzer B, Sc\nhoeffmann K, Böszörmenyi L (2017) Content-based processing and analysis \nof endoscopic images and videos: A survey. Multimed Tools Appl. https://\n \ndoi.\n \norg/\n \n10.\n \n1007/ \ns11042-\n \n016-\n \n4219-z\n 25.\n Münzer B, Leibe\ntseder A, Kletz S, Schoeffmann K (2019) ECAT - endoscopic concept annotation \ntool. In: Kompatsiaris I, Huet B, Mezaris V, Gurrin C, Cheng W, Vrochidis S (eds) MultiMedia \nModeling - 25th International Conference, MMM 2019, Thessaloniki, Greece, January 8-11, 2019, \n6214 Multimedia Tools and Applications (2022) 81:6191–6215\n\n1 3\nProceedings, Part II, Springer, Lecture Notes in Computer Science, vol 11296, pp 571–576. https://  \ndoi.\n \norg/\n \n10.\n \n1007/\n \n978-3-\n \n030-\n \n05716-9_\n \n48,\n 26.\n Nw\noye CI, Mutter D, Marescaux J, Padoy N (2019) Weakly supervised convolutional lstm approach \nfor tool tracking in laparoscopic videos. InternationAl Journal of Computer Assisted Radiology and \nSurgery 14(6):1059–1067\n 27.\n Oza\nwa T, Ishihara S, Fujishiro M, Kumagai Y, Shichijo S, Tada T (2020) Automated endoscopic \ndetection and classification of colorectal polyps using convolutional neural networks. TherapeutiC \nAdvances in Gastroenterology 13:1756284820910659\n 28.\n P\nark SY, Sargent D (2016) Colonoscopic polyp detection using convolutional neural networks. In: \nTourassi GD, Armato SG (eds) Medical Imaging 2016: Computer-Aided Diagnosis, San Diego, \nCalifornia, United States, 27 February - 3 March 2016, International Society for Optics and Pho-\ntonics, pp 978528. https://\n \ndoi.\n \norg/\n \n10.\n \n1117/\n \n12.\n \n22171\n \n48\n 29.\n P\netscharnig S, Schöffmann K (2018) Learning laparoscopic video shot classification for gyneco-\nlogical surgery. Multimedia Tools and Applications 77(7):8061–8079\n 30.\n Piccialli F\n, Di Somma V, Giampaolo F, Cuomo S, Fortino G (2021) A survey on deep learning in \nmedicine: Why, how and when? Information Fusion 66:111–137\n 31.\n R\nai HM, Chatterjee K, Gupta A, Dubey A (2020) A novel deep cnn model for classification of \nbrain tumor from mr images. In: 2020 IEEE 1st international conference for convergence in engi-\nneering (ICCE), pp 134–138.https://\n \ndoi.\n \norg/\n \n10.\n \n1109/\n \nICCE5\n \n0343.\n \n2020.\n \n92907\n \n40\n 32.\n R\nai HM, Chatterjee K, Dashkevich S (2021) Automatic and accurate abnormality detection from \nbrain mr images using a novel hybrid unetresnext-50 deep cnn model. Biomedical Signal Process-\ning and Control 66:102477\n 33.\n R\nen S, He K, Girshick R, Sun J (2017) Faster r-cnn: Towards real-time object detection with region \nproposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 39(6):1137–\n1149. https://\n \ndoi.\n \norg/\n \n10.\n \n1109/\n \nTPAMI.\n \n2016.\n \n25770\n \n31\n 34.\n Sar\naswat L, Ayansina D, Cooper K, Bhattacharya S, Horne A, Bhattacharya S (2018) Impact of \nendometriosis on risk of further gynaecological surgery and cancer: a national cohort study. BJOG\n :\n \nAn International Journal of Obstetrics & Gynaecology 125(1):64–72\n 35.\n Szeg\nedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich \nA (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer \nvision and pattern recognition, pp 1–9\n 36.\n T\nelea A (2004) An image inpainting technique based on the fast marching method. Journal of \nGraphics Tools 9(1):23–34\n 37.\n T\nwinanda AP, Shehata S, Mutter D, Marescaux J, de Mathelin M, Padoy N (2017) EndoNet: A \ndeep architecture for recognition tasks on laparoscopic videos. IEEE Transactions on Medical \nImaging 36(1):86–97. https://\n \ndoi.\n \norg/\n \n10.\n \n1109/\n \nTMI.\n \n2016.\n \n25939\n \n57\n 38.\n V\nisalaxi S, Punnoose D, Muthu TS (2021a) An analogy of endometriosis recognition using \nmachine learning techniques. In: 2021 third international conference on intelligent communication \ntechnologies and virtual mobile networks (ICICV), IEEE, pp 739–746\n 39.\n V\nisalaxi S, Punnoose D, Muthu TS (2021b) Lesion extraction of endometriotic images using open \ncomputer vision. In: 2021 international conference on artificial intelligence and smart systems \n(ICAIS), IEEE, pp 747–751\n 40.\n V\nisalaxia S, Muthua TS (2021) Automated prediction of endometriosis using deep learning. Int J \nNonlinear Anal Appl 12(2):2403–2416\n 41.\n Y\nang C, Zhao Z, Hu S (2020) Image-based laparoscopic tool detection and tracking using convolu-\ntional neural networks: a review of the literature. Computer Assisted Surgery 25(1):15–28\n 42.\n Y\nengera G, Mutter D, Marescaux J, Padoy N (2018) Less is more: Surgical phase recognition with less \nannotations through self-supervised pre-training of cnn-lstm networks. arXiv:\n \n18050\n \n8569\n 43.\n Zadeh SM, F\nrancois T, Calvet L, Chauvet P, Canis M, Bartoli A, Bourdel N (2020) Surgai: deep \nlearning for computerized laparoscopic image understanding in gynaecology. Surgical Endoscopy \n34(12):5377–5383\n6215Multimedia Tools and Applications (2022) 81:6191–6215","source_license":"CC0","license_restricted":false}