Intra- and interobserver reproducibility of transvaginal ultrasound for the detection and measurement of endometriotic lesions of the bowel

P.C. and D.J. designed the study, performed the ultrasound examinations and wrote the manuscript. A.C. contributed to the data collection and contributed to writing the manuscript. T.T. contributed with data interpretation, writing the manuscript and Figures. All authors approved the final version of the manuscript.

No specific funding was received for this study. During the conduction of the study, TT received a research grant from the South‐Eastern Norwegian Health Authority (grant no. 2020083).

Out of 576 patients attending the unit for detailed endometriosis scans during our study period, 50 participants formed the final study sample (Figure  4 ). The repeat ultrasound was well‐tolerated, and no examination had to be aborted due to patient discomfort. Their demographic and clinical characteristics are shown in Table  1 , and indications for attendance in Table  2 . Flow chart summarizing the inclusion and exclusion of eligible women in the study. TVUS, transvaginal ultrasound. Demographic characteristics of the study cohort ( N  = 50). Abbreviations: CI, confidence interval; IQR, interquartile range. Indications for visit to clinic. Abbreviations: CI, confidence interval. Pelvic pain included any of the following: menstrual and nonmenstrual pelvic pain, deep dyspareunia, dyschezia, dysuria. Both the inter‐rater reliability and agreement for diagnosing bowel endometriosis on TVUS were very good (Table  3 ). There was disagreement in only two patients, in whom both operators had identified the same nodules but classified them differently. Operator A had reported infiltration of the bowel muscularis, whereas operator B had described a nodule in the retrocervical area tethered superficially to the bowel serosa. Inter‐rater agreement and reliability for categorical outcomes. Abbreviations: CI, confidence interval; LAVD, lesion to anal verge distance; POD, pouch of Douglas. Analysis using weighted kappa. When assessing the number of bowel nodules, both inter‐rater agreement and reliability reached a comparably high level (Table  3 ), and there was disagreement in only three cases, including the two patients discussed above. In the third case, the more experienced operator diagnosed a second nodule in the sigmoid colon, which operator A did not identify. TVUS also showed very good inter‐rater agreement and reliability for assessing the depth of bowel infiltration, as demonstrated in Table  3 , with only one case of disagreement, where the nodule in question was also located in the sigmoid colon. When assessing the extent of obliteration of the POD, both inter‐rater agreement and reliability were good, even if there was disagreement in five cases (Table  3 ). In three women, operator A described unilateral partial obliteration of the POD, whereas operator B reported no obliteration, and in two cases, operator A reported complete obliteration and operator B bilateral partial obliteration. The intra‐rater reliability for the mean nodule diameter, the maximum length of the nodule and the total length of bowel affected were excellent for both operators and good for LAVD (Table  4 ). The inter‐rater reliability was not as high but was still good for the mean nodule diameter and LAVD, and moderate for the maximum length of the nodule and total length of bowel affected (Table  4 ). Clinically, the Bland Altman limits of agreement demonstrated a narrow range, suggesting good intra‐rater agreement for mean nodule diameter, maximum nodule length and total length of bowel affected, but a wider range for LAVD (Figure  5 ). The intra‐rater agreement for these continuous parameters was overall better than the inter‐rater agreement, which was still good for mean nodule diameter and total length of bowel affected, but again not as good for LAVD, or maximum nodule length (Table  4 ). The inter‐rater agreement and reliability were significantly higher for maximum length of bowel nodule and LAVD when operators were compared in their judgment of whether these measurements were <3 and <8 cm, respectively (Table  3 ). Intra and inter‐rater reliability and agreement for continuous outcomes – ICC and Bland Altman analysis. Abbreviations: BA, Bland Altman; CI, confidence interval; ICC, intraclass correlation coefficient; LAVD, lesion to anal verge distance; SD, standard deviation. Differences calculated as measurement 2 minus measurement 1. Differences calculated as operator B minus operator A. Bland Altman analysis omitting one outlying value with large inter‐rater difference. Bland–Altman plots for measurements on transvaginal ultrasound. The red lines represent the mean differences between the measurements, the blue lines represent the upper and lower limits of agreement (95% confidence interval). All measurements are indicated in millimeters (mm). Images (a) represent interobserver agreement between operators (Op A; Op B); Images (b) represent the intraobserver agreement for Operator A (Op A); Images (c) represent intraobserver agreement for Operator B (Op B) for the respective measurements.

Our study showed a high intra‐ and inter‐rater reproducibility of TVUS in the detection of the presence, number, size and depth of endometriotic nodules in the bowel, whether LAVD was <8 cm, and extent of POD obliteration. These parameters are crucial for planning endometriosis surgery. Only two disagreements occurred between observers regarding the presence of bowel nodules. In one of these two cases, a nodule was located in the sigmoid colon. The literature reports that nodules >25 cm from the anal verge are difficult to visualize using TVUS, 14 requiring a very experienced operator when there is suspicion of endometriotic lesions in the sigmoid. The disagreements in our study were not clinically significant, as they would not affect the surgical approach or assessment of surgical risks. When assessing continuous parameters, such as the maximum nodule length, LAVD and total length of bowel affected, the intra‐rater reliability and agreement were overall higher than the inter‐rater measurements, which is often the case with reproducibility studies. The inter‐rater reliability was only moderate for “total length of bowel involved;” however, there were only four cases with more than one bowel nodule identified by both operators. When operators were assessing whether LAVD was <8 cm or not, inter‐ and intra‐rater reliability and agreement were much higher than when individual LAVD measurements were being compared. Given that the surgical approach and risks of the procedure considerably change if the LAVD is <8 cm vs ≥8 cm, this analysis for LAVD is clinically more relevant than the latter. Although measurement of the LAVD using TVUS is recommended by the IDEA group, there is no standardized guidance on the methodology. 23 A recent study by Aas‐Eng et al. demonstrated good reproducibility of TVUS in measuring LAVD, when compared to intraoperative measurement (IOM) using a rectal probe (ICC 0.81 and good agreement on Bland–Altman plot). 14 A second study by the same group also found TVUS and MRI to have overall similar performance in their measurement of LAVD when compared to IOM. 30 However, their TVUS and IOM measurements do not account for the curvature of the bowel, leading to an underestimation of LAVD in lesions located further from the anal verge, compared to MRI technique which allows for the bowel curvature. Furthermore, they have not assessed the reproducibility of their method. In clinical practice, the nodule length is a significant factor affecting the management when it reaches a cutoff of 30 mm, as the surgical approach becomes more invasive (segmental resection rather than discoid). 3 The inter‐rater agreement and reliability significantly improved when operators assessed whether the length was <30 mm or not, rather than interpreting the exact measurements. Of six existing studies evaluating the intra‐ and interobserver reproducibility of TVUS for the identification +/− measurement of rectosigmoid endometriotic nodules, 15 , 16 , 17 , 18 , 19 , 20 ours was comparable to only three of these, 15 , 19 , 20 due to significant differences in the study design and statistical analysis. Two of these three studies were conducted similarly to ours, other than both their examiners being highly experienced in the TVUS diagnosis of endometriosis. 15 , 19 While one of the studies is more recent, from 2020, 19 the other is significantly older, being published in 2013. 15 This study reported a lower interobserver reliability in the detection of bowel endometriosis compared to our study (κ 0.56 vs. κ 0.96), which could be explained by technological advances in ultrasound and the development of standardized criteria for diagnosing endometriosis since. 23 The more recent study 19 demonstrated results comparable to ours, reporting very good inter‐rater reliability of TVUS for identification of bowel nodules and inter‐ and intra‐rater reliability of measurements of bowel nodules, with κ 0.82 and ICC 0.88, 0.96 and 0.93, respectively. 19 Their results for intra‐ and interobserver agreement of measurement of these nodules were also comparable to ours (Bland Altman limits of agreement for observers A and B and interobserver agreement: −2.8–3.8, −4.0–2.2 and −5.4–4.6, respectively). The third study, conducted by Guerriero et al., compared three highly experienced examiners to three with less than two years' experience, which is similar to the difference in the level of expertise between observer A and B in our study. However, they observed lower levels of reliability for TVUS detection of rectosigmoid endometriosis than in our study, with moderate to good intraobserver reliability (κ values between 0.49–0.96) and fair to very good interobserver reliability (κ values between 0.21–0.87). 20 Different teaching setups for skill development in less experienced investigators, individual differences, and diagnoses being made from stored 3D volumes might be the reason for this discrepancy. That specific training in the ultrasound assessment of deep endometriosis and real‐time analysis appears to be relevant, is supported by the comparison of our results to those of Reid et al. 12 They reported a lower intra‐ and inter‐rater reliability of TVUS in determining POD obliteration compared to our study (κ 0.60–0.95 and 0.46–1.0 for different observers, and κ 0.646, respectively). Their study included four observers, two of whom were fetal medicine experts with no experience in gynecological scanning. Furthermore, the study was based on off‐line prerecorded video sets rather than real‐time scans. Similar issues existed for the study by Menakaya et al. who also reported a lower inter‐rater reliability than our study. 31 Few studies have assessed the inter‐rater reproducibility of TVUS in determining POD obliteration. 12 , 15 , 31 , 32 Holland et al., who reported a higher inter‐rater reliability than in our study (κ 0.947), 15 only classed obliteration into “yes” or “no”. However, reporting partial or complete POD obliteration, as done in this study, is clinically important as it considerably influences the complexity of a surgical case. Leonardi et al. assessed the accuracy of TVUS assessment of POD obliteration between three gynecology trainees compared to an expert in gynecological scanning, but not the intraobserver reproducibility. 32 A strength of our study is that it is the first to assess the invasion depth of bowel nodules, measurement of total length of affected bowel segment and LAVD. The other strengths include the prospective approach with predefined diagnostic parameters, blinding examiners to their own and each other's findings, and that all assessments were performed using real‐time ultrasound examinations. Off‐line image and video analysis for re‐examination are a source of bias in assessing the relevance of operator experience. The reliability of our study was further strengthened by adhering to recommended guidelines on reproducibility studies. 21 One of the limitations of our study was that examiners were not blinded to the fact that they were participating in a research study. Second, both examiners worked in the same institution, a tertiary endometriosis center, and conducted examinations using the same routine, technique and high‐quality ultrasound machines. Therefore, our findings may not be transferable to operators working in a different setting, using different techniques and equipment. Also, there was no corroboration of the imaging finding with surgical findings. However, the accuracy of TVUS in diagnosing deep endometriosis is now considered comparable to that of laparoscopy 11 and has been demonstrated in multiple studies. 13 , 14 , 33 , 34 , 35 When the intraobserver reproducibility was assessed, there was a very short interval between examinations, while in clinical practice several weeks may pass between examinations. It is important to note this limitation when considering our results of intraobserver variability.

Reproducibility studies were important before the introduction of diagnostic tests into clinical practice. 36 Our study demonstrated that both less and more experienced operators can achieve similar results when performing a preoperative sonographic assessment of various parameters of bowel endometriosis and degree of POD obliteration. Further research is required to investigate an optimal sonographic training and quality markers that define proficiency in bowel endometriosis scanning. The use of TVUS in predicting the complexity of surgical cases and assessing the risk of complications should be addressed in future studies.

Deep endometriosis commonly affects the anterior wall of the rectosigmoid colon and can cause severe bowel symptoms. 1 , 2 Symptomatic rectosigmoid endometriosis is often treated surgically. Surgical treatment can, however, be challenging and associated with severe complications, including unexpected bowel injury, anastomotic leak and rectovaginal fistula. 1 The size, number and depth of bowel lesions and lesion‐to‐anal‐verge distance (LAVD) are all considered important parameters in predicting surgical complexity. 3 Superficial lesions can be treated by shaving and might not require bowel resection. When deep lesions are present, excision is necessary, which often requires the support of colorectal surgeons. Lesions measuring <30 mm in length are considered amenable to discoid resection, which has been shown to have a lower complication rate than segmental resection. 3 However, segmental resection is still recommended for larger lesions 3 and multifocal lesions, 4 making the total length of the affected bowel segment a relevant factor. 5 Deeper lesions affecting the submucosal or mucosal layers of the bowel have been shown to be associated with >40% involvement of the bowel circumference. 6 Removing a disc of this depth could increase the risk of bowel stenosis. 3 , 7 Resection of lesions <5–8 cm from the anal verge is associated with an increased risk of postoperative anastomotic leaks, 8 fistula formation 9 and transient neurogenic bladder dysfunction. 10 The extent of pouch of Douglas (POD) obliteration is also a factor which can increase the complexity of a surgical procedure. Therefore, it is important to presurgically determine the degree of bowel and POD involvement to allow for safe and effective preoperative planning and counseling of patients. This can help to manage patient expectations and minimize the risks of intraoperative complications, residual disease, persistent symptoms, and need for repeat surgery. Transvaginal ultrasound (TVUS) is considered comparable to laparoscopy for the diagnosis of pelvic endometriosis. 11 Recent studies have shown that ultrasound measurements of the size of rectosigmoid endometriotic nodules, 12 , 13 LAVD 14 and assessment of POD obliteration 12 , 15 are accurate compared to findings at laparoscopy as a reference standard. Previous studies have assessed the intra‐ and interobserver reproducibility of TVUS in the detection and measurement of endometriotic bowel lesions 16 , 17 , 18 , 19 , 20 and POD obliteration, 15 but not for the number and depth of bowel nodules, the total length of bowel affected by endometriosis when multifocal lesions are present, nor the LAVD. The aim of our study was to examine the intra‐ and inter‐rater reproducibility of TVUS in assessing the presence, number, size and invasion depth of bowel nodules, the total length of bowel affected by endometriosis when >1 lesion is present, LAVD and POD obliteration, between an experienced and less experienced examiner.

TT reports receiving personal fees for lectures on ultrasound from GE Healthcare, Samsung, Medtronic and Merck, outside of this study. The other authors report no conflict of interest.

This was a prospective observational cross‐sectional study carried out at the Department of Obstetrics and Gynecology at University College London Hospital between July 2019 and November 2020. Our study was conducted and reported according to ‘The Guidelines for Reporting Reliability and Agreement Studies’ (GRRAS statement) 21 and guidance from existing literature. 22 Women who were referred to our unit for a detailed TVUS by our specialist endometriosis team were eligible for inclusion. We consecutively included all women that attended the clinic when both operators were present. All participants were examined by the same two operators during a single clinic attendance, using the same ultrasound machine and probe. The patients had not previously had any TVUS examinations in our clinic and were aged over 18 years. Exclusion criteria were being pregnant or postmenopausal (defined as at least 12 months of amenorrhea, which could not be attributed to hormonal treatment, breastfeeding or endocrine disorders). Operator A was a clinical research fellow with approximately 2 years of prior experience in gynecological ultrasound, having performed around 1500 examinations before the commencement of the study. The fellow had received intensive training in advanced gynecological ultrasound, including the diagnosis of pelvic endometriosis and assessment of its severity. Operator B was a senior consultant gynecologist and a level III expert in this field, with over 30 years of gynecological ultrasound experience. The primary outcome of this study was the inter‐rater reproducibility of TVUS for detecting the presence or absence of endometriotic lesions of the bowel. Secondary outcomes included the inter‐rater reproducibility of TVUS for identification of the number, depth and size of bowel endometriotic nodules, LAVD, the lesion length (maximum diameter of lesion measured in the sagittal section along the axis of the bowel), the total length of bowel segment affected by endometriosis in cases of multifocal lesions (sum of the length of all lesions) and POD obliteration. We also studied the intraobserver repeatability of TVUS for the size of nodules and LAVD, the lesion length and the total length of bowel segment affected by endometriosis in cases of multifocal lesions. The latter are deemed the most important measurements in predicting complexity and type of surgical intervention. 3 , 5 A demographic and clinical history was taken prior to the ultrasound examination, and the information was entered into a secure hospital database (Viewpoint Bildverabeitung GmbH), as per routine practice in our clinic. A two‐dimensional (2D) and three‐dimensional (3D) TVUS examination of the pelvis was systematically performed by operator A for each patient using a 4–9‐MhZ probe (Voluson E8, GE Medical Systems), as described below. In addition to the routine pelvic scan findings of the uterus and adnexa, the presence, number, depth and size of endometriotic lesions of the bowel, LAVD and obliteration of the POD were recorded. When all endometriotic lesions of the bowel had been measured and recorded, all lesions were then remeasured and recorded by the same operator. Upon completion of this process, operator B repeated the procedure in the same manner. The measurements were obtained in real‐time and recorded in a concealed fashion by an independent clinical research nurse who was present during the examinations. The research nurse was not involved in the data analysis, and both operators were blinded to their own and each other's results. Endometriosis of the bowel was diagnosed according to the systematic approach described by the International Deep Endometriosis Analysis (IDEA) group 23 (Figure  1 ). The rectosigmoid colon was followed from the anal verge, proximally towards the sigmoid colon. Five layers of the normal bowel wall can be differentiated from each other by their distinct appearance on TVUS. 24 (Figure  2 ). Going from outwards to inwards, they can be described as follows: serosal layer (thin hyperechoic line), muscularis layer (hypoechoic), submucosal layer (hyperechoic), mucosal layer (hypoechoic strip), and interface between the lumen and the mucosal layer (hyperechoic area). Illustration of the female pelvis and the anatomical relation of bowel to internal genitals. Definition of bowel segments according to the International Deep Endometriosis Analysis (IDEA) consensus. The anal canal ends at the dental line, followed by the lower rectum, which is retroperitoneal, thus not visible during diagnostic laparoscopy. The beginning of the upper rectum corresponds approximately to the pouch of Douglas/retrocervical area, where this bowel segment is first only anteriorly intraperitoneal while the posterior wall is still in the retroperitoneal space and immobile. When bending to the left, the rectosigmoid enters the peritoneal cavity also posteriorly, becoming completely intraperitoneally both on the anterior and posterior surface. The rectosigmoid junction is defined to be approximately at the uterine fundus, from where the sigmoid colon continues. The numbers (2, 8, 12) indicate the distance from the anal verge and their corresponding landmarks in centimeters (cm). The anatomy is consistent in‐between people. The dental line is at 2 cm distance from the anal verge; the second anal valve is found at 8 cm and the third anal valve at 12 cm. B, bladder; O, ovary; P, pouch of Douglas; U, uterus. Transvaginal ultrasound, B‐mode, illustrating the different layers of the bowel wall and endometriotic nodules of the bowel. (A) Normal bowel anatomy without endometriosis. (B) Endometriotic nodule (N) confined to the anterior bowel muscularis, with no involvement of other parts of the bowel wall. (C) Endometriotic nodule (N) invading the anterior submucosal layer of the bowel. Note the loss of normal anatomy of the anterior bowel which is replaced by deep endometriosis nodule. The submucosal (SMc) and muscularis (M) layers are only discernible within the posterior bowel wall. In, layer interface; Mc, mucosal layer; M, muscularis layer; N, endometriotic nodule; S, serosal layer; SMc, submucosal layer. The rectosigmoid colon was assessed for the presence of endometriotic nodules, which were diagnosed when hypoechoic, avascular, solid lesions with irregular outer margins were seen (Figure  2B,C ). They were usually tender on palpation with the ultrasound probe, and often adherent to the neighboring pelvic structures, including the uterine serosa, uterosacral ligaments and pelvic side wall. Endometriotic nodules were measured in three orthogonal planes. The average size of each lesion was calculated by taking the mean of all three of these readings. The depth of invasion of the bowel by endometriosis was determined by assessing which layer of the bowel wall was affected (Figure  2 ). Submucosal involvement was diagnosed when nodules breached the muscularis‐submucosal junction with a partial or complete loss of the definition of the anterior submucosal layer (Figure  2C ). The LAVD was defined as the distance between the most distal aspect of the lowermost endometriotic lesion of the bowel and the anal verge (Figure  1 ). It was measured in a stepwise fashion by starting at the most distal point of the lesion and following the hypoechoic line of the muscularis layer of the bowel down to the anal verge, taking measurements successively between identifiable anatomical landmarks (Figure  3 ), and adding them together to calculate the total distance at the end. Illustration of the approach to measure the lesion‐to‐anal verge distance (LAVD). LAVD was measured in a stepwise fashion (A–D), starting at the most distal point of the lesion (A) and following the hypoechoic line of the muscularis layer of the bowel down to the anal verge, taking measurements successively between identifiable anatomical landmarks and adding them together to calculate the total distance (D) at the end. B, bladder; C, cervix; N, endometriotic nodule; V, vaginal wall. Obliteration of the POD was assessed using the well‐described “sliding organs sign”, and it was classified as none, partial or complete. 23 The “flapping sail sign” 25 was used to confirm the presence of thin, filmy adhesions. When filmy adhesions were present, the POD was described as partially obliterated. The inter‐rater reproducibility of TVUS for categorical variables (presence/absence of endometriotic nodules of the bowel, depth of invasion, maximum length of nodules  3 cm, and LAVD  8 cm) was assessed by calculating reliability and agreement using Cohen's kappa (κ) statistic and proportions of agreement respectively. Cohen's weighted kappa (κ) statistic and proportions of agreement were used when measuring the inter‐rater reproducibility of TVUS for identification of the number of nodules present and assessment of obliteration of the POD. We considered a κ value of ≤0.2 to be very poor, 0.21–0.4 poor, 0.41–0.6 moderate, 0.61–0.8 good, and 0.81–1.0 very good. 26 Confidence intervals (CI) for proportions were calculated using the Wilson efficient‐score method, corrected for continuity. The inter‐ and intra‐rater reproducibility of TVUS for measurements of the average size and maximum length of bowel endometriotic nodules, and total length of the bowel segment involved in cases of multifocal lesions, and LAVD were calculated by assessing the reliability and agreement using the intraclass correlation coefficient (ICC) and Bland–Altman limits of agreement method, respectively. The ICC was calculated using a two‐way random‐effects model, with a corresponding 95% CI. Values of 0.9 excellent. 27 The Bland–Altman analysis gave the intervals into which 95% of all differences between measurements should lie. Its interpretation was based on clinical judgment to determine whether the interval demonstrated an acceptable degree of variation between measurements. A previous comparable study reported a κ value of 0.82 for the intraobserver agreement in detecting endometriotic nodules of the bowel. 19 In our study, the κ value in the null hypothesis was set at 0.3, as agreement between the observers was likely to be better than expected by chance and using a value of zero would therefore be inappropriate. 28 Assuming an expected κ value of 0.8, power of 80% and alpha of 5%, a sample size of 23 was required. However, because the proportion of women with either presence or absence of bowel endometriosis was not expected to be equal, we multiplied this minimum sample size by two to accommodate this variation. This resulted in a minimum required sample size of 46. 29 This study was approved by the Joint Research Office (JRO) at University College London Hospital (decision date June 6, 2019). All women gave their consent to be included in the study. All women who are referred to our tertiary endometriosis center have detailed transvaginal ultrasound scans by clinical research fellows, to map the extent of their disease. In all cases, these findings are routinely checked by a consultant with expert skills in the sonographic diagnosis of endometriosis during the same clinic visit. Given this, the JRO assessed that the scans being undertaken for the purpose of this study were part of routine clinical care and further approval from the Health Research Authority, the Research Ethics Committee, or Research and Development, was waived.