Validation of administrative health data for the identification of endometriosis diagnosis

Endometriosis is a common gynecological disorder affecting at least 11% of reproductive-age women in community-based samples ( Buck Louis et al. , 2011 ) but increasing to approximately half of women experiencing pelvic pain ( Schliep et al. , 2015 ) or infertility ( Buck Louis et al. , 2016 ), the two most common comorbidities associated with endometriosis. Emerging research suggests that a diagnosis of endometriosis may be associated with a higher likelihood of a subsequent diagnosis of ovarian cancer ( Wang et al. , 2016 ), autoimmune disease ( Shigesi et al. , 2019 ), or cardiovascular disease ( Mu et al. , 2017 ). The time lag between women’s early reproductive health and mid-life chronic disease risk, in addition to the limited size of many reproductive epidemiologic studies, makes routine electronic health record (EHR) data and administrative health data a promising vehicle for carrying out risk factor research. However, before such research can be undertaken, an understanding of how well administrative health data accurately capture endometriosis within population-based cohorts is needed ( Benchimol et al. , 2016 ). Prior research has focused on assessing the reliability between and within gynecologic surgeons for diagnosing endometriosis ( Schliep et al. , 2012 , 2017 ) and the validity of self-reported endometriosis compared to administrative health records (including medical records and inpatient hospital registries) ( Shafrir et al. , 2021 ). Indeed, our own prior research within the Endometriosis, Natural History, Diagnosis and Outcomes (ENDO) Study found substantial inter- and intra-rater reliability of endometriosis diagnoses between surgeons at different levels of expertise ( Schliep et al. , 2012 ) in addition to increased inter-rater agreement when operative reports and/or pathology reports accompany operative images ( Schliep et al. , 2017 ). Regarding validity, findings to date show a wide range of agreement between self-report of endometriosis and medical records from 32% to 89% ( Missmer et al. , 2004 ; Saha et al. , 2015 , 2017 ; Shafrir et al. , 2021 ). The variability in validation proportions could partially be due to varying populations, questionnaires, and data sources. Recent research evaluated four international cohorts’ self-reported endometriosis (n = 827 total) against clinically and surgically confirmed endometriosis and observed concordance of 84% and ≥95%, respectively ( Shafrir et al. , 2021 ). There is a research gap in comparing surgically confirmed endometriosis diagnoses to diagnoses recorded in administrative health data within nested cohorts. Therefore, we set out to compare diagnoses (test method) of endometriosis and endometriosis typology (superficial endometriosis [SE], ovarian endometriomas [OE], and deep infiltrating endometriosis [DE]) recorded in administrative health data with surgically confirmed diagnoses (reference method) in the same individuals who took part in the US-based ENDO Study ( Buck Louis et al. , 2011 ).

Table 1 describes the patient’s characteristics. Women with endometriosis had lower BMI (26.5 kg/m 2 ) compared to women without endometriosis (29.5 kg/m 2 ). Women with endometriosis were more likely to be nulliparous (52.0%) compared to women without endometriosis (29.7%). No other appreciable differences between diagnoses were identified. Stage I was the most common severity of endometriosis (56.6%) and SE was the most common subtype (82.7%). Demographics of the Utah site Endometriosis, Natural History, Diagnosis, and Outcomes Study (ENDO) operative cohort (n = 412). The percent agreement was relatively high for all comparison categories—87.6% for endometriosis diagnosis and between 83.7% and 91.0% for endometriosis subtypes. There was substantial agreement among overall endometriosis ( K  = 0.75) and SE subtype diagnoses ( K  = 0.65), moderate agreement in the OE subtype ( K  = 0.58), and slight agreement in the DE subtype ( K  = 0.17). All other measures for endometriosis diagnosis were above 0.80, indicating strong agreement. Accuracy measures in the SE and OE subtypes were relatively high. The DE subtype demonstrated the lowest accuracy measures, especially in sensitivity and Κ with moderate AUCR ( Table 2 ). Diagnostic accuracy measures for endometriosis-associated diagnosis codes in the electronic health record. 88% (84%, 91%) 84% (80%, 87%) 91% (88%, 94%) 87% (85%, 88%) 0.88 (0.85, 0.91) 0.84 (0.81, 0.88) 0.87 (0.80, 0.93) 0.56 (0.52, 0.60) 0.88 (0.83, 0.93) 0.86 (0.80, 0.92) 0.82 (0.68, 0.92) 0.12 (0.05, 0.21) 0.87 (0.82, 0.91) 0.83 (0.78, 0.87) 0.92 (0.89, 0.94) 0.99 (0.98, 1.00) 0.91 (0.88, 0.94) 0.92 (0.89, 0.95) 0.98 (0.97, 0.99) 0.87 (0.86, 0.89) 0.83 (0.78, 0.88) 0.72 (0.67, 0.78) 0.51 (0.41, 0.61) 0.71 (0.38, 1.00) 0.75 (0.68, 0.81) 0.65 (0.59, 0.73) 0.58 (0.46, 0.68) 0.17 (0.05, 0.30) SE, superficial endometriosis; OE, ovarian endometriomas; DE, deep infiltrating; AUCR, area under the receiver operating characteristic curve; NPV, negative predictive value; PPV, positive predictive value; Κ, Cohen’s kappa. We found no differences in diagnostic accuracy between specific populations, including Women of Color and White women as well as women of Hispanic ethnicity and non-Hispanic ethnicity ( Table 3 ). Comparison of diagnostic accuracy measures of endometriosis diagnosis between populations. AUCR, area under the receiver operating characteristic curve; Κ , Cohen’s kappa. For the 20 false negative patients, Fig. 1A illustrates the diagnosis codes present in the administrative health data at the time of the ENDO Study. Codes indicating either a laparoscopy or laparotomy or pain were the most common in the false negatives, next to codes representative of other symptoms. For the 31 false positive patients, Fig. 1B illustrates the subtypes of endometriosis present in the administrative health data. SE was the most common subtype present in the false positives. Error analysis results. ( A ) Codes present in the medical records of false negative patients (surgically confirmed disease not reported in the administrative health data at the time of the study). ( B ) Endometriosis subtypes diagnosed in the medical record of false positive patients (without disease per surgical confirmation but a record of disease in the administrative health data at the time of the study). NOS, not otherwise specified.

This study sample comprised participants from the ENDO Study (2007–2009) who were part of the Utah operative cohort (n = 412) and who could be linked to the Utah Population Database (UPDB) (100% linkage) ( Buck Louis et al. , 2011 ). The ENDO Study included adult women undergoing laparoscopy or laparotomy for a variety of surgical indications, including pelvic pain (44%), pelvic mass (16%), irregular menses (13%), fibroids (10%), tubal ligation (10%), and infertility (7%). Surgeries were scheduled independently of and prior to ENDO Study enrollment. Women were excluded from the original ENDO Study if they were previously surgically diagnosed with endometriosis, pregnant, currently breastfeeding for 6 months or more, given injectable hormones within the past 2 years, or diagnosed with cancer other than non-melanoma skin cancer. We used the established clinical gold standard definition for endometriosis, namely, surgically visualized disease ( American Society for Reproductive Medicine, 1997 ; Kennedy et al. , 2005 ). All participating surgeons of the ENDO Study had surgical training in the diagnosis and staging of endometriosis ( Buck Louis et al. , 2011 ; Schliep et al. , 2012 ). Consistent with the observational design, surgeons were not asked to change their practice in any way but were encouraged to obtain specimens for histology. Surgeons completed the revised American Society for Reproductive Medicine (rASRM) standardized operative report immediately after surgery to capture gynecologic and pelvic pathology including endometriosis, uterine fibroids, pelvic adhesions, benign ovarian cysts, neoplasms, and congenital Mullerian anomalies. Endometriosis typology was assessed via the rASRM standardized form ( American Society for Reproductive Medicine, 1997 ). Out of the 173 women with an endometriosis diagnosis, 168 (97%) women had information on lesion location and size recorded on the rASRM form. Women with only superficial lesions on the ovary or peritoneum were considered to have SE, deep lesions (>5 mm invasion) ( Johnson et al. , 2017 ) noted in the peritoneum or obliteration of posterior cul-de-sac were considered to be DE, and deep lesions of any size noted in the ovary were considered to be OE; women who had deep ovarian and peritoneal lesions were considered to have OE + DE. Women without information on lesion location, size, and depth (n = 10) were assumed to have SE, as documented in prior research ( Byun et al. , 2020 ). All participants in this study were diagnosed with endometriosis via laparoscopy or laparotomy as part of the previous ENDO Study ( Buck Louis et al. , 2011 ). These diagnoses became part of the patient’s EHR via the normal administrative process of diagnosis code input and billing. We linked ENDO Study participants to the UPDB ( Smith et al. , 2022 ). The UPDB is a unique database containing linked data on over 11 million individuals, including statewide ambulatory and inpatient records, state vital records, and University of Utah (UU) Health and Intermountain Healthcare (IH) EHRs—including diagnosis and billing codes—capturing the majority of Utah residents. UPDB endometriosis diagnoses were identified via International Classification of Diseases (ICD) 9 and 10 codes (617.X or N80.X). Our categorization was informed by previously published research ( Saavalainen et al. , 2018 ; Rossi et al. , 2021 ) and expert opinion from our physician advisors (C.M.P. and J.B.S.) to determine the subtype of endometriosis based on the ICD code (see Supplementary Table S1 ). Data from the UPDB were restricted to the ENDO Study period (2007–2009). Medical record diagnoses that occurred more than 1 month after the individual’s participation in the ENDO Study were removed. We determined this 1-month threshold with expert opinion (consensus from obstetrics and gynecology providers) and timeline visualization ( Supplementary Fig. S1 ). We isolated specific patient populations in our data to evaluate if disparities in validity exist between different groups. We compared endometriosis diagnoses of Women of Color to White women and women of Hispanic ethnicity to women of Non-Hispanic ethnicity. We compared patient characteristics between the women with and without endometriosis as per the ENDO Study research diagnoses. We used several metrics to measure the accuracy of the administrative health data diagnoses, as compared to surgically confirmed diagnoses, including percent agreement, area under the receiver operating characteristic curve (AUCR), sensitivity, specificity, negative predictive value (NPV), positive predictive value (PPV), and Cohen’s kappa (Κ). Additionally, we compared the endometriosis subtypes diagnosed from each source and calculated the accuracy measures. To calculate 95% confidence intervals, we bootstrapped the data 1000 times, building a distribution of measures. We calculated the confidence intervals from this distribution using the percentile method ( Lee and Fung, 1993 ). We completed an error analysis to identify the patients with incorrect endometriosis diagnosis in the test method. For false negatives (those with surgically confirmed disease not reported in the administrative health data), we identified the diagnosis codes present in the administrative health data at the time of the study—when we expected the endometriosis diagnosis to be present. For false positives (those without disease per surgical confirmation but a record of disease in the administrative health data), we identified the endometriosis subtype present in the administrative health data at the time of the study—when we did not expect an endometriosis diagnosis.

In this study, we found substantial agreement when comparing surgically confirmed endometriosis diagnoses with those found in administrative health data. The sensitivity and specificity of overall endometriosis diagnosis were relatively high. We additionally assessed endometriosis subtypes. While the sensitivity, specificity, and agreement for SE and OE were high, DE had lower sensitivity and agreement with high specificity. Our findings indicate clinically diagnosed endometriosis from administrative health data accurately reflects surgically confirmed diagnoses. In the past, endometriosis research has suffered from small sample sizes—often attributable to lack of availability of participants with endometriosis diagnoses within a specific clinical setting ( Eskenazi et al. , 2001 ; Shafrir et al. , 2021 ). To address this, some researchers have turned to medical records, or administrative health data, to identify women with endometriosis in greater numbers ( Eskenazi et al. , 2001 ; Chao et al. , 2022 ; Penrod et al. , 2023 ). This is a viable method to increase the statistical power of a study; however, it has not been validated for endometriosis diagnosis. Previous studies have validated diagnoses found in the medical record for several conditions, including placenta accreta spectrum disorders, pulmonary hypertension, kidney disease, and heart failure ( Quach et al. , 2010 ; Vlasschaert et al. , 2011 ; Wijeratne et al. , 2022 ; Jotwani et al. , 2024 ). This is the first study, to our knowledge, comparing endometriosis diagnoses from administrative health data to those confirmed surgically in a study setting. Our results demonstrated using diagnoses of endometriosis from administrative health data is both a viable and valid way to increase study size when researching endometriosis. Previous studies have investigated the validity of self-reported endometriosis, as questionnaire-based research may be another method to increase statistical power in endometriosis research ( Saha et al. , 2017 ; Shafrir et al. , 2021 ). Self-report relies on the memory of the participant to accurately report if they were diagnosed with endometriosis. Shafrir et al. (2021) performed a validation study of self-reported endometriosis diagnosis, comparing self-reported endometriosis with diagnoses from the medical record. They found women were able to recall their endometriosis diagnosis well, with 84% confirmation from self-report compared to the medical record. Saha et al. (2017) performed a different but similar study, comparing self-reported endometriosis with a national register. They found a high specificity of 97% but only moderate sensitivity of 62% of self-reported endometriosis. While these results indicate self-report can be a valid method of collecting endometriosis diagnosis status, these studies compared self-report to medical records. Our study went a step further, validating administrative health data, or medical records, against surgical diagnosis. We found differing levels of sensitivity, specificity, and agreement among endometriosis subtypes. SE and OE were well documented in the medical records, resulting in relatively high agreement, sensitivity, and specificity. The sensitivity for DE was low, meaning there were many cases of DE subtype not documented in the medical records with diagnosis codes. However, the specificity of DE was very high, meaning when a case of DE was recorded, this was almost always correct. This lack of documentation of DE has been reported previously ( Shafrir et al. , 2021 ). Vanhie et al. (2016) proposed standardized surgical reporting forms for DE to address deficient standardization and incomplete reporting. From an epidemiological standpoint, high specificity is ideal as we want to be sure that cases studied are true cases. In the future, greater documentation of endometriosis subtypes, especially DE, in administrative health data could allow for more in-depth analyses. While our study has many strengths, we also acknowledge the limitations of this study. We did not include clinical notes in this study as we were only looking at structured administrative health data. Clinical notes have the potential to hold more detailed information, including endometriosis subtype. However, these are traditionally more difficult to utilize for research, often requiring natural language processing pipelines. Additionally, given that participants whose administrative health records we utilized were also part of the ENDO Study, the surgeons may have been more vigilant in diagnostic coding due to the operative forms they completed for the ENDO Study, which may have led to increased validity. However, the surgeries and administrative health data reporting was independent of the ENDO Study with all surgeons (n = 152) within the geographic catchment area eligible to perform ENDO Study surgeries and instructed not to change their practice in any way, and thus we believe our study captures what happens in a usual pelvic laparoscopy in this system. This study may have limited generalizability to the USA overall, as most of the study population was white (88.1%). However, we did evaluate diagnostic accuracy among diverse populations, albeit with smaller sample sizes. Despite the limitations, this study was able to compare surgically confirmed diagnoses from a comprehensive research study with diagnoses from administrative health data as opposed to self-reported diagnoses. The limited exclusion criteria of the original ENDO Study provided a wider range of patients for our current study, including women with many different surgical indications—not just pelvic pain or infertility. We observed substantial agreement between administrative health data and surgically confirmed endometriosis diagnoses overall, and for SE and OE subtypes. Our findings for nearly perfect specificity but low sensitivity for DE corroborates prior research and suggest that medical record-based DE diagnoses may be suitable for etiologic studies—cases identified are true cases—but not for surveillance or detection studies—not all true cases are identified. These findings may provide reassurance to researchers using administrative health data to assess risk factors and long-term health outcomes of endometriosis.