The application of risk models based on machine learning to predict endometriosis-associated ovarian cancer in patients with endometriosis

SW and JL: Conception and design. SW and QF: Provision of study materials or patients. CX: Collection and assembly of data. XC and SW: Data analysis and interpretation. All authors: Manuscript writing and final manuscript approval.

This study was funded by the National Key R&D Program of China (No. 2017YFC1001200) and the CAMS Innovation Fund for Medical Sciences (CIFMS) (No. CIFMS 2021‐I2M‐1‐003).

A total of 6809 patients were finally enrolled for the analysis and risk model construction, 125 (1.84%) of whom were diagnosed with EAOC. Compared with the non‐EAOC group, the EAOC group was significantly older at diagnosis (46.7 ± 9.1 vs. 36.2 ± 8.2 years; p  < 0.001), were older at menopause (49.1 ± 4.3 vs. 47.3 ± 6.2 years; p  = 0.023), had a higher proportion of patients in menopause ( p  < 0.001), and had less parity ( p  < 0.001). As for the chief complaints, a lower proportion of EAOC patients had dysmenorrhea ( p  < 0.001), dyspareunia ( p  = 0.001), or infertility ( p  < 0.001), but a higher proportion had a pelvic mass ( p  < 0.001). Furthermore, a higher proportion of patients with EAOC had personal history of endometrial carcinoma ( p  < 0.001). A history of gynecologic surgery was common in the EAOC group and included benign ovarian cystectomy ( p  = 0.003), oophorectomy ( p  < 0.001), and salpingectomy ( p  = 0.010). Serum carbohydrate antigen 125 (CA125) and CA199 was higher in those patients with EAOC, and the maximum diameter of the tumor was larger. Patient baseline characteristics are summarized in Table  1 . Epidemiologic and clinicopathological characteristics of the patients with endometriosis Note: Data are presented as mean ± standard deviation, n (%), or median (range) unless otherwise indicated. The bold values are those less than 0.05. Abbreviations: AUB, abnormal uterine bleeding; BMI, body mass index; CA125, carbohydrate antigen 125; CA199, carbohydrate antigen 199; CS, cesarean section; DIE, deep invasive endometriosis; EAOC, endometriosis‐associated ovarian cancer; EIN, endometrial intraepithelial neoplasma; GnRHa, gonadotropin‐releasing hormone agonist; NA, not available; SD, standard deviation; US, ultrasound; VD, vaginal delivery. The final logistic regression model included 10 variables (Table  2 ). Seven variables with OR ≥1 were positively related with the occurrence of EAOC. However, chief complaints of dysmenorrhea, preoperative use of gonadotrophin‐releasing hormone agonists (GnRHa), and concurrent leiomyoma or adenomyoma were negatively associated with malignant transformation of endometriosis. This model had an AUC of 0.903 (95% CI 0.857–0.948) with a sensitivity of 89.2% and a specificity of 82.3% in the training dataset (Figure  2A ) and an AUC of 0.891 (95% CI 0.821–0.960) with a sensitivity of 88.9% and a specificity of 76.7% in the testing dataset (Figure  2B ). Variables enrolled in the logistics regression model Abbreviations: CA125, carbohydrate antigen 125; CI, confidence interval; EC, endometrial carcinoma; GnRHa, gonadotropin‐releasing hormone agonist; OR, odds ratio; US, ultrasound. Receiver operating characteristic curve of the logistic regression model for the training dataset (A) and testing dataset (B). (C) Calibration curve of the logistic regression model. (D) Receiver operating characteristic curve of the logistic regression model for patients not using gonadotrophin‐releasing hormone agonists (GnRHa). AUC, area under the receiver operating characteristic curve. To further explore the potency of this model, the calibration of the model was assessed by calibration curve 15 as shown in Figure  2C . The p ‐value of the Hosmer‐Lemeshow goodness‐of‐fit test was 0.4, which was >0.05, whereas the C‐index was 0.891. Thus, the calibration curve also verified the prediction value of this model. Changes in the predictive performance were observed when each variable was removed. The results showed that the use of GnRHa played the most important part among all the variables. Thus, this model was further evaluated in those without the use of GnRHa and revealed an AUC of 0.891 (95% CI 0.850–0.931) (Figure  2D ), with a sensitivity of 84.3% and a specificity of 83.8%. It is noteworthy that only 125 of the total 6809 patients were diagnosed with EAOC. Since the data distribution was extremely unbalanced, models were trained and tested with a series of datasets that were split by ratios from 9:1 to 1:9. Even when leaf nodes were set at the minimum of two, the AUC for the most unbalanced datasets (i.e., 1:9) could still reach over 0.79, indicating that the model derived based on this dataset would be robust (Table  S1 ). The top 10 variables were selected as independent predictors of highest significance from the model with the most optimal hyperparameters. The features of significance for each of the variables were reported in Figure  3 . The features of importance for each variable enrolled in the logistic regression model. BMI, body mass index. The number of nodes was optimized in the derivation dataset to maximize the AUC. The hyperparameters with the highest AUC in the testing dataset, which indicated the performance of the model, were considered as optimal hyperparameters. The AUC with sensitivity and specificity for testing datasets of different hyperparameters are shown in Table  S2 . The gradient‐boosting machine with the most optimal hyperparameters (leaf nodes: 10, training: testing = 7:3) had the largest AUC (0.9417; 95% CI 0.914–0.969) in the testing dataset, with a sensitivity of 86.8% and a specificity of 86.7% (Figure  4 ). Receiver operating characteristic curve of the gradient‐boosting machine with the most optimal hyperparameters. GBDT, gradient‐boosting decision tree. The AUC increased from 0.891 as in the logistic regression model to 0.942 as in the machine learning risk model in the testing dataset (Figure  5 ). The difference was statistically significant ( p  = 0.036) (Table  S3 ). The receiver operating characteristic curve comparison of the two‐risk model for endometriosis‐associated ovarian cancer. GBDT, gradient‐boosting decision tree. The same NLP method was used to extract the prospective dataset from September 1, 2019, to May 30, 2021, and 1788 of 2378 cases were finally enrolled for the validation. The machine learning risk model had an AUC of 0.8758, with a sensitivity of 94.4% and a specificity of 73.8% (Figure  6 ). To facilitate the practice used, we created a setup.exe to help predict the occurrence rate of EAOC in patients with endometriosis. Receiver operating characteristic curve of the gradient boosting machine with the most optimal hyperparameters. AUC, area under the receiver operating characteristic curve.

The prevalence of EAOC was 1.84% in this study (125/6809), which was slightly higher than in the previous report. 2 In this study, we built a risk model through machine learning that could predict the risk of EAOC in patients with endometriosis, with a sensitivity of 86.8% and a specificity of 86.7%. The decision curve analysis showed that the machine learning‐based risk model had greater benefit for the cohort of patients with endometriosis than the logistic regression risk model, and the difference was of statistical significance. During the validation process, the same NLP method was used to collect the prospective dataset, and the prevalence of EAOC was 1.01% (18/1788). The performance of the machine learning‐based risk model was also validated using this prospective dataset, revealing a sensitivity of 94.4% and a specificity of 73.8%. To make this machine learning‐based risk model more practical and convenient, we created a setup.exe that is easy to install and convenient for clinical use. The risk prediction of malignant transformation can be calculated with either a piece or a batch of data. It has been proposed that factors predictive of endometriosis malignancy include increasing age, postmenopausal status, higher levels of carbohydrate antigen 125, larger endometriomas, and long‐standing endometriosis. 16 A previous study in this center explored the risk factors for EAOC among 1038 women with endometrioma aged ≥45 years, revealing that risk factors included menopausal status, tumors ≥8 cm in diameter, and coexisting endometrial disorders. 17 Furthermore, infertility and nulliparity are reported to be associated with a higher risk of endometriosis‐related ovarian cancer. 18 , 19 , 20 , 21 All in all, previous studies have described the difference of epidemiology and clinicopathologic characteristics between those with endometriosis and those with EAOC. However, no consensus was reached concerning these high‐risk factors for EAOC in the previous studies, and no predictive risk model exists for clinical practice. To distinguish malignant transformation of endometriosis, increasing differential methods have been proposed or explored, ranging from the most common and cost‐effective imaging examinations to innovative approaches, including biochemical and technical advancement. The use of magnetic resonance relaxometry as a noninvasive tool to help discriminate EAOC from ovarian endometrioma had a sensitivity of 86% and specificity of 94%. 22 , 23 The sensitivity of serum Smac + HE4 + CA125 was the highest, at up to 98.3%, although specificity was only up to 75%. 24 A relation between methylation status of RASSF2A gene promoter and EAOC has also been revealed. The combination of miR‐16, 21, and 191 may represent a signature unique to EAOCs. 25 Furthermore, proteomic analysis of endometrial fluid and circulating tumor DNA may also be used to detect precursor lesions in EAOC and to investigate the risk of developing EAOC. 26 The total iron levels of cyst fluid may have helped discriminate EAOC from ovarian endometrioma with a sensitivity of 85% and a specificity of 98% when the cut‐off point of total iron was set at 64.8 mg/L. 22 However, the application of these study results was limited by their small sample size, the difficulty of obtaining samples to test, high costs, simultaneous confounding factors, shortage of satisfactory accuracy, or poor clinical practicality. More importantly, the results of previous studies lacked prospective validation. Thus, more practical predictive methods still need to be explored. The early and accurate prediction of malignant transformation of endometriosis remains challenging. More research is urgently needed to establish a reliable risk model for predicting EAOC in patients with endometriosis. 27 To our delight, machine learning has been widely used in clinical differentiation and early diagnosis, such as Parkinson's disease, 28 diabetic kidney disease, 29 non‐alcoholic fatty liver disease, 30 and prostate cancer diagnosis. 31 Compared with traditional methods, the use of machine learning algorithms has advantages for modeling and validation. A random subsampling scheme was used to minimize the estimated bias. Different clinicopathological characteristics have been described in the previous studies between patients with EAOC and those with endometriosis. However, the clinical data were not fully used to predict the occurrence of endometriosis malignancy. In this study, we applied machine learning to construct a risk model to predict endometriosis malignancy, which was the novelty of this pilot study. This model includes not only clinical risk factors related to EAOC, which have been previously reported, but also unreported clinicopathologic factors as it was constructed making full use of these clinical variables. The results from this pilot study showed that the machine learning‐based risk model has a satisfactory predictive effect, and it should be noted that a setup.exe was constructed to facilitate its clinical application. This study has both strengths and limitations. The machine learning‐based risk model is of great generalizability for its features. Although this is only a single tertiary center analysis, the sample size is large enough and was validated in a prospective dataset. The risk model can provide an occurrence rate for EAOC, which previous research could not. The construction of a setup.exe is also a highlight, as this facilitates clinical practice and may even become a reference tool for primary medical institutions. However, its limitations should not be ignored. Although repeated cross‐validations were used to minimize bias, over‐fitting and higher accuracy results may remain. The lack of external validation with independent datasets from other centers for generalization and the biases inherent in retrospective studies and cross‐sectional analysis are additional limitations.

A risk model based on machine learning was constructed to help predict which patients were most likely to experience malignant transformation of endometriosis. It demonstrates that clinical characteristics can be applied to facilitate the early diagnosis of EAOC. This risk model can help gynecologists recognize patients at potential risk of EAOC and help with the monitoring and pursuing of risk‐reducing medical or surgical treatment regimens. However, the model still requires further validation with a larger, prospective, multicenter dataset to provide favorable evidence for its clinical application.

The prevalence of endometriosis has been estimated at 5%–10% among women of childbearing age. 1 It has been reported that approximately 1% of patients with endometriosis experience a malignancy, the most common site (80%) of which was in the ovaries, 2 with the presence of associated endometriosis either in the same ovary or elsewhere. 3 Ovarian cancer along with endometriosis is termed endometriosis‐associated ovarian cancer (EAOC). Patients with endometriosis‐associated clear cell ovarian cancer should also receive retroperitoneal staging and adjuvant therapy. 4 , 5 Compared with epithelial ovarian cancer, EAOC has a younger age onset, an earlier stage of disease and lower grade at diagnosis, decreased recurrence rates (26.9% vs. 41%), and a better overall 5‐year survival (75% vs. 55%). 6 , 7 In spite of being a rare disease and having a better prognosis than epithelial ovarian cancer, EAOC decreases patients' survival. Furthermore, the screening for and monitoring of endometriosis malignancy is more likely to be harmful because of the unnecessary medical care and invasive examinations. 8 With more accurate and noninvasive diagnostic methods for EAOC, the overdiagnosis and overtreatment associated with EAOC may be avoided. 9 Thus, an ability to predict the probability of malignancy among patients with endometriosis remains of great significance. After substantial exploration, endometriosis malignancy‐related risk factors have been gradually revealed. Increasingly innovative methods, including biochemical and imaging technical advancements, have been proposed to predict endometriosis malignancy, but they remain less than satisfactory. Therefore, a more practical, noninvasive, and cost‐effective risk model is urgently needed. The use of artificial intelligence tools may be another promising method to predict EAOC, and has been widely used in clinical practice. The objective of this study was to construct and evaluate a risk‐predicting model based on machine learning depending on epidemiologic and clinicopathologic features extracted from electronic medical records. This risk model can help gynecologists predict which patients are more likely to experience malignancy transformation of endometriosis and to provide appropriate and timely intervention.

None.

Detailed medical records of patients with pathologically confirmed endometriosis were extracted from patients admitted to our national referral hospital for surgery from January 1, 2015, to September 1, 2019, using natural language processing (NLP). With the logistic regression model as a comparison, the machine learning algorithms were adopted to exploit the most appropriate model for predicting the risk of endometriosis malignancy. The same NLP method was used to extract the prospective dataset for validation from September 1, 2019, to May 30, 2021. The flow chart showing this process is presented in Figure  1 . Flow chart showing study processes. NLP, natural language processing. The inclusion criteria were as follows: (1) patients aged ≥18 years and (2) diagnosed with “endometriosis” and concurrent “ovarian cancer”, namely patients with endometriosis confirmed by surgery and/or pathology and concurrent ovarian cancer confirmed by pathology. Patients with repeat cases and those with a diagnosis of endometriosis not confirmed by pathology and/or surgery were excluded. A number of criteria are used to define EAOC. Strict histologic conditions are applied in the Sampson and Scott criteria for diagnosis of cancer resulting from endometriosis, which is malignant transformation in the endometriosis glands resulting in EAOC. 10 , 11 The Van Gorp classification provides broader criteria, 12 with EAOC divided into three categories: (A) ovarian cancers with histologic proof of areas of transition between endometriosis, atypical endometriosis, and endometriosis‐associated carcinoma in accordance with Sampson and Scott's definition; (B) ovarian cancers with endometriosis in the same ovary, but without histologic proof of transition; (C) ovarian cancers with endometriosis at any location in the pelvis. We applied Van Gorp's criteria in this study. Patients’ medical information, including epidemiologic and clinicopathologic data, were extracted using algorithms. NLP techniques were applied to extract information efficiently and accurately, which included Chinese word segmentation, national word identification, part‐of‐speech tagging, semantic analysis, parsing, and relation extraction. Regular expression (RegEx) was also employed to match the keywords. The information was extracted in three steps: (1) sentence segmentation and word tokenization of the medical history or examination reports, (2) part‐of‐speech tagging and normalization of the text processed through step 1, and (3) key information extraction according to the semantic logic between words. 13 A total of 178 repeated cases were excluded from the dataset, and 6809 patients were finally enrolled for the analysis. The patients were divided into those with or without EAOC, and a total of 94 variables were included. Finally, a 6809 × 95 matrix was generated based on this rule, in which each row represented a patient and each column represented a type of clinical feature. Normally distributed continuous variables were described as means ± standard deviations, and non‐normally distributed discrete variables were summarized as median (range) and interquartile range. Categorical variables were expressed as counts with percentages. For univariate analysis, independent t ‐tests were conducted as appropriate to compare continuous data between groups if assumptions of normal distribution were confirmed. The Mann–Whitney U test was used for non‐normally distributed variables, and the chi‐squared test was used for categorical data analysis. p values <0.05 were considered statistically significant. All data analyses were performed using R 3.6.2 (R Core Team) and Python 3.7. Since the data distribution was extremely unbalanced, models were trained and tested on a series of datasets that were split by ratios from 9:1 to 1:9 (starting with 9:1, then 8:2, and so on until 1:9) for trains and tests to validate the robustness of the model. The collected cases were randomly separated into training and testing datasets, and the proportion of cases with or without EAOC were the same in both datasets. While developing the machine learning model, 10‐fold cross validation was applied while the training dataset was divided into 10 equivalent parts containing the same number of cases with EAOC, of which nine were used as the training set and the remaining one as the testing set. This process continued until each part was used once for validation. In the derivation dataset, we used a gradient‐boosting decision tree model powered by LightGBM (v3.0.0) to predict patients with endometriosis resulting in EAOC using all the predictor variables. This method generates a sequence of decision trees, and each tree is used to optimize the outcomes that were predicted poorly by the previous trees. The algorithm allows for the development of a prediction model by first creating a single decision tree that best identifies patients at risk for EAOC. The algorithm then creates a subsequent decision tree designed to identify patients who developed EAOC and who were not predicted accurately by the first tree (i.e., the residuals of the first tree). This process continues iteratively, with each additional tree aiming to better predict cases of EAOC missed by the previous trees. The final model, therefore, consists of a series of trees, with the optimal number of leaf nodes in the derivation dataset. To ensure the comparability of results, the same training and testing datasets were used for the model construction as in the final machine learning algorithms. All covariates with a p ‐value <0.1 were initially enrolled in the multivariate logistic regression analysis. A backward, stepwise procedure was used to select the best combination of potential predictors that were independently associated with the diagnosis of EAOC in the training dataset, and we then evaluated the model with the testing data. The odds ratio (OR) of enrolled variables with 95% confidence interval (CI) were reported. The risk model was assessed by area under the receiver operating characteristic (ROC) curve (AUC) and calibration curve. Changes in the model were also observed when each enrolled variable was removed. The sensitivity and specificity of the final models of the training set and the testing set were both reported. To compare the performances of the two models, the AUCs were compared using DeLong's test. 14 In addition, the established model was also validated with the prospective dataset collected in the same hospital using the same NLP method. This study was approved by the institutional review board from the study center (no. JS‐1532) on March 27, 2018. Informed consent was obtained before enrollment. All procedures performed in the study involving human participants were in accordance with the ethical standards of the institutional and national research committee and with the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards.

Table S1 Table S2 Table S3 Click here for additional data file.