Deep learning prediction model based on multi-modal transvaginal ultrasound scan images for endometrial cancer.

The use of ultrasound images, clinical information, and the data collection protocol was granted approval by the Research Ethics Committee of Tongji Medical College, Huazhong University of Science and Technology (S089). This study was conducted in accordance with the principles outlined in the Helsinki Declaration. As only pre-existing medical data was applied, the requirement for patient consent was waived. All patient information was stored in the hospital system prior to analysis. We identified patients with both uterine benign diseases (including endometrial polyps, adenomyoma, endometriosis, and myomas) and malignant diseases between May 2012 and May 2025 who satisfied the following inclusion criteria: (1) underwent surgery following transvaginal ultrasound examination; (2) possessed gray scale, CDFI, and 3D reconstructed transvaginal ultrasound images. Patients meeting any of the following criteria were excluded: (1) previous history of gynecological malignancy; (2) prior pelvic surgery or receipt of chemotherapy/radiotherapy; (3) ultrasound specialist determined that image evaluation was affected by low image resolution, poor image quality, and significant ultrasound artifacts; (4) inadequate pathological reports or incomplete diagnosis. The patient flowchart is shown in Fig. 1 . The authors collected clinical data of the patients from the database, including age, age at menarche, menopausal status, presence of vaginal bleeding in the past year, parity, history of hypertension, history of diabetes, family history of malignancy, and pathological diagnosis. For patients with endometrial cancer, additional data were collected, including endometrial cancer FIGO staging, presence of cervical stromal invasion, presence of deep myometrial invasion, presence of lymph node metastasis, and classification as high-risk endometrial cancer according to the 2016 ESMO-ESGO-ESTRO guidelines [ 13 ]. Additionally, the authors collected pathological types for all patients based on the pathological diagnosis, which is presented in Table S1 . Fig. 1 The flowchart of the eligibility criteria and procedures for evaluating deep learning models. DL fusion = Deep Learning Model designed for the fusion of three modalities The flowchart of the eligibility criteria and procedures for evaluating deep learning models. DL fusion = Deep Learning Model designed for the fusion of three modalities The ultrasound examinations were carried out using three distinct ultrasound systems: gray scale, color Doppler, and 3D reconstruction. All ultrasound examinations were conducted using three high-end ultrasound systems (Philips EPIQ7, GE Voluson S8, Philips IU22) equipped with transvaginal probes. All images were obtained following a standardized protocol, conducted by ultrasound practitioners with over five years of specialized experience in gynecological diagnostics. 3D ultrasound images were reconstructed offline by capturing a series of continuous 2D B-mode slices acquired through a motorized transvaginal probe sweep. The reconstruction algorithm, supplied by the scanner manufacturer, aligns and interpolates 2D slices into an isotropic volume. A consistent automated reconstruction procedure was applied to all ultrasound systems utilized in this study, ensuring the consistency of the 3D data generation method. Radiologists were certified sonographers, specialized in gynecological scanning, and had a minimum of 5 years of autonomous clinical involvement in pelvic ultrasonography. Endometrial tumors were delineated using the terminology and definitions outlined by the International Endometrial Tumor Analysis (IETA) group [ 14 ]. The DL analysis involved the selection of one gray scale image and its corresponding CDFI image, either in the longitudinal or transverse view, for each patient. The preference was given to the utilization of the longitudinal view image of the uterus, if it was available. Only in cases where the longitudinal view was not available, the transverse view image was utilized. Additionally, since each patient had only one 3D reconstruction image, the unique 3D reconstruction image was chosen for analysis. The collected transvaginal ultrasound images in three modalities were manually segmented along the uterine border by one ultrasound radiologist using the open-source ITK-SNAP software (version 3.8.0, http://www.itksnap.org/pmwiki/pmwiki.php ) [ 15 ]. The segmentation results were reviewed by another ultrasound physician, both of whom had at least 5 years of experience in the field. The clinical information of the patients was blinded to both physicians prior to the segmentation process. The data was collected from two institutions: data from Wuhan Tongji Hospital was used as an internal training set, and data from Women’s Hospital School of Medicine Zhejiang University was used as an external test set. The number of images in the internal training set and external test set is 696 and 78, respectively. The data were divided into benign and malignant categories based on pathological diagnosis. Further, we divided the internal set into the training set and the validation set in a ratio of 8:2, with the proportion of benign and malignant patients in the training set and the test set remaining consistent during division. The image processing procedure is shown in Fig.  2 . To reduce the impact of noise, the areas of interest were marked for these images, and the smallest rectangles covering the region of interest were extracted. The extracted images were considered as input for the DL-based model. During the process of model training, each image in the training set was scaled to 224*224, horizontally randomly flipped, and randomly rotated. In the process of the model test, images from the test set were passed through the trained model to produce a probability value, which was used for benign or malignant decisions according to a predefined cutoff threshold. Fig. 2 The main flow of image processing and prediction. Three types of ultrasound scan images (from top to bottom) were used as the input: presenting gray scale images of the uterus captured from orthogonal planes, color Doppler images with blood flow information, and three-dimensional reconstruction imaging of the uterine cavity for visualization. The areas of interest were marked for these images, and the smallest rectangles covering the region of interest were extracted. The extracted images were considered as input for the deep learning based model The main flow of image processing and prediction. Three types of ultrasound scan images (from top to bottom) were used as the input: presenting gray scale images of the uterus captured from orthogonal planes, color Doppler images with blood flow information, and three-dimensional reconstruction imaging of the uterine cavity for visualization. The areas of interest were marked for these images, and the smallest rectangles covering the region of interest were extracted. The extracted images were considered as input for the deep learning based model The DL models employed in this study utilized DL-based algorithms, which were enhanced versions of the widely used mainstream residual network (ResNet) [ 16 ]. Moreover, a modality fusion strategy was employed to develop the DL fusion model [ 17 ]. The DL model contains three ResNet-18 networks for feature extraction of three US image types. The extracted features were concatenated and transformed by a fully connected layer with 32 hidden nodes. The network architecture is shown in Fig. 3 . In order to minimize the cross-entropy loss for the validation set, the DL model was trained by the SGD (stochastic gradient descent) optimizer. The initial learning rate was set to le-3, and it shrank by a factor of 10 every 5 epochs. The batch size was set to 4 and the epoch was set to 50. After training, the model with the smallest loss on the validation set was considered the final model for prediction. Fig. 3 The deep learning neural network structure of the method based on deep learning techniques. In the deep learning model, for each lesion, three ultrasound scan images with different types were used as input, and features were extracted from them with the model ResNet-18. These features were subsequently concatenated and transformed by a fully connected layer. The output was considered as the predicted probability and transformed by a softmax layer to yield the result of binary classification. The ResNet-18 contains one convolution layer and four groups of convolution layers The deep learning neural network structure of the method based on deep learning techniques. In the deep learning model, for each lesion, three ultrasound scan images with different types were used as input, and features were extracted from them with the model ResNet-18. These features were subsequently concatenated and transformed by a fully connected layer. The output was considered as the predicted probability and transformed by a softmax layer to yield the result of binary classification. The ResNet-18 contains one convolution layer and four groups of convolution layers Twelve radiologists with 3–5 years of work experience independently reviewed the images, drawing preliminary conclusions (benign or malignant). Following this, the diagnostic outcomes and estimated malignancy probabilities from the DL fusion model were shared with the radiologists for reevaluation, resulting in the radiologists’ conclusions with DL fusion assistance. All radiologists were unaware of the pathological diagnosis and clinical details. The histopathological diagnosis of the excised tissue served as the reference standard. The tumor tissues obtained from the surgery were examined at the Department of Pathology of Wuhan Tongji Hospital and Women’s Hospital School of Medicine Zhejiang University, following the guidelines outlined by the World Health Organization [ 18 ]. The classification of the tumors was performed according to the criteria recommended by the International Federation of Gynecology and Obstetrics (FIGO) [ 19 ]. All statistical analyses were conducted using R software version 4.0.5. The normality of the feature distributions in each optimal model was evaluated using the Kolmogorov-Smirnov test (K-S test). For comparisons between variables that followed a normal distribution, the Student’s t-test was used. The Mann-Whitney U-test was employed to evaluate differences in continuous variables, while the chi-squared test or Fisher’s exact test was used to assess disparities in categorical variables. In cases where the variables did not follow a normal distribution, the run test was used. AUC values were compared using the deLong test. To compare the diagnostic performances of models and radiologists’ assessments, metrics including AUC, sensitivity, specificity, positive and negative predictive values, positive and negative likelihood ratios, and diagnostic odds ratios were used. The distribution of selected features was described using the mean, median, and variance values. All statistical tests were two-tailed, and a significance level of p  < 0.05 was considered statistically significant.

A cohort of 696 patients from Wuhan Tongji Hospital was selected as the training (394 benign and 162 malignant) and validation sets (99 benign and 41 malignant), while a cohort of additional 78 patients (55 benign and 23 malignant) from Women’s Hospital School of Medicine, Zhejiang University was used as an external independent test set. All patients underwent pathological confirmation and had access to gray scale, color Doppler, and 3D reconstructed ultrasound images. Tables  1 and 2 and Table S1 summarize the demographic, clinical, and pathological characteristics of the patients in the internal and external datasets. The distribution of benign and malignant patients shows remarkable consistency in both the internal and external datasets. Moreover, the age distribution of patients exhibits similarity, with an average age of 44.84 years in the internal dataset and 44.47 years in the external dataset. Among the endometrial cancer patients from both centers, the majority of cases are in FIGO stage I based on surgical pathology. Additionally, the predominant histological subtype in both centers is endometrioid carcinoma. The analysis of clinical characteristics in benign and malignant patients from two centers revealed notable discrepancies in age, menopausal status, abnormal vaginal bleeding, and the presence of diabetes, consistent with the clinical profile of endometrial cancer typically encountered in clinical practice. Table 1 Patient demographics and clinical characteristics in the training, validation, and external testing sets Characteristic Training and Validation Sets Test Set No. of patients 696 78 Age(y) 44.84 ± 10.76 (21–77) 44.47 ± 11.23 (21–77) Benign Patients Malignant Patients P-value Benign Patients Malignant Patients P-value No. of patients 493(71) 203(29) 55(71) 23(29) Age(y) 41.05 ± 8.93 (21–74) 54.02 ± 9.18 (21–74) < 0.0001 42.16 ± 10.10 (21–68) 49.87 ± 12.38 (28–77) < 0.01 Nulliparous 87(18) 19(9) < 0.01 19(35) 5(22) 0.26 Menopause 55(11) 129(64) < 0.0001 6(11) 9(39) < 0.01 Abnormal Vaginal Bleeding* 223(45) 183(90) < 0.0001 29(53) 21(91) < 0.01 Family History of Tumors 93(19) 32(16) 0.33 6(11) 5(22) 0.21 Hypertension 64(13) 88(43) < 0.0001 6(11) 5(22) 0.21 Diabetes mellitus 11(2) 21(10) < 0.0001 1(2) 4(17) < 0.01 Note: *Symptoms of abnormal vaginal bleeding within one year Patient demographics and clinical characteristics in the training, validation, and external testing sets 44.84 ± 10.76 (21–77) 44.47 ± 11.23 (21–77) 41.05 ± 8.93 (21–74) 54.02 ± 9.18 (21–74) 42.16 ± 10.10 (21–68) 49.87 ± 12.38 (28–77) Note: *Symptoms of abnormal vaginal bleeding within one year Table 2 FIGO Staging and tumor invasion and metastasis indicators of malignant patients in the training, validation, and external testing sets Characteristic Training and Validation Sets Test Set FIGO Staging  Stage IA 114(56) 16(70)  Stage IB 33(16) 3(13)  Stage II 22(11) 1(4) Stage IIIA and Above 34(17) 3(13) Tumor Invasion and Metastasis Indicators  Cervical Stromal Invasion 34(17) 3(13)  Deep Myometrial Invasion 55(27) 4(17)  Lymphatic Node Metastasis 19(9) 1(4)  2016 ESGO High Risk† 73(36) 10(43) Note: † Classification as high-risk endometrial cancer according to the 2016 ESMO-ESGO-ESTRO guidelines FIGO Staging and tumor invasion and metastasis indicators of malignant patients in the training, validation, and external testing sets Note: † Classification as high-risk endometrial cancer according to the 2016 ESMO-ESGO-ESTRO guidelines The application of the optimal cutoff values, obtained by maximizing the Youden index in the ROC curve (DL gray , 0.78; DL CDFI , 0.43; DL 3D , 0.36; and DL fusion , 0.56), facilitated the classification of benign and malignant cases. For the DL fusion model, we compared the AUC values of the ensemble prediction model based on individual ResNet-18 models with the AUC values of each single-modal model in the internal validation set and external test set. In the internal validation set, DL gray , DL CDFI , and DL 3D achieve AUC values of 0.88 (95% CI: 0.82, 0.93), 0.89 (95% CI: 0.84, 0.94), and 0.96 (95% CI: 0.92, 0.99), respectively. The DL fusion model classified lesions based on concatenated features extracted from three ResNet-18 models using three modalities of input ultrasound images, which achieves an AUC of 0.94 (95% CI: 0.90, 0.98) (Fig.  4 , Table S2 ). The diagnostic performance of DL fusion surpasses that of DL gray ( p  = 0.01), while being similar to that of DL CDFI ( p  = 0.11) and DL 3D ( p  = 0.41). Fig. 4 The ROC curves of the DL gray , DL CDFI , DL 3D and DL fusion models in the internal validation dataset. The performance is compared against 12 ultrasound radiologists. Each gray star represents the individual performance of a radiologist, while the black stars depict the average performance of the radiologists The ROC curves of the DL gray , DL CDFI , DL 3D and DL fusion models in the internal validation dataset. The performance is compared against 12 ultrasound radiologists. Each gray star represents the individual performance of a radiologist, while the black stars depict the average performance of the radiologists In the external test set, DL gray , DL CDFI , and DL 3D achieve AUC values of 0.78 (95% CI: 0.67, 0.89), 0.79 (95% CI: 0.69, 0.89), and 0.85 (95% CI: 0.76, 0.93), respectively. The DL fusion model achieves an AUC of 0.92 (95% CI: 0.86, 0.98) (Fig.  5 ; Table  3 ), which is significantly higher than the DL gray model and DL CDFI model ( p  < 0.01 and p  = 0.01, respectively) but similar to the DL 3D model ( p  = 0.14). Additionally, DL fusion demonstrates the highest level of diagnostic accuracy of 0.86 (95% CI: 0.78, 0.94), sensitivity of 0.91(95% CI: 0.85, 0.98), and specificity of 0.84 (95% CI: 0.75, 0.92). Fig. 5 The ROC curves of DL gray , DL CDFI , DL 3D and DL fusion models in the external test dataset. The performance is compared against 12 ultrasound radiologists. Each gray star represents the individual performance of a radiologist, while the black stars depict the average performance of the radiologists The ROC curves of DL gray , DL CDFI , DL 3D and DL fusion models in the external test dataset. The performance is compared against 12 ultrasound radiologists. Each gray star represents the individual performance of a radiologist, while the black stars depict the average performance of the radiologists Table 3 Diagnostic performance of deep learning models and radiologist assessment in the external test set DL gray DL CDFI DL 3D DL fusion Radiologist Assessment AUC 0.78[0.69–0.87] 0.79[0.70–0.88] 0.85[0.77–0.93] 0.92[0.86–0.98]* 0.64[0.55–0.74] Sensitivity 0.83[0.74–0.91] 0.87[0.79–0.94] 0.87[0.79–0.95] 0.91[0.85–0.98] 0.30[0.13–0.52] Specificity 0.64[0.53–0.74] 0.71[0.61–0.81] 0.75[0.65–0.84] 0.84[0.75–0.92] 0.98[0.95-1.00] accuracy 0.69[0.59–0.80] 0.78[0.69–0.88] 0.76[0.66–0.85] 0.86[0.78–0.94] 0.78[0.67–0.87] PPV 0.49[0.37–0.60] 0.59[0.48–0.70] 0.56[0.44–0.67] 0.70[0.60–0.80] 0.88[0.65-1.00] NPV 0.90[0.83–0.97] 0.93[0.87–0.99] 0.93[0.87–0.99] 0.96[0.91-1.00] 0.77[0.67–0.87] * p  < 0.01 compared with DL gray , p  = 0.01 compared with DL CDFI , p  = 0.17 compared with DL 3D , p  < 0.001 compared with Radiologist Assessment using the DeLong test Diagnostic performance of deep learning models and radiologist assessment in the external test set * p  < 0.01 compared with DL gray , p  = 0.01 compared with DL CDFI , p  = 0.17 compared with DL 3D , p  < 0.001 compared with Radiologist Assessment using the DeLong test The radiologist’s assessment results were composite diagnostic conclusions derived from the consensus of the majority of 12 radiologists. The accuracy of the DL fusion model is significantly superior to the comprehensive assessment of radiologists in both the internal validation set (0.90 vs. 0.86, p  < 0.001) (Table S2 ) and the external test set (0.86 vs. 0.78, p  < 0.001) ( Table  3 ) . Moreover, in the internal validation dataset, the DL fusion model exhibits higher sensitivity compared with radiologists (0.73 vs. 0.66, p  < 0.001), while demonstrating similar specificity (0.96 vs. 0.97, p  = 0.51). In the external test set, the DL fusion model demonstrates higher sensitivity compared with radiologists (0.91 vs. 0.30, p  < 0.001), albeit lower specificity (0.84 vs. 0.98, p  < 0.001). To assess the efficacy of DL fusion in assisting radiologists with the diagnosis of endometrial cancer, we compared the diagnostic performance parameters of 12 radiologists with and without diagnostic conclusions of the DL fusion model in the internal validation set and external test set (Table S3 , Table  4 ). With the assistance of the DL fusion model, radiologists demonstrate heightened levels of diagnostic accuracy and sensitivity compared with their no model-assisted status in both the internal validation set (0.89 vs. 0.86, p  = 0.001; 0.66 vs. 0.59, p  < 0.01) and the external test set (0.85 vs. 0.77, p  < 0.001; 0.56 vs. 0.33, p  < 0.001). Table 4 Diagnostic performance of radiologists with or without DL fusion assistance in the external test set AUC Sensitivity Specificity Accuracy PPV NPV Radiologists without DL fusion Radiologist 1 0.77[0.66–0.88] 0.61[0.39–0.78] 0.93[0.85–0.98] 0.83[0.73–0.91] 0.78[0.59–0.97] 0.85[0.76–0.94] Radiologist 2 0.63[0.53–0.73] 0.30[0.13–0.48] 0.96[0.91-1.00] 0.77[0.66–0.86] 0.78[0.51-1.00] 0.77[0.67–0.87] Radiologist 3 0.73[0.62–0.84] 0.52[0.30–0.74] 0.95[0.89-1.00] 0.82[0.72–0.90] 0.80[0.60-1.00] 0.83[0.73–0.92] Radiologist 4 0.62[0.52–0.73] 0.30[0.13–0.48] 0.95[0.87-1.00] 0.76[0.65–0.85] 0.70[0.42–0.98] 0.76[0.66–0.87] Radiologist 5 0.62[0.52–0.73] 0.30[0.13–0.52] 0.95[0.87-1.00] 0.76[0.65–0.85] 0.70[0.42–0.98] 0.76[0.66–0.87] Radiologist 6 0.62[0.52–0.73] 0.30[0.13–0.48] 0.95[0.87-1.00] 0.76[0.65–0.85] 0.70[0.42–0.98] 0.76[0.66–0.87] Radiologist 7 0.61[0.52–0.69] 0.22[0.04–0.39] 1.00[1.00–1.00] 0.77[0.66–0.86] 1.00[1.00–1.00] 0.75[0.65–0.85] Radiologist 8 0.62[0.51–0.72] 0.30[0.13–0.52] 0.93[0.85–0.98] 0.74[0.63–0.84] 0.64[0.35–0.92] 0.76[0.66–0.86] Radiologist 9 0.59[0.50–0.68] 0.22[0.09–0.39] 0.96[0.91-1.00] 0.74[0.63–0.84] 0.71[0.38-1.00] 0.75[0.65–0.85] Radiologist 10 0.60[0.51–0.69] 0.22[0.04–0.39] 0.98[0.95-1.00] 0.76[0.65–0.85] 0.83[0.54-1.00] 0.75[0.65–0.85] Radiologist 11 0.65[0.56–0.75] 0.3[0.13–0.48] 1.00[1.00–1.00] 0.79[0.69–0.88] 1.00[1.00–1.00] 0.77[0.68–0.87] Radiologist 12 0.66[0.56–0.77] 0.35[0.17–0.57] 0.98[0.95-1.00] 0.79[0.69–0.88] 0.89[0.68-1.00] 0.78[0.69–0.88] Radiologists’ mean 0.65[0.61–0.68] 0.33[0.25–0.41] 0.96[0.94–0.98] 0.77[0.76–0.79] 0.79[0.72–0.87] 0.78[0.76–0.80] Radiologists with DL fusion Radiologist 1 0.79[0.68–0.90] 0.65[0.48–0.83] 0.93[0.85–0.98] 0.85[0.75–0.92] 0.79[0.61–0.97] 0.86[0.78–0.95] Radiologist 2 0.73[0.62–0.84] 0.48[0.26–0.70] 0.98[0.95-1.00] 0.83[0.73–0.91] 0.92[0.76-1.00] 0.82[0.73–0.91] Radiologist 3 0.77[0.67–0.88] 0.57[0.35–0.78] 0.98[0.95-1.00] 0.86[0.76–0.93] 0.93[0.79-1.00] 0.84[0.75–0.93] Radiologist 4 0.88[0.79–0.97] 0.78[0.61–0.96] 0.98[0.95-1.00] 0.92[0.84–0.97] 0.95[0.85-1.00] 0.92[0.84–0.99] Radiologist 5 0.84[0.74–0.94] 0.70[0.52–0.87] 0.98[0.95-1.00] 0.90[0.81–0.95] 0.94[0.83-1.00] 0.89[0.81–0.97] Radiologist 6 0.73[0.62–0.84] 0.48[0.26–0.70] 0.98[0.95-1.00] 0.83[0.73–0.91] 0.92[0.76-1.00] 0.82[0.73–0.91] Radiologist 7 0.80[0.70–0.91] 0.61[0.39–0.83] 1.00[1.00–1.00] 0.88[0.79–0.95] 1.00[1.00–1.00] 0.86[0.77–0.94] Radiologist 8 0.70[0.59–0.80] 0.39[0.22–0.61] 1.00[1.00–1.00] 0.82[0.72–0.90] 1.00[1.00–1.00] 0.80[0.70–0.89] Radiologist 9 0.73[0.62–0.84] 0.48[0.30–0.70] 0.98[0.95-1.00] 0.83[0.73–0.91] 0.92[0.76-1.00] 0.82[0.73–0.91] Radiologist 10 0.71[0.60–0.81] 0.43[0.22–0.65] 0.98[0.95-1.00] 0.82[0.72–0.90] 0.91[0.74-1.00] 0.81[0.71–0.90] Radiologist 11 0.80[0.69–0.90] 0.61[0.39–0.83] 0.98[0.95-1.00] 0.87[0.78–0.94] 0.93[0.81-1.00] 0.86[0.77–0.94] Radiologist 12 0.75[0.64–0.86] 0.57[0.35–0.78] 0.93[0.85–0.98] 0.82[0.72–0.90] 0.76[0.56–0.93] 0.84[0.74–0.93] Radiologists’ mean 0.77[0.73–0.80] 0.56[0.49–0.63] 0.98[0.96–0.99] 0.85[0.83–0.89] 0.91[0.87–0.96] 0.84[0.82–0.87] Diagnostic performance of radiologists with or without DL fusion assistance in the external test set In order to enhance the model’s interpretability, we applied Gradient-weighted Class Activation Mapping (GRAD-CAM) to visualize the final classification layer. GRAD-CAM highlights the regions of interest in ultrasound images that contribute to the model’s predictions and uses a color gradient from red (highest attention) to blue (lowest attention) to differentiate, providing insights into the model’s decision-making process. The visual representation in Fig.  6 presents representative GRAD-CAM images for different models of benign uterine diseases and endometrial cancer. These visual results confirm that the model focuses primarily on the endometrial region and its neighboring myometrial layer. Fig. 6 Representative examples of the GRAD-CAM heatmaps generated from the deep learning model. The GRAD-CAM heatmaps overlaid on the original ultrasound images are shown for benign diseases ( A-C ) and endometrial cancer ( D-F ) patients in gray scale, CDFI, and 3D ultrasound modals Representative examples of the GRAD-CAM heatmaps generated from the deep learning model. The GRAD-CAM heatmaps overlaid on the original ultrasound images are shown for benign diseases ( A-C ) and endometrial cancer ( D-F ) patients in gray scale, CDFI, and 3D ultrasound modals

Endometrial cancer is the most common gynecological malignancy in developed countries [ 1 ]. Early diagnosis is critical to the prognosis of patients with endometrial cancer. The 5-year overall survival rate for patients with endometrial cancer confined to the uterus is approximately 95%, whereas patients with regional and distant metastasis experience reduced survival rates of around 70% and below 20%, respectively [ 2 ]. Transvaginal ultrasound examination is commonly utilized as an initial evaluation method for suspected endometrial cancer patients due to its simple operation and cost-effectiveness compared with CT and MRI scans [ 3 ]. However, ultrasound imaging characteristics of early-stage endometrial tumors display resemblances to those of endometrial hyperplasia. This resemblance presents challenges in the differential diagnosis between benign and malignant endometrial diseases among post-menopausal women [ 4 ]. Accordingly, the conclusive diagnosis of endometrial cancer still relies on invasive diagnostic procedures like endometrial biopsy, hysteroscopy, and dilation and curettage (D&C) [ 5 ], which could lead to escalated expenditures, patient discomfort, and relevant complications. Additionally, these invasive procedures carry the risk of missing tumor lesions, limit the assessment of tumor infiltration extent, and increase the potential for tumor dissemination and uterine perforation [ 5 , 6 ]. Therefore, it is crucial to maximize the diagnostic performance of radiological techniques to guide the selection of subsequent treatment approaches and reduce the financial burden and risks associated with overtreatment. Deep learning (DL) networks have become a common approach in the development of tumor diagnostic models [ 7 ]. DL networks enable the non-invasive extraction of quantifiable features and subtle differences in imaging characteristics that are imperceptible to radiologists, thereby aiding in clinical decision-making. Furthermore, the adoption of nonlinear network architecture enables effective feature learning, leading to a reduced number of network parameters and mitigated overfitting [ 8 ]. Ultrasound-based DL networks have been successfully applied in the diagnosis of various cancers, including thyroid cancer [ 9 ], breast cancer [ 10 ], and ovarian cancer [ 11 ]. However, ultrasound-based artificial intelligence diagnostic models specifically designed for endometrial cancer are still lacking currently, requiring further exploration and development [ 12 ]. In this study, we developed a multi-modal DL algorithm for automated classification of benign and malignant endometrial diseases based on gray scale, color Doppler, and 3D reconstructed ultrasound images, and validated its diagnostic performance with internal and external datasets. Furthermore, we conducted a comparative analysis to evaluate its diagnostic performance in comparison with subjective radiologist assessments, and examined whether the utilization of predictive models could enhance the accuracy of radiologists in diagnosing endometrial cancer.

DL algorithms have garnered increasing attention in the fields of image recognition and disease prediction. In this study, we developed single DL models based on gray scale, color Doppler, and 3D reconstruction transvaginal ultrasound modalities, as well as a novel fusion model combining these three modalities (DL fusion ), for distinguishing between benign and malignant endometrial diseases. In general, the DL fusion model exhibits excellent performance and outperforms three single-modal models. Furthermore, the DL fusion model exceeds the average level of subjective assessment by 12 ultrasound radiologists in terms of diagnostic accuracy, as evidenced by both the internal validation and external test sets. More importantly, a substantial improvement in the diagnostic accuracy of ultrasound radiologists is observed with the assistance of the DL fusion model. In single-modality models, we found that DL 3D is generally superior to the DL gray or DL CDFI . This phenomenon may be related to the enhanced tissue contrast and improved clarity of tissue boundaries provided by three-dimensional reconstructed ultrasound in comparison to two-dimensional ultrasound, thus emphasizing the advantages of three-dimensional ultrasound in diagnosing conditions within the uterine cavity [ 20 ]. Additionally, evidence from studies suggests that three-dimensional reconstructed ultrasound can be a preferred method for initial assessment of endometrial cancer over two-dimensional ultrasound due to its enhanced diagnostic accuracy, which is substantially consistent with our findings [ 21 , 22 ]. We have also noticed that even in the DL fusion model, the PPV is still not high enough (70%). This could be attributed to the relatively high proportion of benign cases compared to malignant cases (approximately 2.4:1) in this study, despite the notable sensitivity and specificity of DL fusion . Consequently, there is a higher likelihood of misdiagnosing benign diseases as malignant, leading to an inadequate PPV. Similar findings arise in research with comparable sample sizes and compositions to this study [ 23 – 27 ]. In clinical practice, postmenopausal bleeding stands out as the most prevalent symptom of endometrial cancer, representing approximately two-thirds of gynecological patient consultations. Nonetheless, only 9% of cases end up confirmed as endometrial cancer [ 28 ]. Hence, this DL fusion model is not intended for general screening of gynecological patient cohorts to avoid misdiagnosing a substantial number of benign uterine disease cases as endometrial cancer. On the contrary, the DL fusion model is better suited to assist radiologists in enhancing image diagnostic performance and aiding clinicians in making more personalized clinical decisions for patients. Although the sensitivity of diagnosing endometrial cancer is significantly raised with DL fusion assistance, it is not yet at an ideal level. This observation may be related to the clinical characteristics of endometrial cancer. Since patients often seek medical attention at an early stage, the ultrasound imaging characteristics of early endometrial tumors closely resemble those of endometrial hyperplasia [ 4 ]. The difficulty arises from the fact that the image diagnoses provided by the DL fusion model may conflict with the previous clinical experiences of radiologists. This deviation might prompt some radiologists to rely on individual judgment in certain diagnostic cases out of their commitment to patient care. We also note that the study was designed as a one-time assistance scenario; in practice, ongoing interaction and training with the tool may lead to greater trust and sensitivity improvements over time [ 29 ]. In our future research, our goal is to devise a continuous learning mechanism that incorporates feedback from radiologists aiming to minimize decision discrepancies and enhance diagnostic performance. An ideal early detection tool for endometrial cancer should be patient-friendly, cost-effective, and able to accurately identify high-risk patients for subsequent diagnosis and treatment while reassuring low-risk women. As the currently accepted preferred screening method, ultrasound still presents several limitations. Results can be subjective and inconsistent since radiographic assessment of disease by specialists relies most commonly on visual evaluation. Furthermore, ultrasound has a relatively low specificity, and is susceptible to other conditions such as coexisting myomas, adenomyosis, axial uterus, or previous surgery history. In contrast, the DL model that we developed automatically recognizes complex patterns and synthesizes information from the three imaging modalities to provide quantitative assessments of radiographic characteristics, greatly improving the diagnostic performance, which has been proven both in our internal validation and external test datasets. Our results suggest that the application of this model may provide a more accurate and reproducible imaging evaluation for early diagnosis of endometrial cancer. Furthermore, our model holds great potential as a powerful assistant for ultrasound doctors. Comparing the efficacy of radiologists in interpreting ultrasound images for endometrial cancer diagnosis before and after the DL fusion model assistance, we find that the model significantly helps enhance the diagnostic accuracy and sensitivity of radiologists. Accordingly, it can serve as a tool to provide guidance for ultrasound doctors in practical clinical work, especially for those with less experience. Besides, traditional ultrasound diagnostic techniques largely depend on the specialized knowledge and visual assessments of radiologists, requiring their focused attention. Less experienced radiologists may face challenges in diagnostic efficiency and accuracy due to their limited experience. The DL algorithm in this study can promptly provide diagnoses based on gathered multimodal images, thereby reducing the cognitive burden and time spent on uncertain diagnoses. This process alleviates the cognitive demands on radiologists and aids doctors in conducting diagnoses with enhanced efficiency and accuracy. This implies that in underserved remote areas with limited healthcare resources, the utilization of the DL fusion model may help meet medical needs. Currently, artificial intelligence-based radiomics models in endometrial cancer are primarily used for risk stratification assessment or survival prognosis prediction, such as the identification of deep myometrial invasion and lymphovascular space invasion, and the prediction of pelvic lymph node metastasis and 5-year progression-free survival (PFS) [ 12 , 27 , 30 – 34 ]. To the best of our knowledge, the application of radiomics models in ultrasound diagnosis of endometrial cancer is still scarce. In addition, the majority of studies on imaging diagnostic models focus exclusively on single imaging modalities. Indeed, the integration of different imaging modalities through multimodal fusion techniques can offer a more comprehensive range of imaging features compared with single-modality approaches. Previous studies investigating imaging diagnostic models for various types of cancer consistently indicated that multimodal fusion models demonstrated superior diagnostic performance [ 35 – 37 ]. DL models are susceptible to overfitting during training, and discrepancies frequently exist between external test sets and internal validation sets in terms of ultrasound machines, ultrasound image acquisition time, and image clarity. In contrast, the three-modal fusion model employed in this study exhibits minimal differences in diagnostic performance between the external test set and internal validation set, highlighting the robustness of the multi-modal DL fusion model. As the first DL prediction model based on multi-modal transvaginal ultrasound scan images for endometrial cancer, our work fills a gap in the field of early diagnosis of endometrial cancer with artificial intelligence technology. This study has certain limitations. Firstly, its retrospective design limits the strength of the conclusions drawn, and a prospective validation would provide more robust evidence. Secondly, due to the difficulties in collecting images from all three modalities along with comprehensive clinical information, our sample size is relatively small. Thirdly, the limited size of the external test set prevents us from validating the diagnostic performance of the DL model on a larger sample size. Fourthly, manual segmentation was performed by one reader. The segmented mask can significantly affect the results of the model, but inter-reader variability cannot be assessed because of single annotator. Lastly, clinical parameters were not included in the training process of DL model, leading to a limitation in the improvement of the diagnostic performance. DL models that integrate multimodal ultrasound imaging with clinical characteristics for the diagnosis of endometrial cancer warrant further development.

Our research endeavors to develop a DL model that leverages the fusion of three modalities, including gray scale, color Doppler, and 3D reconstruction, to classify benign and malignant endometrial diseases accurately. Our results demonstrate that the model exhibits superior diagnostic performance compared with the average proficiency of ultrasound doctors in both internal and external datasets, and showcases its potential to assist ultrasound radiologists in improving overall diagnostic accuracy and making treatment decisions.

Below is the link to the electronic supplementary material. Supplementary Material 1 Supplementary Material 1