Development of a novel multimodal deep learning approach to improve diagnostic precision in ovarian cancer.

Table 2 presents the subject-level classification performance. Among traditional CNN models, SENet achieved the best performance with 78.33% accuracy, 74.49% sensitivity, 81.90% specificity, and 0.83 AUC. ResNet50 showed comparable results (77.83% accuracy, 0.83 AUC). Vision Transformer models significantly underperformed, with standard ViT achieving only 59.11% accuracy and Swin Transformer showing 63.05% accuracy. Subject-level classification between benign and malignant tumors. The Ensemble model approach combining Swin Transformer and DenseNet-121 achieved 75.86% accuracy with balanced sensitivity 72.73% and specificity 78.85%. However, the multimodal model incorporating Bio-Clinical BERT demonstrated the best performance, achieving 81.77% accuracy, 79.59% sensitivity, 83.81% specificity, and the highest AUC of 0.88. The backward selection analysis revealed the substantial contribution of textual features to diagnostic performance and demonstrated the synergistic value of different report components ( Table 3 ). When evaluated independently, the Bio-Clinical BERT model processing only textual information achieved limited performance with 53.90% (95% CI: 47.08%, 60.69%) accuracy, 52.58% (95% CI: 43.25%, 62.63%) sensitivity, 54.81% (95% CI: 45.24%, 64.03%) specificity, and 0.54 (95% CI: 0.46, 0.62) AUC, indicating that text alone was insufficient for reliable classification. However, the progressive integration of textual components with the hybrid image model (Swin Transformer + DenseNet-121) showed marked performance improvements. The baseline hybrid model achieved 76.24% (95% CI: 69.91%, 81.58%) accuracy with 73.47% (95% CI: 63.96%, 81.20%) sensitivity, 78.85% (95% CI: 70.04%, 85.59%) specificity, and 0.83 (95% CI: 0.77, 0.89) AUC. Adding structured data components (patient age and ultrasound route) enhanced performance to 79.21% (95% CI: 72.09%, 84.23%) accuracy, 75.26% (95% CI: 65.82%, 82.77%) sensitivity, 82.86% (95% CI: 74.52%, 88.87%) specificity, and 0.87 (95% CI: 0.82, 0.91) AUC, demonstrating the diagnostic value of basic clinical parameters. Subject-level backward selection for textual variables. The analysis of individual textual components revealed distinct contributions from different report sections. When incorporating uterine findings (Part A) along with age and route, the model achieved 80.20% (95% CI: 74.16%, 85.11%) accuracy, 74.23% (95% CI: 64.72%, 81.89%) sensitivity, 85.71% (95% CI: 77.76%, 91.15%) specificity, and 0.88 (95% CI: 0.82, 0.93) AUC. Alternatively, including ovarian tumor descriptions (Part B) with structured data resulted in 82.18% (95% CI: 76.31%, 86.84%) accuracy, 80.41% (95% CI: 71.42%, 87.01%) sensitivity, 83.81% (95% CI: 75.58%, 89.63%) specificity, and 0.88 (95% CI: 0.83, 0.93) AUC, suggesting that ovarian-specific descriptions provided more direct diagnostic value than uterine findings. The complete multimodal model incorporating all textual components (age, route, uterine findings, and ovarian descriptions) achieved optimal performance with 81.77% (95% CI: 75.89%, 86.48%) accuracy, 79.59% (95% CI: 70.57%, 86.38%) sensitivity, 83.81% (95% CI: 75.59%, 89.64%) specificity, and 0.88 (95% CI: 0.83, 0.93) AUC. Notably, the combined model’s performance was intermediate between the individual component models, indicating that while both uterine and ovarian descriptions contribute valuable information, their combined effect demonstrates complementary rather than purely additive diagnostic value. The incremental value of textual features is provided in Table 3 . Notably, to facilitate better interpretation and visualization of the images analyzed by our proposed model, Grad-CAM examples are presented in the supplementary data (Fig. S1 ). Furthermore, we assessed the clinical interpretation of the text used in the prediction model ( Table S2 ). To evaluate the generalizability of our proposed model, we conducted an external validation study using a new dataset of 280 cases from 3 independent medical centers. After excluding 12 cases due to corrupted images or unreadable reports, we evaluated the model on a final external cohort of 268 effective cases (133 benign and 135 malignant). Our model achieved excellent diagnostic performance on this unseen data, with an accuracy of 88.81%, sensitivity of 0.926, and specificity of 0.850. In general, our prediction model works well in the external samples collected from different medical centers, suggesting the integration of clinical text with ultrasound images provides rich contextual information that helps the model adapt to the variations in imaging equipment and reporting styles found in different clinical settings. Lastly, we compared the prediction performances of our proposed model with that from the IOTA scores. Analyzing the same samples for the external cohort, the performance metrics for the IOTA scores were: sensitivity 0.741, specificity 0.986, and accuracy 0.864. Notably, our integrated model demonstrated superior sensitivity and overall accuracy compared to the IOTA system, although the IOTA scores yielded higher specificity. In the context of ovarian cancer diagnosis, the primary clinical objective is to identify high-risk individuals; therefore, we consider sensitivity to be the more critical metric to minimize missed diagnoses. Given our model’s higher sensitivity and improved overall accuracy, we believe that integrating clinical text with imaging data provides a more robust classification tool for ovarian cancer.

At two tertiary referring centers, retrospective review of women surgically treated for ovarian tumors at the National Taiwan University Hospital (NTUH) system (Main Hospital and Hsinchu Branch) from 1 January 2011 to 31 December 2021 was conducted. The study included 1062 patients and 1342 ultrasound images. Baseline characteristics of subject-level and image-level data are provided in Table 1 . The average age of our study population was 43.26 years. Based on the ground truth diagnosis (i.e. the diagnosis of the examining clinician), 612 (57.63%) patients in this study were classified as the benign tumor group, and 450 (42.37%) patients were classified as the malignant (borderline) group. To evaluate the generalizability of our prediction model, we collected an independent validation cohort of 280 samples from 2 external institutions: National Taiwan University Hospital (NTUH), Hsinchu Biomedical Science Park Branch and the National Taiwan University Cancer Center. Characteristics of study participants from the NTUH database. a inflammatory cyst * 8, functional cyst * 5, fibroma/fibrothecoma * 9, rare tumors * 5. b germ cell tumor * 6, rare tumors * . Patients were selected for this study if they had a preliminary diagnosis of a benign or malignant ovarian tumor before surgery, underwent either transvaginal or transabdominal ultrasound within 1 month preceding the operation with images available for review, and had both their surgery and pathological confirmation performed at the NTUH system. The most representative ultrasound images that corresponded to the pathological findings were selected by two senior gynecologists. If more than one typical tumor feature was present, all representative images were retained. The image data was available for review in Digital Imaging and Communications in Medicine (DICOM) format, which is an international standard format and communication protocol specifically designed for medical imaging. We designed a clinical text preprocessing pipeline to extract semantically meaningful feature representations from ultrasound reports while ensuring data privacy and experimental reproducibility. The processed clinical features included three categories: ultrasound examination route, patient age, and free-text diagnostic reports. Categorical variable processing: the ultrasound examination route was transformed using one-hot encoding. Missing values were imputed with the label “Unknown” to maintain data integrity. Numerical variable processing: patient age underwent z-score normalization to ensure comparable scales across feature dimensions. Text feature extraction: free-text ultrasound reports (REPORT_TEXT column) were processed using the Bio_ClinicalBERT model. The preprocessing workflow included the following four steps. Handling missing data: null reports were replaced with empty strings. Tokenization: text was tokenized using the Bio_ClinicalBERT tokenizer with a maximum sequence length set to 128 tokens. Standardization: padding and truncation were applied to standardize input lengths across all samples. Feature embedding: we extracted the 768-dimensional embedding vector of the [CLS] token from the final hidden layer to serve as the global semantic representation for each report. The textual information was categorized into two parts, including structured and unstructured data. The structured data included patient age, recorded as a numerical feature representing the patient’s age in years at the time of ultrasound examination, and ultrasound route, which was categorically encoded to distinguish between transabdominal and transvaginal imaging approaches. The unstructured data consisted of free-text narrative reports authored by clinical fellows within 24 hours following each ultrasound examination. These clinical reports typically encompassed two main components: findings describing the uterus position, measured size dimensions, and endometrial characteristics (referred to below as Part A); and information related to the ovaries (Part B), including detailed morphological characteristics of identified ovarian masses, including location, size, composition (cystic, solid, or complex), wall characteristics (thin, thick, or irregular), internal content (clear, echogenic, septations, or papillary projections), and vascular assessment using Doppler evaluation. Each report concluded with an impression statement containing the interpreting fellow’s summary or differential diagnosis, providing valuable clinical context for the multimodal analysis. The dataset was divided into training, validation, and testing sets using a subject-level stratified approach to ensure robust model evaluation and prevent the data leakage inherent in multimodal architectures. A critical consideration in our study design was that individual patients could have multiple ultrasound images acquired during a single examination session, but all images from the same patient shared a single comprehensive ultrasound report. This one-to-many relationship between text reports and images necessitated subject-level splitting to avoid data leakage that could occur in our multimodal framework if different images from the same patient appeared in both training and testing sets while sharing identical textual variables. The dataset splitting process followed a two-stage hierarchical approach to address this multimodal data structure ( Fig. 1 ). This subject-level splitting approach ensured that all images from a given subject, along with their shared ultrasound report, remained within the same subset (training, validation, or testing), preventing the model from potentially memorizing subject-specific characteristics or exploiting shared textual information that could lead to overly optimistic performance estimates and compromised model generalizability. Flowchart of the subject-level data splitting strategy. The dataset was first split by patient ID into a held-out test set (15%) and a development set (85%). The development set was further utilized for five-fold stratified cross-validation. This rigorous separation ensures that no patient’s data appears in both training and testing phases simultaneously. To validate the effectiveness of our splitting methodology in preventing data leakage, we designed a control experiment evaluating the impact of overlap rates between training and testing sets. The experiment involved creating 11 different dataset configurations with overlap rates ranging from 0% to 100%, where the overlap rate represented the proportion of patients appearing in both training and testing sets. When traditional image-level splitting was employed, regression analysis revealed a significant positive correlation between overlap rate and model performance (coefficient: 0.12, 95% CI: 0.02–0.23, P  = .0251), indicating artificially inflated performance metrics due to data leakage. In contrast, subject-level splitting showed no significant correlation between overlap rate and performance (coefficient: −0.02, 95% CI: −0.16–0.12, P  = .777), confirming the robustness of our methodology against data leakage and validating our approach for reliable multimodal model evaluation. To extract the effective diagnostic region of interest (ROI) from original DICOM images and eliminate potential interference, we established a standardized preprocessing pipeline consisting of the following five steps. Image binarization: images were first converted to grayscale. We then applied Otsu’s algorithm to automatically determine the optimal segmentation threshold, effectively separating the ultrasound imaging region from the background. Morphological refinement: a morphological closing operation followed by an opening operation was performed using a 5 × 5 kernel. This step aimed to repair ROI edge discontinuities caused by acoustic shadowing and to remove isolated speckle noise from the background. ROI localization: we calculated the connected components of the binary mask and selected the largest component as the primary imaging region, thereby excluding text labels and artifacts. Padding and normalization: a bounding box was generated based on the largest connected component and expanded outwards by 15 pixels to prevent the loss of edge information. Aspect-ratio-preserving resize: finally, images were resized to a resolution of 224 × 224 using zero-padding (letterboxing). This ensured that the anatomical aspect ratios remained consistent across all samples input into the neural network, preventing geometric distortion from affecting diagnostic features. A series of data augmentation strategies were implemented on the training dataset to enhance the model’s generalization capabilities and simulate the variability inherent in real-world ultrasound imaging ( Fig. 2 ). Initially, to standardize input and focus on the lesion area, potential ROIs were automatically localized using a threshold-based segmentation approach. Given the heterogeneity in original image dimensions, these ROIs were uniformly resized to 224 × 224 pixels while preserving their original aspect ratio. Subsequently, augmentation techniques specific to ultrasound imaging characteristics were introduced: (i) simulation of speckle noise, a typical multiplicative noise pattern inherent to the physics of ultrasound imaging; (ii) simulation of acoustic shadowing, representing signal loss artifacts that occur when specific tissues impede sound waves; and (iii) simulation of posterior acoustic enhancement, mimicking the increased signal intensity behind structures with low attenuation. Representative ultrasound images before and after augmentation. The panel illustrates four distinct histological categories: benign, endometrioma, borderline, and malignant. Notably, the augmented images exhibit enhanced contrast and brightness. Furthermore, image enhancement and standardization techniques addressed issues such as insufficient contrast and blurred details. Finally, standard geometric transformations were incorporated, including minor random translations, random scaling (with a scaling factor ranging from 0.95 to 1.05), and random rotations (within a range of −15 to +15 degrees). These augmentation methods collectively aim to expand the diversity of the training data and improve the model’s robustness to various imaging variations. Examples of ultrasound images before and after augmentation are provided in Fig. 2 . Our multimodal deep learning architecture integrates complementary visual and textual information through a sophisticated fusion framework that combines three specialized neural network branches for comprehensive ovarian tumor analysis. The model processes two primary inputs: standard 2D ultrasound images and their corresponding clinical text reports, which undergo specific preprocessing before feature extraction. For image processing, we implemented dual parallel branches to capture diverse visual representations from the ultrasound data. The first branch employs DenseNet-121, a convolutional neural network (CNN) characterized by dense connectivity patterns that introduce direct connections from any layer to all subsequent layers within dense blocks, enabling effective capture of fine-grained local features crucial for identifying subtle textural changes such as septations or papillary projections [ 23 ]. The DenseNet-121 model, pretrained on ImageNet, has its original fully connected classifier replaced with an identity mapping to extract a 1024-dimensional feature vector, which subsequently passes through dropout ( P  = .5) and batch normalization layers for regularization. The second image branch utilizes Swin Transformer, a hierarchical vision transformer that employs shifted window mechanisms to model long-range dependencies essential for understanding overall tumor morphology and spatial relationships [ 24 ]. The Swin Transformer model, also pretrained on ImageNet, generates a 768-dimensional feature vector after removing its classification head and applying dropout ( P  = .5) and batch normalization. For textual information processing, clinical reports are first processed using Bio-Clinical BERT, a specialized language model initialized by BioBERT and trained on MIMIC-III clinical notes to understand medical terminology and clinical reasoning patterns [ 25 ]. The resulting text embeddings are refined through a dedicated multilayer perceptron consisting of sequential transformations: linear projection from BERT output to 512 dimensions, layer normalization, Gaussian error linear unit (GELU) activation, dropout ( P  = .1), further reduction to 256 dimensions with identical processing layers, and final projection to a 128-dimensional representation. The feature fusion mechanism concatenates the processed outputs from all three branches—DenseNet (1024 dimensions), Swin Transformer (768 dimensions), and clinical text (128 dimensions)—into a unified 1920-dimensional feature vector. This concatenated representation undergoes final processing through a fusion multilayer perceptron that progressively reduces dimensionality from 1920 to 512 [with rectified linear unit (ReLU) activation and 0.5 dropout], then to 256 dimensions (with ReLU activation and 0.3 dropout), before the final linear classification layer maps the 256-dimensional fused features to output logits for binary classification. The entire architecture is trained end-to-end using focal loss and AdamW optimizer with a learning rate of 0.00005, weight decay of 0.0001, maximum 150 epochs, and early stopping patience of 30 epochs to prevent overfitting while optimizing diagnostic performance. The architecture of the multimodal model is shown in Fig. 3 . All codes used in this study are deposited in the following link: https://github.com/Keras870924/OV_Multimodal . Architecture of multimodal model. Schematic overview of the proposed multimodal deep learning architecture. The model consists of two parallel processing streams: (1) an image branch that utilizes two pretrained networks, Swin Transformer and DenseNet121 to extract diverse visual features from ultrasound images. (2) A text branch that employs Bio-Clinical BERT to encode clinical variables and report text. Features from all three networks are concatenated and passed through fully connected layers to generate the final classification. To evaluate the effectiveness of our multimodal approach, we systematically compared our proposed model against several baseline architectures, including traditional CNN models (ResNet-50, DenseNet-121, GoogleNet, and SENet), transformer-based models (ViT and Swin Transformer), and an ensemble model that combines DenseNet-121 and Swin Transformer features without textual information. This comparative analysis enabled us to quantify the performance improvements achieved through multimodal integration. To further investigate the contribution of different textual components, we employed a backward selection approach to evaluate the relative importance of various text features. This analysis systematically examined the impact of removing specific components of the ultrasound reports, including uterine findings (Part A) and ovarian tumor descriptions (Part B), to determine their individual and combined contributions to the final diagnostic performance. The backward selection methodology enables us to identify which textual variables provide the most valuable diagnostic information and validate the necessity of incorporating comprehensive clinical reports rather than simplified structured data alone. All computational experiments were executed on a workstation equipped with NVIDIA GPU(s) utilizing CUDA acceleration ( Table S1 ). The framework was implemented in Python using PyTorch for deep learning development. To ensure strict reproducibility and evaluate the robustness of our results, experiments were conducted using five randomly selected seed values: 42, 123, 456, 789, and 2025. We designed a multimodal fusion framework that integrates three distinct feature extraction branches: Image Branch 1 (Transformer-based): a Swin Transformer-Tiny (Swin-T) backbone, pretrained on ImageNet-1K, was employed to extract 768-dimensional global features. Image Branch 2 (CNN-based): a DenseNet-121 backbone, also pretrained on ImageNet-1K, was utilized to extract 1024-dimensional local dense features. Text Branch: the Bio_ClinicalBERT model was used to encode clinical reports into 768-dimensional semantic embeddings. For evaluating the performance of the proposed image classification model in the context of computer-assisted cancer diagnosis, we prioritized accuracy, sensitivity, specificity, and AUC due to their clinical relevance. These metrics provide crucial insights into the model’s diagnostic capabilities. The specific definitions are: Accuracy (ACC): the overall proportion of correct classifications. formula: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $\mathrm{ACC}=\left(\mathrm{TP}+\mathrm{TN}\right)/\left(\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}\right)$\end{document} Sensitivity (SE) / True Positive Rate: the proportion of actual positive cases correctly identified. formula: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $\mathrm{SE}=\mathrm{TP}/\left(\mathrm{TP}+\mathrm{FN}\right)$\end{document} Specificity (SP) / True Negative Rate: the proportion of actual negative cases correctly identified. formula: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $\mathrm{SP}=\mathrm{TN}/\left(\mathrm{TN}+\mathrm{FP}\right)$\end{document} AUC: it is derived from the receiver operating characteristic curve, which plots the sensitivity/true positive rate against the specificity/false positive rate at various threshold settings. formula: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} $\mathrm{AUC}={\int}_0^1 SE\left({SP}^{-1}(x)\right) dx$\end{document} TP, TN, FP, and FN denote the counts of true positives, true negatives, false positives, and false negatives, respectively.

In conclusion, this study demonstrated the considerable potential of multimodal deep learning to improve diagnostic accuracy for ovarian tumors. By offering clinicians a promising adjunctive tool, this approach holds the prospect of facilitating more accurate preoperative assessments, enabling more informed decision-making, and ultimately contributing to better patient outcomes. Traditional ultrasound diagnosis of ovarian cancer is highly operator-dependent. We developed a deep learning model that used both ultrasound images and text. This multimodal model showed 81.7% diagnostic accuracy.

This study introduces and evaluates a novel multimodal deep learning model that integrates features from ultrasound images with their corresponding textual interpretation reports to improve the diagnostic accuracy of ovarian tumors. Specifically, the model focuses on the critical task of differentiating benign lesions from malignant neoplasms, inclusive of borderline tumors. The proposed multimodal approach demonstrated robust performance, achieving an overall accuracy of 81.77% and an AUC of 0.88. Notably, this performance surpasses that observed with traditional operator-dependent diagnostic methods and unimodal deep learning architectures relying solely on either image or text data (discussed below). These findings suggest that the fusion of visual and semantic information offers synergistic benefits, providing a pathway towards more precise preoperative diagnoses. This advancement holds significant potential to substantially improve clinical decision-making processes and patient outcomes. Comparison of multimodal fusion strategies. During the development of our multimodal architecture, we initially explored more sophisticated alignment mechanisms between image and text features. Two advanced cross-modal attention architectures were designed and implemented: Basic Cross-Modal Attention and Bidirectional Cross-Modal Attention. The Basic Cross-Modal Attention mechanism employed a multi-head attention framework where text features serve as queries to attend to image features (keys and values), while Bidirectional Cross-Modal Attention includes both image-to-text attention and text-to-image attention. Despite theoretical advantages, both attention mechanisms resulted in severe overfitting during training. The main factors contributing to this phenomenon were parameter explosion and the cross-modal attention layers, which introduced substantial parameter overhead. The Basic Cross-Modal Attention model added additional parameters (image projection: 256 × 256, text projection: 128 × 256, three attention matrices: 256 × 256 each, output projection: 256 × 256, and gating mechanism: 128 × 1). The Bidirectional version nearly doubled these overhead parameters. This finding led us to adopt the simpler concatenation-based fusion approach, which proved more suitable for our dataset constraints while still achieving superior performance compared to unimodal baselines. We conducted an ablation study comparing our concatenation method with two sophisticated attention-based mechanisms: Cross-Attention and Bidirectional Attention. As shown in Table 4 , while attention mechanisms increased model capacity, they led to a noticeable decline in performance. These results indicate that for our specific clinical dataset, the increased parameter overhead of attention layers (adding ~1.15 M to 2.01 M parameters) led to overfitting. The simpler concatenation approach proved more robust, effectively capturing synergistic visual and textual features without introducing excessive variance. We further evaluated the backbone choice of image feature extractor in this study by replacing DenseNet-121 with ResNet-50. As shown in Table 5 , the results confirm that DenseNet-121 provides superior diagnostic precision for ovarian tumor classification. Performance comparison of image backbones. We have conducted a detailed subject-level ablation study on the independent testing set ( Table S3 ), which demonstrates the incremental performance gains provided by each augmentation layer. These strategies were specifically designed to mimic fundamental ultrasound physics; for instance, the simulation of Speckle Noise reflects the multiplicative noise inherent in backscattered sound waves, while Acoustic Shadowing and Enhancement represent common signal-loss or gain artifacts encountered during clinical examinations. Our results show that while baseline geometric transforms established a solid performance level, the inclusion of ultrasound-specific noise simulation (Row 4) was critical for increasing the AUC from 0.87 to 0.89. This improvement suggests that by exposing the model to these realistic physical artifacts during training, we enhance its ability to focus on robust diagnostic features even when faced with the image quality variability typical of real-world clinical data. The final model, incorporating all these strategies along with Cutmix, achieved the highest subject-level accuracy of 81.68%. Our multimodal deep learning model demonstrated robust performance, achieving an accuracy of 81.77% and an AUC of 0.88 on the internal testing set. This represents a significant improvement over our image-only ensemble baseline (accuracy: 75.86%, AUC: 0.83), confirming that the integration of clinical text provides critical diagnostic context. To further validate its clinical utility, we performed a head-to-head comparison with the IOTA Simple Rules on an external validation cohort. Our model achieved a superior sensitivity of 0.926 and an overall accuracy of 88.81%, compared to 0.741 and 86.44% for the IOTA system, respectively. While the IOTA system maintained higher specificity, our model’s high sensitivity is particularly advantageous for preoperative triage to ensure high-risk malignant cases are not missed. To provide a broader context for our findings, we have summarized a structured comparison of our model with recent state-of-the-art studies ( Table S4 ). While landmark studies, such as the multicenter study by Gao et al. (2022) [ 19 ], utilized massive datasets of over 500 000 images to achieve high performance, our multimodal framework reaches a comparable subject-level AUC of 0.88 using a significantly more focused dataset of 1342 images. This is achieved by effectively leveraging the semantic nuances within clinical narrative reports, which compensate for the smaller imaging sample size. Furthermore, unlike models that rely on structured clinical indicators such as serum markers (CA125) or menopausal status, our approach prioritizes data routinely available at the time of initial triage [ 26 ]. This design offers a data-efficient and practical pathway for specialized centers to improve preoperative diagnostic accuracy during early-stage assessment. A recent study utilized multi-omics data for cancer subtype classification using a graph theory-based model. Their results demonstrated that this model outperformed other approaches, including conventional statistical models and machine learning methods, across 10 different The Cancer Genome Atlas (TCGA) cancer types. However, a significant limitation of this model is the strict requirement for complete omics data without missing variables, a condition is often difficult to achieve in real-world clinical settings [ 27 ]. Similarly, a study on hypopharyngeal cancer demonstrated that a multimodal approach—concurrently analyzing radiomics features and clinical variables—outperformed traditional imaging-only methods [ 28 ]. These results corroborate our conclusion that integrating imaging and clinical data yields superior predictive performance compared to models relying on a single data source. In addition, a chest X-ray study demonstrated that CNN models embedded in an automated server could efficiently identify pathological cases [ 29 ]. Crucially, these models were capable of multi-class classification, predicting multiple specific disease types rather than being restricted to binary outcomes. In conclusion, these recent studies collectively suggest that developing deep learning approaches using multimodal data is a promising and feasible strategy for advancing disease diagnosis and progression prediction. The choice of combining DenseNet-121 and Swin Transformer for image feature extraction was strategically motivated by their complementary strengths. DenseNet-121’s dense connectivity pattern excels at capturing fine-grained local features crucial for identifying subtle textural changes in ultrasound images, such as septations or papillary projections. Conversely, Swin Transformer’s hierarchical structure with shifted windows enables effective modeling of long-range dependencies, essential for understanding the overall tumor morphology and its relationship to surrounding structures. The poor performance of standalone Vision Transformer models (ViT: 59.00% accuracy, Swin: 62.87% accuracy) highlights the importance of sample size and the limitations of pure attention-based architectures in medical imaging. Unlike natural images, ultrasound images contain significant speckle noise and artifacts that may disrupt the self-attention mechanism. The hybrid approach of combining CNN and Transformer architectures leveraged the noise-robustness of convolutions while benefiting from the global context modeling of transformers. The integration of Bio-Clinical BERT for text processing proved particularly valuable, as evidenced by the incremental performance gains when adding textual features. The pretraining on MIMIC-III clinical notes enabled the model to understand medical terminology and clinical reasoning patterns. Our ablation studies revealed that both general uterine findings (Part A) and specific tumor descriptions (Part B) contributed synergistically to the final diagnosis, suggesting that the model learned to contextualize tumor characteristics within the broader clinical picture. The analysis of image characteristics from DICOM metadata revealed significant heterogeneity in equipment and imaging parameters across our dataset ( Table 6 ). DICOM data were sourced from more than six distinct manufacturers, encompassing six different equipment models and seven varying image dimensions. Despite this variability, the model maintained robust performance, suggesting good generalization to different ultrasound systems. This is particularly relevant for deployment in diverse clinical settings where standardization of imaging equipment may not be feasible. Characteristics of images from DICOM metadata. To provide a comprehensive evaluation of our model’s performance, we have included receiver operating characteristic (ROC) curves for the five-fold cross-validation, the final test set, and baseline comparisons in the Supplementary Data. Overall, these results demonstrate that our proposed model maintains robust performance across all evaluated scenarios. To provide a clearer interpretation, we performed a two-pronged interpretability analysis on the Bio-Clinical BERT component of our multimodal framework. The results are summarized in Figs S2 – S4 , which help to provide better understanding of the prediction performances of our proposed model. This study has several important limitations that should be considered when interpreting the results and planning future research. Model development and generalizability. While the primary development and training of our multimodal model utilized a retrospective dataset from a single tertiary health system (NTUH), we have mitigated potential selection bias by conducting external validation using a dataset from three independent medical centers. Despite the robust performance observed in this external cohort (accuracy 88.81%), the training phase still reflects the clinical practice patterns and reporting styles specific to our institution. Further prospective, multicenter studies remain essential to fully assess the model’s clinical utility across broader healthcare settings with diverse imaging protocols and patient demographics. Data distribution and robustness. The dataset exhibited a class distribution of ~57.63% benign versus 42.37% malignant cases. Although we addressed this using weighted random sampling and focal loss to prioritize the minority class, such imbalance might still influence the model’s performance on rare pathological subtypes. However, our follow-up analysis yielded a high area under the precision-recall curve (AUPR) of 86.45%, suggesting that the model maintains high precision and recall across classes despite the existing distribution. Operator dependency and image quality. Ultrasound imaging remains highly operator-dependent, and our study could not control for sonographer experience or imaging conditions. The model’s performance on suboptimal images or those acquired by less experienced operators remains uncertain. Additionally, we excluded cases without available images or with poor image quality, potentially biasing our results toward better quality examinations. Text report variability. The free-text ultrasound reports showed considerable variability in structure, completeness, and terminology across different reporting physicians. While Bio-Clinical BERT partially addressed this challenge, the model’s performance might degrade with reports using nonstandard terminology or incomplete descriptions. The absence of structured reporting templates during the study period limited standardization. Limited clinical data. Our model incorporated age and ultrasound reports but lacked other potentially valuable clinical information such as CA-125 levels, menopausal status, family history, or symptoms. These factors significantly influence clinical decision-making, and their absence may limit the model’s clinical utility. Integration of comprehensive clinical data could further improve diagnostic accuracy. We also explored the possibility to separate the borderline tumors into an independent category, and thus the prediction model has three outcomes, including benign, borderline, and malignant. The results are summarized in Table S5 and Fig.   S5 . While the overall accuracy expectedly decreased when moving from a binary to a more complex three-class task, the model maintained a robust macro AUC of 82.91% on the independent testing set. As shown in Fig. S5 , the model excels at identifying malignant (AUC = 0.871) and benign (AUC = 0.854) cases. While borderline tumors remain the most challenging to classify (AUC = 0.762). The confusion matrix indicates that the majority of misclassifications for borderline tumors occur between the borderline and benign categories, which mirrors the difficulty faced by clinicians in real-world practice. Borderline tumors often present with predominantly cystic morphology and relatively smooth inner walls, features that may closely resemble benign lesions. In addition, papillary projections or subtle solid components in borderline tumors can be small or indistinct, making them difficult to reliably identify on imaging alone. As a result, even experienced clinicians may encounter diagnostic uncertainty between benign and borderline tumors.

Ovarian cancer significantly impacts women worldwide, presenting challenges with high mortality rates and a pressing need for improved diagnostic tools [ 1 , 2 ]. In Taiwan, the incidence of ovarian cancer has been increasing annually, and it remains the leading cause of death among malignant gynecologic tumors in women [ 3 ]. According to the 2022 Annual Report of the Cancer Registry from the Health Promotion Administration, Ministry of Health and Welfare, Taiwan, ~1859 women were diagnosed with malignant tumors of the ovary, fallopian tube, and broad ligament, accounting for 26.6% of all female reproductive organ (ICD-O-3 C51-58) cases and causing a total of 765 deaths. Ovarian tumors in the pelvis must usually be larger than 10 cm to be felt in the abdomen, and the standard abdominal discomfort symptoms are often similar to gastrointestinal symptoms [ 4 ]. Hence, ovarian tumors are often challenging to detect at an early stage. The commonly used tumor serum marker CA-125 has had its diagnostic value questioned when differentiating between benign and malignant ovarian tumors [ 5 ]. This is because elevated levels of CA-125 can also occur in various benign gynecological conditions, such as uterine fibroids, pelvic inflammatory disease, pregnancy, and endometriosis. Nearly half of the ovarian cancer patients are diagnosed at the third stage or above, so the prognosis is generally poor. About 47.9% of patients have advanced ovarian cancer (stages III and IV) [ 6 ]. Although effective treatment is available at earlier stages, the reported recurrence rates are high [ 7 ], and the 5-year survival rate is ~30%–40% [ 8 ]. The current standard treatment options for ovarian cancer, including debulking surgery and adjuvant chemotherapy, are well established [ 9 , 10 ]. However, the therapeutic strategies for benign versus malignant ovarian tumors are significantly different. Treatment for benign ovarian tumors is mainly minimally invasive surgery, which only removes the tumor while preserving the normal ovarian cortex. This can preserve ovulation and fertility, which is especially important for young patients. In contrast, the current standard treatment for malignant ovarian tumors is tumor debulking surgery and adjuvant chemotherapy, which usually cannot preserve fertility. Misclassification before the surgery can lead to the undertreatment of malignancies or unnecessarily aggressive surgeries for benign conditions. Thus, improving the accuracy of preoperative diagnoses is an important clinical challenge. Medical imaging and tumor visualization are crucial in the diagnostic process, emphasizing the need for precision in these techniques. Ultrasonography can provide information about the tumor, such as its size, consistency, and nature, and is the most frequently used imaging method for evaluating ovarian tumors. Additionally, color Doppler sonography enhances differentiation by assessing the vascular patterns and blood flow within the tumors. However, subjective judgment among different ultrasound operators may influence the interpretation of ovarian tumor images, thereby affecting the treatment decisions made by clinicians. Therefore, there is a need to develop more precise and effective diagnostic techniques to enhance patient care and outcomes. Several global guidelines and scoring systems have been proposed to standardize ultrasound reporting and enhance the consistency of diagnosis. The International Ovarian Tumor Analysis (IOTA) group proposed the “Simple Rules,” which contain five benign and five malignant image features [ 11 ], and the Assessment of Different Neoplasia in the AdneXa (ADNEX) model ( https://www.evidencio.com/models/show/946 ) integrates ultrasound features with clinical factors such as age [ 12 ] and CA-125 levels for risk calculation of malignancy [ 13 ]. Furthermore, the American College of Radiology (ACR) provided the Ovarian-Adnexal Reporting and Data System (O-RADS) [ 14 ], which establishes a standardized risk stratification framework and management recommendations. Although these systems have significantly improved the objectivity and precision of ultrasound diagnoses, interpretation still relies on the operator’s experience, and some challenging cases are still hard to classify. Recent research has explored the application of deep learning technology for the diagnosis of ovarian tumors, utilizing a diverse range of medical imaging modalities. These include magnetic resonance imaging (MRI), which provides detailed soft tissue contrast [ 15 , 16 ]; computed tomography (CT), known for its cross-sectional views [ 17 ]; histopathological imaging, which examines tissue samples at the cellular level [ 18 ]; and ultrasound [ 19–22 ], a widely used technique for real-time imaging. For example, a multicenter study in China demonstrated that a deep convolutional neural network, trained on more than 500 000 ovarian cancer images, achieved an area under the curve (AUC) of 0.870 in an external dataset; this currently serves as the benchmark study [ 19 ]. Another ultrasound study using deep neural networks on ~3000 images demonstrated a sensitivity of 0.96—slightly higher than that of ultrasound experts—without compromising specificity [ 20 ]. Additionally, a separate study utilized texture analysis to extract features and subsequently developed prediction models by integrating support vector machine and random forest algorithms [ 21 ]. Overall, these deep learning models showed prediction performance comparable to that of human experts. Despite the potential of deep learning to improve diagnostic precision, its application to ovarian tumor ultrasound images for differential diagnosis remains in the realm of future work. The lack of large-scale, validated models that integrate multimodal clinical data, including ultrasound images and pathological outcomes, is a critical gap. While existing deep learning approaches for ovarian tumor classification predominantly rely on single-modal architectures using either imaging or clinical features independently, there remains a significant gap in understanding how clinical text data can enhance diagnostic accuracy when systematically integrated with ultrasound image features. Current models have not fully explored the synergistic potential of combining visual ultrasound characteristics with structured clinical information, such as patient demographics, symptomatology, and laboratory findings. Ultrasonography, as the most basic and commonly used method to distinguish benign and malignant ovarian tumors in clinical practice, presents an urgent need for exploration of such multimodal integration approaches. This gap motivates our investigation into multimodal frameworks that can leverage both ultrasound imaging and clinical text data synergistically. Therefore, this study aimed to develop a multimodal model that integrates ultrasound image and clinical text information to assist in diagnosing malignant ovarian tumors.