Multimodal Deep Learning Based on Ultrasound Images and Clinical Data for Better Ovarian Cancer Diagnosis.

This study aimed to develop and validate a multimodal deep learning model that leverages 2D grayscale ultrasound (US) images alongside readily available clinical data to improve diagnostic performance for ovarian cancer (OC). A retrospective analysis was conducted involving 1899 patients who underwent preoperative US examinations and subsequent surgeries for adnexal masses between 2019 and 2024. A multimodal deep learning model was constructed for OC diagnosis and extracting US morphological features from the images. The model’s performance was evaluated using metrics such as receiver operating characteristic (ROC) curves, accuracy, and F1 score. The multimodal deep learning model exhibited superior performance compared to the image-only model, achieving areas under the curves (AUCs) of 0.9393 (95% CI 0.9139–0.9648) and 0.9317 (95% CI 0.9062–0.9573) in the internal and external test sets, respectively. The model significantly improved the AUCs for OC diagnosis by radiologists and enhanced inter-reader agreement. Regarding US morphological feature extraction, the model demonstrated robust performance, attaining accuracies of 86.34% and 85.62% in the internal and external test sets, respectively. Multimodal deep learning has the potential to enhance the diagnostic accuracy and consistency of radiologists in identifying OC. The model’s effective feature extraction from ultrasound images underscores the capability of multimodal deep learning to automate the generation of structured ultrasound reports. Similar content being viewed by others Data Availability The data that support the findings of this study were available upon request from the corresponding author. The data were not publicly available due to privacy or ethical restrictions. Abbreviations - ACR : - American College of Radiology - ADNEX : - Assessment of Different Neoplasia in the adnexa - AUC : - Area under the ROC curve - BERT : - Bidirectional Encoder Representations from Transformers - BMI : - Body mass index - CA125 : - Cancer antigen 125 - ChatGPT : - Chat Generative Pre-Trained Transformer - CI : - Confidence interval - CNN : - Convolutional neural network - CT : - Computed tomography - DL : - Deep learning - Grad-CAM : - Gradient-weighted class activation mapping - ICC : - Intraclass correlation coefficients - IOTA SR : - International Ovarian Tumor Analysis Simple Rules - MMoE : - Multigate mixture of experts - MRI : - Magnetic resonance imaging - OC : - Ovarian cancer - O-RADS : - Ovarian-Adnexal Reporting and Data System - ResNet 50 : - Residual Network 50 - ROC : - Receiver operating characteristic - SITA : - Soft image-text alignment - US : - Ultrasound

Bray F, Laversanne M, Sung H, Ferlay J, Siegel RL, Soerjomataram I, et al. Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin 74:229-263, 2024 Xing L, Wang Z, Feng Y, Luo H, Dai G, Sang L, et al. The biological roles of CD47 in ovarian cancer progression. Cancer Immunol Immunother 73:145, 2024 Mansour S, Hamed S, Kamal R. Spectrum of Ovarian Incidentalomas: Diagnosis and Management. Br J Radiol 96:20211325, 2023 Vilendecic Z, Radojevic M, Stefanovic K, Dotlic J, Likic Ladjevic I, Dugalic S, et al. Accuracy of IOTA Simple Rules, IOTA ADNEX Model, RMI, and Subjective Assessment for Preoperative Adnexal Mass Evaluation: The Experience of a Tertiary Care Referral Hospital. Gynecol Obstet Invest 88:116-122, 2023 Timmerman D, Testa AC, Bourne T, Ameye L, Jurkovic D, Van Holsbeke C, et al. Simple ultrasound-based rules for the diagnosis of ovarian cancer. Ultrasound Obstet Gynecol 31:681-690, 2008 Van Calster B, Van Hoorde K, Valentin L, Testa AC, Fischerova D, Van Holsbeke C, et al. Evaluating the risk of ovarian cancer before surgery using the ADNEX model to differentiate between benign, borderline, early and advanced stage invasive, and secondary metastatic tumours: prospective multicentre diagnostic study. BMJ 349:g5920, 2014 Andreotti RF, Timmerman D, Strachowski LM, Froyman W, Benacerraf BR, Bennett GL, et al. O-RADS US Risk Stratification and Management System: A Consensus Guideline from the ACR Ovarian-Adnexal Reporting and Data System Committee. Radiology 294:168-185, 2020 Wu M, Cai S, Zhu L, Yang D, Huang S, Huang X, et al. Diagnostic performance of a modified O-RADS classification system for adnexal lesions incorporating clinical features. Abdom Radiol (NY), 2024 Xie W, Lin W, Li P, Lai H, Wang Z, Liu P, et al. Developing a deep learning model for predicting ovarian cancer in Ovarian-Adnexal Reporting and Data System Ultrasound (O-RADS US) Category 4 lesions: A multicenter study. J Cancer Res Clin Oncol 150:346, 2024 Shen L, Sadowski EA, Gupta A, Maturen KE, Patel-Lippmann KK, Zafar HM, et al. The Ovarian-Adnexal Reporting and Data System (O-RADS) US Score Effect on Surgical Resection Rate. Radiology 313:e240044, 2024 Vo TQN, Tran DT, Nguyen TTN, Vo VD, Le MT, Nguyen VQH. Diagnostic performances of the Ovarian Adnexal Reporting and Data System, the Risk of Ovarian Malignancy Algorithm, and the Copenhagen Index in the preoperative prediction of ovarian cancer: a prospective cohort study. J Gynecol Oncol, 2024 Dan Q, Xu Z, Burrows H, Bissram J, Stringer JSA, Li Y. Diagnostic performance of deep learning in ultrasound diagnosis of breast cancer: a systematic review. NPJ Precis Oncol 8:21, 2024 Shao J, Feng J, Li J, Liang S, Li W, Wang C. Novel tools for early diagnosis and precision treatment based on artificial intelligence. Chin Med J Pulm Crit Care Med 1:148-160, 2023 Chen H, Yang BW, Qian L, Meng YS, Bai XH, Hong XW, et al. Deep Learning Prediction of Ovarian Malignancy at US Compared with O-RADS and Expert Assessment. Radiology 304:106-113, 2022 Gao Y, Zeng S, Xu X, Li H, Yao S, Song K, et al. Deep learning-enabled pelvic ultrasound images for accurate diagnosis of ovarian cancer in China: a retrospective, multicentre, diagnostic study. Lancet Digit Health 4:e179-e187, 2022 Stahlschmidt SR, Ulfenborg B, Synnergren J. Multimodal deep learning for biomedical data fusion: a review. Brief Bioinform 23, 2022 Gao Z, Liu X, Kang Y, Hu P, Zhang X, Yan W, et al. Improving the Prognostic Evaluation Precision of Hospital Outcomes for Heart Failure Using Admission Notes and Clinical Tabular Data: Multimodal Deep Learning Model. J Med Internet Res 26:e54363, 2024 Volinsky-Fremond S, Horeweg N, Andani S, Barkey Wolf J, Lafarge MW, de Kroon CD, et al. Prediction of recurrence risk in endometrial cancer with multimodal deep learning. Nat Med 30:1962-1973, 2024 Wu Y, Fan L, Shao H, Li J, Yin W, Yin J, et al. Evaluation of a novel ensemble model for preoperative ovarian cancer diagnosis: Clinical factors, O-RADS, and deep learning radiomics. Transl Oncol 54:102335, 2025 Xiang H, Xiao Y, Li F, Li C, Liu L, Deng T, et al. Development and validation of an interpretable model integrating multimodal information for improving ovarian cancer diagnosis. Nat Commun 15:2681, 2024 Strachowski LM, Jha P, Phillips CH, Blanchette Porter MM, Froyman W, Glanc P, et al. O-RADS US v2022: An Update from the American College of Radiology's Ovarian-Adnexal Reporting and Data System US Committee. Radiology 308:e230685, 2023 Liu F, Zhu T, Wu X, Yang B, You C, Wang C, et al. A medical multimodal large language model for future pandemics. NPJ Digit Med 6:226, 2023 He K, Zhang X, Ren S, Sun JJI. Deep Residual Learning for Image Recognition. 2016 Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol 6 long and short papers: Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT 2019), 2–7 June 2019, Minneapolis, Minnesota, USA, 2019. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 33:159-174, 1977 Talaat FM, El-Sappagh S, Alnowaiser K, Hassan E. Improved prostate cancer diagnosis using a modified ResNet50-based deep learning architecture. BMC Med Inform Decis Mak 24:23, 2024 Zhang Y, Liu YL, Nie K, Zhou J, Chen Z, Chen JH, et al. Deep Learning-based Automatic Diagnosis of Breast Cancer on MRI Using Mask R-CNN for Detection Followed by ResNet50 for Classification. Acad Radiol 30 Suppl 2:S161-S171, 2023 Afshar M, Gao Y, Gupta D, Croxford E, Demner-Fushman D. On the role of the UMLS in supporting diagnosis generation proposed by Large Language Models. J Biomed Inform 157:104707, 2024 Lu Q, Wen A, Nguyen T, Liu H. Enhancing Clinical Relevance of Pretrained Language Models Through Integration of External Knowledge: Case Study on Cardiovascular Diagnosis From Electronic Health Records. JMIR AI 3:e56932, 2024 Wang H, Zhu J, Zou D, Rao Q, Han L, Lu H, et al. Multicenter study of ovarian cancer score for diagnosing ovarian cancer. Gynecol Oncol 193:58-64, 2025 Wang Z, Luo S, Chen J, Jiao Y, Cui C, Shi S, et al. Multi-modality deep learning model reaches high prediction accuracy in the diagnosis of ovarian cancer. iScience 27:109403, 2024 Luo L, Chen H, Xiao Y, Zhou Y, Wang X, Vardhanabhuti V, et al. Rethinking Annotation Granularity for Overcoming Shortcuts in Deep Learning-based Radiograph Diagnosis: A Multicenter Study. Radiol Artif Intell 4:e210299, 2022 Sadeghi MH, Sina S, Omidi H, Farshchitabrizi AH, Alavi M. Deep learning in ovarian cancer diagnosis: a comprehensive review of various imaging modalities. Pol J Radiol 89:e30-e48, 2024 Zech JR, Badgeley MA, Liu M, Costa AB, Titano JJ, Oermann EK. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study. PLoS Med 15:e1002683, 2018 Phillips CH, Patel-Lippmann K, Huang J, Strachowski LM, Maturen KE. Ovarian-Adnexal Reporting and Data System Ultrasound v2022: From Origin to Everyday Use. Radiol Clin North Am 63:29-44, 2025 Tawfik DS, Profit J, Morgenthaler TI, Satele DV, Sinsky CA, Dyrbye LN, et al. Physician Burnout, Well-being, and Work Unit Safety Grades in Relationship to Reported Medical Errors. Mayo Clin Proc 93:1571-1580, 2018 Wu SH, Tong WJ, Li MD, Hu HT, Lu XZ, Huang ZR, et al. Collaborative Enhancement of Consistency and Accuracy in US Diagnosis of Thyroid Nodules Using Large Language Models. Radiology 310:e232255, 2024 Jiang H, Xia S, Yang Y, Xu J, Hua Q, Mei Z, et al. Transforming free-text radiology reports into structured reports using ChatGPT: A study on thyroid ultrasonography. Eur J Radiol 175:111458, 2024 Liu C, Wei M, Qin Y, Zhang M, Jiang H, Xu J, et al. Harnessing Large Language Models for Structured Reporting in Breast Ultrasound: A Comparative Study of Open AI (GPT-4.0) and Microsoft Bing (GPT-4). Ultrasound Med Biol 50:1697–1703, 2024 Frosolini A, Catarzi L, Benedetti S, Latini L, Chisci G, Franz L, et al. The Role of Large Language Models (LLMs) in Providing Triage for Maxillofacial Trauma Cases: A Preliminary Study. Diagnostics (Basel) 14, 2024

The authors thank all participants who made valuable contributions to this study, including all the patients and experts of the Department of Ultrasound, Fourth Affiliated Hospital of Harbin Medical University. The authors thank citexs (www.citexs.com) for English language editing. Funding This work was supported by the Natural Science Foundation of Heilongjiang Province, China (Grant No. LH2022H033) and the Research Program Da ‘ai Longjing Charity Foundation of Heilongjiang Province (Grant No. HX2020-20). Author information Authors and Affiliations Contributions All authors contributed to the study conception and design. Material preparation and data collection were performed by Chang Su, Kuo Miao, and Xiaoqiu Dong. Data analysis were performed by Liwei Zhang, Xuemei Yu, Zhiyao Guo, Daoshuang Li, Mingda Xu, and Qiming Zhang. The first draft of the manuscript was written by Chang Su, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript. Corresponding author Ethics declarations Ethical Approval This study was performed in line with the principles of the Declaration of Helsinki. Approval was granted by the Ethics Committee of the Fourth Affiliated Hospital of Harbin Medical University. Consent to Participate Written informed consent was waived by the Institutional Review Board of the Fourth Affiliated Hospital of Harbin Medical University. Consent for Publication The authors affirm that human research participants provided informed consent for publication of the images in Figs. 4 and 6. Competing interests The authors declare no competing interests. Additional information Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Supplementary Information Below is the link to the electronic supplementary material. Rights and permissions Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law. About this article Cite this article Su, C., Miao, K., Zhang, L. et al. Multimodal Deep Learning Based on Ultrasound Images and Clinical Data for Better Ovarian Cancer Diagnosis. J Digit Imaging. Inform. med. 39, 1168–1180 (2026). https://doi.org/10.1007/s10278-025-01566-8 Received: Revised: Accepted: Published: Version of record: Issue date: DOI: https://doi.org/10.1007/s10278-025-01566-8