An integrated model incorporating deep learning, radiomics and clinical and US features to diagnose central lymph node metastasis in patients with papillary thyroid cancer

An integrated model incorporating deep learning, radiomics and clinical and US features to diagnose central lymph node metastasis in patients with papillary thyroid cancer | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article An integrated model incorporating deep learning, radiomics and clinical and US features to diagnose central lymph node metastasis in patients with papillary thyroid cancer Yang Gao, Weizhen Wang, Yuan Yang, Ziting Xu, Yue Lin, Ting Lang, and 5 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3218322/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 12 Jan, 2024 Read the published version in BMC Cancer → Version 1 posted 10 You are reading this latest preprint version Abstract Objective To evaluate the value of an integrated model incorporating deep learning (DL), radiomics and clinical and US imaging features for diagnosing central lymph node metastasis (CLNM) in patients with papillary thyroid cancer (PTC). Methods This retrospective study reviewed 613 patients with clinicopathologically confirmed PTC from two institutions. The DL model and radiomics model were developed using primary lesion images and then integrated with clinical and US features selected by multivariate analysis to generate an integrated model. The performance was compared with junior and senior radiologists on the independent test set. SHapley Additive exPlanations (SHAP) plot and Gradient-weighted Class Activation Mapping (Grad-CAM) were used for the visualized explanation of the model. Results The integrated model yielded the best performance with an AUC of 0.841. surpassing that of the radiomics model (0.706, p < 0.001) and the DL model (0.819, p = 0.26). Compared to junior and senior radiologists, the integrated model reduced the missed CLNM rate from 57.89% and 44.74–27.63%, and decreased the rate of unnecessary central lymph node dissection (CLND) from 29.87% and 27.27–18.18%, respectively. SHAP analysis revealed that the DL features played a primary role in the diagnosis of CLNM, while clinical and US features (such as extrathyroidal extension, tumor size, age, gender, and multifocality) provided additional support. Grad-CAM indicated that the model exhibited a stronger focus on thyroid capsule in patients with CLNM. Conclusion Integrated model can effectively decrease the incidence of missed CLNM and unnecessary CLND. The application of the integrated model can help improve the acceptance of AI-assisted US diagnosis among radiologists. Ultrasonography Papillary Thyroid Carcinoma Lymph Node Metastasis Deep Learning Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Introduction In recent years, the incidence of thyroid cancer has increased significantly worldwide, with papillary thyroid carcinoma (PTC) accounting for most cases [ 1 ]. PTC is characterized by early metastasis to cervical lymph nodes (LNs), particularly in the central region [ 2 ]. The reported rate of central LN metastasis (CLNM) in patients is approximately 50% [ 3 ], which is a known risk factor for recurrence and adversely affects overall survival [ 4 , 5 ]. The necessity of prophylactic central LN dissection (pCLND) remains a subject of debate in thyroid cancer treatment. In China, the latest guidelines recommend routine pCLND at least ipsilateral to the lesion [ 6 ]. While pCLND can effectively reduce the need for reoperation in cases of recurrence, it also leads to unnecessary CLND procedures. US is the most commonly used method for preoperative LN assessment in PTC [ 7 ]. However, its sensitivity in identifying CLNM ranges from only 26–47%, which is insufficient for accurate assessment [ 8 ]. Hence, a more sensitive preoperative assessment of CLNM is crucial for patients with PTC to reduce unnecessary CLND. In recent years, artificial intelligence (AI), especially deep learning (DL) and radiomics, have shown powerful analytical capabilities in extracting intricate and multi-layered features from medical images [ 9 , 10 ]. Radiomics focuses on extracting traditional image features for classification and prediction, whereas high-dimensional features generated by a DL model can represent deep image information of tumours [ 11 ]. We previously reported a preliminary small sample study of CLNM using radiomics, which acquired good performance [ 12 ]. To date, most studies have independently employed DL and radiomics features, and far fewer studies have attempted to fuse these two features from US images. It is worth noting that features extracted by DL models may be sensitive to global translation, rotation, and scaling while radiomics features such as intensity features are not [ 13 , 14 ], Therefore, we hypothesize that radiomics features and DL features extracted from US images could be complementary, and their combination may yield improved prediction outcomes. However, unlike radiologists who incorporate clinical and US information to make diagnoses, most AI models only provide output results without revealing their decision-making process. This lack of transparency is considered one of the reasons why radiologists are skeptical about the clinical application of AI models. Previous studies have highlighted the significance of clinical and US characteristics (e.g., age, gender, and tumor size) in distinguishing CLNM [ 15 ]. Nevertheless, the lack of information such as age and gender in the images, and data pre-processing such as resizing and normalisation, makes detecting these information challenging in machine learning [ 16 ]. By integrating clinical and US features into AI models, it may be possible to improve the predictive efficacy of the models as well as the acceptance from radiologists. Hence, this study aimed to develop and validate whether an integrated model incorporating DL, radiomics and clinical and US features can improve the performance to diagnose CLNM in patients with PTC, in order to reduce the miss rate of CLNM, unnecessary CLND and improve the acceptance of AI-assisted US diagnosis for radiologists. Patients and Methods Patients The Ethics Committees of Nanfang Hospital of South Medical University and the First People’s Hospital of Foshan (NFEC-202008-K6) approved this retrospective study. The requirement for informed consent was waived. The inclusion and exclusion criteria were as follows: Inclusion Criteria Patients were enrolled if they satisfied all the following inclusion criteria: (1) were confirmed to have PTC after lobectomy or total thyroidectomy; (2) underwent CLND with a pathological examination; (3) the thyroid US examination was performed at our hospital within one month before the operation; Exclusion Criteria (1) had other malignancies or distant metastases at diagnosis; (2) received preoperative head and neck therapies such as radiotherapy, chemotherapy, or radiofrequency ablation; (3) with missing data; (4) with poor image quality. After undergoing a rigorous inclusion and exclusion process, datasets of 613 patients treated in our clinical centres from March 2019 to July 2020 were included. The participant recruitment flow is shown in Fig. 1 . The participants were randomly divided into training and independent test cohorts for further analysis. Ultrasonography and image analysis The choice of US machine was not limited, and most data was obtained using devices such as Siemens Sequoia, Supersonic Aixplorer, and Toshiba Aplio 500, stored in the DICOM format. The risk factors for CLNM were identified by including the following variables: gender, age, and US features of thyroid tumors following the C-TIRADS and ATA guidelines [ 7 , 17 ]. These features encompassed tumour size, hypoechoic solid composition, multifocality, aspect ratio, posterior acoustic attenuation, tumour location, extrathyroidal extension (ETE), acoustic halo, microcalcification, and the internal tumour vascularity. Age was dichotomized at 55 years following the 8th American Joint Commission on Cancer staging system. In cases with multifocality, the largest nodule was chosen as the representative. The tumor vascularity was graded from 0 to 3 by colour Doppler flow imaging (CDFI) following the Adler standard [ 18 ]. The US features were re-evaluated by two radiologists with four and seven years of experience in thyroid US diagnosis. Both radiologists were blinded to clinical information and pathological diagnosis. The agreement between them was assessed, and in case of any disagreement, a senior radiologist with over 20 years of experience made the final decision. The preoperative examination of LNs was conducted on all patients by a team of five radiologists, comprising two senior radiologists with 15 and 17 years of experience, and three junior radiologists with 3, 5, and 6 years of experience, respectively. The diagnostic accuracy of the two groups of radiologists, differing in seniority, was determined by comparing the LNs status reported by US with the corresponding postoperative pathological results. Based on the ACR TI-RADS [ 19 ], LNs exhibiting one or more suspicious US features (roundness, loss of the normal echogenic hilum, internal microcalcifications, cystic changes, hyperechogenicity, or presence of peripheral flow) were classified as US-reported CLNM. Region of interest segmentation and Development of the radiomics model The manual segmentation of the regions of interest (ROIs) was performed by a radiologist with five years of experience, Subsequently, a second radiologist with seven years of experience reviewed the drawn ROIs. The final ROIs was determined after discussion when there was disagreement between the two readers. Radiomics features, including intensity, shape texture, wavelet features, and Laplacian of Gaussian features, were extracted from the US images by delineating the ROIs. A total of 783 features were extracted using the open-source python package “pyradiomics” [ 20 ]. The least absolute shrinkage and selection operator (LASSO) logistic regression analysis method was employed to select the radiomics feature on the training dataset. The support vector machine (SVM) algorithm was utilized to establish the prediction model, which was subsequently trained through 10-fold cross-validation in training set to optimize the hyperparameters. Development of the deep learning model 33% of the training images were randomly selected to form a validation cohort to guide the choice of hyperparameters. Supplemental Table S1 shows the result of four tested algorithms. The best-performing model (ResNet50) was adopted as the base model and pre-trained on ImageNet-21k [ 21 ]. Group normalization and weight standardization were used instead of batch normalization, following the tricks proposed for big data transfer [ 22 ]. Rectangular ROIs were cropped from the raw US images using a tumor segmentation mask. These ROIs were then resized to 224 × 224 pixels and normalized. During the training stage, the ROIs were fed into the network to update the model parameters. The cross-entropy of the outputs and labels as the loss function were used to alleviate the influence of class imbalance in differentiating CLNM and online data augmentation was used to avoid overfitting and sample imbalance. Development and explanation of the integrated prediction model In the training cohort, multivariate logistic regression analysis and likelihood ratio tests for positive selection were employed to identify clinical and US characteristics that could effectively differentiate the presence of CLNM. These selected factors were then integrated with the features extracted from the SVM and ResNet50 models to create the final prediction model. The flowchart outlining the integrated prediction model can be seen in Fig. 2 . To assess the performance of the integrated model, the model's performance was compared with that of the radiomics model, DL model, and junior and senior radiologists on independent test set. In addition, the visualized explanation methods named SHapley Additive exPlanations (SHAP) plot and Gradient-weighted Class Activation Mapping (Grad-CAM) were applied to improve the clinical explanation of our model. We used Grad-CAM to extract the areas of interest and generate saliency maps for the DL model, while the SHAP plot was used to calculate the contribution value of each variable to the integrated model. These visualization methods aim to improve the clinical understanding and explanation of our model's predictions. Statistical analysis Statistical analysis was performed using IBM SPSS Statistics for Windows, Version 20.0 (IBM Corp.). Categorical variables are presented as numbers and percentages and analyzed using the chi-squared or Fisher's exact test. The Mann-Whitney U test analyzed continuous variables, and Kappa statistics analyzed the inter-observer agreement. The performance of predictive models was evaluated by the receiver operating characteristic (ROC) curve analysis and the area under curve (AUC). DeLong's test compared the combined prediction model and other methods in predicting CLNM. Other performance measures, including accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV), were also assessed. The two-sided statistical significance was set at 0.05. Results Patient demographics and feature selection Patient characteristics and US features of the thyroid nodules in the training and test cohorts were shown in Table 1 . The training cohort included 460 patients (136 males, 324 females) with a mean age of 40.70 ± 11.16 years (range, 11–73 years). The independent test cohort included 153 patients (45 males and 108 females) with a mean age of 42.59 ± 11.33 years (range, 13–69 years). These two datasets were comparable as there were no significant differences. The inter-observer consistency was satisfactory, with Kappa coefficients between 0.82 and 0.92 ( Supplemental Table S2 ). Table 1 Demographic comparison between training and independent test cohorts Characteristics and US features Training cohort (n = 460) Independent test cohort (n = 153) p value Size (mean ± SD) 1.24 ± 0.92 1.16 ± 0.69 0.433 Sex 0.971 Male 136(75.1%) 45(24.9%) Female 324(75.0%) 108(25.0%) Age 0.484 ≤ 55 415(75.5%) 135(24.5%) ༞55 45(71.4%) 18(28.6%) Tumour location 0.711 Right lobe 228(73.8%) 81(26.2%) Left lobe 213(76.6%) 65(23.4%) Isthmus 19(73.1%) 7(26.9%) Tumour position 0.095 Upper 152(81.7%) 34(18.3%) Mid 164(71.9%) 64(28.1%) Lower 125(72.3%) 48(27.7%) Isthmus 19(73.1%) 7(26.9%) Solid composition with hypoechoic echo 0.110 Present 417(74.2%) 145(25.8%) Absent 43(84.3%) 8(15.7%) Tumour multifocality 0.769 Present 83(76.1%) 26(23.9%) Absent 377(74.8%) 127(25.2%) Aspect ratio 0.198 Present 201(726%) 76(27.4%) Absent 259(77.1%) 77(22.9%) Microcalcification 0.769 Present 360(75.3%) 118(24.7%) Absent 100(74.1%) 35(25.9%) Tumour vascularity 0.320 0–1 401(74.4%) 138(25.6%) 2–3 59(79.7%) 15(20.3%) Acoustic halo 0.804 Present 42(73.7%) 15(26.3%) Absent 418(75.2%) 138(24.8%) ETE 0.947 Present 119(74.8%) 40(25.2%) Absent 341(75.1%) 113(24.9%) Posterior acoustic attenuation 0.282 Present 67(79.8%) 17(20.2%) Absent 393(74.3%) 136(25.7%) CLNM in the pathology outcomes. 0.982 Present 228 (75.0%) 76 (25.0%) Absent 232 (75.1%) 77 (24.9%) Abbreviations: US, ultrasound; ETE, extrathyroidal extension; CLNM, central lymph node metastasis. To better understand the relationship between CLNM and clinical and US features, a multivariate analysis was performed in the training cohort. The results showed that age, sex, tumour size, multifocality, and ETE were independent risk factors for CLNM (Table 2 ). Table 2 Independent risk factors after multiple logistic regression analysis Ultrasound features β Odds ratio (95% CI) p value Prediction of CLNM status Size 0.533 1.704 (1.225–2.370) 0.002 Age Sex -1.301 -0.598 0.272 (0.125–0.595) 0.550 (0.353–0.857) 0.001 0.008 Tumour multifocality 0.738 2.092 (1.224–3.577) 0.007 ETE 1.418 4.130 (2.434–7.008) < 0.001 Abbreviations: CLNM, central lymph node metastasis; US, ultrasound; CI: confidence interval; ETE, extrathyroidal extension Diagnostic performance of CLNM-predicting model We successfully built a radiomics model, a DL model, and an integrated model. In the testing set, our result showed that the DL model exhibited higher sensitivity (75.00% vs 52.63%) but slightly lower specificity (71.43% vs 74.03%) compared to the radiomics model (Table 3 ). By combining radiomics, DL and clinical features, the integrated model showed good predictive efficacy (the specificity and sensitivity were 81.82% and 72.37%, and the PPV and NPV were 79.71% and 75.00%). Meanwhile, the integrated model had most outstanding performance with the AUC of 0.841, which was superior to the radiomics model (0.841 vs 0.706, p < 0.001) as well as the DL model (0.841 vs 0.819, p = 0.26) (Fig. 3 ). These findings highlight the superior performance of the integrated model over the individual models. Table 3 Performance comparison of different AI models in prediction of CLNM Test cohort AUC 95%CI ACC (%) SEN (%) SPE (%) PPV (%) NPV (%) ResNet 0.8189 [0.7542, 0.8835] 73.20 75.00 71.43 72.15 74.32 SVM 0.7061* [0.6246, 0.7875] 63.40 52.63 74.03 66.67 61.29 Integrated Model 0.8406 [0.7792, 0.9020] 77.12 72.37 81.82 79.71 75.00 Abbreviations: CLNM, central lymph node metastasis; ACC, accuracy; SEN, sensitivity; SPE, specificity; PPV, positive predictive value; NPV, negative predictive value; AUC, area under the receiver operating curve; CI, confidence interval *Compared with integrated model, p < 0.05 Performance comparison among integrated model and radiologists The results indicated that the integrated model exhibited a significantly higher AUC compared to both junior and senior radiologists (0.841 vs. 0.561 and 0.640, p < 0.001). In comparison to the junior and senior radiologists, the integrated model demonstrated a decrease in the missed CLNM rate by 30.26% and 17.11% respectively. Additionally, the rate of unnecessary CLND decreased by 11.69% and 9.09%. A detailed comparison of the integrated model and radiologists were summarized in Table 4 . These results indicated that integrated model could improve the efficiency of metastatic LNs detection and reduce the rate of unnecessary CLND. Table 4 Performance comparison of radiologists and integrated model in prediction of CLNM Test cohort AUC 95%CI Undetected CLNM (%) Unnecessary CLND (%) ACC (%) SEN (%) SPE (%) PPV (%) NPV (%) Junior Radiologists 0.5612* [0.4852, 0.6371] 57.89 (44/76) 29.87 (23/77) 56.21 42.11 70.13 58.18 55.10 Senior Radiologists 0.6400* [0.5646, 0.7153] 44.74 (34/76) 27.27 (21/77) 64.05 55.26 72.73 66.67 62.22 Integrated Model 0.8406 [0.7792, 0.9020] 27.63 (21/76) 18.18 (14/77) 77.12 72.37 81.82 79.71 75.00 Abbreviations: CLNM, central lymph node metastasis; CLND, central lymph node dissection; ACC, accuracy; SEN, sensitivity; SPE, specificity; PPV, positive predictive value; NPV, negative predictive value; AUC, area under the receiver operating curve; CI, confidence interval *Compared with integrated model, p < 0.05 Explanation of the integrated model To better compensate for the problem of "cognitive opacity" of AI models, we utilized SHAP plots to illustrate the contribution of each key parameter in the integrated model. The result showed that the DL model contributed the most to CLNM prediction, followed by ETE, tumour size, age, gender, and multifocality. The radiomics model played a relatively minor role within the integrated model (Fig. 4 ). In Fig. 5 , two representative examples were presented to demonstrate how each key parameter contributed to the personalized decision-making process in the integrated model. Furthermore, we employed Grad-CAM to identify the areas of interest for the DL model. Figure 6 showcased several representative cases, indicating that the areas of interest were predominantly located around the thyroid capsule, consistent with the radiologists focusing on areas significantly associated with CLNM. Discussion In this study, we developed an integrated model for predicting CLNM that incorporated deep learning, radiomics, and clinical and US features. Our integrated model outperformed models based solely on radiomics or DL features, as well as junior and senior radiologists. The integrated model decreased the rate of missed CLNM and unnecessary CLND, thus improving preoperative CLND decision-making. Furthermore, the integrated model's visual explanation aligned with radiologists' typical judgments, which contributed to the acceptance of AI-assisted US diagnosis. Currently, one of the primary objectives of US in patients with PTC is to provide preoperative guidance for CLND by detecting the presence of CLNM. However, the presence of air interference in the trachea and esophagus, along with the small size of LNs, leads to unsatisfactory diagnostic accuracy [ 23 ]. Encouragingly, radiomics and DL methods can effectively reveal information that is imperceptible to the human eye, thereby enhancing diagnostic capabilities. Previous studies focusing solely on either radiomics or DL methods in diagnosing CLNM have yielded favourable results [ 24 , 25 ]. However, our findings indicated limitations in the diagnostic efficacy of standalone DL and radiomics models. The DL model exhibited higher sensitivity, while the radiomics model showed higher specificity, indicating a distinction between traditional image features extracted by radiomics and the high-dimensional features extracted by DL. These observations motivated us to develop an integrated model that combines both types of features, resulting in superior performance compared to models based solely on radiomics or DL features. Furthermore, when compared with junior and senior radiologists, the integrated model significantly reduced the missed rate of CLNM by 30.26% and 17.11%, respectively, and decreased the rate of unnecessary CLND by 11.69% and 9.09%. Our findings indicate that the utilization of this model in clinical practice can be beneficial for PTC patients. Radiologists also can benefit from the integrated model, as it can serve as a valuable second opinion during the diagnosis of CLNM, assisting them in making more precise judgments and boosting their diagnostic confidence. Consistent with the results of previous studies [ 26 ], we conducted a screening of clinical and US factors associated with CLNM during routine diagnostic work. These factors were then integrated into our AI model, resulting in improved efficacy. Upon further analysis using the SHAP plot, the integrated model demonstrated that the clinical and US factors provided valuable additional information. Among these crucial factors, ETE had the highest contribution, indicating that tumor cells could breach the thyroid capsule and enter the lymphatic system, leading to the development of metastatic LNs [ 27 ]. Additionally, tumor size, gender, age, and multifocality were also found to be associated with CLNM [ 28 ]. Interestingly, our findings revealed that the integrated model focused primarily on the thyroid capsule, which aligns with the areas of emphasis for radiologists when assessing CLNM. These results suggest that the clinical and US factors incorporated into the integrated model, as well as the regions of the model's interest, are generally consistent with radiologists' judgments, thereby providing the model with some clinical explainability. Overall, the visual explanation provided by the integrated model not only aligns with radiologists' usual judgments but also the integrated model demonstrates higher diagnostic efficacy compared to radiologists. This enhances the clinical acceptance of AI-assisted US diagnosis among radiologists. In contrast to the integrated models derived from CT or MRI images, where radiomics features played a prominent role [ 29 , 30 ], our findings indicated that the contribution of radiomics features to our integrated model was relatively modest. This discrepancy may arise from the fact that some of the features extracted from US images through radiomics, such as shape, grayscale, and texture, can also be obtained through DL methods. Additionally, during US imaging, noise can be generated due to variations in signal intensity, which can degrade image quality and affect the extraction of certain radiomics features. Consequently, these circumstances may account for the relatively limited contribution of radiomics to the model. Several limitations should be acknowledged in this study. Firstly, due to the interference of anatomical structures and the small size of the central LNs, US images of central LNs were not included in the analysis. Secondly, although the incorporation of clinical and US features enhances the acceptance of AI-assisted US diagnosis by radiologists, the interpretability of features learned by the DL and radiomics model remains limited. Future advancements in the field of interpretable AI will inspire further exploration. Finally, the results obtained may be influenced by the limited amount of data utilized. Further investigation of the value of integrated models in prospective studies with larger sample sizes is warranted. In conclusion, the integrated model demonstrated superior performance compared to models relying solely on radiomics or DL features, exceeding the diagnostic capabilities of both junior and senior radiologists. The application of integrated models can significantly reduce missed CLNMs and unnecessary CLNDs along with increasing radiologists' acceptance of AI-assisted US diagnoses. Abbreviations AI Artificial intelligence AUC Area under curve CLND Central lymph node dissection CLNM Central lymph node metastases DL Deep learning Grad-CAM Gradient-weighted Class Activation Mapping LN Lymph node PTC Papillary thyroid carcinoma ROI Region of interest SHAP SHapley Additive exPlanation SVM Support vector machine Declarations Ethics approval and consent to participate The study has been performed in accordance with the Declaration of Helsinki and was approved by The Ethics Committees of Nanfang Hospital of South Medical University and the First People’s Hospital of Foshan (NFEC-202008-K6). The requirement for informed consent was waived by the Ethics Committee of Nanfang Hospital of South Medical University because of the retrospective nature of the study. Consent for publication Not applicable. Availability of data and materials The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request. Competing interests The authors declare no conflicts of interest. Funding This study was funded by the National Natural Science Foundation of China (Grant Nos. 82271998 and 82071949) Authors’ contributions YJ and YG were contributed to the conception and design of the study. WJ organized the database. WY and YY were responsible for the software and statistics. YG and WZ took charge of the writing of this paper. ST, YS, and TL to data collection. ZT, LY for data analysis. All authors have read and approved the manuscript Acknowledgements Not applicable. References Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, et al. Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries. CA Cancer J Clin. 2021;71:209–49. White ML, Gauger PG, Doherty GM. Central lymph node dissection in differentiated thyroid cancer. World J Surg. 2007;31:895–904. Wang TS, Sosa JA. Thyroid surgery for differentiated thyroid cancer — recent advances and future directions. Nat Rev Endocrinol. 2018;14:670–83. Medas F, Canu GL, Cappellacci F, Anedda G, Conzo G, Erdas E, et al. Prophylactic central lymph node dissection improves disease-free survival in patients with intermediate and high risk differentiated thyroid carcinoma: A retrospective analysis on 399 patients. Cancers (Basel). 2020;12:1–12. Adam MA, Pura J, Goffredo P, Dinan MA, Reed SD, Scheri RP, et al. Presence and number of lymph node metastases are associated with compromised survival for patients younger than age 45 years with papillary thyroid cancer. J Clin Oncol. 2015;33:2370–5. Chinese Society of Endocrinology; Thyroid and Metabolism Surgery Group of the Chinese Society of Surgery;Chinese Society of Nuclear Medicine. Chinese Society of Ultrasound in Medicine. Guidelines for the diagnosis and management of thyroid nodules and differentiated thyroid cancer (Second edition). Chin J Endocrinol Metab. 2023;39:181–226. Haugen BR, Alexander EK, Bible KC, Doherty GM, Mandel SJ, Nikiforov YE, et al. 2015 American Thyroid Association Management Guidelines for Adult Patients with Thyroid Nodules and Differentiated Thyroid Cancer: The American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer. Thyroid. 2016;26:1–133. Zhao H, Li H. Meta-analysis of ultrasound for cervical lymph nodes in papillary thyroid cancer: Diagnosis of central and lateral compartment nodal metastases. Eur J Radiol. 2019;112:14–21. Lambin P, Leijenaar RTH, Deist TM, Peerlings J, De Jong EEC, Van Timmeren J, et al. Radiomics: The bridge between medical imaging and personalized medicine. Nat Rev Clin Oncol. 2017;14:749–62. Liu S, Wang Y, Yang X, Lei B, Liu L, Li SX, et al. Deep Learn Med Ultrasound Analysis: Rev Eng. 2019;5:261–75. Levine AB, Schlosser C, Grewal J, Coope R, Jones SJM, Yip S. Rise of the Machines: Advances in Deep Learning for Cancer Diagnosis. Trends in cancer. 2019;5:157–69. Wang W, Li Y. [Radiomics for prediction of central lymph node metastasis in the neck in patients with thyroid papillary carcinoma]. Nan Fang Yi Ke Da Xue Xue Bao. 2019;39:1094–8. Wang S, Hou Y, Li Z, Dong J, Tang C. Combining ConvNets with hand-crafted features for action recognition based on an HMM-SVM classifier. Multimed Tools Appl. 2018;77:18983–98. Antropova N, Huynh BQ, Giger ML. A deep feature fusion methodology for breast cancer diagnosis demonstrated on three imaging modality datasets. Med Phys. 2017;44:5162–71. Dai Q, Liu D, Tao Y, Ding C, Li S, Zhao C, et al. Nomograms based on preoperative multimodal ultrasound of papillary thyroid carcinoma for predicting central lymph node metastasis. Eur Radiol. 2022;32:4596–608. Wang J, Yuan C, Han C, Wen Y, Lu H, Liu C, et al. IMAL-Net: Interpretable multi-task attention learning network for invasive lung adenocarcinoma screening in CT images. Med Phys. 2021;48:7913–29. Zhou J, Yin L, Wei X, Zhang S, Song Y, Luo B, et al. 2020 Chinese guidelines for ultrasound malignancy risk stratification of thyroid nodules: the C-TIRADS. Endocrine. 2020;70:256–79. Adler DD, Carson PL, Rubin JM, Quinn-Reid D. of Breast Cancer: Preliminary Findings. Ultrasound Med Biol. 1990;16:553–9. Tessler FN, Middleton WD, Grant EG, Hoang JK, Berland LL, Teefey SA, et al. ACR Thyroid Imaging, Reporting and Data System (TI-RADS): White Paper of the ACR TI-RADS Committee. J Am Coll Radiol. 2017;14:587–95. van Griethuysen JJM, Fedorov A, Parmar C, Hosny A, Aucoin N, Narayan V, et al. Computational Radiomics System to Decode the Radiographic Phenotype. Cancer Res. 2017;77:e104–7. Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L. ImageNet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. 2009. p. 248–55. Kolesnikov A, Beyer L, Zhai X, Puigcerver J, Yung J, Gelly S et al. Big Transfer (BiT): General Visual Representation Learning. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics). 2020;12350 LNCS:491–507. Hwang HS, Orloff LA. Efficacy of preoperative neck ultrasound in the detection of cervical lymph node metastasis from thyroid cancer. Laryngoscope. 2011;121:487–91. Zhou SC, Liu TT, Zhou J, Huang YX, Guo Y, Yu JH, et al. An Ultrasound Radiomics Nomogram for Preoperative Prediction of Central Neck Lymph Node Metastasis in Papillary Thyroid Carcinoma. Front Oncol. 2020;10:1–13. Wu Y, Rao K, Liu J, Han C, Gong L, Chong Y, et al. Machine Learning Algorithms for the Prediction of Central Lymph Node Metastasis in Patients With Papillary Thyroid Cancer. Front Endocrinol (Lausanne). 2020;11:1–9. Zheng X, Yao Z, Huang Y, Yu Y, Wang Y, Liu Y, et al. Deep learning radiomics can predict axillary lymph node status in early-stage breast cancer. Nat Commun. 2020;11:1236. Zou Y, Shi Y, Liu J, Cui G, Yang Z, Liu M, et al. A Comparative Analysis of Six Machine Learning Models Based on Ultrasound to Distinguish the Possibility of Central Cervical Lymph Node Metastasis in Patients With Papillary Thyroid Carcinoma. Front Oncol. 2021;11:1–10. Xia E, Chi Y, Jin L, Shen Y, Hirachan S, Bhandari A, et al. Preoperative prediction of lymph node metastasis in patients with papillary thyroid carcinoma by an artificial intelligence algorithm. Am J Transl Res. 2021;13:7695–704. Paul R, Hawkins SH, Schabath MB, Gillies RJ, Hall LO, Goldgof DB. Predicting malignant nodules by fusing deep features with classical radiomics features. J Med Imaging. 2018;5:1. Hu X, Gong J, Zhou W, Li H, Wang S, Wei M et al. Computer-aided diagnosis of ground glass pulmonary nodule by fusing deep learning and radiomics features. Phys Med Biol. 2021;66. Additional Declarations No competing interests reported. Supplementary Files Supplementary.docx Cite Share Download PDF Status: Published Journal Publication published 12 Jan, 2024 Read the published version in BMC Cancer → Version 1 posted Editorial decision: Revision requested 07 Nov, 2023 Reviews received at journal 21 Oct, 2023 Reviewers agreed at journal 11 Oct, 2023 Reviews received at journal 26 Sep, 2023 Reviewers agreed at journal 15 Sep, 2023 Reviewers invited by journal 15 Sep, 2023 Editor assigned by journal 15 Sep, 2023 Editor invited by journal 28 Aug, 2023 Submission checks completed at journal 28 Aug, 2023 First submitted to journal 30 Jul, 2023 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3218322","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":229274068,"identity":"3638c1c4-d3eb-4d9d-997b-319356e77bce","order_by":0,"name":"Yang Gao","email":"","orcid":"","institution":"Nanfang Hospital, Southern Medical University","correspondingAuthor":false,"prefix":"","firstName":"Yang","middleName":"","lastName":"Gao","suffix":""},{"id":229274069,"identity":"cd6d60e3-ccca-42f9-9af8-266fd52f16f3","order_by":1,"name":"Weizhen Wang","email":"","orcid":"","institution":"Nanfang Hospital, Southern Medical University","correspondingAuthor":false,"prefix":"","firstName":"Weizhen","middleName":"","lastName":"Wang","suffix":""},{"id":229274070,"identity":"16ccf0ee-b185-4ab2-a6b6-f690450c4d64","order_by":2,"name":"Yuan Yang","email":"","orcid":"","institution":"Southern Medical University","correspondingAuthor":false,"prefix":"","firstName":"Yuan","middleName":"","lastName":"Yang","suffix":""},{"id":229274071,"identity":"2977506e-b508-4cd4-abea-4c8b6b9ab952","order_by":3,"name":"Ziting Xu","email":"","orcid":"","institution":"Nanfang Hospital, Southern Medical University","correspondingAuthor":false,"prefix":"","firstName":"Ziting","middleName":"","lastName":"Xu","suffix":""},{"id":229274072,"identity":"3524dfbf-fe71-4a9f-84c2-66f43405c886","order_by":4,"name":"Yue Lin","email":"","orcid":"","institution":"Nanfang Hospital, Southern Medical University","correspondingAuthor":false,"prefix":"","firstName":"Yue","middleName":"","lastName":"Lin","suffix":""},{"id":229274073,"identity":"de46d73a-bb5e-4176-a0ba-e3f2d32e3f6b","order_by":5,"name":"Ting Lang","email":"","orcid":"","institution":"Nanfang Hospital, Southern Medical University","correspondingAuthor":false,"prefix":"","firstName":"Ting","middleName":"","lastName":"Lang","suffix":""},{"id":229274074,"identity":"ab0948e0-1531-45ec-a8c8-4b4cfb37bc44","order_by":6,"name":"Shangtong Lei","email":"","orcid":"","institution":"Nanfang Hospital, Southern Medical University","correspondingAuthor":false,"prefix":"","firstName":"Shangtong","middleName":"","lastName":"Lei","suffix":""},{"id":229274075,"identity":"7f3cdc9b-a2ed-4ea9-a1f0-14a404d49fce","order_by":7,"name":"Yisheng Xiao","email":"","orcid":"","institution":"the First People’s Hospital of Foshan","correspondingAuthor":false,"prefix":"","firstName":"Yisheng","middleName":"","lastName":"Xiao","suffix":""},{"id":229274076,"identity":"7ef1ef21-9c5e-4e46-9bd3-ee9e788155c1","order_by":8,"name":"Wei Yang","email":"","orcid":"","institution":"Southern Medical University","correspondingAuthor":false,"prefix":"","firstName":"Wei","middleName":"","lastName":"Yang","suffix":""},{"id":229274077,"identity":"c1233c57-ffc4-4338-aebf-e8d2df5c6ac4","order_by":9,"name":"Weijun Huang","email":"","orcid":"","institution":"the First People’s Hospital of Foshan","correspondingAuthor":false,"prefix":"","firstName":"Weijun","middleName":"","lastName":"Huang","suffix":""},{"id":229274078,"identity":"911a4ce7-2b7f-4ba8-9cff-479454b6d4eb","order_by":10,"name":"Yingjia Li","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAtUlEQVRIiWNgGAWjYBACPgaGhAMMFRCOBFFa2MBazpCohYGBsY0kLRIJDw/8nFcXbXCA+eBtHga7PGK0JBzs3XY4d8MBtmRrHobkYqK0HODddgCohcdMmofhQGIDUbb8nVMH1ML/jXgth3kbmEG2sBGphedBwmGZY4dzZx5mM7acY5BMWAs/e07yxzc1dbl9x5sf3nhTYUdYC4NATgKEwQwiDAiqB1lz/AAxykbBKBgFo2AkAwAUpD2DTff/XwAAAABJRU5ErkJggg==","orcid":"","institution":"Southern Medical University","correspondingAuthor":true,"prefix":"","firstName":"Yingjia","middleName":"","lastName":"Li","suffix":""}],"badges":[],"createdAt":"2023-07-30 16:59:11","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3218322/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3218322/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1186/s12885-024-11838-1","type":"published","date":"2024-01-12T15:02:00+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":42433983,"identity":"b53f0561-3252-49b7-b005-15066fd40701","added_by":"auto","created_at":"2023-08-31 14:44:24","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":162365,"visible":true,"origin":"","legend":"\u003cp\u003eFlow Chart of participants recruitment.\u003c/p\u003e\n\u003cp\u003eUS, ultrasound; pCLND, prophylactic central lymph node dissection; PTC, papillary thyroid carcinoma\u003c/p\u003e","description":"","filename":"Figure1.png","url":"https://assets-eu.researchsquare.com/files/rs-3218322/v1/928cd3fe57e5cf582eca2e96.png"},{"id":42436462,"identity":"a6ad8de1-97bb-4cae-8750-bbc9c106aa62","added_by":"auto","created_at":"2023-08-31 14:52:24","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":235296,"visible":true,"origin":"","legend":"\u003cp\u003eThe flow chart of the artificial intelligence integrated model.\u003c/p\u003e","description":"","filename":"Figure2.png","url":"https://assets-eu.researchsquare.com/files/rs-3218322/v1/5f79f5009780633d927a7722.png"},{"id":42431395,"identity":"509e50c7-be61-4523-adb8-e93625a69070","added_by":"auto","created_at":"2023-08-31 14:36:24","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":78601,"visible":true,"origin":"","legend":"\u003cp\u003eDiagnostic performance comparison among artificial intelligence models and radiologists in the independent testing cohort.\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-3218322/v1/994eac0109d163807f6df545.png"},{"id":42431392,"identity":"31a63148-9973-4193-9f8c-165c0065143c","added_by":"auto","created_at":"2023-08-31 14:36:24","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":61670,"visible":true,"origin":"","legend":"\u003cp\u003eThe SHAP plot reflected the contribution of each parameter to diagnose central lymph node metastasis in the integrated model.\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-3218322/v1/344ca8acd5df3706cf3f423c.png"},{"id":42431396,"identity":"28dd10c7-33b4-4bc7-b6cb-eea2522f3ba5","added_by":"auto","created_at":"2023-08-31 14:36:24","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":203864,"visible":true,"origin":"","legend":"\u003cp\u003eTwo representative cases for the real output of the integrated model.\u003c/p\u003e\n\u003cp\u003ea. A 31-year-old female suffering from PTC with CLNM. The radiomics model outputs a probability of 48.30% for CLNM. The deep learning model outputs a probability of 81.77%, and the integrated model fuses the risk factors and gives a final probability of 94.32%. The result is inconsistent with the radiologist's diagnosis, so the radiologist is recommended to conduct a second scan and then consult the classification provided by the integrated model. b A 28-year-old female suffering from PTC without CLNM. The radiomics model represented a probability of 46.20% for CLNM. The deep learning model and the integrated model output probabilities of 15.10% and 5.91%, respectively. The result is consistent with the diagnosis of the radiologist.\u003c/p\u003e\n\u003cp\u003eCLNM, central lymph node metastasis; ETE, extrathyroidal extension; PTC, papillary thyroid carcinoma\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-3218322/v1/69a6776cfe1aba0c012520da.png"},{"id":42433984,"identity":"fe0d5aa3-5e7f-4a57-8fa5-955e2031aaf7","added_by":"auto","created_at":"2023-08-31 14:44:24","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":970251,"visible":true,"origin":"","legend":"\u003cp\u003eRepresentative examples of the saliency maps. A. Saliency maps of one CLNM case evaluated by integrated model. The red colour highlighted the activation region associated with the thyroid capsule, consistent with the radiologists' concentration on areas significantly associated with CLNM. B. Saliency maps of a case without CLNM evaluated by integrated model.\u003c/p\u003e\n\u003cp\u003eCLNM, central lymph node metastasis\u003c/p\u003e","description":"","filename":"Figure6.png","url":"https://assets-eu.researchsquare.com/files/rs-3218322/v1/985dddb1b2de2dd76a049fdb.png"},{"id":49628918,"identity":"221ebfdf-51d0-465a-99fa-8ee4847ccb7c","added_by":"auto","created_at":"2024-01-15 15:09:50","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2242508,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3218322/v1/e82fffc3-00fa-44c3-949c-f58630d4f8b4.pdf"},{"id":42431398,"identity":"f0767428-3be4-4a9e-9aad-783f3ff25ee8","added_by":"auto","created_at":"2023-08-31 14:36:24","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":19768,"visible":true,"origin":"","legend":"","description":"","filename":"Supplementary.docx","url":"https://assets-eu.researchsquare.com/files/rs-3218322/v1/cd207e871aaa2bd809b15e91.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"An integrated model incorporating deep learning, radiomics and clinical and US features to diagnose central lymph node metastasis in patients with papillary thyroid cancer","fulltext":[{"header":"Introduction","content":"\u003cp\u003eIn recent years, the incidence of thyroid cancer has increased significantly worldwide, with papillary thyroid carcinoma (PTC) accounting for most cases [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. PTC is characterized by early metastasis to cervical lymph nodes (LNs), particularly in the central region [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. The reported rate of central LN metastasis (CLNM) in patients is approximately 50% [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e], which is a known risk factor for recurrence and adversely affects overall survival [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. The necessity of prophylactic central LN dissection (pCLND) remains a subject of debate in thyroid cancer treatment. In China, the latest guidelines recommend routine pCLND at least ipsilateral to the lesion [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. While pCLND can effectively reduce the need for reoperation in cases of recurrence, it also leads to unnecessary CLND procedures. US is the most commonly used method for preoperative LN assessment in PTC [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. However, its sensitivity in identifying CLNM ranges from only 26\u0026ndash;47%, which is insufficient for accurate assessment [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. Hence, a more sensitive preoperative assessment of CLNM is crucial for patients with PTC to reduce unnecessary CLND.\u003c/p\u003e \u003cp\u003eIn recent years, artificial intelligence (AI), especially deep learning (DL) and radiomics, have shown powerful analytical capabilities in extracting intricate and multi-layered features from medical images [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. Radiomics focuses on extracting traditional image features for classification and prediction, whereas high-dimensional features generated by a DL model can represent deep image information of tumours [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. We previously reported a preliminary small sample study of CLNM using radiomics, which acquired good performance [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. To date, most studies have independently employed DL and radiomics features, and far fewer studies have attempted to fuse these two features from US images. It is worth noting that features extracted by DL models may be sensitive to global translation, rotation, and scaling while radiomics features such as intensity features are not [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e, \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e], Therefore, we hypothesize that radiomics features and DL features extracted from US images could be complementary, and their combination may yield improved prediction outcomes.\u003c/p\u003e \u003cp\u003eHowever, unlike radiologists who incorporate clinical and US information to make diagnoses, most AI models only provide output results without revealing their decision-making process. This lack of transparency is considered one of the reasons why radiologists are skeptical about the clinical application of AI models. Previous studies have highlighted the significance of clinical and US characteristics (e.g., age, gender, and tumor size) in distinguishing CLNM [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. Nevertheless, the lack of information such as age and gender in the images, and data pre-processing such as resizing and normalisation, makes detecting these information challenging in machine learning [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]. By integrating clinical and US features into AI models, it may be possible to improve the predictive efficacy of the models as well as the acceptance from radiologists.\u003c/p\u003e \u003cp\u003eHence, this study aimed to develop and validate whether an integrated model incorporating DL, radiomics and clinical and US features can improve the performance to diagnose CLNM in patients with PTC, in order to reduce the miss rate of CLNM, unnecessary CLND and improve the acceptance of AI-assisted US diagnosis for radiologists.\u003c/p\u003e"},{"header":"Patients and Methods","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003ePatients\u003c/h2\u003e \u003cp\u003e The Ethics Committees of Nanfang Hospital of South Medical University and the First People\u0026rsquo;s Hospital of Foshan (NFEC-202008-K6) approved this retrospective study. The requirement for informed consent was waived. The inclusion and exclusion criteria were as follows:\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003eInclusion Criteria\u003c/h2\u003e \u003cp\u003ePatients were enrolled if they satisfied all the following inclusion criteria: (1) were confirmed to have PTC after lobectomy or total thyroidectomy; (2) underwent CLND with a pathological examination; (3) the thyroid US examination was performed at our hospital within one month before the operation;\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003eExclusion Criteria\u003c/h2\u003e \u003cp\u003e(1) had other malignancies or distant metastases at diagnosis; (2) received preoperative head and neck therapies such as radiotherapy, chemotherapy, or radiofrequency ablation; (3) with missing data; (4) with poor image quality.\u003c/p\u003e \u003cp\u003eAfter undergoing a rigorous inclusion and exclusion process, datasets of 613 patients treated in our clinical centres from March 2019 to July 2020 were included. The participant recruitment flow is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. The participants were randomly divided into training and independent test cohorts for further analysis.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003eUltrasonography and image analysis\u003c/h2\u003e \u003cp\u003eThe choice of US machine was not limited, and most data was obtained using devices such as Siemens Sequoia, Supersonic Aixplorer, and Toshiba Aplio 500, stored in the DICOM format. The risk factors for CLNM were identified by including the following variables: gender, age, and US features of thyroid tumors following the C-TIRADS and ATA guidelines [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. These features encompassed tumour size, hypoechoic solid composition, multifocality, aspect ratio, posterior acoustic attenuation, tumour location, extrathyroidal extension (ETE), acoustic halo, microcalcification, and the internal tumour vascularity. Age was dichotomized at 55 years following the 8th American Joint Commission on Cancer staging system. In cases with multifocality, the largest nodule was chosen as the representative. The tumor vascularity was graded from 0 to 3 by colour Doppler flow imaging (CDFI) following the Adler standard [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e]. The US features were re-evaluated by two radiologists with four and seven years of experience in thyroid US diagnosis. Both radiologists were blinded to clinical information and pathological diagnosis. The agreement between them was assessed, and in case of any disagreement, a senior radiologist with over 20 years of experience made the final decision.\u003c/p\u003e \u003cp\u003eThe preoperative examination of LNs was conducted on all patients by a team of five radiologists, comprising two senior radiologists with 15 and 17 years of experience, and three junior radiologists with 3, 5, and 6 years of experience, respectively. The diagnostic accuracy of the two groups of radiologists, differing in seniority, was determined by comparing the LNs status reported by US with the corresponding postoperative pathological results. Based on the ACR TI-RADS [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e], LNs exhibiting one or more suspicious US features (roundness, loss of the normal echogenic hilum, internal microcalcifications, cystic changes, hyperechogenicity, or presence of peripheral flow) were classified as US-reported CLNM.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003eRegion of interest segmentation and Development of the radiomics model\u003c/h2\u003e \u003cp\u003eThe manual segmentation of the regions of interest (ROIs) was performed by a radiologist with five years of experience, Subsequently, a second radiologist with seven years of experience reviewed the drawn ROIs. The final ROIs was determined after discussion when there was disagreement between the two readers.\u003c/p\u003e \u003cp\u003eRadiomics features, including intensity, shape texture, wavelet features, and Laplacian of Gaussian features, were extracted from the US images by delineating the ROIs. A total of 783 features were extracted using the open-source python package \u0026ldquo;pyradiomics\u0026rdquo; [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]. The least absolute shrinkage and selection operator (LASSO) logistic regression analysis method was employed to select the radiomics feature on the training dataset. The support vector machine (SVM) algorithm was utilized to establish the prediction model, which was subsequently trained through 10-fold cross-validation in training set to optimize the hyperparameters.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eDevelopment of the deep learning model\u003c/h2\u003e \u003cp\u003e33% of the training images were randomly selected to form a validation cohort to guide the choice of hyperparameters. \u003cb\u003eSupplemental Table \u003cspan refid=\"MOESM1\" class=\"InternalRef\"\u003eS1\u003c/span\u003e\u003c/b\u003e shows the result of four tested algorithms. The best-performing model (ResNet50) was adopted as the base model and pre-trained on ImageNet-21k [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]. Group normalization and weight standardization were used instead of batch normalization, following the tricks proposed for big data transfer [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]. Rectangular ROIs were cropped from the raw US images using a tumor segmentation mask. These ROIs were then resized to 224 \u0026times; 224 pixels and normalized. During the training stage, the ROIs were fed into the network to update the model parameters. The cross-entropy of the outputs and labels as the loss function were used to alleviate the influence of class imbalance in differentiating CLNM and online data augmentation was used to avoid overfitting and sample imbalance.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003eDevelopment and explanation of the integrated prediction model\u003c/h2\u003e \u003cp\u003eIn the training cohort, multivariate logistic regression analysis and likelihood ratio tests for positive selection were employed to identify clinical and US characteristics that could effectively differentiate the presence of CLNM. These selected factors were then integrated with the features extracted from the SVM and ResNet50 models to create the final prediction model. The flowchart outlining the integrated prediction model can be seen in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e. To assess the performance of the integrated model, the model's performance was compared with that of the radiomics model, DL model, and junior and senior radiologists on independent test set.\u003c/p\u003e \u003cp\u003eIn addition, the visualized explanation methods named SHapley Additive exPlanations (SHAP) plot and Gradient-weighted Class Activation Mapping (Grad-CAM) were applied to improve the clinical explanation of our model. We used Grad-CAM to extract the areas of interest and generate saliency maps for the DL model, while the SHAP plot was used to calculate the contribution value of each variable to the integrated model. These visualization methods aim to improve the clinical understanding and explanation of our model's predictions.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003eStatistical analysis\u003c/h2\u003e \u003cp\u003eStatistical analysis was performed using IBM SPSS Statistics for Windows, Version 20.0 (IBM Corp.).\u003c/p\u003e \u003cp\u003eCategorical variables are presented as numbers and percentages and analyzed using the chi-squared or Fisher's exact test. The Mann-Whitney U test analyzed continuous variables, and Kappa statistics analyzed the inter-observer agreement. The performance of predictive models was evaluated by the receiver operating characteristic (ROC) curve analysis and the area under curve (AUC). DeLong's test compared the combined prediction model and other methods in predicting CLNM. Other performance measures, including accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV), were also assessed. The two-sided statistical significance was set at 0.05.\u003c/p\u003e \u003c/div\u003e"},{"header":"Results","content":"\u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003ePatient demographics and feature selection\u003c/h2\u003e \u003cp\u003ePatient characteristics and US features of the thyroid nodules in the training and test cohorts were shown in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. The training cohort included 460 patients (136 males, 324 females) with a mean age of 40.70\u0026thinsp;\u0026plusmn;\u0026thinsp;11.16 years (range, 11\u0026ndash;73 years). The independent test cohort included 153 patients (45 males and 108 females) with a mean age of 42.59\u0026thinsp;\u0026plusmn;\u0026thinsp;11.33 years (range, 13\u0026ndash;69 years). These two datasets were comparable as there were no significant differences. The inter-observer consistency was satisfactory, with Kappa coefficients between 0.82 and 0.92 (\u003cb\u003eSupplemental Table S2\u003c/b\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eDemographic comparison between training and independent test cohorts\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCharacteristics\u003c/p\u003e \u003cp\u003eand US features\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTraining cohort\u003c/p\u003e \u003cp\u003e(n\u0026thinsp;=\u0026thinsp;460)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eIndependent test cohort\u003c/p\u003e \u003cp\u003e(n\u0026thinsp;=\u0026thinsp;153)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cem\u003ep\u003c/em\u003e value\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSize (mean\u0026thinsp;\u0026plusmn;\u0026thinsp;SD)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e1.24\u0026thinsp;\u0026plusmn;\u0026thinsp;0.92\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1.16\u0026thinsp;\u0026plusmn;\u0026thinsp;0.69\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.433\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.971\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMale\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e136(75.1%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e45(24.9%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFemale\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e324(75.0%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e108(25.0%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAge\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.484\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u0026le;\u0026thinsp;55\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e415(75.5%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e135(24.5%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e༞55\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e45(71.4%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e18(28.6%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTumour location\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.711\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRight lobe\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e228(73.8%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e81(26.2%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLeft lobe\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e213(76.6%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e65(23.4%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eIsthmus\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e19(73.1%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e7(26.9%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTumour position\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.095\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eUpper\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e152(81.7%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e34(18.3%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMid\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e164(71.9%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e64(28.1%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLower\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e125(72.3%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e48(27.7%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eIsthmus\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e19(73.1%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e7(26.9%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSolid composition with hypoechoic echo\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.110\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePresent\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e417(74.2%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e145(25.8%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAbsent\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e43(84.3%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e8(15.7%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTumour multifocality\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.769\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePresent\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e83(76.1%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e26(23.9%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAbsent\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e377(74.8%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e127(25.2%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAspect ratio\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.198\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePresent\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e201(726%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e76(27.4%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAbsent\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e259(77.1%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e77(22.9%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMicrocalcification\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.769\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePresent\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e360(75.3%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e118(24.7%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAbsent\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e100(74.1%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e35(25.9%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTumour vascularity\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.320\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e0\u0026ndash;1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e401(74.4%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e138(25.6%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2\u0026ndash;3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e59(79.7%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e15(20.3%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAcoustic halo\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.804\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePresent\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e42(73.7%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e15(26.3%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAbsent\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e418(75.2%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e138(24.8%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eETE\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.947\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePresent\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e119(74.8%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e40(25.2%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAbsent\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e341(75.1%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e113(24.9%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePosterior acoustic attenuation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.282\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePresent\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e67(79.8%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e17(20.2%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAbsent\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e393(74.3%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e136(25.7%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCLNM in the pathology outcomes.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.982\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePresent\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e228 (75.0%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e76 (25.0%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAbsent\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e232 (75.1%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e77 (24.9%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"4\"\u003eAbbreviations: US, ultrasound; ETE, extrathyroidal extension; CLNM, central lymph node metastasis.\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eTo better understand the relationship between CLNM and clinical and US features, a multivariate analysis was performed in the training cohort. The results showed that age, sex, tumour size, multifocality, and ETE were independent risk factors for CLNM (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eIndependent risk factors after multiple logistic regression analysis\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eUltrasound features\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eβ\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eOdds ratio (95% CI)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cem\u003ep\u003c/em\u003e value\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003ePrediction of CLNM status\u003c/b\u003e\u003c/p\u003e \u003cp\u003eSize\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.533\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1.704 (1.225\u0026ndash;2.370)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.002\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAge\u003c/p\u003e \u003cp\u003eSex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e-1.301\u003c/p\u003e \u003cp\u003e-0.598\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.272 (0.125\u0026ndash;0.595)\u003c/p\u003e \u003cp\u003e0.550 (0.353\u0026ndash;0.857)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.001\u003c/p\u003e \u003cp\u003e0.008\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTumour multifocality\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.738\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e2.092 (1.224\u0026ndash;3.577)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.007\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eETE\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e1.418\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e4.130 (2.434\u0026ndash;7.008)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u0026lt;\u0026thinsp;0.001\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"4\"\u003eAbbreviations: CLNM, central lymph node metastasis; US, ultrasound; CI: confidence interval; ETE, extrathyroidal extension\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003eDiagnostic performance of CLNM-predicting model\u003c/h2\u003e \u003cp\u003eWe successfully built a radiomics model, a DL model, and an integrated model. In the testing set, our result showed that the DL model exhibited higher sensitivity (75.00% vs 52.63%) but slightly lower specificity (71.43% vs 74.03%) compared to the radiomics model (Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). By combining radiomics, DL and clinical features, the integrated model showed good predictive efficacy (the specificity and sensitivity were 81.82% and 72.37%, and the PPV and NPV were 79.71% and 75.00%). Meanwhile, the integrated model had most outstanding performance with the AUC of 0.841, which was superior to the radiomics model (0.841 vs 0.706, p\u0026thinsp;\u0026lt;\u0026thinsp;0.001) as well as the DL model (0.841 vs 0.819, p\u0026thinsp;=\u0026thinsp;0.26) (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). These findings highlight the superior performance of the integrated model over the individual models.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003ePerformance comparison of different AI models in prediction of CLNM\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"8\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTest cohort\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAUC\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003e95%CI\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eACC (%)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSEN (%)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eSPE (%)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003ePPV (%)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e \u003cp\u003eNPV (%)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResNet\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.8189\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e[0.7542, 0.8835]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e73.20\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e75.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e71.43\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e72.15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e74.32\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSVM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.7061*\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e[0.6246, 0.7875]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e63.40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e52.63\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e74.03\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e66.67\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e61.29\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eIntegrated Model\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.8406\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e[0.7792, 0.9020]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e77.12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e72.37\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e81.82\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e79.71\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e75.00\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"8\"\u003eAbbreviations: CLNM, central lymph node metastasis; ACC, accuracy; SEN, sensitivity; SPE, specificity; PPV, positive predictive value; NPV, negative predictive value; AUC, area under the receiver operating curve; CI, confidence interval *Compared with integrated model, p\u0026thinsp;\u0026lt;\u0026thinsp;0.05\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003ePerformance comparison among integrated model and radiologists\u003c/h2\u003e \u003cp\u003eThe results indicated that the integrated model exhibited a significantly higher AUC compared to both junior and senior radiologists (0.841 vs. 0.561 and 0.640, p\u0026thinsp;\u0026lt;\u0026thinsp;0.001). In comparison to the junior and senior radiologists, the integrated model demonstrated a decrease in the missed CLNM rate by 30.26% and 17.11% respectively. Additionally, the rate of unnecessary CLND decreased by 11.69% and 9.09%. A detailed comparison of the integrated model and radiologists were summarized in Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e. These results indicated that integrated model could improve the efficiency of metastatic LNs detection and reduce the rate of unnecessary CLND.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003ePerformance comparison of radiologists and integrated model in prediction of CLNM\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"10\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c10\" colnum=\"10\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTest cohort\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAUC\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003e95%CI\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eUndetected CLNM (%)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eUnnecessary CLND (%)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eACC (%)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eSEN (%)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e \u003cp\u003eSPE (%)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c9\"\u003e \u003cp\u003ePPV (%)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c10\"\u003e \u003cp\u003eNPV (%)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eJunior Radiologists\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.5612*\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e[0.4852, 0.6371]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e57.89\u003c/p\u003e \u003cp\u003e(44/76)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e29.87\u003c/p\u003e \u003cp\u003e(23/77)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e56.21\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e42.11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e70.13\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e58.18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e55.10\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSenior Radiologists\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.6400*\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e[0.5646, 0.7153]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e44.74\u003c/p\u003e \u003cp\u003e(34/76)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e27.27\u003c/p\u003e \u003cp\u003e(21/77)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e64.05\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e55.26\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e72.73\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e66.67\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e62.22\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eIntegrated Model\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.8406\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e[0.7792, 0.9020]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e27.63\u003c/p\u003e \u003cp\u003e(21/76)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e18.18\u003c/p\u003e \u003cp\u003e(14/77)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e77.12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e72.37\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e81.82\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e79.71\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e75.00\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"10\"\u003eAbbreviations: CLNM, central lymph node metastasis; CLND, central lymph node dissection; ACC, accuracy; SEN, sensitivity; SPE, specificity; PPV, positive predictive value; NPV, negative predictive value; AUC, area under the receiver operating curve; CI, confidence interval *Compared with integrated model, p\u0026thinsp;\u0026lt;\u0026thinsp;0.05\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003eExplanation of the integrated model\u003c/h2\u003e \u003cp\u003eTo better compensate for the problem of \"cognitive opacity\" of AI models, we utilized SHAP plots to illustrate the contribution of each key parameter in the integrated model. The result showed that the DL model contributed the most to CLNM prediction, followed by ETE, tumour size, age, gender, and multifocality. The radiomics model played a relatively minor role within the integrated model (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e). In Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e, two representative examples were presented to demonstrate how each key parameter contributed to the personalized decision-making process in the integrated model. Furthermore, we employed Grad-CAM to identify the areas of interest for the DL model. Figure\u0026nbsp;\u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e6\u003c/span\u003e showcased several representative cases, indicating that the areas of interest were predominantly located around the thyroid capsule, consistent with the radiologists focusing on areas significantly associated with CLNM.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eIn this study, we developed an integrated model for predicting CLNM that incorporated deep learning, radiomics, and clinical and US features. Our integrated model outperformed models based solely on radiomics or DL features, as well as junior and senior radiologists. The integrated model decreased the rate of missed CLNM and unnecessary CLND, thus improving preoperative CLND decision-making. Furthermore, the integrated model's visual explanation aligned with radiologists' typical judgments, which contributed to the acceptance of AI-assisted US diagnosis.\u003c/p\u003e \u003cp\u003eCurrently, one of the primary objectives of US in patients with PTC is to provide preoperative guidance for CLND by detecting the presence of CLNM. However, the presence of air interference in the trachea and esophagus, along with the small size of LNs, leads to unsatisfactory diagnostic accuracy [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e]. Encouragingly, radiomics and DL methods can effectively reveal information that is imperceptible to the human eye, thereby enhancing diagnostic capabilities. Previous studies focusing solely on either radiomics or DL methods in diagnosing CLNM have yielded favourable results [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e, \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]. However, our findings indicated limitations in the diagnostic efficacy of standalone DL and radiomics models. The DL model exhibited higher sensitivity, while the radiomics model showed higher specificity, indicating a distinction between traditional image features extracted by radiomics and the high-dimensional features extracted by DL. These observations motivated us to develop an integrated model that combines both types of features, resulting in superior performance compared to models based solely on radiomics or DL features. Furthermore, when compared with junior and senior radiologists, the integrated model significantly reduced the missed rate of CLNM by 30.26% and 17.11%, respectively, and decreased the rate of unnecessary CLND by 11.69% and 9.09%. Our findings indicate that the utilization of this model in clinical practice can be beneficial for PTC patients. Radiologists also can benefit from the integrated model, as it can serve as a valuable second opinion during the diagnosis of CLNM, assisting them in making more precise judgments and boosting their diagnostic confidence.\u003c/p\u003e \u003cp\u003eConsistent with the results of previous studies [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e], we conducted a screening of clinical and US factors associated with CLNM during routine diagnostic work. These factors were then integrated into our AI model, resulting in improved efficacy. Upon further analysis using the SHAP plot, the integrated model demonstrated that the clinical and US factors provided valuable additional information. Among these crucial factors, ETE had the highest contribution, indicating that tumor cells could breach the thyroid capsule and enter the lymphatic system, leading to the development of metastatic LNs [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e]. Additionally, tumor size, gender, age, and multifocality were also found to be associated with CLNM [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]. Interestingly, our findings revealed that the integrated model focused primarily on the thyroid capsule, which aligns with the areas of emphasis for radiologists when assessing CLNM. These results suggest that the clinical and US factors incorporated into the integrated model, as well as the regions of the model's interest, are generally consistent with radiologists' judgments, thereby providing the model with some clinical explainability. Overall, the visual explanation provided by the integrated model not only aligns with radiologists' usual judgments but also the integrated model demonstrates higher diagnostic efficacy compared to radiologists. This enhances the clinical acceptance of AI-assisted US diagnosis among radiologists.\u003c/p\u003e \u003cp\u003eIn contrast to the integrated models derived from CT or MRI images, where radiomics features played a prominent role [\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e], our findings indicated that the contribution of radiomics features to our integrated model was relatively modest. This discrepancy may arise from the fact that some of the features extracted from US images through radiomics, such as shape, grayscale, and texture, can also be obtained through DL methods. Additionally, during US imaging, noise can be generated due to variations in signal intensity, which can degrade image quality and affect the extraction of certain radiomics features. Consequently, these circumstances may account for the relatively limited contribution of radiomics to the model.\u003c/p\u003e \u003cp\u003eSeveral limitations should be acknowledged in this study. Firstly, due to the interference of anatomical structures and the small size of the central LNs, US images of central LNs were not included in the analysis. Secondly, although the incorporation of clinical and US features enhances the acceptance of AI-assisted US diagnosis by radiologists, the interpretability of features learned by the DL and radiomics model remains limited. Future advancements in the field of interpretable AI will inspire further exploration. Finally, the results obtained may be influenced by the limited amount of data utilized. Further investigation of the value of integrated models in prospective studies with larger sample sizes is warranted.\u003c/p\u003e \u003cp\u003eIn conclusion, the integrated model demonstrated superior performance compared to models relying solely on radiomics or DL features, exceeding the diagnostic capabilities of both junior and senior radiologists. The application of integrated models can significantly reduce missed CLNMs and unnecessary CLNDs along with increasing radiologists' acceptance of AI-assisted US diagnoses.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cp\u003eAI Artificial intelligence\u003c/p\u003e\u003cp\u003eAUC Area under curve\u003c/p\u003e\u003cp\u003eCLND Central lymph node dissection\u003c/p\u003e\u003cp\u003eCLNM Central lymph node metastases\u003c/p\u003e\u003cp\u003eDL Deep learning\u003c/p\u003e\u003cp\u003eGrad-CAM Gradient-weighted Class Activation Mapping\u003c/p\u003e\u003cp\u003eLN Lymph node\u003c/p\u003e\u003cp\u003ePTC Papillary thyroid carcinoma\u003c/p\u003e\u003cp\u003eROI Region of interest\u003c/p\u003e\u003cp\u003eSHAP SHapley Additive exPlanation\u003c/p\u003e\u003cp\u003eSVM Support vector machine\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eEthics approval and consent to participate\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe study has been performed in accordance with the Declaration of Helsinki and was approved by The Ethics Committees of Nanfang Hospital of South Medical University and the First People\u0026rsquo;s Hospital of Foshan (NFEC-202008-K6). The requirement for informed consent was waived by the Ethics Committee of Nanfang Hospital of South Medical University because of the retrospective nature of the study.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent for publication\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAvailability of data and materials\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors declare no conflicts of interest.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis study was funded by the National Natural Science Foundation of China (Grant Nos. 82271998 and 82071949)\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthors\u0026rsquo; contributions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eYJ and YG were contributed to the conception and design of the study. WJ organized the database. WY and YY were responsible for the software and statistics. YG and WZ took charge of the writing of this paper. ST, YS, and TL to data collection. ZT, LY for data analysis. All authors have read and approved the manuscript\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eSung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, et al. Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries. CA Cancer J Clin. 2021;71:209\u0026ndash;49.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWhite ML, Gauger PG, Doherty GM. Central lymph node dissection in differentiated thyroid cancer. World J Surg. 2007;31:895\u0026ndash;904.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang TS, Sosa JA. Thyroid surgery for differentiated thyroid cancer \u0026mdash; recent advances and future directions. Nat Rev Endocrinol. 2018;14:670\u0026ndash;83.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMedas F, Canu GL, Cappellacci F, Anedda G, Conzo G, Erdas E, et al. Prophylactic central lymph node dissection improves disease-free survival in patients with intermediate and high risk differentiated thyroid carcinoma: A retrospective analysis on 399 patients. Cancers (Basel). 2020;12:1\u0026ndash;12.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAdam MA, Pura J, Goffredo P, Dinan MA, Reed SD, Scheri RP, et al. Presence and number of lymph node metastases are associated with compromised survival for patients younger than age 45 years with papillary thyroid cancer. J Clin Oncol. 2015;33:2370\u0026ndash;5.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChinese Society of Endocrinology; Thyroid and Metabolism Surgery Group of the Chinese Society of Surgery;Chinese Society of Nuclear Medicine. Chinese Society of Ultrasound in Medicine. Guidelines for the diagnosis and management of thyroid nodules and differentiated thyroid cancer (Second edition). Chin J Endocrinol Metab. 2023;39:181\u0026ndash;226.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHaugen BR, Alexander EK, Bible KC, Doherty GM, Mandel SJ, Nikiforov YE, et al. 2015 American Thyroid Association Management Guidelines for Adult Patients with Thyroid Nodules and Differentiated Thyroid Cancer: The American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer. Thyroid. 2016;26:1\u0026ndash;133.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhao H, Li H. Meta-analysis of ultrasound for cervical lymph nodes in papillary thyroid cancer: Diagnosis of central and lateral compartment nodal metastases. Eur J Radiol. 2019;112:14\u0026ndash;21.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLambin P, Leijenaar RTH, Deist TM, Peerlings J, De Jong EEC, Van Timmeren J, et al. Radiomics: The bridge between medical imaging and personalized medicine. Nat Rev Clin Oncol. 2017;14:749\u0026ndash;62.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu S, Wang Y, Yang X, Lei B, Liu L, Li SX, et al. Deep Learn Med Ultrasound Analysis: Rev Eng. 2019;5:261\u0026ndash;75.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLevine AB, Schlosser C, Grewal J, Coope R, Jones SJM, Yip S. Rise of the Machines: Advances in Deep Learning for Cancer Diagnosis. Trends in cancer. 2019;5:157\u0026ndash;69.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang W, Li Y. [Radiomics for prediction of central lymph node metastasis in the neck in patients with thyroid papillary carcinoma]. Nan Fang Yi Ke Da Xue Xue Bao. 2019;39:1094\u0026ndash;8.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang S, Hou Y, Li Z, Dong J, Tang C. Combining ConvNets with hand-crafted features for action recognition based on an HMM-SVM classifier. Multimed Tools Appl. 2018;77:18983\u0026ndash;98.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAntropova N, Huynh BQ, Giger ML. A deep feature fusion methodology for breast cancer diagnosis demonstrated on three imaging modality datasets. Med Phys. 2017;44:5162\u0026ndash;71.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDai Q, Liu D, Tao Y, Ding C, Li S, Zhao C, et al. Nomograms based on preoperative multimodal ultrasound of papillary thyroid carcinoma for predicting central lymph node metastasis. Eur Radiol. 2022;32:4596\u0026ndash;608.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang J, Yuan C, Han C, Wen Y, Lu H, Liu C, et al. IMAL-Net: Interpretable multi-task attention learning network for invasive lung adenocarcinoma screening in CT images. Med Phys. 2021;48:7913\u0026ndash;29.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhou J, Yin L, Wei X, Zhang S, Song Y, Luo B, et al. 2020 Chinese guidelines for ultrasound malignancy risk stratification of thyroid nodules: the C-TIRADS. Endocrine. 2020;70:256\u0026ndash;79.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAdler DD, Carson PL, Rubin JM, Quinn-Reid D. of Breast Cancer: Preliminary Findings. Ultrasound Med Biol. 1990;16:553\u0026ndash;9.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTessler FN, Middleton WD, Grant EG, Hoang JK, Berland LL, Teefey SA, et al. ACR Thyroid Imaging, Reporting and Data System (TI-RADS): White Paper of the ACR TI-RADS Committee. J Am Coll Radiol. 2017;14:587\u0026ndash;95.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003evan Griethuysen JJM, Fedorov A, Parmar C, Hosny A, Aucoin N, Narayan V, et al. Computational Radiomics System to Decode the Radiographic Phenotype. Cancer Res. 2017;77:e104\u0026ndash;7.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDeng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L. ImageNet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. 2009. p.\u0026nbsp;248\u0026ndash;55.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKolesnikov A, Beyer L, Zhai X, Puigcerver J, Yung J, Gelly S et al. Big Transfer (BiT): General Visual Representation Learning. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics). 2020;12350 LNCS:491\u0026ndash;507.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHwang HS, Orloff LA. Efficacy of preoperative neck ultrasound in the detection of cervical lymph node metastasis from thyroid cancer. Laryngoscope. 2011;121:487\u0026ndash;91.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhou SC, Liu TT, Zhou J, Huang YX, Guo Y, Yu JH, et al. An Ultrasound Radiomics Nomogram for Preoperative Prediction of Central Neck Lymph Node Metastasis in Papillary Thyroid Carcinoma. Front Oncol. 2020;10:1\u0026ndash;13.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWu Y, Rao K, Liu J, Han C, Gong L, Chong Y, et al. Machine Learning Algorithms for the Prediction of Central Lymph Node Metastasis in Patients With Papillary Thyroid Cancer. Front Endocrinol (Lausanne). 2020;11:1\u0026ndash;9.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZheng X, Yao Z, Huang Y, Yu Y, Wang Y, Liu Y, et al. Deep learning radiomics can predict axillary lymph node status in early-stage breast cancer. Nat Commun. 2020;11:1236.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZou Y, Shi Y, Liu J, Cui G, Yang Z, Liu M, et al. A Comparative Analysis of Six Machine Learning Models Based on Ultrasound to Distinguish the Possibility of Central Cervical Lymph Node Metastasis in Patients With Papillary Thyroid Carcinoma. Front Oncol. 2021;11:1\u0026ndash;10.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eXia E, Chi Y, Jin L, Shen Y, Hirachan S, Bhandari A, et al. Preoperative prediction of lymph node metastasis in patients with papillary thyroid carcinoma by an artificial intelligence algorithm. Am J Transl Res. 2021;13:7695\u0026ndash;704.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePaul R, Hawkins SH, Schabath MB, Gillies RJ, Hall LO, Goldgof DB. Predicting malignant nodules by fusing deep features with classical radiomics features. J Med Imaging. 2018;5:1.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHu X, Gong J, Zhou W, Li H, Wang S, Wei M et al. Computer-aided diagnosis of ground glass pulmonary nodule by fusing deep learning and radiomics features. Phys Med Biol. 2021;66.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-cancer","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"bcan","sideBox":"Learn more about [BMC Cancer](http://bmccancer.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/bcan/default.aspx","title":"BMC Cancer","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Ultrasonography, Papillary Thyroid Carcinoma, Lymph Node Metastasis, Deep Learning","lastPublishedDoi":"10.21203/rs.3.rs-3218322/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3218322/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eObjective\u003c/h2\u003e \u003cp\u003eTo evaluate the value of an integrated model incorporating deep learning (DL), radiomics and clinical and US imaging features for diagnosing central lymph node metastasis (CLNM) in patients with papillary thyroid cancer (PTC).\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eThis retrospective study reviewed 613 patients with clinicopathologically confirmed PTC from two institutions. The DL model and radiomics model were developed using primary lesion images and then integrated with clinical and US features selected by multivariate analysis to generate an integrated model. The performance was compared with junior and senior radiologists on the independent test set. SHapley Additive exPlanations (SHAP) plot and Gradient-weighted Class Activation Mapping (Grad-CAM) were used for the visualized explanation of the model.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eThe integrated model yielded the best performance with an AUC of 0.841. surpassing that of the radiomics model (0.706, p\u0026thinsp;\u0026lt;\u0026thinsp;0.001) and the DL model (0.819, p\u0026thinsp;=\u0026thinsp;0.26). Compared to junior and senior radiologists, the integrated model reduced the missed CLNM rate from 57.89% and 44.74\u0026ndash;27.63%, and decreased the rate of unnecessary central lymph node dissection (CLND) from 29.87% and 27.27\u0026ndash;18.18%, respectively. SHAP analysis revealed that the DL features played a primary role in the diagnosis of CLNM, while clinical and US features (such as extrathyroidal extension, tumor size, age, gender, and multifocality) provided additional support. Grad-CAM indicated that the model exhibited a stronger focus on thyroid capsule in patients with CLNM.\u003c/p\u003e\u003ch2\u003eConclusion\u003c/h2\u003e \u003cp\u003eIntegrated model can effectively decrease the incidence of missed CLNM and unnecessary CLND. The application of the integrated model can help improve the acceptance of AI-assisted US diagnosis among radiologists.\u003c/p\u003e","manuscriptTitle":"An integrated model incorporating deep learning, radiomics and clinical and US features to diagnose central lymph node metastasis in patients with papillary thyroid cancer","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2023-08-31 14:36:19","doi":"10.21203/rs.3.rs-3218322/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2023-11-07T05:22:47+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2023-10-21T12:42:58+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"4bb00a97-0ef1-4658-8517-bcc7ec67b1af","date":"2023-10-11T11:19:03+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2023-09-26T12:50:54+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"6a5cdeb0-2280-4f1a-be58-01718b2ddb5b","date":"2023-09-15T14:30:06+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2023-09-15T06:54:57+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2023-09-15T06:53:05+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2023-08-28T06:48:30+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2023-08-28T06:45:54+00:00","index":"","fulltext":""},{"type":"submitted","content":"BMC Cancer","date":"2023-07-30T16:51:07+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-cancer","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"bcan","sideBox":"Learn more about [BMC Cancer](http://bmccancer.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/bcan/default.aspx","title":"BMC Cancer","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"f530260d-4b63-4213-955f-bafc6cb18c84","owner":[],"postedDate":"August 31st, 2023","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2024-01-15T15:07:20+00:00","versionOfRecord":{"articleIdentity":"rs-3218322","link":"https://doi.org/10.1186/s12885-024-11838-1","journal":{"identity":"bmc-cancer","isVorOnly":false,"title":"BMC Cancer"},"publishedOn":"2024-01-12 15:02:00","publishedOnDateReadable":"January 12th, 2024"},"versionCreatedAt":"2023-08-31 14:36:19","video":"","vorDoi":"10.1186/s12885-024-11838-1","vorDoiUrl":"https://doi.org/10.1186/s12885-024-11838-1","workflowStages":[]},"version":"v1","identity":"rs-3218322","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3218322","identity":"rs-3218322","version":["v1"]},"buildId":"_2-kVJe1T_tPrBINL-cwx","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}