Deep learning from ultrasound images of breast cancer sentinel lymph nodes to predict metastasis

Deep learning from ultrasound images of breast cancer sentinel lymph nodes to predict metastasis | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Deep learning from ultrasound images of breast cancer sentinel lymph nodes to predict metastasis Yunhao Luo, Zhuo Wei, Jie Chen, Wenbin Cao, Zhengquan Feng, Chaonan Li, and 5 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4437751/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Objectives This study aims to develop a non-invasive assessment in breast cancer sentinel lymph node (SLN) using deep learning. Materials and methods Continuously retrospective patients with breast cancer who have undergone both contrast-enhanced ultrasound (CEUS) and two-dimensional ultrasound (TDUS) for sentinel lymph node examination. Those patients were randomly divided into training set, validation set, and internal test set in a ratio of 8:1:1. A Re-parameterization Visual Geometry Group-Convolutional Block Attention Module (RepVGG-CBAM) model was constructed based on the RepVGG network, embedding the CBAM attention mechanism. The area under the receiver operating characteristic curve (AUC) was used to evaluate diagnostic performance. Results In the test set, the AUC were experts in TDUS, CEUS, and combination ultrasound (CBUS), model in TDUS, CEUS and CBUS were 0.794, 0.806, 0.774, 0.861, 0.851, 0.842 respectively. The difference in AUC between Experts in TDUS (0.794) and Model in TDUS (0.861) was statistically significant ( p = 0.043). The difference in AUC between Experts in TDUS (0.794) and Model in CEUS (0.851) was statistically significant ( p< 0.01). The difference in AUC between Experts in CBUS (0.774) and Model in TDUS (0.861) was statistically significant ( p = 0.007). The difference in AUC between Experts in CBUS (0.774) and Model in CEUS (0.851) was statistically significant ( p< 0.001). Conclusions An algorithm model was developed to determine the SLN metastasis status of breast cancer patients. Breast cancer Sentinel lymph nodes Contrast-enhanced ultrasound RepVGG CNN. Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Key Points AI Model Surpasses Expert Diagnostic Performance: The AI model developed in this study, based on the RepVGG-CBAM, has demonstrated a higher diagnostic efficiency in determining the sentinel lymph node (SLN) metastasis status in breast cancer patients compared to human experts. Application of Deep Learning in Medical Image Analysis: By integrating contrast-enhanced ultrasound (CEUS) and two-dimensional ultrasound (TDUS) images, the AI model is capable of directly learning the ultrasound image features of SLN to ascertain the presence of metastasis, which has not been previously reported. Enhanced Clinical Accuracy and Reduced Need for Invasive Surgery: The AI model aids in reducing the false-negative rate in determining SLN metastasis and improves the accuracy of preoperative SLN assessment, potentially reducing the need for invasive SLN biopsies in the future. Introduction Breast cancer has now become the most common malignancy worldwide and the second leading cause of death from cancer in women [ 1 ]. The status of axillary lymph nodes (ALN) is one of the most important factors for evaluating the clinical staging, treatment modality, and prognosis of breast cancer patients [ 2 – 4 ]. The sentinel lymph node (SLN) is the first lymph node in the lymphatic system to receive lymphatic drainage and is the first site for ALN metastasis in breast cancer patients [ 5 ]. It is clinically important to accurately determine the presence of SLN metastasis preoperatively, and the standard procedure is sentinel lymph node biopsy (SLNB) [ 6 ]. Studies have shown that 80–85% of early-stage breast cancer patients do not have ALN metastasis, resulting in most patients undergoing invasive excisional biopsy, increasing intraoperative waiting time for biopsy results, and also indicating that the existing preoperative clinical assessment is not accurate enough [ 7 ]. The SOUND study further confirmed that the 5-year distant disease-free survival rate, axillary recurrence rate, distant recurrence rate, disease-free survival rate and overall survival rate of breast cancer patients with breast cancer less than 2 cm in diameter and negative preoperative axillary ultrasound results were similar between SLNB and no axillary surgery. Future axillary management of breast cancer patients is gradually becoming exempt, therefore, accurate evaluation of SLN is crucial [ 8 ]. A multicenter study [ 9 ] showed that preoperative contrast-enhanced ultrasound (CEUS) can be used to accurately identify and localize SLN in breast cancer patients, and previous studies in our center [ 10 , 11 ] also confirmed that CEUS guidance can accurately localize SLN, but it is still difficult to determine the presence of metastasis in SLN [ 12 – 18 ]. With the development of artificial intelligence (AI), deep learning models, especially convolutional neural network (CNN), have been shown to be comparable to human experts in determining the ALN status of early-stage breast cancer [ 19 ]. AI models based on deep learning can extract and recognize detailed features in medical images that are invisible to human experts, and have been widely used in medical image segmentation and intelligent prediction fields [ 20 , 21 ]. However, research on directly learning SLN ultrasound images using deep learning models to determine the presence of metastasis has not been reported. Therefore, in this study, we aim to construct separate datasets of two-dimensional ultrasound (TDUS) and CEUS images of SLNs and design an algorithm model to determine SLN metastasis. Materials and methods Basic research information This study was approved and exempted from informed consent by the Ethics Committee of Sichuan Provincial People's Hospital (Approval Number: 2022.261). This study is a retrospective study and informed consents were waived. This work complied with the Declaration of Helsinki. This study included SLNs images, including TDUS and CEUS images, of pathologically diagnosed breast cancer patients from June 2017 to October 2022 in Sichuan Provincial People's Hospital. All SLNs were performed with surgical resection pathology findings as the gold standard. The overall research process was shown in Fig. 1 . Inclusion Criteria: 1. Breast cancer patients with SLNB including T1-3 and ductal carcinoma in situ (DCIS) scheduled for total mastectomy. 2. All patients were pathologically confirmed as having malignant breast tumors by preoperative puncture biopsy. 3. All patients underwent contrast-enhanced ultrasound through the lymphatic pathway to identify and locate the SLN prior to surgery. Exclusion Criteria: 1. ALN was confirmed as metastatic carcinoma by biopsy before surgery; 2. Previous breast or axillary radiotherapy or chemotherapy; 3. Advanced breast cancer (T4a-c) and inflammatory breast cancer (T4d); 4. Bilateral breast cancer patients; 5. Male breast cancer patients; 6. Ultrasound image quality cannot reach the requirements of AI analysis. Access to medical images Philips iU-Elite and GE logic-E9 ultrasonic diagnostic devices, equipped with CEUS display system, were selected for images acquisition. L12-5 (Philips) and ML6-15 (GE) linear array probes were used for TDUS examination, and L9-3 (Philips) and 9L (GE) linear array probes were used for CEUS examination. SonoVue (Bracco spa, Milan, Italy) with a low mechanical index (MI) of 0.07 (Philips) and 0.16 (GE) was used as the ultrasound contrast agent. The SLNs of breast cancer patients were located by injecting the ultrasound contrast agent into the areola region [ 9 – 11 ]. TDUS and CEUS images data acquisition were performed at the longest diameter level of each SLN, both including at least one static image and at least 15 seconds of dynamic video, and the image acquisition was performed by three experts (Jun Luo, Wenbin Cao, and Yunhao Luo. with more than 15 years, 7 years, 5 years of experience in breast ultrasound diagnosis, respectively). The ultrasound images to be included in the study were reviewed by a senior expert Jun Luo) for image quality, and when the diagnosis was more difficult, the senior expert (Qin Chen. with more than 20 years of experience in breast ultrasound diagnosis) reviewed the images, and cases with poor image quality were excluded. Artificial intelligence data preprocessing of medical images Deep learning researchers (Jianhua Deng, Zhuo Wei, and Zhengquan Feng, with more than 15 years, 3 years, and 2 years of experience in AI image processing, respectively) collaborated with medical experts (Wenbin Cao, Yunhao Luo) to annotate the images. The annotations were then reviewed by senior experts (Jun Luo, Qin Chen) to ensure accurate and error-free delineation of the SLN regions. Finally, all the SLNs images were randomly divided into training set, verification set and internal test set according to the ratio of 8:1:1. Due to the limited number of images in the dataset, the offline data enhancement method was adopted in this study to expand the dataset, and the images were flipped horizontally, rotated 180°, and flipped horizontally after 180° rotation, and the final training set reached four times the original dataset. The TDUS images and CEUS images collected in this study are shown in Fig. 2 . Deep Convolutional Neural Network The establishment of the AI model was completed by the researchers (Jianhua Deng, Zhuo Wei, and Zhengquan Feng). The model was based on the Re-parameterization Visual Geometry Group (RepVGG) network [ 22 ] and incorporates spatial attention and channel attention mechanisms. As shown in Fig. 3 , the Convolutional Block Attention Module (CBAM) [ 23 ] attention mechanism module was embedded between the convolutional layers and ReLU activation layers after Stage 0 and Stage 4 of RepVGG-A0, resulting in the neural network model used in this study, RepVGG-CBAM. Taking TDUS images as an example, the AI analysis process consisted of training and inference stages. During the training process, the dataset were classified as negative (no metastasis) or positive (metastasis) according to the SLN gold standard. Under various parameter settings, the data was input into the neural network model for training, and the best model was automatically saved. Finally, the trained model was obtained for the inference stage. During the inference process, unknown SLN images were input into the inference model, and relevant parameters were set to obtain the model's output result, indicating whether it was negative or positive. Artificial Intelligence Model Training 1. Experimental environment. This study was conducted in the Digital Information Systems Laboratory at the University of Electronic Science and Technology of China. The experimental hardware was configured as follows: Linux operating system version was Ubuntu 18.10 LTS, CPU model was Inteli9-9900K, memory size was 64G, two GPU graphics cards model was Nvidia GeForce RTX 2080 Ti, graphics card memory was 12G. The software configurations were: Python 3.7, Pytorch 1.2.0, CUDA 10.1, Torchvision 0.4.0, and tensorboardX 2.6. 2. Experimental parameters. Following the idea of migration learning, the weights of the pre-trained model of RepVGG-A0 version on ImageNET were used as the initial weights. During the training process, the epoch was set to 1000, the batch_size was set to 16, the initial learning rate was set to 0.01, and the learning rate was jointly adjusted by a combination of equal-step decay and cosine annealing decay during the training process, and the optimizer selected SGD. Medical image evaluation Three experts (Yunhao Luo, Wenbin Cao, and Jun Luo) jointly determined the test set data as benign or malignant. If the three diagnoses diverged and disagreed, the senior expert (Qin Chen.) made the diagnosis. All experts were blinded to clinical and histologic outcomes. Images meeting the negative diagnostic criteria were considered as no metastasis, and those meeting the suspicious positive and positive diagnostic criteria were considered as metastasis. The diagnosis criteria of the test set experts were as follows: 1. For TDUS images, this study was divided into three lymph node statuses as shown in Fig. 4 . (1) Negative: long diameter/short diameter > 2, uniform cortex, clear corticomedullary demarcation, identifiable lymphatic portal, no abnormal echogenicity in the cortex, and lymphatic portal-type blood flow signal was visible on CDFI. (2) Suspicious positive: ①uneven cortical thickening, the thicker part was > 2 times compared with other cortical thickness measurements; ②uniform cortical thickening, but the morphology was slightly full, the long diameter/short diameter was < 2, the corticomedullary demarcation was clear, the lymphatic portal-type was identifiable, there was no abnormal echogenicity in the cortex, and the lymphatic portal-type blood flow signal was visible on CDFI. (3) Positive: ①complete morphology, long/short diameter < 2, indistinct corticomedullary demarcation, disappearance of lymphatic portals, non-lymphatic flow signal was visible on CDFI; ②abnormal cortical echogenicity (calcification, liquefaction, abnormal hyperechoic). 2. For CEUS images, this study was divided into three lymph node statuses as shown in Fig. 5 . (1) Negative: uniform enhancement. (2) Suspicious positive: non-uniform enhancement and cortical enhancement defects; (3) Positive: ①no enhancement (except for lymphatic vessel interruption leading to inability to visualize lymph nodes); ②circumferential enhancement with uniform or uneven enhancement within it. Internal test set The internal test set used experts and models to test TDUS, CEUS, and combination ultrasound(CBUS), for a total of 6 test groups. CBUS means combines TDUS and CEUS images to determine together. If a case in the TDUS or CEUS group was determined to be metastatic, the CBUS group for that case was also determined to be metastatic. Statistical Methods SPSS 22.0 was used for statistical analysis. The central tendency of quantitative data was expressed as mean, and the degree of dispersion was expressed as standard deviation. The relative amounts of qualitative information were expressed as rates. The accuracy, sensitivity, specificity, positive predictive value, negative predictive value, and the area under the receiver operating characteristic curve (AUC) of experts and AI models in diagnosing SLN metastasis of breast cancer were calculated by comparing of AUC differences using the Delong test. Results Patient Characteristics The basic information of the included patients was shown in Table 1 . A total of 972 patients were included. Other pathologic types included invasive mucinous carcinoma, invasive micropapillary carcinoma, tubule carcinoma, secretory carcinoma, adenoid cystic carcinoma, mammary degenerative carcinoma and mixed breast cancer. Table 1 Characteristics of Patients Variable Training set (n = 772) Validation set (n = 100) internal test set (n = 100) P Age 0.787 51.36 ± 10.62 50.68 ± 10.84 50.86 ± 12.42 Left or right 0.695 Left 391 48 54 Right 381 52 46 Metastasis 0.163 No 628 88 78 Yes 144 12 22 Pathological types of tumor 0.233 DCIS 99 17 18 IDC 586 69 75 ILC 21 1 1 Others 66 13 6 T stage of tumor 0.288 T1mi 24 5 5 T1a 20 6 4 T1b 77 7 7 T1c 299 33 40 T2 243 32 24 T3 10 0 1 Tis 99 17 18 Molecular typing of tumor 0.066 Basal-like 80 9 6 HER2 over-expression 94 6 11 Luminal A 138 11 17 Luminal B HER2 (-) 270 35 37 Luminal B HER2 (+) 91 22 11 Tis 99 17 18 DCIS:Ductal carcinoma in situ; IDC:Infiltrating ductal carcinoma; ILC:Infiltrating lobular carcinoma Inference Results of the experts and RepVGG-CBAM Model on the Test Set The trained models were separately inferred and tested on the internal test set. The test results when the binary cutoff value is set to 0.2. The Receiver Operating Characteristic (ROC) curves of the experts and model test results were shown in Fig. 6 . The experimental performance were shown in Table 2 . The paired-sample area difference under the ROC curves were shown in Table 3 . Table 2 Experimental performance of Experts and RepVGG-CBAM model in the test set. Evaluation methods AUC (95%Cl) SENS (95%Cl) SPEC (95%Cl) PPV (95%Cl) NPV (95%Cl) ACC (95%Cl) Experts in TDUS 0.794 0.699 0.889 0.818 0.597 0.948 0.769 0.660 0.857 0.500 0.389 0.611 0.938 0.860 0.974 0.780 0.686 0.857 Experts in CEUS 0.806 0.704 0.908 0.727 0.498 0.893 0.885 0.792 0.946 0.640 0.477 0.776 0.920 0.853 0.958 0.850 0.765 0.914 Experts in CBUS 0.774 0.678 0.871 0.818 0.597 0.948 0.731 0.618 0.825 0.462 0.361 0.565 0.934 0.853 0.972 0.750 0.653 0.831 Model in TDUS 0.861 0.778 0.944 0.864 0.651 0.971 0.859 0.762 0.927 0.633 0.494 0.754 0.957 0.886 0.985 0.860 0.776 0.921 Model in CEUS 0.851 0.762 0.941 0.818 0.597 0.942 0.885 0.792 0.946 0.667 0.512 0.792 0.945 0.876 0.977 0.870 0.788 0.929 Model in CBUS 0.842 0.757 0.927 0.864 0.651 0.971 0.821 0.717 0.898 0.576 0.451 0.692 0.955 0.881 0.984 0.830 0.742 0.898 The data in brackets are the 95% confidence intervals. Test set, n = 100. Abbreviations: AUC: area under the receiver operating characteristic curve; ACC: accuracy; SENS: sensitivity; SPEC: specificity; PPV: positive predictive value; NPV: negative predictive value. Table 3 Paired-Sample Area Difference Under the ROC Curves Test Result Pair(s) Asymptotic AUC Std. Error Asymptotic 95% Confidence Interval z P Difference Difference Lower Bound Upper Bound Experts in TDUS - Experts in CEUS -0.309 0.757 -0.012 0.311 -0.090 0.065 Experts in TDUS - Experts in CBUS 1.755 0.079 0.019 0.305 -0.002 0.041 Experts in TDUS - Model in TDUS -2.025 0.043 -0.068 0.296 -0.133 -0.002 Experts in TDUS - Model in CEUS -2.586 <0.01 -0.058 0.301 -0.101 -0.014 Experts in TDUS - Model in CBUS -1.524 0.128 -0.048 0.298 -0.111 0.014 Experts in CEUS - Experts in CBUS 0.839 0.401 0.031 0.312 -0.042 0.105 Experts in CEUS - Model in TDUS -1.330 0.184 -0.055 0.303 -0.137 0.026 Experts in CEUS - Model in CEUS -1.391 0.164 -0.045 0.307 -0.109 0.019 Experts in CEUS - Model in CBUS -0.881 0.378 -0.036 0.304 -0.116 0.044 Experts in CBUS - Model in TDUS -2.685 0.007 -0.087 0.297 -0.150 -0.023 Experts in CBUS - Model in CEUS -3.742 <0.001 -0.077 0.301 -0.117 -0.037 Experts in CBUS - Model in CBUS -2.195 0.028 -0.068 0.299 -0.128 -0.007 Model in TDUS - Model in CEUS 0.340 0.734 0.010 0.292 -0.047 0.067 Model in TDUS - Model in CBUS 1.755 0.079 0.019 0.287 -0.002 0.041 Model in CEUS - Model in CBUS 0.350 0.727 0.009 0.293 -0.043 0.062 Discussion In this study, the research team successfully established an AI prediction model using the RepVGG-CBAM model to determine SLN metastasis in breast cancer patients. The results on the test set were satisfactory, with the diagnostic performance exceeding the expert level. For SLN metastasis, the higher sensitivity were of greater clinical importance. Compared with experts diagnosis, in TDUS, CEUS and CBUS, the AI prediction models established in this study showed higher sensitivity. To our knowledge, this was the first study to use deep learning algorithms to directly learn ultrasound images of SLN in breast cancer to determine SLN metastasis. Zhou et al [ 21 ] conducted a deep learning study on TDUS features of primary breast cancer lesions to indirectly predict ALN metastasis, and achieved good diagnostic performance with an AUC of 0.89. Zha et al [ 24 ] predicted SLN metastasis in breast cancer by combining various clinical and pathological factors of the primary lesion, and achieved an AUC of 0.833 on the test set. However, the aforementioned studies did not learn the imaging characteristics of the lymph nodes themselves. In clinical practice, analysis of lymph node imaging, especially SLN, directly and accurately determines the presence or absence of metastasis. Our study precisely addresses this issue. Zheng et al [ 19 ] used AI to directly learn multimodal ultrasound images of ALN to predict metastasis, further improving diagnostic efficiency. However, ALN includes non-SLN, which may increase the false negative rate for predicting ALN metastasis, and the study did not include contrast-enhanced imaging, which can improve the diagnosis of SLN metastasis [ 15 ]. In this study, the research team obtained TDUS and CEUS of SLN after lymph node enhancement and established an AI model to directly learn and diagnose SLN metastasis. The study simulated a real clinical environment. SLNB has ushered in a minimally invasive era compared to traditional axillary lymph node dissection (ALND). However, SLNB required intraoperative waiting for frozen pathology results, which increased operative time and medical costs. In addition, frozen pathology results may not fully reflect the true presence or absence of SLN metastasis, thus increasing the risk of secondary surgery [ 25 , 26 ]. Studies have shown that 15–20% of patients have SLN metastasis [ 7 ]. In this study, the SLN metastasis rate was 18.31% (178/972), which is similar to previous research reports [ 27 ]. The use of the model developed by the research team can further reduce the false negative rate in determining SLNs metastases and improve the accuracy of preoperative SLNs assessment. For patients identified by the model as having SLNs metastases, it serves as a reminder for clinicians to perform meticulous operations during surgery to avoid missing suspicious SLNs. For the images identified as false negative by RepVGG-CBAM in the test set, including 4 patients (1 form CEUS, 3 form CBUS ). All the patients underwent ALND after SLNB. Postoperative pathology confirmed that the total number of axillary lymph node metastases on the affected side were 1 to 2, including 2 patients with only 1 micrometastasis. Studies have shown that for patients with 1 to 2 axillary lymph node metastases, axillary radiotherapy can achieve the same long-term prognosis as surgical resection [ 26 ]. Therefore, for patients planning to undergo postoperative axillary radiotherapy, SLN diagnosis using the model is expected to prevent some patients from undergoing invasive SLNB in the future, which deserves further study. SLN metastasis can be divided into isolated tumor cell clusters (ITC), micrometastasis, and macrometastasis. Because ultrasound images cannot observe the microvascular perfusion of SLNs, the ultrasound features of ITC and micrometastasis often appeared normal. In actual clinical work, experts cannot accurately determine micrometastasis or even ITC, as well as other non-morphological features. The model in this study also misdiagnosed for two cases of micrometastasis. Due to the extremely limited number of such cases there are certain limitations. However, with the advancement of clinical diagnostic techniques, updates in AI algorithms, and an increase in the number of cases, the combination of molecular biology and advanced AI models holds promise for overcoming the current research limitations in the future. AI has achieved practical results in the diagnosis of breast cancer and lymph node metastasis in its drainage area, several studies [ 19 , 20 , 28 – 30 ] have shown that AI has achieved the same or higher diagnostic efficiency than clinical experts. Compared to commonly used models such as ResNet [ 31 ], the RepVGG model was chosen as the foundation for this study. Its uniqueness lies in the use of the reparameterization concept, which converts the three-branch network structure during the training phase into an equivalent single-branch network during the inference phase. This approach significantly improved inference speed without compromising accuracy. In addition, considering the challenges of the ultrasound images used in this study, such as high noise and low image quality, the researchers incorporated the CBAM attention mechanism into the network structure of RepVGG during model design. This effectively enhanced the feature extraction capability of the neural network, thereby improving classification accuracy. In the context of image classification, the attention mechanism mimics human visual attention by further extracting and processing important "eye-catching" feature information. This helped the neural network focus on more important features, resulting in better performance. The main reason for the significant increase in the number of training data in the CEUS dataset compared to the TDUS dataset was that the storage time of CEUS image videos was longer than that of TDUS videos, which was necessary to meet the requirements of actual clinical work. Therefore, more CEUS images were obtained through video segmentation. In this study, the negative data volume in the CEUS dataset was much larger than the positive data volume. This was because the study only included patients with cN0 status which reflected the clinical reality. To prevent overfitting of the model, the described data augmentation methods were applied to the positive data in this study, resulting in a relatively balanced distribution of the final training dataset. In binary classification tasks, the output of a neural network was a probability value. In this study, the output of the model represented the probability of SLN positivity. Therefore, a threshold was needed, where if the output probability was below the threshold, it was classified as negative, and if it was above the threshold, it was classified as positive. The optimal range for the threshold can be determined based on the AUC. Considering the goal of the study to identify all true positives and minimize false negatives, a binary classification threshold of 0.2 was chosen after conducting comparative experiments. This means that if the model's output probability of SLN positivity for a particular image was less than 0.2, it was classified as negative, and if it was equal to or greater than 0.2, it was classified as positive. This study also has several limitations. Firstly, this was a retrospective study, and the data analysis was limited to images obtained in the past. Secondly, this was a single-center study with a small sample size, and future multicenter studies are needed to fully validate and optimize the model. Furthermore, although all ultrasounds were performed by experienced experts using a standardized procedure, real-world clinical work depends on the different conditions of different patients, and the quality of images may vary somewhat. Conclusion In this study, an algorithm model was designed and developed to determine the SLN metastasis status of breast cancer patients. The diagnostic efficiency of the model exceeds that of experts. Abbreviations Sentinel lymph node (SLN) Contrast-enhanced ultrasound (CEUS) two-dimensional ultrasound (TDUS) RepVGG-CBAM (Re-parameterization Visual Geometry Group - Convolutional Block Attention Module) area under the curve (AUC) axillary lymph nodes (ALN) sentinel lymph node biopsy (SLNB) convolutional neural network (CNN) ductal carcinoma in situ (DCIS) mechanical index (MI) axillary lymph node dissection (ALND) isolated tumor cell clusters (ITC) Declarations Acknowledgements Not applicable. Author contribution J.L. and J.H.D. guarantor of integrity of the entire study. Y.H.L., Z.W., J.L. and J.H.D. study concepts and design. Y.H.L. and Z.W. literature research. Y.H.L., Z.W., J.C., and J.L clinical studies. Y.H.L., Z.W., J.C., W.B.C., Z.Q.F.,C.N.L., Y.Y.L., Q.C.and J. L. experimental studies / data analysis. Y.H.L. and Z.W. statistical analysis. Y.H.L., Z.W., J.L. and J.H.D. manuscript preparation. Y.H.L., Z.W., J.L. and J.H.D. manuscript editing. Funding This study has received funding by Sichuan Provincial Health Commission (universal application) Project Number: 21PJ072, Sichuan Provincial Department of Science and Technology (key research and development project) Project Number: 2023YFS0263. Data availability All authors have reviewed the final version of the paper and would like to take public responsibility for its content. The datasets generated and/or analyzed during the current study are not (yet) publicly available due to use of datasets in an additional study, but are available from the corresponding author on reasonable request. Declarations Ethics approval and consent to participate This study was approved and exempted from informed consent by the Ethics Committee of Sichuan Provincial People's Hospital (Approval Number: 2022.261). This study is a retrospective study and informed consents were waived. This work complied with the Declaration of Helsinki.. Consent for publication Not applicable. Competing interests The authors declare that they have no competing interests. References Siegel RL, Miller KD, Wagle NS, Jemal A. Cancer statistics, 2023. CA Cancer J Clin. 2023;73(1):17–48. Tafreshi NK, Kumar V, Morse DL, Gatenby RA. Molecular and functional imaging of breast cancer. Cancer Control. 2010;17(3):143–55. Gradishar WJ, Moran MS, Abraham J, Abramson V, Aft R, Agnese D, Allison KH, Anderson B, Burstein HJ, Chew H, et al. NCCN Guidelines(R) Insights: Breast Cancer, Version 4.2023. J Natl Compr Canc Netw. 2023;21(6):594–608. Hughes KS, Schnaper LA, Berry D, Cirrincione C, McCormick B, Shank B, Wheeler J, Champion LA, Smith TJ, Smith BL, et al. Lumpectomy plus tamoxifen with or without irradiation in women 70 years of age or older with early breast cancer. N Engl J Med. 2004;351(10):971–7. Yang WT, Goldberg BB. Microbubble contrast-enhanced ultrasound for sentinel lymph node detection: ready for prime time? AJR Am J Roentgenol. 2011;196(2):249–50. Lyman GH, Somerfield MR, Bosserman LD, Perkins CL, Weaver DL, Giuliano AE. Sentinel Lymph Node Biopsy for Patients With Early-Stage Breast Cancer: American Society of Clinical Oncology Clinical Practice Guideline Update. J Clin Oncol. 2017;35(5):561–4. de Boer M, van Deurzen CH, van Dijck JA, Borm GF, van Diest PJ, Adang EM, Nortier JW, Rutgers EJ, Seynaeve C, Menke-Pluymers MB, et al. Micrometastases or isolated tumor cells and the outcome of breast cancer. N Engl J Med. 2009;361(7):653–63. Gentilini OD, Botteri E, Sangalli C, Galimberti V, Porpiglia M, Agresti R, Luini A, Viale G, Cassano E, Peradze N et al. Sentinel Lymph Node Biopsy vs No Axillary Surgery in Patients With Small Breast Cancer and Negative Results on Ultrasonography of Axillary Lymph Nodes. JAMA Oncol 2023, 9(11). Li J, Li H, Guan L, Lu Y, Zhan W, Dong Y, Gu P, Liu J, Cheng W, Na Z, et al. The value of preoperative sentinel lymph node contrast-enhanced ultrasound for breast cancer: a large, multicenter trial. BMC Cancer. 2022;22(1):455. Luo J, Feng L, Zhou Q, Chen Q, Liu J, Wu C, Luo J, Chen J, Wu H, Deng W. The value of contrast-enhanced ultrasound in determining the location of sentinel lymph nodes in breast cancer. Cancer Imaging. 2021;21(1):28. Luo Y, Chen J, Feng L, Cao W, Wu H, Ma M, He F, Luo J, Wu C, Liu J, et al. Study on Sentinel Lymph Node and Its Lymphatic Drainage Pattern of Breast Cancer by Contrast-Enhanced Ultrasound. J Ultrasound Med. 2022;41(11):2727–37. Cox K, Taylor-Phillips S, Sharma N, Weeks J, Mills P, Sever A, Lim A, Haigh I, Hashem M, de Silva T, et al. Enhanced pre-operative axillary staging using intradermal microbubbles and contrast-enhanced ultrasound to detect and biopsy sentinel lymph nodes in breast cancer: a potential replacement for axillary surgery. Br J Radiol. 2018;91(1082):20170626. Sever AR, Mills P, Weeks J, Jones SE, Fish D, Jones PA, Mali W. Preoperative needle biopsy of sentinel lymph nodes using intradermal microbubbles and contrast-enhanced ultrasound in patients with breast cancer. AJR Am J Roentgenol. 2012;199(2):465–70. Dellaportas D, Koureas A, Contis J, Lykoudis PM, Vraka I, Psychogios D, Kondi-Pafiti A, Voros DK. Contrast-Enhanced Color Doppler Ultrasonography for Preoperative Evaluation of Sentinel Lymph Node in Breast Cancer Patients. Breast Care. 2015;10(5):331–5. Omoto K, Matsunaga H, Take N, Hozumi Y, Takehara M, Omoto Y, Shiozawa M, Mizunuma H, Harashima H, Taniguchi N, et al. Sentinel node detection method using contrast-enhanced ultrasonography with sonazoid in breast cancer: preliminary clinical study. Ultrasound Med Biol. 2009;35(8):1249–56. Zhao J, Zhang J, Zhu Q-L, Jiang Y-X, Sun Q, Zhou Y-D, Wang M-Q, Meng Z-L, Mao X-X. The value of contrast-enhanced ultrasound for sentinel lymph node identification and characterisation in pre-operative breast cancer patients: A prospective study. Eur Radiol. 2017;28(4):1654–61. Nielsen Moody A, Bull J, Culpan AM, Munyombwe T, Sharma N, Whitaker M, Wolstenhulme S. Preoperative sentinel lymph node identification, biopsy and localisation using contrast enhanced ultrasound (CEUS) in patients with breast cancer: a systematic review and meta-analysis. Clin Radiol. 2017;72(11):959–71. Sharma N, Cox K. Axillary Nodal Staging with Contrast-Enhanced Ultrasound. Curr Breast Cancer Rep. 2017;9(4):259–63. Zheng X, Yao Z, Huang Y, Yu Y, Wang Y, Liu Y, Mao R, Li F, Xiao Y, Wang Y, et al. Deep learning radiomics can predict axillary lymph node status in early-stage breast cancer. Nat Commun. 2020;11(1):1236. Hosny A, Parmar C, Quackenbush J, Schwartz LH, Aerts HJWL. Artificial intelligence in radiology. Nat Rev Cancer. 2018;18(8):500–10. Zhou LQ, Wu XL, Huang SY, Wu GG, Ye HR, Wei Q, Bao LY, Deng YB, Li XR, Cui XW, et al. Lymph Node Metastasis Prediction from Primary Breast Cancer US Images Using Deep Learning. Radiology. 2020;294(1):19–28. Ding X, Zhang X, Ma N, Han J, Ding G, Sun J. Repvgg: Making vgg-style convnets great again. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition: 2021 ; 2021: 13733–13742. Sanghyun W, Jongchan P, Joon-Young L, In. S: CBAM: Convolutional Block Attention Module Proceedings of the European Conference on Computer Vision (ECCV). 2018. Zha H-l, Zong M, Liu X-p, Pan J-z, Wang H, Gong H-y, Xia T-s. Liu X-a, Li C-y: Preoperative ultrasound-based radiomics score can improve the accuracy of the Memorial Sloan Kettering Cancer Center nomogram for predicting sentinel lymph node metastasis in breast cancer. Eur J Radiol 2021, 135. Giuliano AE, Hunt KK, Ballman KV, Beitsch PD, Whitworth PW, Blumencranz PW, Leitch AM, Saha S, McCall LM, Morrow M. Axillary dissection vs no axillary dissection in women with invasive breast cancer and sentinel node metastasis: a randomized clinical trial. JAMA. 2011;305(6):569–75. Giuliano AE, Ballman KV, McCall L, Beitsch PD, Brennan MB, Kelemen PR, Ollila DW, Hansen NM, Whitworth PW, Blumencranz PW et al. Effect of Axillary Dissection vs No Axillary Dissection on 10-Year Overall Survival Among Women With Invasive Breast Cancer and Sentinel Node Metastasis. JAMA 2017, 318(10). Rivadeneira DE, Simmons RM, Christos PJ, Hanna K, Daly JM, Osborne MP. Predictive factors associated with axillary lymph node metastases in T1a and T1b breast carcinomas: analysis in more than 900 patients. J Am Coll Surg. 2000;191(1):1–6. discussion 6–8. Guo X, Liu Z, Sun C, Zhang L, Wang Y, Li Z, Shi J, Wu T, Cui H, Zhang J, et al. Deep learning radiomics of ultrasonography: Identifying the risk of axillary non-sentinel lymph node involvement in primary breast cancer. EBioMedicine. 2020;60:103018. Dihge L, Vallon-Christersson J, Hegardt C, Saal LH, Häkkinen J, Larsson C, Ehinger A, Loman N, Malmberg M, Bendahl P-O, et al. Prediction of Lymph Node Metastasis in Breast Cancer by Gene Expression and Clinicopathological Models: Development and Validation within a Population-Based Cohort. Clin Cancer Res. 2019;25(21):6368–81. Yu Y, He Z, Ouyang J, Tan Y, Chen Y, Gu Y, Mao L, Ren W, Wang J, Lin L, et al. Magnetic resonance imaging radiomics predicts preoperative axillary lymph node metastasis to support surgical decisions and is associated with tumor microenvironment in invasive breast cancer: A machine learning, multicenter study. EBioMedicine. 2021;69:103460. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition : 2016; 2016: 770–778. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4437751","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":308108606,"identity":"844f6f8d-3874-44ed-a305-9a788b978a12","order_by":0,"name":"Yunhao Luo","email":"","orcid":"","institution":"Chengdu University of Traditional Chinese Medicine","correspondingAuthor":false,"prefix":"","firstName":"Yunhao","middleName":"","lastName":"Luo","suffix":""},{"id":308108607,"identity":"6e2ee41c-7b5c-4bd3-96b7-20e00991c0dd","order_by":1,"name":"Zhuo Wei","email":"","orcid":"","institution":"University of Electronic Science and Technology of China","correspondingAuthor":false,"prefix":"","firstName":"Zhuo","middleName":"","lastName":"Wei","suffix":""},{"id":308108608,"identity":"f929e49f-bb87-42e6-8678-b7a222dcbe09","order_by":2,"name":"Jie Chen","email":"","orcid":"","institution":"Sichuan Academy of Medical Sciences \u0026 Sichuan Provincial People's Hospital","correspondingAuthor":false,"prefix":"","firstName":"Jie","middleName":"","lastName":"Chen","suffix":""},{"id":308108609,"identity":"325df580-25bf-400b-9cb4-f9b1ca650f77","order_by":3,"name":"Wenbin Cao","email":"","orcid":"","institution":"Sichuan Academy of Medical Sciences \u0026 Sichuan Provincial People's Hospital","correspondingAuthor":false,"prefix":"","firstName":"Wenbin","middleName":"","lastName":"Cao","suffix":""},{"id":308108610,"identity":"260155e0-a23c-424f-a400-bd9cffa49acd","order_by":4,"name":"Zhengquan Feng","email":"","orcid":"","institution":"University of Electronic Science and Technology of China","correspondingAuthor":false,"prefix":"","firstName":"Zhengquan","middleName":"","lastName":"Feng","suffix":""},{"id":308108611,"identity":"341602c0-4c64-4c72-820a-f3ec394b040c","order_by":5,"name":"Chaonan Li","email":"","orcid":"","institution":"Chengdu University of Traditional Chinese Medicine","correspondingAuthor":false,"prefix":"","firstName":"Chaonan","middleName":"","lastName":"Li","suffix":""},{"id":308108612,"identity":"083df0cb-44b9-4e28-8a2a-b303aeae4a93","order_by":6,"name":"Yuyan Liu","email":"","orcid":"","institution":"University of Electronic Science and Technology of China","correspondingAuthor":false,"prefix":"","firstName":"Yuyan","middleName":"","lastName":"Liu","suffix":""},{"id":308108613,"identity":"1bcf5da9-5b96-491f-8e09-9171913757db","order_by":7,"name":"Qin Chen","email":"","orcid":"","institution":"Sichuan Academy of Medical Sciences \u0026 Sichuan Provincial People's Hospital","correspondingAuthor":false,"prefix":"","firstName":"Qin","middleName":"","lastName":"Chen","suffix":""},{"id":308108614,"identity":"22810342-0ca3-47df-8872-21a330a4bcbe","order_by":8,"name":"Jing Luo","email":"","orcid":"","institution":"Sichuan Academy of Medical Sciences \u0026 Sichuan Provincial People's Hospital","correspondingAuthor":false,"prefix":"","firstName":"Jing","middleName":"","lastName":"Luo","suffix":""},{"id":308108615,"identity":"d7b475db-2f0e-4668-93e9-942c2d8a14c9","order_by":9,"name":"Jianhua Deng","email":"","orcid":"","institution":"University of Electronic Science and Technology of China","correspondingAuthor":false,"prefix":"","firstName":"Jianhua","middleName":"","lastName":"Deng","suffix":""},{"id":308108616,"identity":"95d21bf2-b63f-4580-a852-8d181aa61342","order_by":10,"name":"Jun Luo","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAsUlEQVRIiWNgGAWjYBACPoYDIOqAHBt7+wHitLBBtRjz8ZxJIFYLGBxInCfhYECkFsbjDz8X7riT3ibBkMDwo2IbMbacMZaeeeZZbpt04wHGnjO3idLCxszbdji3TeZAAjNjG1Fajj8DaUlnk0gwIFbLATOQlgRStID9ctiwDRjIB4nyC78EOMQOy8u3tx988KOCCC0MEgcYmBkbIOwDRKgHWdOA0DIKRsEoGAWjACsAAKFKPcBuncBsAAAAAElFTkSuQmCC","orcid":"","institution":"Sichuan Academy of Medical Sciences \u0026 Sichuan Provincial People's Hospital","correspondingAuthor":true,"prefix":"","firstName":"Jun","middleName":"","lastName":"Luo","suffix":""}],"badges":[],"createdAt":"2024-05-17 15:55:24","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4437751/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4437751/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":57864947,"identity":"8c498f12-35aa-41e0-8d21-268d64f38d55","added_by":"auto","created_at":"2024-06-06 15:35:06","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":1924952,"visible":true,"origin":"","legend":"\u003cp\u003eFlowchart of the overall research process.\u003c/p\u003e","description":"","filename":"figure1.png","url":"https://assets-eu.researchsquare.com/files/rs-4437751/v1/15602c696d79f4d85aadb501.png"},{"id":57864949,"identity":"58c4cebe-9934-4a02-8f8a-015af8455d11","added_by":"auto","created_at":"2024-06-06 15:35:06","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":3053533,"visible":true,"origin":"","legend":"\u003cp\u003eData set image display.\u003c/p\u003e\n\u003cp\u003eA, TDUS original image data display form; \u0026nbsp;B, Manually labeled SLN area of the original TDUS image; C, SLN image of TDUS dataset filled to 224×224 with black border; D, CEUS original image data display form; E, Manually labeled SLN area of the original CEUS image; F, SLN image of CEUS dataset filled to 224×224 with black border.\u003c/p\u003e","description":"","filename":"figure2.png","url":"https://assets-eu.researchsquare.com/files/rs-4437751/v1/7d54c71976eb0e4b0f9b0381.png"},{"id":57864945,"identity":"bcb22238-0b02-41b6-9a8d-039962b3b06d","added_by":"auto","created_at":"2024-06-06 15:35:06","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":1441932,"visible":true,"origin":"","legend":"\u003cp\u003eArtificial Intelligence Analysis Process.\u003c/p\u003e\n\u003cp\u003eSLN-P, sentinel lymph node positive. SLN-N, sentinel lymph node negative\u003c/p\u003e","description":"","filename":"figure3.png","url":"https://assets-eu.researchsquare.com/files/rs-4437751/v1/2d4cec6e261de53c6595d07d.png"},{"id":57865559,"identity":"b0469373-7375-4d6d-8cb8-d43f2da29c9c","added_by":"auto","created_at":"2024-06-06 15:43:06","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":3926802,"visible":true,"origin":"","legend":"\u003cp\u003eTDUS evaluation criteria.\u003c/p\u003e\n\u003cp\u003eA, Normal lymph node B-mode ultrasound image; B, Normal lymph node portal-type blood flow signal; C, Lymph node cortical inhomogeneous thickening; D, Lymph node cortical homogeneous thickening; E, Lymph node corticomedullary border was unclear, full morphology; F, Lymph node cortical abnormal echogenicity.\u003c/p\u003e","description":"","filename":"figure4.png","url":"https://assets-eu.researchsquare.com/files/rs-4437751/v1/d5628189df9a5bfcbbc9bd2b.png"},{"id":57864950,"identity":"d31a8502-000b-485c-aff8-dae621f0700c","added_by":"auto","created_at":"2024-06-06 15:35:06","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":4933936,"visible":true,"origin":"","legend":"\u003cp\u003eCEUS evaluation criteria.\u003c/p\u003e\n\u003cp\u003eA, Uniform enhancement of lymph nodes; B, Uneven enhancement of lymph nodes; C, No enhancement of lymph nodes; D, Circumferential enhancement of lymph nodes.\u003c/p\u003e","description":"","filename":"figure5.png","url":"https://assets-eu.researchsquare.com/files/rs-4437751/v1/2c944cb70243d85838cd5489.png"},{"id":57864946,"identity":"61f2d106-4522-4eeb-bf9b-078233c8f048","added_by":"auto","created_at":"2024-06-06 15:35:06","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":1161728,"visible":true,"origin":"","legend":"\u003cp\u003eThe ROC curves of different image for predicting SLNs metastasis.\u003c/p\u003e","description":"","filename":"figure6.png","url":"https://assets-eu.researchsquare.com/files/rs-4437751/v1/94b0192a7e462dbc6e876b29.png"},{"id":74156463,"identity":"e14d31c6-d2f0-41ad-b9a8-b9cc2b149bb2","added_by":"auto","created_at":"2025-01-18 21:46:30","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":23235756,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4437751/v1/531b3a2c-bb94-40a6-b107-aa5f82d7f541.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Deep learning from ultrasound images of breast cancer sentinel lymph nodes to predict metastasis","fulltext":[{"header":"Key Points","content":"\u003cp\u003e\u003cstrong\u003eAI Model Surpasses Expert Diagnostic Performance:\u0026nbsp;\u003c/strong\u003eThe AI model developed in this study, based on the RepVGG-CBAM, has demonstrated a higher diagnostic efficiency in determining the sentinel lymph node (SLN) metastasis status in breast cancer patients compared to human experts.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eApplication of Deep Learning in Medical Image Analysis:\u0026nbsp;\u003c/strong\u003eBy integrating contrast-enhanced ultrasound (CEUS) and two-dimensional ultrasound (TDUS) images, the AI model is capable of directly learning the ultrasound image features of SLN to ascertain the presence of metastasis, which has not been previously reported.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEnhanced Clinical Accuracy and Reduced Need for Invasive Surgery:\u0026nbsp;\u003c/strong\u003eThe AI model aids in reducing the false-negative rate in determining SLN metastasis and improves the accuracy of preoperative SLN assessment, potentially reducing the need for invasive SLN biopsies in the future.\u003c/p\u003e"},{"header":"Introduction","content":"\u003cp\u003eBreast cancer has now become the most common malignancy worldwide and the second leading cause of death from cancer in women [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. The status of axillary lymph nodes (ALN) is one of the most important factors for evaluating the clinical staging, treatment modality, and prognosis of breast cancer patients [\u003cspan additionalcitationids=\"CR3\" citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. The sentinel lymph node (SLN) is the first lymph node in the lymphatic system to receive lymphatic drainage and is the first site for ALN metastasis in breast cancer patients [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. It is clinically important to accurately determine the presence of SLN metastasis preoperatively, and the standard procedure is sentinel lymph node biopsy (SLNB) [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. Studies have shown that 80\u0026ndash;85% of early-stage breast cancer patients do not have ALN metastasis, resulting in most patients undergoing invasive excisional biopsy, increasing intraoperative waiting time for biopsy results, and also indicating that the existing preoperative clinical assessment is not accurate enough [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. The \u003cb\u003eSOUND\u003c/b\u003e study further confirmed that the 5-year distant disease-free survival rate, axillary recurrence rate, distant recurrence rate, disease-free survival rate and overall survival rate of breast cancer patients with breast cancer less than 2 cm in diameter and negative preoperative axillary ultrasound results were similar between SLNB and no axillary surgery. Future axillary management of breast cancer patients is gradually becoming exempt, therefore, accurate evaluation of SLN is crucial [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. A multicenter study [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e] showed that preoperative contrast-enhanced ultrasound (CEUS) can be used to accurately identify and localize SLN in breast cancer patients, and previous studies in our center [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e] also confirmed that CEUS guidance can accurately localize SLN, but it is still difficult to determine the presence of metastasis in SLN [\u003cspan additionalcitationids=\"CR13 CR14 CR15 CR16 CR17\" citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eWith the development of artificial intelligence (AI), deep learning models, especially convolutional neural network (CNN), have been shown to be comparable to human experts in determining the ALN status of early-stage breast cancer [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. AI models based on deep learning can extract and recognize detailed features in medical images that are invisible to human experts, and have been widely used in medical image segmentation and intelligent prediction fields [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]. However, research on directly learning SLN ultrasound images using deep learning models to determine the presence of metastasis has not been reported. Therefore, in this study, we aim to construct separate datasets of two-dimensional ultrasound (TDUS) and CEUS images of SLNs and design an algorithm model to determine SLN metastasis.\u003c/p\u003e"},{"header":"Materials and methods","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eBasic research information\u003c/h2\u003e \u003cp\u003e This study was approved and exempted from informed consent by the Ethics Committee of Sichuan Provincial People's Hospital (Approval Number: 2022.261). This study is a retrospective study and informed consents were waived. This work complied with the Declaration of Helsinki. This study included SLNs images, including TDUS and CEUS images, of pathologically diagnosed breast cancer patients from June 2017 to October 2022 in Sichuan Provincial People's Hospital. All SLNs were performed with surgical resection pathology findings as the gold standard. The overall research process was shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eInclusion Criteria: 1. Breast cancer patients with SLNB including T1-3 and ductal carcinoma in situ (DCIS) scheduled for total mastectomy. 2. All patients were pathologically confirmed as having malignant breast tumors by preoperative puncture biopsy. 3. All patients underwent contrast-enhanced ultrasound through the lymphatic pathway to identify and locate the SLN prior to surgery.\u003c/p\u003e \u003cp\u003eExclusion Criteria: 1. ALN was confirmed as metastatic carcinoma by biopsy before surgery; 2. Previous breast or axillary radiotherapy or chemotherapy; 3. Advanced breast cancer (T4a-c) and inflammatory breast cancer (T4d); 4. Bilateral breast cancer patients; 5. Male breast cancer patients; 6. Ultrasound image quality cannot reach the requirements of AI analysis.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003eAccess to medical images\u003c/h2\u003e \u003cp\u003ePhilips iU-Elite and GE logic-E9 ultrasonic diagnostic devices, equipped with CEUS display system, were selected for images acquisition. L12-5 (Philips) and ML6-15 (GE) linear array probes were used for TDUS examination, and L9-3 (Philips) and 9L (GE) linear array probes were used for CEUS examination. SonoVue (Bracco spa, Milan, Italy) with a low mechanical index (MI) of 0.07 (Philips) and 0.16 (GE) was used as the ultrasound contrast agent.\u003c/p\u003e \u003cp\u003eThe SLNs of breast cancer patients were located by injecting the ultrasound contrast agent into the areola region [\u003cspan additionalcitationids=\"CR10\" citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. TDUS and CEUS images data acquisition were performed at the longest diameter level of each SLN, both including at least one static image and at least 15 seconds of dynamic video, and the image acquisition was performed by three experts (Jun Luo, Wenbin Cao, and Yunhao Luo. with more than 15 years, 7 years, 5 years of experience in breast ultrasound diagnosis, respectively). The ultrasound images to be included in the study were reviewed by a senior expert Jun Luo) for image quality, and when the diagnosis was more difficult, the senior expert (Qin Chen. with more than 20 years of experience in breast ultrasound diagnosis) reviewed the images, and cases with poor image quality were excluded.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003eArtificial intelligence data preprocessing of medical images\u003c/h2\u003e \u003cp\u003eDeep learning researchers (Jianhua Deng, Zhuo Wei, and Zhengquan Feng, with more than 15 years, 3 years, and 2 years of experience in AI image processing, respectively) collaborated with medical experts (Wenbin Cao, Yunhao Luo) to annotate the images. The annotations were then reviewed by senior experts (Jun Luo, Qin Chen) to ensure accurate and error-free delineation of the SLN regions. Finally, all the SLNs images were randomly divided into training set, verification set and internal test set according to the ratio of 8:1:1. Due to the limited number of images in the dataset, the offline data enhancement method was adopted in this study to expand the dataset, and the images were flipped horizontally, rotated 180\u0026deg;, and flipped horizontally after 180\u0026deg; rotation, and the final training set reached four times the original dataset. The TDUS images and CEUS images collected in this study are shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003eDeep Convolutional Neural Network\u003c/h2\u003e \u003cp\u003eThe establishment of the AI model was completed by the researchers (Jianhua Deng, Zhuo Wei, and Zhengquan Feng). The model was based on the Re-parameterization Visual Geometry Group (RepVGG) network [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e] and incorporates spatial attention and channel attention mechanisms. As shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e, the Convolutional Block Attention Module (CBAM) [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e] attention mechanism module was embedded between the convolutional layers and ReLU activation layers after Stage 0 and Stage 4 of RepVGG-A0, resulting in the neural network model used in this study, RepVGG-CBAM.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eTaking TDUS images as an example, the AI analysis process consisted of training and inference stages. During the training process, the dataset were classified as negative (no metastasis) or positive (metastasis) according to the SLN gold standard. Under various parameter settings, the data was input into the neural network model for training, and the best model was automatically saved. Finally, the trained model was obtained for the inference stage. During the inference process, unknown SLN images were input into the inference model, and relevant parameters were set to obtain the model's output result, indicating whether it was negative or positive.\u003c/p\u003e \u003cp\u003e \u003cb\u003eArtificial Intelligence Model Training\u003c/b\u003e \u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003e1. Experimental environment.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eThis study was conducted in the Digital Information Systems Laboratory at the University of Electronic Science and Technology of China. The experimental hardware was configured as follows: Linux operating system version was Ubuntu 18.10 LTS, CPU model was Inteli9-9900K, memory size was 64G, two GPU graphics cards model was Nvidia GeForce RTX 2080 Ti, graphics card memory was 12G. The software configurations were: Python 3.7, Pytorch 1.2.0, CUDA 10.1, Torchvision 0.4.0, and tensorboardX 2.6.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e2. Experimental parameters.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eFollowing the idea of migration learning, the weights of the pre-trained model of RepVGG-A0 version on ImageNET were used as the initial weights. During the training process, the epoch was set to 1000, the batch_size was set to 16, the initial learning rate was set to 0.01, and the learning rate was jointly adjusted by a combination of equal-step decay and cosine annealing decay during the training process, and the optimizer selected SGD.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003eMedical image evaluation\u003c/h2\u003e \u003cp\u003eThree experts (Yunhao Luo, Wenbin Cao, and Jun Luo) jointly determined the test set data as benign or malignant. If the three diagnoses diverged and disagreed, the senior expert (Qin Chen.) made the diagnosis. All experts were blinded to clinical and histologic outcomes. Images meeting the negative diagnostic criteria were considered as no metastasis, and those meeting the suspicious positive and positive diagnostic criteria were considered as metastasis.\u003c/p\u003e \u003cp\u003eThe diagnosis criteria of the test set experts were as follows:\u003c/p\u003e \u003cp\u003e1. For TDUS images, this study was divided into three lymph node statuses as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e(1) Negative: long diameter/short diameter\u0026thinsp;\u0026gt;\u0026thinsp;2, uniform cortex, clear corticomedullary demarcation, identifiable lymphatic portal, no abnormal echogenicity in the cortex, and lymphatic portal-type blood flow signal was visible on CDFI.\u003c/p\u003e \u003cp\u003e(2) Suspicious positive: ①uneven cortical thickening, the thicker part was \u0026gt;\u0026thinsp;2 times compared with other cortical thickness measurements; ②uniform cortical thickening, but the morphology was slightly full, the long diameter/short diameter was \u0026lt;\u0026thinsp;2, the corticomedullary demarcation was clear, the lymphatic portal-type was identifiable, there was no abnormal echogenicity in the cortex, and the lymphatic portal-type blood flow signal was visible on CDFI.\u003c/p\u003e \u003cp\u003e(3) Positive: ①complete morphology, long/short diameter\u0026thinsp;\u0026lt;\u0026thinsp;2, indistinct corticomedullary demarcation, disappearance of lymphatic portals, non-lymphatic flow signal was visible on CDFI; ②abnormal cortical echogenicity (calcification, liquefaction, abnormal hyperechoic).\u003c/p\u003e \u003cp\u003e2. For CEUS images, this study was divided into three lymph node statuses as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e(1) Negative: uniform enhancement.\u003c/p\u003e \u003cp\u003e(2) Suspicious positive: non-uniform enhancement and cortical enhancement defects;\u003c/p\u003e \u003cp\u003e(3) Positive: ①no enhancement (except for lymphatic vessel interruption leading to inability to visualize lymph nodes); ②circumferential enhancement with uniform or uneven enhancement within it.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eInternal test set\u003c/h2\u003e \u003cp\u003eThe internal test set used experts and models to test TDUS, CEUS, and combination ultrasound(CBUS), for a total of 6 test groups. CBUS means combines TDUS and CEUS images to determine together. If a case in the TDUS or CEUS group was determined to be metastatic, the CBUS group for that case was also determined to be metastatic.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003eStatistical Methods\u003c/h2\u003e \u003cp\u003eSPSS 22.0 was used for statistical analysis. The central tendency of quantitative data was expressed as mean, and the degree of dispersion was expressed as standard deviation. The relative amounts of qualitative information were expressed as rates. The accuracy, sensitivity, specificity, positive predictive value, negative predictive value, and the area under the receiver operating characteristic curve (AUC) of experts and AI models in diagnosing SLN metastasis of breast cancer were calculated by comparing of AUC differences using the Delong test.\u003c/p\u003e \u003c/div\u003e"},{"header":"Results","content":"\u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003ePatient Characteristics\u003c/h2\u003e \u003cp\u003eThe basic information of the included patients was shown in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. A total of 972 patients were included. Other pathologic types included invasive mucinous carcinoma, invasive micropapillary carcinoma, tubule carcinoma, secretory carcinoma, adenoid cystic carcinoma, mammary degenerative carcinoma and mixed breast cancer.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eCharacteristics of Patients\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eVariable\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTraining set (n\u0026thinsp;=\u0026thinsp;772)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eValidation set (n\u0026thinsp;=\u0026thinsp;100)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003einternal test set (n\u0026thinsp;=\u0026thinsp;100)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cem\u003eP\u003c/em\u003e\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAge\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.787\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e51.36\u0026thinsp;\u0026plusmn;\u0026thinsp;10.62\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e50.68\u0026thinsp;\u0026plusmn;\u0026thinsp;10.84\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e50.86\u0026thinsp;\u0026plusmn;\u0026thinsp;12.42\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLeft or right\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.695\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLeft\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e391\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e48\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e54\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRight\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e381\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e52\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e46\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMetastasis\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.163\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNo\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e628\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e88\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e78\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eYes\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e144\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e22\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePathological types of tumor\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.233\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDCIS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e99\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eIDC\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e586\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e69\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e75\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eILC\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e21\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOthers\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e66\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e13\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eT stage of tumor\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.288\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eT1mi\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e24\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eT1a\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e20\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eT1b\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e77\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eT1c\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e299\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e33\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eT2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e243\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e32\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e24\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eT3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e10\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTis\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e99\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMolecular typing of tumor\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.066\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBasal-like\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e80\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHER2 over-expression\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e94\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLuminal A\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e138\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLuminal B HER2 (-)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e270\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e35\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e37\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLuminal B HER2 (+)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e91\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e22\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTis\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e99\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"5\"\u003eDCIS:Ductal carcinoma in situ; IDC:Infiltrating ductal carcinoma; ILC:Infiltrating lobular carcinoma\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003eInference Results of the experts and RepVGG-CBAM Model on the Test Set\u003c/h2\u003e \u003cp\u003eThe trained models were separately inferred and tested on the internal test set. The test results when the binary cutoff value is set to 0.2. The Receiver Operating Characteristic (ROC) curves of the experts and model test results were shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e6\u003c/span\u003e. The experimental performance were shown in Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e. The paired-sample area difference under the ROC curves were shown in Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eExperimental performance of Experts and RepVGG-CBAM model in the test set.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"19\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c10\" colnum=\"10\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c11\" colnum=\"11\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c12\" colnum=\"12\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c13\" colnum=\"13\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c14\" colnum=\"14\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c15\" colnum=\"15\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c16\" colnum=\"16\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c17\" colnum=\"17\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c18\" colnum=\"18\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c19\" colnum=\"19\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEvaluation methods\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c4\" namest=\"c2\"\u003e \u003cp\u003eAUC (95%Cl)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c7\" namest=\"c5\"\u003e \u003cp\u003eSENS (95%Cl)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c10\" namest=\"c8\"\u003e \u003cp\u003eSPEC (95%Cl)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c13\" namest=\"c11\"\u003e \u003cp\u003ePPV (95%Cl)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c16\" namest=\"c14\"\u003e \u003cp\u003eNPV (95%Cl)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c19\" namest=\"c17\"\u003e \u003cp\u003eACC (95%Cl)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eExperts in TDUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.794\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.699\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.889\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.818\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.597\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.948\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e0.769\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e0.660\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e0.857\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c11\"\u003e \u003cp\u003e0.500\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c12\"\u003e \u003cp\u003e0.389\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c13\"\u003e \u003cp\u003e0.611\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c14\"\u003e \u003cp\u003e0.938\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c15\"\u003e \u003cp\u003e0.860\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c16\"\u003e \u003cp\u003e0.974\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c17\"\u003e \u003cp\u003e0.780\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c18\"\u003e \u003cp\u003e0.686\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c19\"\u003e \u003cp\u003e0.857\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eExperts in CEUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.806\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.704\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.908\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.727\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.498\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.893\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e0.885\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e0.792\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e0.946\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c11\"\u003e \u003cp\u003e0.640\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c12\"\u003e \u003cp\u003e0.477\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c13\"\u003e \u003cp\u003e0.776\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c14\"\u003e \u003cp\u003e0.920\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c15\"\u003e \u003cp\u003e0.853\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c16\"\u003e \u003cp\u003e0.958\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c17\"\u003e \u003cp\u003e0.850\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c18\"\u003e \u003cp\u003e0.765\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c19\"\u003e \u003cp\u003e0.914\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eExperts in CBUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.774\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.678\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.871\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.818\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.597\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.948\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e0.731\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e0.618\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e0.825\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c11\"\u003e \u003cp\u003e0.462\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c12\"\u003e \u003cp\u003e0.361\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c13\"\u003e \u003cp\u003e0.565\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c14\"\u003e \u003cp\u003e0.934\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c15\"\u003e \u003cp\u003e0.853\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c16\"\u003e \u003cp\u003e0.972\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c17\"\u003e \u003cp\u003e0.750\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c18\"\u003e \u003cp\u003e0.653\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c19\"\u003e \u003cp\u003e0.831\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel in TDUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.861\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.778\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.944\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.864\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.651\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.971\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e0.859\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e0.762\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e0.927\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c11\"\u003e \u003cp\u003e0.633\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c12\"\u003e \u003cp\u003e0.494\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c13\"\u003e \u003cp\u003e0.754\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c14\"\u003e \u003cp\u003e0.957\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c15\"\u003e \u003cp\u003e0.886\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c16\"\u003e \u003cp\u003e0.985\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c17\"\u003e \u003cp\u003e0.860\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c18\"\u003e \u003cp\u003e0.776\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c19\"\u003e \u003cp\u003e0.921\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel in CEUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.851\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.762\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.941\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.818\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.597\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.942\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e0.885\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e0.792\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e0.946\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c11\"\u003e \u003cp\u003e0.667\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c12\"\u003e \u003cp\u003e0.512\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c13\"\u003e \u003cp\u003e0.792\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c14\"\u003e \u003cp\u003e0.945\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c15\"\u003e \u003cp\u003e0.876\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c16\"\u003e \u003cp\u003e0.977\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c17\"\u003e \u003cp\u003e0.870\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c18\"\u003e \u003cp\u003e0.788\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c19\"\u003e \u003cp\u003e0.929\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel in CBUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.842\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.757\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.927\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.864\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.651\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.971\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e0.821\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e0.717\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e0.898\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c11\"\u003e \u003cp\u003e0.576\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c12\"\u003e \u003cp\u003e0.451\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c13\"\u003e \u003cp\u003e0.692\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c14\"\u003e \u003cp\u003e0.955\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c15\"\u003e \u003cp\u003e0.881\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c16\"\u003e \u003cp\u003e0.984\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c17\"\u003e \u003cp\u003e0.830\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c18\"\u003e \u003cp\u003e0.742\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c19\"\u003e \u003cp\u003e0.898\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"19\"\u003eThe data in brackets are the 95% confidence intervals. Test set, n\u0026thinsp;=\u0026thinsp;100. Abbreviations: AUC: area under the receiver operating characteristic curve; ACC: accuracy; SENS: sensitivity; SPEC: specificity; PPV: positive predictive value; NPV: negative predictive value.\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003ePaired-Sample Area Difference Under the ROC Curves\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"7\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eTest Result Pair(s)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eAsymptotic\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAUC\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eStd. Error\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"2\" nameend=\"c7\" namest=\"c6\"\u003e \u003cp\u003eAsymptotic 95% Confidence Interval\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003ez\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eP\u003c/em\u003e\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eDifference\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eDifference\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eLower Bound\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eUpper Bound\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eExperts in TDUS - Experts in CEUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e-0.309\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.757\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e-0.012\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.311\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e-0.090\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.065\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eExperts in TDUS - Experts in CBUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1.755\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.079\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.019\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.305\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e-0.002\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.041\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eExperts in TDUS - Model in TDUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e-2.025\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.043\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e-0.068\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.296\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e-0.133\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e-0.002\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eExperts in TDUS - Model in CEUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e-2.586\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u0026lt;0.01\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e-0.058\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.301\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e-0.101\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e-0.014\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eExperts in TDUS - Model in CBUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e-1.524\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.128\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e-0.048\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.298\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e-0.111\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.014\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eExperts in CEUS - Experts in CBUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.839\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.401\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.031\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.312\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e-0.042\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.105\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eExperts in CEUS - Model in TDUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e-1.330\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.184\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e-0.055\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.303\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e-0.137\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.026\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eExperts in CEUS - Model in CEUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e-1.391\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.164\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e-0.045\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.307\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e-0.109\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.019\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eExperts in CEUS - Model in CBUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e-0.881\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.378\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e-0.036\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.304\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e-0.116\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.044\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eExperts in CBUS - Model in TDUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e-2.685\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.007\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e-0.087\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.297\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e-0.150\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e-0.023\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eExperts in CBUS - Model in CEUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e-3.742\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u0026lt;0.001\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e-0.077\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.301\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e-0.117\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e-0.037\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eExperts in CBUS - Model in CBUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e-2.195\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.028\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e-0.068\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.299\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e-0.128\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e-0.007\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel in TDUS - Model in CEUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.340\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.734\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.010\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.292\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e-0.047\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.067\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel in TDUS - Model in CBUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1.755\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.079\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.019\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.287\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e-0.002\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.041\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel in CEUS - Model in CBUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.350\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.727\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.009\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.293\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e-0.043\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0.062\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"Discussion","content":"\u003cp\u003eIn this study, the research team successfully established an AI prediction model using the RepVGG-CBAM model to determine SLN metastasis in breast cancer patients. The results on the test set were satisfactory, with the diagnostic performance exceeding the expert level. For SLN metastasis, the higher sensitivity were of greater clinical importance. Compared with experts diagnosis, in TDUS, CEUS and CBUS, the AI prediction models established in this study showed higher sensitivity. To our knowledge, this was the first study to use deep learning algorithms to directly learn ultrasound images of SLN in breast cancer to determine SLN metastasis.\u003c/p\u003e \u003cp\u003eZhou et al [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e] conducted a deep learning study on TDUS features of primary breast cancer lesions to indirectly predict ALN metastasis, and achieved good diagnostic performance with an AUC of 0.89. Zha et al [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e] predicted SLN metastasis in breast cancer by combining various clinical and pathological factors of the primary lesion, and achieved an AUC of 0.833 on the test set. However, the aforementioned studies did not learn the imaging characteristics of the lymph nodes themselves. In clinical practice, analysis of lymph node imaging, especially SLN, directly and accurately determines the presence or absence of metastasis. Our study precisely addresses this issue. Zheng et al [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e] used AI to directly learn multimodal ultrasound images of ALN to predict metastasis, further improving diagnostic efficiency. However, ALN includes non-SLN, which may increase the false negative rate for predicting ALN metastasis, and the study did not include contrast-enhanced imaging, which can improve the diagnosis of SLN metastasis [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. In this study, the research team obtained TDUS and CEUS of SLN after lymph node enhancement and established an AI model to directly learn and diagnose SLN metastasis. The study simulated a real clinical environment.\u003c/p\u003e \u003cp\u003eSLNB has ushered in a minimally invasive era compared to traditional axillary lymph node dissection (ALND). However, SLNB required intraoperative waiting for frozen pathology results, which increased operative time and medical costs. In addition, frozen pathology results may not fully reflect the true presence or absence of SLN metastasis, thus increasing the risk of secondary surgery [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e]. Studies have shown that 15\u0026ndash;20% of patients have SLN metastasis [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. In this study, the SLN metastasis rate was 18.31% (178/972), which is similar to previous research reports [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e]. The use of the model developed by the research team can further reduce the false negative rate in determining SLNs metastases and improve the accuracy of preoperative SLNs assessment. For patients identified by the model as having SLNs metastases, it serves as a reminder for clinicians to perform meticulous operations during surgery to avoid missing suspicious SLNs.\u003c/p\u003e \u003cp\u003eFor the images identified as false negative by RepVGG-CBAM in the test set, including 4 patients (1 form CEUS, 3 form CBUS ). All the patients underwent ALND after SLNB. Postoperative pathology confirmed that the total number of axillary lymph node metastases on the affected side were 1 to 2, including 2 patients with only 1 micrometastasis. Studies have shown that for patients with 1 to 2 axillary lymph node metastases, axillary radiotherapy can achieve the same long-term prognosis as surgical resection [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e]. Therefore, for patients planning to undergo postoperative axillary radiotherapy, SLN diagnosis using the model is expected to prevent some patients from undergoing invasive SLNB in the future, which deserves further study.\u003c/p\u003e \u003cp\u003eSLN metastasis can be divided into isolated tumor cell clusters (ITC), micrometastasis, and macrometastasis. Because ultrasound images cannot observe the microvascular perfusion of SLNs, the ultrasound features of ITC and micrometastasis often appeared normal. In actual clinical work, experts cannot accurately determine micrometastasis or even ITC, as well as other non-morphological features. The model in this study also misdiagnosed for two cases of micrometastasis. Due to the extremely limited number of such cases there are certain limitations. However, with the advancement of clinical diagnostic techniques, updates in AI algorithms, and an increase in the number of cases, the combination of molecular biology and advanced AI models holds promise for overcoming the current research limitations in the future.\u003c/p\u003e \u003cp\u003eAI has achieved practical results in the diagnosis of breast cancer and lymph node metastasis in its drainage area, several studies [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e, \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e, \u003cspan additionalcitationids=\"CR29\" citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e] have shown that AI has achieved the same or higher diagnostic efficiency than clinical experts. Compared to commonly used models such as ResNet [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e], the RepVGG model was chosen as the foundation for this study. Its uniqueness lies in the use of the reparameterization concept, which converts the three-branch network structure during the training phase into an equivalent single-branch network during the inference phase. This approach significantly improved inference speed without compromising accuracy. In addition, considering the challenges of the ultrasound images used in this study, such as high noise and low image quality, the researchers incorporated the CBAM attention mechanism into the network structure of RepVGG during model design. This effectively enhanced the feature extraction capability of the neural network, thereby improving classification accuracy. In the context of image classification, the attention mechanism mimics human visual attention by further extracting and processing important \"eye-catching\" feature information. This helped the neural network focus on more important features, resulting in better performance.\u003c/p\u003e \u003cp\u003eThe main reason for the significant increase in the number of training data in the CEUS dataset compared to the TDUS dataset was that the storage time of CEUS image videos was longer than that of TDUS videos, which was necessary to meet the requirements of actual clinical work. Therefore, more CEUS images were obtained through video segmentation. In this study, the negative data volume in the CEUS dataset was much larger than the positive data volume. This was because the study only included patients with cN0 status which reflected the clinical reality. To prevent overfitting of the model, the described data augmentation methods were applied to the positive data in this study, resulting in a relatively balanced distribution of the final training dataset.\u003c/p\u003e \u003cp\u003eIn binary classification tasks, the output of a neural network was a probability value. In this study, the output of the model represented the probability of SLN positivity. Therefore, a threshold was needed, where if the output probability was below the threshold, it was classified as negative, and if it was above the threshold, it was classified as positive. The optimal range for the threshold can be determined based on the AUC. Considering the goal of the study to identify all true positives and minimize false negatives, a binary classification threshold of 0.2 was chosen after conducting comparative experiments. This means that if the model's output probability of SLN positivity for a particular image was less than 0.2, it was classified as negative, and if it was equal to or greater than 0.2, it was classified as positive.\u003c/p\u003e \u003cp\u003eThis study also has several limitations. Firstly, this was a retrospective study, and the data analysis was limited to images obtained in the past. Secondly, this was a single-center study with a small sample size, and future multicenter studies are needed to fully validate and optimize the model. Furthermore, although all ultrasounds were performed by experienced experts using a standardized procedure, real-world clinical work depends on the different conditions of different patients, and the quality of images may vary somewhat.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eIn this study, an algorithm model was designed and developed to determine the SLN metastasis status of breast cancer patients. The diagnostic efficiency of the model exceeds that of experts.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cp\u003eSentinel lymph node (SLN)\u003c/p\u003e\n\u003cp\u003eContrast-enhanced ultrasound (CEUS)\u003c/p\u003e\n\u003cp\u003etwo-dimensional ultrasound (TDUS)\u003c/p\u003e\n\u003cp\u003eRepVGG-CBAM (Re-parameterization Visual Geometry Group - Convolutional Block Attention Module)\u003c/p\u003e\n\u003cp\u003earea under the curve (AUC)\u003c/p\u003e\n\u003cp\u003eaxillary lymph nodes (ALN)\u003c/p\u003e\n\u003cp\u003esentinel lymph node biopsy (SLNB)\u003c/p\u003e\n\u003cp\u003econvolutional neural network (CNN)\u003c/p\u003e\n\u003cp\u003eductal carcinoma in situ (DCIS)\u003c/p\u003e\n\u003cp\u003emechanical index (MI)\u003c/p\u003e\n\u003cp\u003eaxillary lymph node dissection (ALND)\u003c/p\u003e\n\u003cp\u003eisolated tumor cell clusters (ITC)\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor contribution\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eJ.L. and J.H.D. guarantor of integrity of the entire study. Y.H.L., Z.W., J.L. and J.H.D. study concepts and design. Y.H.L. and Z.W. literature research. Y.H.L., Z.W., J.C., and J.L clinical studies. Y.H.L., Z.W., J.C., W.B.C., Z.Q.F.,C.N.L., Y.Y.L., Q.C.and J. L. experimental studies / data analysis. Y.H.L. and Z.W. statistical analysis. Y.H.L., Z.W., J.L. and J.H.D. manuscript preparation. Y.H.L., Z.W., J.L. and J.H.D. manuscript editing.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis study has received funding by Sichuan Provincial Health Commission (universal application) Project Number: 21PJ072, Sichuan Provincial Department of Science and Technology (key research and development project) Project Number: 2023YFS0263.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData availability\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAll authors have reviewed the final version of the paper and would like to take public responsibility for its content. The datasets generated and/or analyzed during the current study are not (yet) publicly available due to use of datasets in an additional study, but are available from the corresponding author on reasonable request.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDeclarations\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEthics approval and consent to participate\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis study was approved and exempted from informed consent by the Ethics Committee of Sichuan Provincial People\u0026apos;s Hospital (Approval Number: 2022.261). This study is a retrospective study and informed consents were waived. This work complied with the Declaration of Helsinki..\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent for publication\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors declare that they have no competing interests.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eSiegel RL, Miller KD, Wagle NS, Jemal A. Cancer statistics, 2023. CA Cancer J Clin. 2023;73(1):17\u0026ndash;48.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTafreshi NK, Kumar V, Morse DL, Gatenby RA. Molecular and functional imaging of breast cancer. Cancer Control. 2010;17(3):143\u0026ndash;55.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGradishar WJ, Moran MS, Abraham J, Abramson V, Aft R, Agnese D, Allison KH, Anderson B, Burstein HJ, Chew H, et al. NCCN Guidelines(R) Insights: Breast Cancer, Version 4.2023. J Natl Compr Canc Netw. 2023;21(6):594\u0026ndash;608.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHughes KS, Schnaper LA, Berry D, Cirrincione C, McCormick B, Shank B, Wheeler J, Champion LA, Smith TJ, Smith BL, et al. Lumpectomy plus tamoxifen with or without irradiation in women 70 years of age or older with early breast cancer. N Engl J Med. 2004;351(10):971\u0026ndash;7.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYang WT, Goldberg BB. Microbubble contrast-enhanced ultrasound for sentinel lymph node detection: ready for prime time? AJR Am J Roentgenol. 2011;196(2):249\u0026ndash;50.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLyman GH, Somerfield MR, Bosserman LD, Perkins CL, Weaver DL, Giuliano AE. Sentinel Lymph Node Biopsy for Patients With Early-Stage Breast Cancer: American Society of Clinical Oncology Clinical Practice Guideline Update. J Clin Oncol. 2017;35(5):561\u0026ndash;4.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ede Boer M, van Deurzen CH, van Dijck JA, Borm GF, van Diest PJ, Adang EM, Nortier JW, Rutgers EJ, Seynaeve C, Menke-Pluymers MB, et al. Micrometastases or isolated tumor cells and the outcome of breast cancer. N Engl J Med. 2009;361(7):653\u0026ndash;63.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGentilini OD, Botteri E, Sangalli C, Galimberti V, Porpiglia M, Agresti R, Luini A, Viale G, Cassano E, Peradze N et al. Sentinel Lymph Node Biopsy vs No Axillary Surgery in Patients With Small Breast Cancer and Negative Results on Ultrasonography of Axillary Lymph Nodes. JAMA Oncol 2023, 9(11).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi J, Li H, Guan L, Lu Y, Zhan W, Dong Y, Gu P, Liu J, Cheng W, Na Z, et al. The value of preoperative sentinel lymph node contrast-enhanced ultrasound for breast cancer: a large, multicenter trial. BMC Cancer. 2022;22(1):455.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLuo J, Feng L, Zhou Q, Chen Q, Liu J, Wu C, Luo J, Chen J, Wu H, Deng W. The value of contrast-enhanced ultrasound in determining the location of sentinel lymph nodes in breast cancer. Cancer Imaging. 2021;21(1):28.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLuo Y, Chen J, Feng L, Cao W, Wu H, Ma M, He F, Luo J, Wu C, Liu J, et al. Study on Sentinel Lymph Node and Its Lymphatic Drainage Pattern of Breast Cancer by Contrast-Enhanced Ultrasound. J Ultrasound Med. 2022;41(11):2727\u0026ndash;37.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCox K, Taylor-Phillips S, Sharma N, Weeks J, Mills P, Sever A, Lim A, Haigh I, Hashem M, de Silva T, et al. Enhanced pre-operative axillary staging using intradermal microbubbles and contrast-enhanced ultrasound to detect and biopsy sentinel lymph nodes in breast cancer: a potential replacement for axillary surgery. Br J Radiol. 2018;91(1082):20170626.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSever AR, Mills P, Weeks J, Jones SE, Fish D, Jones PA, Mali W. Preoperative needle biopsy of sentinel lymph nodes using intradermal microbubbles and contrast-enhanced ultrasound in patients with breast cancer. AJR Am J Roentgenol. 2012;199(2):465\u0026ndash;70.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDellaportas D, Koureas A, Contis J, Lykoudis PM, Vraka I, Psychogios D, Kondi-Pafiti A, Voros DK. Contrast-Enhanced Color Doppler Ultrasonography for Preoperative Evaluation of Sentinel Lymph Node in Breast Cancer Patients. Breast Care. 2015;10(5):331\u0026ndash;5.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOmoto K, Matsunaga H, Take N, Hozumi Y, Takehara M, Omoto Y, Shiozawa M, Mizunuma H, Harashima H, Taniguchi N, et al. Sentinel node detection method using contrast-enhanced ultrasonography with sonazoid in breast cancer: preliminary clinical study. Ultrasound Med Biol. 2009;35(8):1249\u0026ndash;56.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhao J, Zhang J, Zhu Q-L, Jiang Y-X, Sun Q, Zhou Y-D, Wang M-Q, Meng Z-L, Mao X-X. The value of contrast-enhanced ultrasound for sentinel lymph node identification and characterisation in pre-operative breast cancer patients: A prospective study. Eur Radiol. 2017;28(4):1654\u0026ndash;61.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNielsen Moody A, Bull J, Culpan AM, Munyombwe T, Sharma N, Whitaker M, Wolstenhulme S. Preoperative sentinel lymph node identification, biopsy and localisation using contrast enhanced ultrasound (CEUS) in patients with breast cancer: a systematic review and meta-analysis. Clin Radiol. 2017;72(11):959\u0026ndash;71.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSharma N, Cox K. Axillary Nodal Staging with Contrast-Enhanced Ultrasound. Curr Breast Cancer Rep. 2017;9(4):259\u0026ndash;63.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZheng X, Yao Z, Huang Y, Yu Y, Wang Y, Liu Y, Mao R, Li F, Xiao Y, Wang Y, et al. Deep learning radiomics can predict axillary lymph node status in early-stage breast cancer. Nat Commun. 2020;11(1):1236.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHosny A, Parmar C, Quackenbush J, Schwartz LH, Aerts HJWL. Artificial intelligence in radiology. Nat Rev Cancer. 2018;18(8):500\u0026ndash;10.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhou LQ, Wu XL, Huang SY, Wu GG, Ye HR, Wei Q, Bao LY, Deng YB, Li XR, Cui XW, et al. Lymph Node Metastasis Prediction from Primary Breast Cancer US Images Using Deep Learning. Radiology. 2020;294(1):19\u0026ndash;28.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDing X, Zhang X, Ma N, Han J, Ding G, Sun J. Repvgg: Making vgg-style convnets great again. In: \u003cem\u003eProceedings of the IEEE/CVF conference on computer vision and pattern recognition: 2021\u003c/em\u003e; 2021: 13733\u0026ndash;13742.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSanghyun W, Jongchan P, Joon-Young L, In. S: CBAM: Convolutional Block Attention Module Proceedings of the European Conference on Computer Vision (ECCV). 2018.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZha H-l, Zong M, Liu X-p, Pan J-z, Wang H, Gong H-y, Xia T-s. Liu X-a, Li C-y: Preoperative ultrasound-based radiomics score can improve the accuracy of the Memorial Sloan Kettering Cancer Center nomogram for predicting sentinel lymph node metastasis in breast cancer. Eur J Radiol 2021, 135.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGiuliano AE, Hunt KK, Ballman KV, Beitsch PD, Whitworth PW, Blumencranz PW, Leitch AM, Saha S, McCall LM, Morrow M. Axillary dissection vs no axillary dissection in women with invasive breast cancer and sentinel node metastasis: a randomized clinical trial. JAMA. 2011;305(6):569\u0026ndash;75.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGiuliano AE, Ballman KV, McCall L, Beitsch PD, Brennan MB, Kelemen PR, Ollila DW, Hansen NM, Whitworth PW, Blumencranz PW et al. Effect of Axillary Dissection vs No Axillary Dissection on 10-Year Overall Survival Among Women With Invasive Breast Cancer and Sentinel Node Metastasis. JAMA 2017, 318(10).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRivadeneira DE, Simmons RM, Christos PJ, Hanna K, Daly JM, Osborne MP. Predictive factors associated with axillary lymph node metastases in T1a and T1b breast carcinomas: analysis in more than 900 patients. J Am Coll Surg. 2000;191(1):1\u0026ndash;6. discussion 6\u0026ndash;8.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGuo X, Liu Z, Sun C, Zhang L, Wang Y, Li Z, Shi J, Wu T, Cui H, Zhang J, et al. Deep learning radiomics of ultrasonography: Identifying the risk of axillary non-sentinel lymph node involvement in primary breast cancer. EBioMedicine. 2020;60:103018.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDihge L, Vallon-Christersson J, Hegardt C, Saal LH, H\u0026auml;kkinen J, Larsson C, Ehinger A, Loman N, Malmberg M, Bendahl P-O, et al. Prediction of Lymph Node Metastasis in Breast Cancer by Gene Expression and Clinicopathological Models: Development and Validation within a Population-Based Cohort. Clin Cancer Res. 2019;25(21):6368\u0026ndash;81.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYu Y, He Z, Ouyang J, Tan Y, Chen Y, Gu Y, Mao L, Ren W, Wang J, Lin L, et al. Magnetic resonance imaging radiomics predicts preoperative axillary lymph node metastasis to support surgical decisions and is associated with tumor microenvironment in invasive breast cancer: A machine learning, multicenter study. EBioMedicine. 2021;69:103460.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHe K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: \u003cem\u003eProceedings of the IEEE conference on computer vision and pattern recognition\u003c/em\u003e: 2016; 2016: 770\u0026ndash;778.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Breast cancer, Sentinel lymph nodes, Contrast-enhanced ultrasound, RepVGG, CNN.","lastPublishedDoi":"10.21203/rs.3.rs-4437751/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4437751/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eObjectives\u003c/h2\u003e \u003cp\u003eThis study aims to develop a non-invasive assessment in breast cancer sentinel lymph node (SLN) using deep learning.\u003c/p\u003e\u003ch2\u003eMaterials and methods\u003c/h2\u003e \u003cp\u003eContinuously retrospective patients with breast cancer who have undergone both contrast-enhanced ultrasound (CEUS) and two-dimensional ultrasound (TDUS) for sentinel lymph node examination. Those patients were randomly divided into training set, validation set, and internal test set in a ratio of 8:1:1. A Re-parameterization Visual Geometry Group-Convolutional Block Attention Module (RepVGG-CBAM) model was constructed based on the RepVGG network, embedding the CBAM attention mechanism. The area under the receiver operating characteristic curve (AUC) was used to evaluate diagnostic performance.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eIn the test set, the AUC were experts in TDUS, CEUS, and combination ultrasound (CBUS), model in TDUS, CEUS and CBUS were 0.794, 0.806, 0.774, 0.861, 0.851, 0.842 respectively. The difference in AUC between Experts in TDUS (0.794) and Model in TDUS (0.861) was statistically significant (\u003cem\u003ep\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.043). The difference in AUC between Experts in TDUS (0.794) and Model in CEUS (0.851) was statistically significant (\u003cem\u003ep\u0026lt;\u003c/em\u003e0.01). The difference in AUC between Experts in CBUS (0.774) and Model in TDUS (0.861) was statistically significant (\u003cem\u003ep\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.007). The difference in AUC between Experts in CBUS (0.774) and Model in CEUS (0.851) was statistically significant (\u003cem\u003ep\u0026lt;\u003c/em\u003e0.001).\u003c/p\u003e\u003ch2\u003eConclusions\u003c/h2\u003e \u003cp\u003eAn algorithm model was developed to determine the SLN metastasis status of breast cancer patients.\u003c/p\u003e","manuscriptTitle":"Deep learning from ultrasound images of breast cancer sentinel lymph nodes to predict metastasis","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-06-06 15:35:01","doi":"10.21203/rs.3.rs-4437751/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"ca8813ab-8e96-4016-9c5c-e9e6bdaaef56","owner":[],"postedDate":"June 6th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-01-18T21:38:12+00:00","versionOfRecord":[],"versionCreatedAt":"2024-06-06 15:35:01","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4437751","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4437751","identity":"rs-4437751","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}