Using the Deep Convolutional Neural Network to Evaluate Thyroid Nodules With Atypia of Undetermined Significance/follicular Lesion of Undetermined Significance Cytology: Multicenter Study

Using the Deep Convolutional Neural Network to Evaluate Thyroid Nodules With Atypia of Undetermined Significance/follicular Lesion of Undetermined Significance Cytology: Multicenter Study | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Using the Deep Convolutional Neural Network to Evaluate Thyroid Nodules With Atypia of Undetermined Significance/follicular Lesion of Undetermined Significance Cytology: Multicenter Study Inyoung Youn, Eunjung Lee, Jung Hyun Yoon, Hye Sun Lee, Mi-Ri Kwon, and 8 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-400151/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 8 You are reading this latest preprint version Abstract To compare the diagnostic performances of physicians and a deep convolutional neural network (CNN) predicting malignancy with ultrasonography images of thyroid nodules with atypia of undetermined significance (AUS)/follicular lesion of undetermined significance (FLUS) results on fine-needle aspiration (FNA). This study included 202 patients with 202 nodules ≥ 1cm AUS/FLUS on FNA, and underwent surgery in one of 3 different institutions. Diagnostic performances were compared between 8 physicians (4 radiologists, 4 endocrinologists) with varying experience levels and CNN, and AUS/FLUS subgroups were analyzed. Interobserver variability was assessed among the 8 physicians. Of the 202 nodules, 158 were AUS, and 44 were FLUS; 86 were benign, and 116 were malignant. The area under the curves (AUCs) of the 8 physicians and CNN were 0.680-0.722 and 0.666, without significant differences ( P > 0.05). In the subgroup analysis, the AUCs for the 8 physicians and CNN were 0.657–0.768 and 0.652 for AUS, 0.469-0.674 and 0.622 for FLUS. Interobserver agreements were moderate (k=0.543), substantial (k=0.652), and moderate (k=0.455) among the 8 physicians, 4 radiologists, and 4 endocrinologists. For thyroid nodules with AUS/FLUS cytology, the diagnostic performance of CNN to differentiate malignancy with US images was comparable to that of physicians with variable experience levels. Cancer Biology Oncology convolutional neural network (CNN) deep learning thyroid nodule atypia of undetermined significance/follicular lesion of undetermined significance (AUS/FLUS) Bethesda system Biopsy Fine-Needle Figures Figure 1 Figure 2 Figure 3 Introduction Thyroid nodules occur commonly with incidence rates going up to 68% 1 , and ultrasonography (US) is the primary screening method used to detect these nodules with high sensitivity and specificity. Fine-needle aspiration (FNA) is an easy, relatively safe, and highly accurate diagnostic tool that can be performed under US-guidance to identify benign and malignant nodules based on US findings. The Bethesda system is a standardized, category-based reporting system for thyroid cytopathology, and widely used to interpret FNA results 2 . The nodules with Bethesda class III lesions, otherwise known as atypia of undetermined significance (AUS) or follicular lesion of undetermined significance (FLUS), have a malignancy risk of 6–18%, and management plans vary widely from clinical observation, US follow up, repeat FNA or core needle biopsy, molecular test to thyroid surgery 2 , 3 . Although thyroid US examination has been shown to help stratify the risk of Bethesda class III lesions 3 , 4 , US assessment is limited in application due to its inherent limitations of poorly reproducible tests 5 . Recently, machine learning and deep learning methods have been developed, and rapidly become a methodology of choice for medical image analysis 6 , 7 . Deep convolutional neural network (CNN) trained with an automated process using raw image pixels rather than engineered features extracted by experts of traditional machine learning algorithm 7 . Recently, we developed a computer-aided program that uses a deep convolutional neural network (CNN) to diagnose thyroid nodules according to US features 8 . This CNN can be an objective, operator-independent method to identify benign lesions and malignancy, and these advantages are thought to be especially helpful for nodules with AUS/FLUS cytology on FNA in predicting malignant risk and determining the next management step. The purpose of this study was to compare the diagnostic performances of physicians with varying experience levels and CNN to predict malignancy using US images of thyroid nodules with Bethesda class III results on FNA. Results Table 1 summarized the demographic features of the included 202 nodules. There were 86 (42.6%) benign nodules and 116 (57.4%) malignancies confirmed after surgery. The pathologic results after surgery were shown in Table 2 . Of 202 nodules, preoperative FNA found 158 with AUS cytology and 44 with FLUS cytology. There was no statistical difference between the benign and malignant nodules for sex and age. Malignant nodules had significantly smaller size than benign ones (P = 0.009), and higher cancer probabilities than benign nodules using CNN (P < 0.001). Table 1 Summary of the demographic features. Total Benign Malignancy P value No. of nodules 202 86 (42.6%) 116 (57.4%) Sex 0.416 Male 48 18 (20.9%) 30 (25.9%) Female 154 68 (79.1%) 86 (74.1%) Mean age (years) a 47.9 ± 13.3 47.0 ± 14.8 0.669 Cytologic result < 0.001 No. of AUS Institution A b 158 78 50 (58.1%) 23 108 (93.1%) 55 Institution B b 43 14 29 Institution C b 37 13 24 No. of FLUS Institution A b 44 34 36 (41.9%) 29 8 (6.9%) 5 Institution B b 1 1 0 Institution C b 9 6 3 Median size (IQR, mm) c 19.5 (13–32) 13.5 (11–23) 0.009 Median cancer probability calculated by CNN (IQR, %) c 36.5 (18.7–69.5) 67.7 (30.2–89.9) < 0.001 a The independent two sample t-test. b We collected consecutive patients from three institutions, and the numbers of patients recruited from each hospital was expressed as Institution A, B, and C. c The Mann-Whitney U test. AUS atypia of undetermined significance, FLUS follicular lesion of undetermined significance, IQR interquartile range, CNN deep convolutional neural network. Table 2 Pathologic results after surgery. Pathologic result AUS FLUS Total Benign Adenomatous hyperplasia 22 (44.0) 12 (3.3) 34 (39.5) Follicular adenoma 19 (38.0) 20 (55.6) 39 (45.3) Hurthle cell adenoma 3 (6.0) 3 (8.3) 6 (7.0) Noninvasive follicular thyroid neoplasm with papillary-like nuclear feature 3 (6.0) 1 (2.8) 4 (4.7) Hyaline trabecular tumor 1 (2.0) - 1 (1.2) Localized fibrosis 1 (2.0) - 1 (1.2) Lymphocytic thyroiditis 1 (2.0) - 1 (1.2) Total 50 36 86 Malignancy Papillary thyroid carcinoma 99 (91.7) 4 (50.0) 103 (88.8) Follicular carcinoma 8 (7.4) 3 (37.5) 11 (9.5) Poorly differentiated carcinoma 1 (0.9) 1 (12.5) 2 (1.7) Total 108 8 116 Data in parentheses are percentages. AUS atypia of undetermined significance, FLUS follicular lesion of undermined significance. The diagnostic performances of the 8 physicians and CNN were compared in Table 3 . The sensitivity, specificity, and AUC of the 8 physicians were 24.1–50.9%, 81.4–98.8%, and 0.680–0.722, respectively (Table 3 , Fig. 1 ). The calculated sensitivity, specificity, and AUC of CNN were 59.5%, 69.8%, and 0.666, respectively, using an estimated cut-off value of 54.1% (Table 3 , Fig. 1 ). CNN showed significantly higher sensitivity than 6 physicians, but not over Radiologist 4 (50.0%; P = 0.082) and Endocrinologist 1 (50.9%; P = 0.137). CNN showed significantly lower specificity than all 8 physicians (P 0.05). Table 3 Diagnostic performances of the 8 physicians and deep convolutional neural network. Sensitivity P value a Specificity P value a AUC P value b Total 202 nodules R1 37.9% (29.1–46.8%) < .001 96.5% (92.6–100%) < .001 0.709 (0.643–0.776) 0.279 R2 44.8% (35.8–53.9%) 0.008 95.3% (90.9–99.8%) < .001 0.717 (0.649–0.784) 0.187 R3 47.4% (38.3–56.5%) 0.020 89.5% (83.1–96.0%) < 0.001 0.688 (0.62–0.757) 0.568 R4 50.0% (40.9–59.1%) 0.082 90.7% (84.6–96.8%) < .001 0.722 (0.654–0.789) 0.145 E1 50.9% (41.8–60.0%) 0.137 81.4% (73.3–89.6%) 0.015 0.680 (0.612–0.749) 0.742 E2 39.7% (30.8–48.6%) 0.001 89.5% (83.1–96.0%) 0.001 0.695 (0.629–0.760) 0.500 E3 24.1% (16.4–31.9%) < .001 98.8% (96.6–100%) < .001 0.709 (0.642–0.775) 0.305 E4 42.2% (33.3–51.2%) 0.001 87.2% (80.2–94.3%) 0.002 0.692 (0.624–0.761) 0.494 CNN 59.5% (50.5–68.4%) 69.8% (60.1–79.5%) 0.666 (0.592–0.740) AUS (n = 158) R1 39.8% (30.6–49.0%) < .001 96.0% (90.6–100%) < .001 0.732 (0.658–0.806) 0.111 R2 47.2% (37.8–56.6%) 0.011 98.0% (94.2–100%) < .001 0.768 (0.699–0.837) 0.011 R3 50.0% (40.6–59.4%) 0.029 86.0% (76.4–95.6%) 0.008 0.698 (0.618–0.778) 0.336 R4 52.8% (43.4–62.2%) 0.110 84.0% (73.8–94.2%) 0.008 0.705 (0.624–0.786) 0.253 E1 52.8% (43.4–62.2%) 0.128 76.0% (64.2–87.8%) 0.123 0.657 (0.574–0.741) 0.913 E2 42.6% (33.3–51.9%) 0.001 86.0% (76.4–95.6%) 0.008 0.685 (0.605–0.765) 0.525 E3 25.0% (16.8–33.2%) < .001 98.0% (94.2–100%) < .001 0.730 (0.654–0.806) 0.110 E4 44.4% (35.1–53.8%) 0.002 82.0% (71.4–92.6%) 0.037 0.675 (0.59–0.759) 0.628 CNN 62.0% (52.9–71.2%) 66.0% (52.9–79.1%) 0.652 (0.563–0.741) FLUS (n = 44) R1 12.5% (0-35.4%) 0.046 97.2% (91.9–100%) 0.011 0.469 (0.234–0.703) 0.435 R2 12.5% (0-35.4%) 0.046 91.7% (82.6–100%) 0.119 0.634 (0.372–0.895) 0.902 R3 12.5% (0-35.4%) 0.046 94.4% (87.0-100%) 0.046 0.535 (0.313–0.757) 0.493 R4 12.5% (0-35.4%) 0.046 100% (100–100%) 0.001 0.535 (0.290–0.780) 0.699 E1 25.0% (0–55.0%) 0.128 88.9% (78.6–99.2%) 0.239 0.587 (0.371–0.803) 0.857 E2 0% (0–0%) 0.001 94.4% (87.0-100%) 0.046 0.509 (0.320–0.697) 0.528 E3 12.5% (0-35.4%) 0.046 100% (100–100%) 0.001 0.674 (0.465–0.882) 0.803 E4 12.5% (0-35.4%) 0.046 94.4% (87.0-100%) 0.046 0.615 (0.420–0.809) 0.970 CNN 62.5% (29.0–96.0%) 77.8% (64.2–91.4%) 0.808 0.622 (0.355–0.888) a compared with the results of the convolutional neural network (CNN) using by generalized estimating equation. b compared with the results of the CNN using by DeLong’s test. R radiologist, E endocrinologist, CNN deep convolutional neural network, AUS atypia of undetermined significance, FLUS follicular lesion of undetermined significance. In the 158 nodules of the AUS group, the sensitivity, specificity, and AUC of the 8 physicians ranged 25.0-52.8%, 76.0–98.0%, and 0.657–0.768, respectively, while the sensitivity, specificity, and AUC value of CNN was 62.0%, 66.0%, and 0.652 with a cut-off value of 54.1% (Table 3 , Fig. 1 ). CNN showed significantly higher sensitivity than 6 physicians (ranges, 25.0–50.0%; P < 0.05) but not over Radiologist 4 (52.8%; P = 0.110) and Endocrinologist 1 (52.8%; P = 0.128). CNN showed significantly lower specificity than 7 physicians (ranges, 82.0–98.0%; P < 0.050), but not lower than Endocrinologist 1 (76.0%; P = 0.123), and CNN had relatively lower AUC values than all 8 physicians, but this difference was only significant in Radiologist 2 (P = 0.011). In the 44 nodules of the FLUS group, the sensitivity, specificity, and AUC of the 8 physicians were 0–25.0%, 88.9–100%, and 0.469–0.674, respectively. The sensitivity, specificity, and AUC value of CNN was 62.5%, 77.8%, and 0.622, respectively, with an estimated cut-off value of 15.9% (Table 3 , Fig. 1 ). CNN showed significantly higher sensitivity than 7 physicians (ranges, 0-12.5%; P < 0.050) but not over Endocrinologist 1 (25.0%; P = 0.128). CNN showed significantly lower specificity than 6 physicians (P 0.050). For interobserver variability, the 8 physicians showed moderate agreement (k = 0.543; 95% confidence interval [CI], 0.381–0.414), the 4 radiologists substantial agreement (k = 0.652; 95% CI, 0.596–0.709), and the 4 endocrinologists moderate agreement (k = 0.455; 95% CI, 0.399–0.511). In the subgroup analysis for the 158 nodules with AUS cytology, the 8 physicians showed moderate agreement (k = 0.523; 95% CI, 0.493–0.552), the 4 radiologists substantial agreement (k = 0.624; 95% CI, 0.560–0.687), and the 4 endocrinologists moderate agreement (k = 0.447; 95% CI, 0.383–0.511). The 8 physicians showed fair agreement (k = 0.349; 95% CI, 0.293–0.405), substantial agreement (k = 0.647; 95% CI, 0.526–0.767), and slight agreement (k = 0.106; 95% CI, 0.015–0.226) for the 44 nodules with FLUS cytology. Discussion The AUS/FLUS cytology includes a heterogeneous and broad spectrum of diagnoses which contain more pronounced cells with architectural and/or nuclear atypia than benign lesions but not enough of these cells to be considered malignant, and have a malignancy risk of 6–18% after NIFTP is removed which can make it difficult for clinicians to reach a decision on further management 2 . For nodules of this category, we can perform repeat FNA/CNB or molecular tests as supplementary evaluation methods instead of proceeding to surgery; however, even results from repeated FNA show the same cytology in 10–30% of the nodules 9 . In nodules with AUS/FLUS cytology, US features can help stratify the malignancy risk of thyroid nodules 3,4,10−12 . A meta-analysis study showed that the more suspicious US features a nodule has, the more likely it is to be malignant 3 , with similar results being observed in nodules with AUS cytology, but not in those with FLUS cytology 10 , 11 . However, the US examination itself is highly subjective, operator dependent and less reproducible than other imaging methods 5 , 13 . CNN is a typical deep learning algorithm based on feature recognition 14 – 16 . It can extract regular features automatically from 2D images including thyroid US to achieve good diagnostic results; thus, CNN is more objective and highly reproducible compared to US when assisting diagnosis 14,17−20 . Several recent studies have shown comparable diagnostic performance between radiologists and CNN for evaluating thyroid nodules on US 17 – 20 . This study mainly aimed to suggest a possible supportive role of CNN for predicting malignancy in AUS/FLUS lesions. Past studies have compared the diagnostic performances of CNN and human physicians, but to our knowledge, all of the physicians in these past studies were radiologists 17,19−21 . Our study compared the diagnostic performances of 8 physicians and CNN for diagnosing thyroid malignancy and the physicians in our study were a heterogeneous group of 4 radiologists and 4 endocrinologists with variable levels of experience. Recently, machine learning and deep learning methods have been developed, and CNN showed the highest accuracy and specificity when machine learning models were compared to differentiate Bethesda category III nodules from Bethesda IV/V/VI nodules using US images 22 . This previous study was performed to make decisions on treatment, and showed the US characteristics of the ACR TI-RADS system assigned by each radiologist, but diagnostic accuracy was not compared between clinicians and the machine learning approaches. Our study is meaningful because as far as we know, it is the first to compare the diagnostic performance of clinicians and CNN to predict malignancy in thyroid nodules with AUS/FLUS cytology. In this study, the AUC of CNN was similar to those of the 8 physicians for diagnosing malignancy. CNN showed higher sensitivity and lower specificity for diagnosing malignancy in AUS/FLUS lesions than the 8 physicians and these results were comparable to those of other recent studies with higher sensitivity and lower specificity for CNN compared to radiologists 17 , 20 , 21 , 23 . However, our results for both CNN and radiologists showed relatively lower sensitivity, higher specificity, and lower AUC values than other studies 17 , 20 , 21 . Our study only included nodules with AUS/FLUS confirmed at FNA. Furthermore, the structures of CNNs are varying in each study and used cut-off values to make the decision based on the probability results from CNNs (there are diverse approaches to determine the cut-off value) are different. In comparison, other studies included thyroid nodules without considering their cytologic results of FNA. Thus, the absolute values of the diagnostic performances are affected by these differences. Rather than weighing the absolute values of the diagnostic performances, it would be more appropriate to check and compare trends. Moreover, most of our study population consisted of AUS nodules (78.2%), and CNN also showed similar diagnostic performances with AUS/FLUS. Interobserver variability is a very important issue because US is highly subjective and operator dependent as mentioned above, and diagnosis using captured JPEG images is more subjective 5 , 13 . There was a study evaluating the interobserver variability of three radiologists with various experience levels (a resident, a fellow, and a staff), and moderate agreement was observed for each US characteristic (k = 0.473–0.634) except for shape (k = 0.034) 21 . Ko et al. reported fair interobserver variability between two radiologists using TI-RADS by Kwak et al., and criteria by Kim et al. 20 . We only analyzed risk levels according to the ACR TI-RADS system for interobserver variability, and did not analyze each US feature. Our results showed moderate interobserver variability among the 8 physicians. Substantial agreement was observed between the 4 radiologists, which is slightly superior to the interobserver variability of all 8 physicians and also the interobserver variability of 4 endocrinologists. Our 4 radiologists had different levels of experience with thyroid US, but their daily work exposed them much more to US images, making them also much more familiar with US images and the ACR TI-RADS system than endocrinologists. Our study has several limitations. First, there was selection bias due to its retrospective study design. Second, the total sample size was not large despite it being a multicenter study, and the number of FLUS cytology nodules was only 44 (21.8%), which is relatively small for generalizing its findings to an entire population. Third, the malignancy rate after surgery was 57.4%, much higher than the rate recommended by the Bethesda system 2 . For AUS/FLUS cytology, excision can be considered when repeated FNA/CNB or molecular tests are not helpful or nodules show suspicious US characteristics. We used the inclusion criteria of surgery-performed lesions only, thus, a higher malignancy rate is expected. Fourth, we only compared the risk levels of the ACR TI-RADS system without considering each US feature, which again was a point of conflict between the 8 physicians (Supplementary Table 1). The diagnostic performance of CNN was comparable to that of physicians with variable experience levels in differentiating malignancy from thyroid nodules with AUS/FLUS cytology on US. Methods This multicenter study was based on patient data collected from three tertiary referral institutions in South Korea. The institutional review boards (IRB) of all three institutions approved this retrospective observational study and the need of informed consent was waived for the review of patient images and records by three IRBs (Kangbuk Samsung Hospital Institutional Review Board, 2020-03-020; Yonsei University Health System, Severance Hospital, Institutional Review Board, 4-2020-0106; and Seoul National University College of Medicine/ Seoul National University Hospital Institutional Review Board, 1911-039-1076). This study was performed in accordance with relevant guidelines and regulations. We collected 3,590 consecutive patients who underwent thyroid surgery at each hospital (Institution A, Jan 2014 to Jun 2019, n = 1,938; Institution B, Jan 2019 to Sep 2019, n = 1,311; and Institution C, Jan 2017 to Jun 2019, n = 341; Fig. 2 ). In these patients, we searched for nodules ≥ 1cm that were confirmed as Bethesda category III on FNA and surgically excised. Finally, 202 nodules in 202 patients were included in this study (A, n = 112; B, n = 44; and C, n = 46; Fig. 2 ). US Examinations and Imaging Interpretation. US examinations were performed using several types of US machines (Supplementary Information 1). One clinician at each hospital reviewed the preoperative thyroid US images, selected the most representative image of each thyroid nodule, and saved them as JPEG files (Fig. 3 ). A square region-of-interest (ROI) was then drawn to cover each whole nodule using the Microsoft Paint program (version 6.1; Microsoft Corporation, Redmond, WA, USA). The saved images from the 3 hospitals were randomly mixed and numbered by an experienced radiologist (Fig. 3 ). They were independently reviewed by the following 8 physicians, none who had information on the cytopathologic results of each thyroid nodule: 2 faculty radiologists (7 and 10 years of experience in thyroid imaging), 2 less experienced radiologists (2 and 4 years of experience), 2 faculty endocrinologists (more than 5 years of experience), and 2 less experienced endocrinologists (1 year of experience). Before reviewing the captured images, all of 8 physicians were trained using the user’s guide by ACR TI-RADS 24 . The 8 physicians evaluated the following US features using the TI-RADS system proposed by the ACR 24 : composition (cystic or almost completely cystic, spongiform, mixed cystic and solid, solid or almost completely solid), echogenicity (anechoic, hyperechoic or isoechoic, hypoechoic, very hypoechoic), shape (wider-than-taller, taller-than-wide), margin (smooth, ill-defined, lobulated or irregular, extrathyroidal extension), and echogenic foci (none or large comet-tail artifacts, macrocalcifications, peripheral calcifications, punctate echogenic foci). Eight physicians determined malignancy risk using the ACR TI-RADS system and the assigned risk levels ranged from TI-RADS (TR) 1 (benign, 0 points), TR2 (not suspicious, 2 points), TR3 (mildly suspicious, 3 points), TR4 (moderately suspicious, 4–6 points), to TR5 (highly suspicious, 7 or more points) (Supplementary Table 2) 24 . Deep Convolutional Neural Network. In this study, we used a computer-aided diagnosis (CAD) program to differentiate malignancy from benign lesions, which was recently developed with 13,560 US images of thyroid nodules using a deep convolutional neural network 8 (Supplementary Information 2 and Supplementary Fig. 1). Statistical Analysis. We collected data on the final diagnosis of each thyroid nodule after surgery that had been recorded in the electronic medical records of each hospital. Cancer probabilities were calculated using CNN, and were presented as percentages (0 ~ 100%). Categorical data were summarized as frequencies and percentages, and continuous variables were presented as means ± standard deviations or median (interquartile range). The Shapiro-Wilk test was performed to assess the normality of continuous variables. We evaluated differences in variables using the independent two-sample t-test, Mann-Whitney U test, Chi-square test, or Fisher’s exact test. Sensitivities and specificities of the 8 physicians and CNN for predicting malignancy were evaluated and compared by generalized estimating equation (GEE). Of the risk levels of the ACR TI-RADS system, we used a cut-off point of TR 5 for the 8 physicians. The cut-off values of CNN were determined with Youden’s index. A receiver operating characteristic (ROC) curve analysis and areas under the curve (AUCs) were compared by DeLong’s test. The diagnostic performances of the 8 physicians and CNN were evaluated in each AUS and FLUS group, and also compared using the ROC curve analysis. We evaluated interobserver variability among all 8 physicians using Fleiss’ Kappa, and then divided the physicians into 2 groups to also compare interobserver variability among the 4 radiologists and among the 4 endocrinologists separately with Fleiss’ Kappa. A kappa value (k) of less than 0 indicated no agreement; 0-0.20, slight agreement; 0.21–0.40, fair agreement; 0.41–0.60, moderate agreement; 0.61–0.80, substantial agreement; and 0.81-1.00, almost perfect agreement 25 . All P values were calculated using the two-tailed t-test and a P < 0.05 was considered to indicate statistical significance. All statistical analyses were performed using commercially available statistical software (SAS, version 9.4, SAS Inc., Cary, NC, USA) and R Statistical Package (Institute for Statistics and Mathematics, Vienna, Austria, ver 4.0.2, www.R-project.org ). Declarations Acknowledgements This study was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2019R1A2C1002375 and 2021R1A2C2007492). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. We collected consecutive patients from three institutions, and the numbers of patients recruited from each hospital was expressed as follows: Institution A, Kangbuk Samsung Hospital; Institution B, Severance Hospital; Institution C, Seoul National University Hospital Author contributions statement S.W.C. and J.Y.K. designed the study, E.L. developed CNN, I.Y., S.W.C., and J.Y.K. reviewed and captured images, J.Y.K. randomly mixed and numbered the image, I.Y., J.H.Y., M.K., J.M., S.K., S.K., K.J., and S.W.C. reviewed the captured images, H.S.L., Y.J.P., and D.J.P. analyzed the results, I.Y., J.Y.K. wrote the manuscript, S.W.C. and J.Y.K. contributed equally to the work as corresponding authors, all authors reviewed the manuscript. Competing interests The authors declare no competing interests. References Keh, S. M., El-Shunnar, S. K., Palmer, T. & Ahsan, S. F. Incidence of malignancy in solitary thyroid nodules. J Laryngol Otol 129 , 677-681, https://doi.org/10.1017/S0022215115000882 (2015). Cibas, E. S. & Ali, S. Z. The 2017 Bethesda System for Reporting Thyroid Cytopathology. Thyroid 27 , 1341-1346, https://doi.org/10.1089/thy.2017.0500 (2017). Gao, L. Y. et al. Ultrasound is helpful to differentiate Bethesda class III thyroid nodules: A PRISMA-compliant systematic review and meta-analysis. Medicine (Baltimore) 96 , e6564, https://doi.org/10.1097/MD.0000000000006564 (2017). Lee, J. H. et al. Risk Stratification of Thyroid Nodules With Atypia of Undetermined Significance/Follicular Lesion of Undetermined Significance (AUS/FLUS) Cytology Using Ultrasonography Patterns Defined by the 2015 ATA Guidelines. Ann Otol Rhinol Laryngol 126 , 625-633, https://doi.org/10.1177/0003489417719472 (2017). Choi, S. H., Kim, E. K., Kwak, J. Y., Kim, M. J. & Son, E. J. Interobserver and intraobserver variations in ultrasound assessment of thyroid nodules. Thyroid 20 , 167-172, https://doi.org/10.1089/thy.2008.0354 (2010). Litjens, G. et al. A survey on deep learning in medical image analysis. Med Image Anal 42 , 60-88, https://doi.org/10.1016/j.media.2017.07.005 (2017). Li, X. et al. Diagnosis of thyroid cancer using deep convolutional neural network models applied to sonographic images: a retrospective, multicohort, diagnostic study. Lancet Oncol 20 , 193-201, https://doi.org/10.1016/S1470-2045(18)30762-9 (2019). Koh, J. et al. Diagnosis of thyroid nodules on ultrasonography by a deep convolutional neural network. Sci Rep 10 , 15245, https://doi.org/10.1038/s41598-020-72270-6 (2020). Haugen, B. R. et al. 2015 American Thyroid Association Management Guidelines for Adult Patients with Thyroid Nodules and Differentiated Thyroid Cancer: The American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer. Thyroid 26 , 1-133, https://doi.org/10.1089/thy.2015.0020 (2016). Yoon, J. H., Kwon, H. J., Kim, E. K., Moon, H. J. & Kwak, J. Y. Subcategorization of atypia of undetermined significance/follicular lesion of undetermined significance (AUS/FLUS): a study applying Thyroid Imaging Reporting and Data System (TIRADS). Clin Endocrinol (Oxf) 85 , 275-282, https://doi.org/10.1111/cen.12987 (2016). Baser, H. et al. Diagnostic accuracy of Thyroid Imaging Reporting and Data System in the prediction of malignancy in nodules with atypia and follicular lesion of undetermined significance cytologies. Clin Endocrinol (Oxf) 86 , 584-590, https://doi.org/10.1111/cen.13274 (2017). Ha, E. J., Na, D. G., Moon, W.-J., Lee, Y. H. & Choi, N. Diagnostic Performance of Ultrasound-Based Risk-Stratification Systems for Thyroid Nodules: Comparison of the 2015 American Thyroid Association Guidelines with the 2016 Korean Thyroid Association/Korean Society of Thyroid Radiology and 2017 American College of Radiology Guidelines. Thyroid 28 , 1532-1537, https://doi.org/10.1089/thy.2018.0094 (2018). Kim, S. H. et al. Observer variability and the performance between faculties and residents: US criteria for benign and malignant thyroid nodules. Korean J Radiol 11 , 149-155, https://doi.org/10.3348/kjr.2010.11.2.149 (2010). Ma, J., Wu, F., Zhu, J., Xu, D. & Kong, D. A pre-trained convolutional neural network based method for thyroid nodule diagnosis. Ultrasonics 73 , 221-230, https://doi.org/10.1016/j.ultras.2016.09.011 (2017). Lee, E. et al. Differentiation of thyroid nodules on US using features learned and extracted from various convolutional neural networks. Sci Rep 9 , 19854, https://doi.org/10.1038/s41598-019-56395-x (2019). Park, V. Y. et al. Diagnosis of Thyroid Nodules: Performance of a Deep Learning Convolutional Neural Network Model vs. Radiologists. Sci Rep 9 , 17843, https://doi.org/10.1038/s41598-019-54434-1 (2019). Gao, L. et al. Computer-aided system for diagnosing thyroid nodules on ultrasound: A comparison with radiologist-based clinical assessments. Head Neck 40 , 778-783, https://doi.org/10.1002/hed.25049 (2018). Jeong, E. Y. et al. Computer-aided diagnosis system for thyroid nodules on ultrasonography: diagnostic performance and reproducibility based on the experience level of operators. Eur Radiol 29 , 1978-1985, https://doi.org/10.1007/s00330-018-5772-9 (2019). Jin, Z. et al. Ultrasound Computer-Aided Diagnosis (CAD) Based on the Thyroid Imaging Reporting and Data System (TI-RADS) to Distinguish Benign from Malignant Thyroid Nodules and the Diagnostic Performance of Radiologists with Different Diagnostic Experience. Med Sci Monit 26 , e918452, https://doi.org/10.12659/MSM.918452 (2020). Ko, S. Y. et al. Deep convolutional neural network for the diagnosis of thyroid nodules on ultrasound. Head Neck 41 , 885-891, https://doi.org/10.1002/hed.25415 (2019). Chung, S. R. et al. Computer-Aided Diagnosis System for the Evaluation of Thyroid Nodules on Ultrasonography: Prospective Non-Inferiority Study according to the Experience Level of Radiologists. Korean J Radiol 21 , 369-376, https://doi.org/10.3348/kjr.2019.0581 (2020). Zhu, Y., Sang, Q., Jia, S., Wang, Y. & Deyer, T. Deep neural networks could differentiate Bethesda class III versus class IV/V/VI. Ann Transl Med 7 , 231, https://doi.org/10.21037/atm.2018.07.03 (2019). Choi, Y. J. et al. A Computer-Aided Diagnosis System Using Artificial Intelligence for the Diagnosis and Characterization of Thyroid Nodules on Ultrasound: Initial Clinical Assessment. Thyroid 27 , 546-552, https://doi.org/10.1089/thy.2016.0372 (2017). Tessler, F. N., Middleton, W. D. & Grant, E. G. Thyroid Imaging Reporting and Data System (TI-RADS): A User's Guide. Radiology 287 , 29-36, https://doi.org/10.1148/radiol.2017171240 (2018). Landis, J. R. & Koch, G. G. The measurement of observer agreement for categorical data. Biometrics 33 , 159-174, https://doi.org/10.2307/2529310 (1977). Additional Declarations Competing interest reported. This study was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2019R1A2C1002375 and 2021R1A2C2007492). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Supplementary Files Supplementarymaterial.docx Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Major revision 17 Aug, 2021 Reviews received at journal 02 Aug, 2021 Reviewers agreed at journal 29 Jul, 2021 Reviewers invited by journal 18 Jul, 2021 Editor assigned by journal 16 Jul, 2021 Editor invited by journal 15 Apr, 2021 Submission checks completed at journal 15 Apr, 2021 First submitted to journal 06 Apr, 2021 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About In Review Editorial Policies Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-400151","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":21459607,"identity":"8e4e1816-45e9-4a2a-83ea-250a1103dbca","order_by":0,"name":"Inyoung Youn","email":"","orcid":"","institution":"Kangbuk Samsung Hospital, Sungkyunkwan University School of Medicine","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Inyoung","middleName":"","lastName":"Youn","suffix":""},{"id":21459608,"identity":"d4a9e63c-ac0e-4b8d-90bb-38067c08d0fb","order_by":1,"name":"Eunjung Lee","email":"","orcid":"","institution":"Yonsei University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Eunjung","middleName":"","lastName":"Lee","suffix":""},{"id":21459609,"identity":"e0e2d7ea-dc76-4fa9-a723-afa12bcae7ea","order_by":2,"name":"Jung Hyun Yoon","email":"","orcid":"","institution":"Yonsei University College of Medicine","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Jung","middleName":"Hyun","lastName":"Yoon","suffix":""},{"id":21459610,"identity":"63c40032-128f-4de8-8bf7-3996f3cc5ae6","order_by":3,"name":"Hye Sun Lee","email":"","orcid":"","institution":"Yonsei University College of Medicine","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Hye","middleName":"Sun","lastName":"Lee","suffix":""},{"id":21459611,"identity":"77f4df09-8617-47ef-bccf-b6fa50eaefe2","order_by":4,"name":"Mi-Ri Kwon","email":"","orcid":"","institution":"Kangbuk Samsung Hospital, Sungkyunkwan University School of Medicine","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Mi-Ri","middleName":"","lastName":"Kwon","suffix":""},{"id":21459612,"identity":"a0ebfa14-ac71-4e92-9f87-137601718ab4","order_by":5,"name":"Juhee Moon","email":"","orcid":"","institution":"Kangbuk Samsung Hospital, Sungkyunkwan University School of Medicine","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Juhee","middleName":"","lastName":"Moon","suffix":""},{"id":21459613,"identity":"cd03b475-50a2-47bf-a4e7-495f304fea13","order_by":6,"name":"Sunyoung Kang","email":"","orcid":"","institution":"Seoul National University Hospital","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Sunyoung","middleName":"","lastName":"Kang","suffix":""},{"id":21459614,"identity":"0a96a070-241d-4c5a-b992-2b58ac87217e","order_by":7,"name":"Seulki Kwon","email":"","orcid":"","institution":"Seoul National University Hospital","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Seulki","middleName":"","lastName":"Kwon","suffix":""},{"id":21459615,"identity":"973ef8d5-6cb2-48d7-a0dd-62604570b654","order_by":8,"name":"Kyong Yeun Jung","email":"","orcid":"","institution":"Eulji University Hospital","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Kyong","middleName":"Yeun","lastName":"Jung","suffix":""},{"id":21459616,"identity":"f1402d4f-404c-45f8-a985-e6192de203fc","order_by":9,"name":"Young Joo Park","email":"","orcid":"","institution":"Seoul National University Hospital","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Young","middleName":"Joo","lastName":"Park","suffix":""},{"id":21459617,"identity":"7bb84251-22a4-4218-9d6f-30c48c806274","order_by":10,"name":"Do Joon Park","email":"","orcid":"","institution":"Seoul National University Hospital","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Do","middleName":"Joon","lastName":"Park","suffix":""},{"id":21459618,"identity":"64b850de-f1c1-4e47-9f87-061efba29d73","order_by":11,"name":"Sun Wook Cho","email":"","orcid":"","institution":"Seoul National University Hospital","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Sun","middleName":"Wook","lastName":"Cho","suffix":""},{"id":21459619,"identity":"bf95b90d-5f71-4621-9019-6a6eaa2a413c","order_by":12,"name":"Jin Young Kwak","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAq0lEQVRIiWNgGAWjYDCCAwwMBgkMNgwMEiRqSSNRCxAcJkEL3/HmBwUPd5xP3HC7/QHDjxoitEieOWZgkHjmduKGO2cMGHuOEaHF4EYOg0FiG1ALkMHA20C8lnNALekPGP+SoOUAUEuCATNRtkD80pZsPPNGjsFhGWL8AgyxZ4Y/2+xk+26kP3z4hpgQAwI2AyDhCHLSAeI0MDAwPwAS9sSqHgWjYBSMghEIACyvPyrrWRcuAAAAAElFTkSuQmCC","orcid":"","institution":"Yonsei University College of Medicine","correspondingAuthor":true,"submittingAuthor":false,"prefix":"","firstName":"Jin","middleName":"Young","lastName":"Kwak","suffix":""}],"badges":[],"createdAt":"2021-04-07 01:44:03","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-400151/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-400151/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":8108792,"identity":"3e4653a2-7110-4dff-8532-d65ca1ca17a4","added_by":"auto","created_at":"2021-04-16 22:06:26","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":188588,"visible":true,"origin":"","legend":"Comparing diagnostic performances between the 8 physicians and CNN using the receiver operating characteristic analysis for the atypia of undetermined significance (AUS)/follicular lesion of undetermined significance (FLUS, A), only AUS (B), and only FLUS (C) groups. Data in parentheses are the AUC results of each physician or CNN.\nCNN deep convolutional neural network, AUS atypia of undetermined significance, FLUS follicular lesion of undetermined significance, R radiologist, E endocrinologist.","description":"","filename":"Figure1.jpg","url":"https://assets-eu.researchsquare.com/files/rs-400151/v1/b03310af04bf6b5932f942bb.jpg"},{"id":8108719,"identity":"059cbc75-944b-41b7-97a5-5ad3ed90cc3e","added_by":"auto","created_at":"2021-04-16 22:03:16","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":206865,"visible":true,"origin":"","legend":"Diagram of the study group which included patients from 3 different hospitals.\nFNA fine-needle aspiration, AUS atypia of undetermined significance, FLUS follicular lesion of undetermined significance. ","description":"","filename":"Figure2.jpg","url":"https://assets-eu.researchsquare.com/files/rs-400151/v1/bdb4598f9a9e9c47426ef142.jpg"},{"id":8108721,"identity":"58c09cd0-ad75-466e-985d-908015038c72","added_by":"auto","created_at":"2021-04-16 22:03:16","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":175160,"visible":true,"origin":"","legend":"Deep convolutional neural network (CNN) processing using ultrasonography (US) images of malignant thyroid nodules with atypia of undetermined significance (AUS, A) or follicular lesion of undetermined significance (FLUS, B) results on fine-needle aspiration (FNA). A, A captured thyroid US image of a yellow square region-of-interest covering the whole thyroid nodule in a 71-year-old man. There was a 10mm-sized thyroid nodule diagnosed as AUS on US-guided FNA. The cancer probability calculated by CNN was 90.9%. The patient underwent surgery, and pathology confirmed papillary carcinoma. B, A captured thyroid US image of a yellow square region-of-interest covering the whole nodule in a 57-year-old woman. There was a 12mm-sized thyroid nodule diagnosed as FLUS on US-guided FNA. The cancer probability calculated by CNN was 88.1%. The patient underwent surgery, and pathology confirmed encapsulated angioinvasive follicular carcinoma.","description":"","filename":"Figure3.jpg","url":"https://assets-eu.researchsquare.com/files/rs-400151/v1/a264880de4b2c7614ac1ea74.jpg"},{"id":13686814,"identity":"2c2f968a-d4de-4607-a757-d42058d5513c","added_by":"auto","created_at":"2021-09-17 12:17:41","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":592457,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-400151/v1/1e4bfd55-1632-4eea-a3d4-f86d644cdf96.pdf"},{"id":8108720,"identity":"d22c5c84-c3b4-42d3-afbb-875e60591879","added_by":"auto","created_at":"2021-04-16 22:03:16","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":49434,"visible":true,"origin":"","legend":"","description":"","filename":"Supplementarymaterial.docx","url":"https://assets-eu.researchsquare.com/files/rs-400151/v1/0491f3c11a3aa41a9b15cf99.docx"}],"financialInterests":"Competing interest reported. This study was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2019R1A2C1002375 and 2021R1A2C2007492). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.","formattedTitle":"\u003cp\u003eUsing the Deep Convolutional Neural Network to Evaluate Thyroid Nodules With Atypia of Undetermined Significance/follicular Lesion of Undetermined Significance Cytology: Multicenter Study\u003c/p\u003e","fulltext":[{"header":"Introduction","content":" \u003cp\u003eThyroid nodules occur commonly with incidence rates going up to 68%\u003csup\u003e1\u003c/sup\u003e, and ultrasonography (US) is the primary screening method used to detect these nodules with high sensitivity and specificity. Fine-needle aspiration (FNA) is an easy, relatively safe, and highly accurate diagnostic tool that can be performed under US-guidance to identify benign and malignant nodules based on US findings.\u003c/p\u003e \u003cp\u003eThe Bethesda system is a standardized, category-based reporting system for thyroid cytopathology, and widely used to interpret FNA results \u003csup\u003e\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e. The nodules with Bethesda class III lesions, otherwise known as atypia of undetermined significance (AUS) or follicular lesion of undetermined significance (FLUS), have a malignancy risk of 6\u0026ndash;18%, and management plans vary widely from clinical observation, US follow up, repeat FNA or core needle biopsy, molecular test to thyroid surgery\u003csup\u003e\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e,\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e. Although thyroid US examination has been shown to help stratify the risk of Bethesda class III lesions\u003csup\u003e\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e,\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u003c/sup\u003e, US assessment is limited in application due to its inherent limitations of poorly reproducible tests\u003csup\u003e\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eRecently, machine learning and deep learning methods have been developed, and rapidly become a methodology of choice for medical image analysis\u003csup\u003e\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e,\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e\u003c/sup\u003e. Deep convolutional neural network (CNN) trained with an automated process using raw image pixels rather than engineered features extracted by experts of traditional machine learning algorithm\u003csup\u003e\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e\u003c/sup\u003e. Recently, we developed a computer-aided program that uses a deep convolutional neural network (CNN) to diagnose thyroid nodules according to US features\u003csup\u003e\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u003c/sup\u003e. This CNN can be an objective, operator-independent method to identify benign lesions and malignancy, and these advantages are thought to be especially helpful for nodules with AUS/FLUS cytology on FNA in predicting malignant risk and determining the next management step.\u003c/p\u003e \u003cp\u003eThe purpose of this study was to compare the diagnostic performances of physicians with varying experience levels and CNN to predict malignancy using US images of thyroid nodules with Bethesda class III results on FNA.\u003c/p\u003e "},{"header":"Results","content":"\u003cp\u003eTable\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e summarized the demographic features of the included 202 nodules. There were 86 (42.6%) benign nodules and 116 (57.4%) malignancies confirmed after surgery. The pathologic results after surgery were shown in Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e. Of 202 nodules, preoperative FNA found 158 with AUS cytology and 44 with FLUS cytology. There was no statistical difference between the benign and malignant nodules for sex and age. Malignant nodules had significantly smaller size than benign ones (P\u0026thinsp;=\u0026thinsp;0.009), and higher cancer probabilities than benign nodules using CNN (P\u0026thinsp;\u0026lt;\u0026thinsp;0.001).\u003c/p\u003e\n\u003cdiv class=\"gridtable\"\u003e\n\u003ctable id=\"Tab1\" border=\"1\"\u003e\u003ccaption\u003e\n\u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e\n\u003cdiv class=\"CaptionContent\"\u003e\n\u003cp\u003eSummary of the demographic features.\u003c/p\u003e\n\u003c/div\u003e\n\u003c/caption\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth align=\"left\"\u003e\u0026nbsp;\u003c/th\u003e\n\u003cth colspan=\"2\" align=\"left\"\u003e\n\u003cp\u003eTotal\u003c/p\u003e\n\u003c/th\u003e\n\u003cth colspan=\"2\" align=\"left\"\u003e\n\u003cp\u003eBenign\u003c/p\u003e\n\u003c/th\u003e\n\u003cth colspan=\"2\" align=\"left\"\u003e\n\u003cp\u003eMalignancy\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eP value\u003c/p\u003e\n\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eNo. of nodules\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\n\u003cp\u003e202\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\n\u003cp\u003e86 (42.6%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\n\u003cp\u003e116 (57.4%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eSex\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.416\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eMale\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\n\u003cp\u003e48\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\n\u003cp\u003e18 (20.9%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\n\u003cp\u003e30 (25.9%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eFemale\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\n\u003cp\u003e154\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\n\u003cp\u003e68 (79.1%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\n\u003cp\u003e86 (74.1%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eMean age (years)\u003csup\u003ea\u003c/sup\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\n\u003cp\u003e47.9\u0026thinsp;\u0026plusmn;\u0026thinsp;13.3\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\n\u003cp\u003e47.0\u0026thinsp;\u0026plusmn;\u0026thinsp;14.8\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.669\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eCytologic result\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u0026lt;\u0026thinsp;0.001\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eNo. of AUS\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eInstitution A\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd rowspan=\"3\" align=\"left\"\u003e\n\u003cp\u003e158\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e78\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd rowspan=\"3\" align=\"left\"\u003e\n\u003cp\u003e50 (58.1%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e23\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd rowspan=\"3\" align=\"left\"\u003e\n\u003cp\u003e108 (93.1%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e55\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd rowspan=\"3\" align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eInstitution B\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e43\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e14\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e29\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eInstitution C\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e37\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e13\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e24\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eNo. of FLUS\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eInstitution A\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd rowspan=\"3\" align=\"left\"\u003e\n\u003cp\u003e44\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e34\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd rowspan=\"3\" align=\"left\"\u003e\n\u003cp\u003e36 (41.9%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e29\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd rowspan=\"3\" align=\"left\"\u003e\n\u003cp\u003e8 (6.9%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e5\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd rowspan=\"3\" align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eInstitution B\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e1\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e1\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eInstitution C\u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e9\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e6\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e3\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eMedian size (IQR, mm)\u003csup\u003ec\u003c/sup\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\n\u003cp\u003e19.5 (13\u0026ndash;32)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\n\u003cp\u003e13.5 (11\u0026ndash;23)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.009\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eMedian cancer probability calculated by CNN (IQR, %)\u003csup\u003ec\u003c/sup\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\n\u003cp\u003e36.5 (18.7\u0026ndash;69.5)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"2\" align=\"left\"\u003e\n\u003cp\u003e67.7 (30.2\u0026ndash;89.9)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u0026lt;\u0026thinsp;0.001\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003ctfoot\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"8\"\u003e\u003csup\u003ea\u003c/sup\u003eThe independent two sample t-test.\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"8\"\u003e\u003csup\u003eb\u003c/sup\u003eWe collected consecutive patients from three institutions, and the numbers of patients recruited from each hospital was expressed as Institution A, B, and C.\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"8\"\u003e\u003csup\u003ec\u003c/sup\u003eThe Mann-Whitney U test.\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"8\"\u003e\u003cem\u003eAUS\u003c/em\u003e atypia of undetermined significance, \u003cem\u003eFLUS\u003c/em\u003e follicular lesion of undetermined significance, \u003cem\u003eIQR\u003c/em\u003e interquartile range, \u003cem\u003eCNN\u003c/em\u003e deep convolutional neural network.\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tfoot\u003e\n\u003c/table\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"gridtable\"\u003e\n\u003ctable id=\"Tab2\" border=\"1\"\u003e\u003ccaption\u003e\n\u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e\n\u003cdiv class=\"CaptionContent\"\u003e\n\u003cp\u003ePathologic results after surgery.\u003c/p\u003e\n\u003c/div\u003e\n\u003c/caption\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003ePathologic result\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eAUS\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eFLUS\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003e\u003cem\u003eTotal\u003c/em\u003e\u003c/p\u003e\n\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"4\" align=\"left\"\u003e\n\u003cp\u003eBenign\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eAdenomatous hyperplasia\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e22 (44.0)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e12 (3.3)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e34 (39.5)\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eFollicular adenoma\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e19 (38.0)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e20 (55.6)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e39 (45.3)\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eHurthle cell adenoma\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e3 (6.0)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e3 (8.3)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e6 (7.0)\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eNoninvasive follicular thyroid neoplasm with papillary-like nuclear feature\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e3 (6.0)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e1 (2.8)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e4 (4.7)\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eHyaline trabecular tumor\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e1 (2.0)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e-\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e1 (1.2)\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eLocalized fibrosis\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e1 (2.0)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e-\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e1 (1.2)\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eLymphocytic thyroiditis\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e1 (2.0)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e-\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e1 (1.2)\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cem\u003eTotal\u003c/em\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e50\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e36\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e86\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"4\" align=\"left\"\u003e\n\u003cp\u003eMalignancy\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003ePapillary thyroid carcinoma\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e99 (91.7)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e4 (50.0)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e103 (88.8)\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eFollicular carcinoma\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e8 (7.4)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e3 (37.5)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e11 (9.5)\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003ePoorly differentiated carcinoma\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e1 (0.9)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e1 (12.5)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e2 (1.7)\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cem\u003eTotal\u003c/em\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e108\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e8\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e116\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003ctfoot\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"4\"\u003eData in parentheses are percentages.\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"4\"\u003e\u003cem\u003eAUS\u003c/em\u003e atypia of undetermined significance, \u003cem\u003eFLUS\u003c/em\u003e follicular lesion of undermined significance.\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tfoot\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe diagnostic performances of the 8 physicians and CNN were compared in Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e. The sensitivity, specificity, and AUC of the 8 physicians were 24.1\u0026ndash;50.9%, 81.4\u0026ndash;98.8%, and 0.680\u0026ndash;0.722, respectively (Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e, Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e). The calculated sensitivity, specificity, and AUC of CNN were 59.5%, 69.8%, and 0.666, respectively, using an estimated cut-off value of 54.1% (Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e, Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e). CNN showed significantly higher sensitivity than 6 physicians, but not over Radiologist 4 (50.0%; P\u0026thinsp;=\u0026thinsp;0.082) and Endocrinologist 1 (50.9%; P\u0026thinsp;=\u0026thinsp;0.137). CNN showed significantly lower specificity than all 8 physicians (P\u0026thinsp;\u0026lt;\u0026thinsp;0.05). CNN had similar AUC values compared to the 8 physicians, without statistical difference (P\u0026thinsp;\u0026gt;\u0026thinsp;0.05).\u003c/p\u003e\n\u003cdiv class=\"gridtable\"\u003e\n\u003ctable id=\"Tab3\" border=\"1\"\u003e\u003ccaption\u003e\n\u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e\n\u003cdiv class=\"CaptionContent\"\u003e\n\u003cp\u003eDiagnostic performances of the 8 physicians and deep convolutional neural network.\u003c/p\u003e\n\u003c/div\u003e\n\u003c/caption\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth align=\"left\"\u003e\u0026nbsp;\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eSensitivity\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eP value \u003csup\u003ea\u003c/sup\u003e\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eSpecificity\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eP value \u003csup\u003ea\u003c/sup\u003e\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eAUC\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eP value \u003csup\u003eb\u003c/sup\u003e\u003c/p\u003e\n\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"7\" align=\"left\"\u003e\n\u003cp\u003eTotal 202 nodules\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eR1\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e37.9% (29.1\u0026ndash;46.8%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u0026lt;\u0026thinsp;.001\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e96.5% (92.6\u0026ndash;100%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u0026lt;\u0026thinsp;.001\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.709 (0.643\u0026ndash;0.776)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.279\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eR2\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e44.8% (35.8\u0026ndash;53.9%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.008\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e95.3% (90.9\u0026ndash;99.8%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u0026lt;\u0026thinsp;.001\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.717 (0.649\u0026ndash;0.784)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.187\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eR3\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e47.4% (38.3\u0026ndash;56.5%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.020\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e89.5% (83.1\u0026ndash;96.0%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u0026lt;\u0026thinsp;0.001\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.688 (0.62\u0026ndash;0.757)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.568\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eR4\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e50.0% (40.9\u0026ndash;59.1%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.082\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e90.7% (84.6\u0026ndash;96.8%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u0026lt;\u0026thinsp;.001\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.722 (0.654\u0026ndash;0.789)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.145\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eE1\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e50.9% (41.8\u0026ndash;60.0%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.137\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e81.4% (73.3\u0026ndash;89.6%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.015\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.680 (0.612\u0026ndash;0.749)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.742\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eE2\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e39.7% (30.8\u0026ndash;48.6%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.001\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e89.5% (83.1\u0026ndash;96.0%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.001\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.695 (0.629\u0026ndash;0.760)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.500\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eE3\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e24.1% (16.4\u0026ndash;31.9%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u0026lt;\u0026thinsp;.001\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e98.8% (96.6\u0026ndash;100%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u0026lt;\u0026thinsp;.001\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.709 (0.642\u0026ndash;0.775)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.305\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eE4\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e42.2% (33.3\u0026ndash;51.2%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.001\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e87.2% (80.2\u0026ndash;94.3%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.002\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.692 (0.624\u0026ndash;0.761)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.494\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eCNN\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e59.5% (50.5\u0026ndash;68.4%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e69.8% (60.1\u0026ndash;79.5%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.666 (0.592\u0026ndash;0.740)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"7\" align=\"left\"\u003e\n\u003cp\u003eAUS (n\u0026thinsp;=\u0026thinsp;158)\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eR1\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e39.8% (30.6\u0026ndash;49.0%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u0026lt;\u0026thinsp;.001\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e96.0% (90.6\u0026ndash;100%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u0026lt;\u0026thinsp;.001\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.732 (0.658\u0026ndash;0.806)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.111\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eR2\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e47.2% (37.8\u0026ndash;56.6%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.011\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e98.0% (94.2\u0026ndash;100%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u0026lt;\u0026thinsp;.001\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.768 (0.699\u0026ndash;0.837)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.011\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eR3\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e50.0% (40.6\u0026ndash;59.4%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.029\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e86.0% (76.4\u0026ndash;95.6%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.008\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.698 (0.618\u0026ndash;0.778)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.336\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eR4\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e52.8% (43.4\u0026ndash;62.2%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.110\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e84.0% (73.8\u0026ndash;94.2%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.008\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.705 (0.624\u0026ndash;0.786)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.253\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eE1\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e52.8% (43.4\u0026ndash;62.2%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.128\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e76.0% (64.2\u0026ndash;87.8%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.123\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.657 (0.574\u0026ndash;0.741)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.913\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eE2\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e42.6% (33.3\u0026ndash;51.9%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.001\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e86.0% (76.4\u0026ndash;95.6%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.008\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.685 (0.605\u0026ndash;0.765)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.525\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eE3\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e25.0% (16.8\u0026ndash;33.2%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u0026lt;\u0026thinsp;.001\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e98.0% (94.2\u0026ndash;100%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u0026lt;\u0026thinsp;.001\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.730 (0.654\u0026ndash;0.806)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.110\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eE4\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e44.4% (35.1\u0026ndash;53.8%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.002\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e82.0% (71.4\u0026ndash;92.6%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.037\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.675 (0.59\u0026ndash;0.759)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.628\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eCNN\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e62.0% (52.9\u0026ndash;71.2%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e66.0% (52.9\u0026ndash;79.1%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.652 (0.563\u0026ndash;0.741)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"7\" align=\"left\"\u003e\n\u003cp\u003eFLUS (n\u0026thinsp;=\u0026thinsp;44)\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eR1\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e12.5% (0-35.4%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.046\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e97.2% (91.9\u0026ndash;100%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.011\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.469 (0.234\u0026ndash;0.703)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.435\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eR2\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e12.5% (0-35.4%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.046\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e91.7% (82.6\u0026ndash;100%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.119\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.634 (0.372\u0026ndash;0.895)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.902\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eR3\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e12.5% (0-35.4%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.046\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e94.4% (87.0-100%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.046\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.535 (0.313\u0026ndash;0.757)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.493\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eR4\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e12.5% (0-35.4%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.046\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e100% (100\u0026ndash;100%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.001\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.535 (0.290\u0026ndash;0.780)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.699\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eE1\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e25.0% (0\u0026ndash;55.0%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.128\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e88.9% (78.6\u0026ndash;99.2%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.239\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.587 (0.371\u0026ndash;0.803)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.857\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eE2\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0% (0\u0026ndash;0%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.001\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e94.4% (87.0-100%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.046\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.509 (0.320\u0026ndash;0.697)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.528\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eE3\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e12.5% (0-35.4%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.046\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e100% (100\u0026ndash;100%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.001\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.674 (0.465\u0026ndash;0.882)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.803\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eE4\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e12.5% (0-35.4%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.046\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e94.4% (87.0-100%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.046\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.615 (0.420\u0026ndash;0.809)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.970\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eCNN\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e62.5% (29.0\u0026ndash;96.0%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e77.8% (64.2\u0026ndash;91.4%)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.808\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.622 (0.355\u0026ndash;0.888)\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003ctfoot\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"7\"\u003e\u003csup\u003ea\u003c/sup\u003ecompared with the results of the convolutional neural network (CNN) using by generalized estimating equation.\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"7\"\u003e\u003csup\u003eb\u003c/sup\u003ecompared with the results of the CNN using by DeLong\u0026rsquo;s test.\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"7\"\u003e\u003cem\u003eR\u003c/em\u003e radiologist, \u003cem\u003eE\u003c/em\u003e endocrinologist, \u003cem\u003eCNN\u003c/em\u003e deep convolutional neural network, \u003cem\u003eAUS\u003c/em\u003e atypia of undetermined significance, \u003cem\u003eFLUS\u003c/em\u003e follicular lesion of undetermined significance.\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tfoot\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eIn the 158 nodules of the AUS group, the sensitivity, specificity, and AUC of the 8 physicians ranged 25.0-52.8%, 76.0\u0026ndash;98.0%, and 0.657\u0026ndash;0.768, respectively, while the sensitivity, specificity, and AUC value of CNN was 62.0%, 66.0%, and 0.652 with a cut-off value of 54.1% (Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e, Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e). CNN showed significantly higher sensitivity than 6 physicians (ranges, 25.0\u0026ndash;50.0%; P\u0026thinsp;\u0026lt;\u0026thinsp;0.05) but not over Radiologist 4 (52.8%; P\u0026thinsp;=\u0026thinsp;0.110) and Endocrinologist 1 (52.8%; P\u0026thinsp;=\u0026thinsp;0.128). CNN showed significantly lower specificity than 7 physicians (ranges, 82.0\u0026ndash;98.0%; P\u0026thinsp;\u0026lt;\u0026thinsp;0.050), but not lower than Endocrinologist 1 (76.0%; P\u0026thinsp;=\u0026thinsp;0.123), and CNN had relatively lower AUC values than all 8 physicians, but this difference was only significant in Radiologist 2 (P\u0026thinsp;=\u0026thinsp;0.011).\u003c/p\u003e\n\u003cp\u003eIn the 44 nodules of the FLUS group, the sensitivity, specificity, and AUC of the 8 physicians were 0\u0026ndash;25.0%, 88.9\u0026ndash;100%, and 0.469\u0026ndash;0.674, respectively. The sensitivity, specificity, and AUC value of CNN was 62.5%, 77.8%, and 0.622, respectively, with an estimated cut-off value of 15.9% (Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e, Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e). CNN showed significantly higher sensitivity than 7 physicians (ranges, 0-12.5%; P\u0026thinsp;\u0026lt;\u0026thinsp;0.050) but not over Endocrinologist 1 (25.0%; P\u0026thinsp;=\u0026thinsp;0.128). CNN showed significantly lower specificity than 6 physicians (P\u0026thinsp;\u0026lt;\u0026thinsp;0.050) but not lower than Radiologist 2 (91.7%, P\u0026thinsp;=\u0026thinsp;0.119) and Endocrinologist 1 (88.9%, P\u0026thinsp;=\u0026thinsp;0.239). AUC values did not differ between the 8 physicians and CNN (P\u0026thinsp;\u0026gt;\u0026thinsp;0.050).\u003c/p\u003e\n\u003cp\u003eFor interobserver variability, the 8 physicians showed moderate agreement (k\u0026thinsp;=\u0026thinsp;0.543; 95% confidence interval [CI], 0.381\u0026ndash;0.414), the 4 radiologists substantial agreement (k\u0026thinsp;=\u0026thinsp;0.652; 95% CI, 0.596\u0026ndash;0.709), and the 4 endocrinologists moderate agreement (k\u0026thinsp;=\u0026thinsp;0.455; 95% CI, 0.399\u0026ndash;0.511). In the subgroup analysis for the 158 nodules with AUS cytology, the 8 physicians showed moderate agreement (k\u0026thinsp;=\u0026thinsp;0.523; 95% CI, 0.493\u0026ndash;0.552), the 4 radiologists substantial agreement (k\u0026thinsp;=\u0026thinsp;0.624; 95% CI, 0.560\u0026ndash;0.687), and the 4 endocrinologists moderate agreement (k\u0026thinsp;=\u0026thinsp;0.447; 95% CI, 0.383\u0026ndash;0.511). The 8 physicians showed fair agreement (k\u0026thinsp;=\u0026thinsp;0.349; 95% CI, 0.293\u0026ndash;0.405), substantial agreement (k\u0026thinsp;=\u0026thinsp;0.647; 95% CI, 0.526\u0026ndash;0.767), and slight agreement (k\u0026thinsp;=\u0026thinsp;0.106; 95% CI, 0.015\u0026ndash;0.226) for the 44 nodules with FLUS cytology.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eThe AUS/FLUS cytology includes a heterogeneous and broad spectrum of diagnoses which contain more pronounced cells with architectural and/or nuclear atypia than benign lesions but not enough of these cells to be considered malignant, and have a malignancy risk of 6\u0026ndash;18% after NIFTP is removed which can make it difficult for clinicians to reach a decision on further management\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e. For nodules of this category, we can perform repeat FNA/CNB or molecular tests as supplementary evaluation methods instead of proceeding to surgery; however, even results from repeated FNA show the same cytology in 10\u0026ndash;30% of the nodules\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e9\u003c/span\u003e\u003c/sup\u003e. In nodules with AUS/FLUS cytology, US features can help stratify the malignancy risk of thyroid nodules\u003csup\u003e3,4,10\u0026minus;12\u003c/sup\u003e. A meta-analysis study showed that the more suspicious US features a nodule has, the more likely it is to be malignant\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e, with similar results being observed in nodules with AUS cytology, but not in those with FLUS cytology\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e10\u003c/span\u003e,\u003cspan class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e. However, the US examination itself is highly subjective, operator dependent and less reproducible than other imaging methods\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e5\u003c/span\u003e,\u003cspan class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\n\u003cp\u003eCNN is a typical deep learning algorithm based on feature recognition\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e14\u003c/span\u003e\u0026ndash;\u003cspan class=\"CitationRef\"\u003e16\u003c/span\u003e\u003c/sup\u003e. It can extract regular features automatically from 2D images including thyroid US to achieve good diagnostic results; thus, CNN is more objective and highly reproducible compared to US when assisting diagnosis \u003csup\u003e14,17\u0026minus;20\u003c/sup\u003e. Several recent studies have shown comparable diagnostic performance between radiologists and CNN for evaluating thyroid nodules on US\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e17\u003c/span\u003e\u0026ndash;\u003cspan class=\"CitationRef\"\u003e20\u003c/span\u003e\u003c/sup\u003e. This study mainly aimed to suggest a possible supportive role of CNN for predicting malignancy in AUS/FLUS lesions. Past studies have compared the diagnostic performances of CNN and human physicians, but to our knowledge, all of the physicians in these past studies were radiologists\u003csup\u003e17,19\u0026minus;21\u003c/sup\u003e. Our study compared the diagnostic performances of 8 physicians and CNN for diagnosing thyroid malignancy and the physicians in our study were a heterogeneous group of 4 radiologists and 4 endocrinologists with variable levels of experience.\u003c/p\u003e\n\u003cp\u003eRecently, machine learning and deep learning methods have been developed, and CNN showed the highest accuracy and specificity when machine learning models were compared to differentiate Bethesda category III nodules from Bethesda IV/V/VI nodules using US images\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e22\u003c/span\u003e\u003c/sup\u003e. This previous study was performed to make decisions on treatment, and showed the US characteristics of the ACR TI-RADS system assigned by each radiologist, but diagnostic accuracy was not compared between clinicians and the machine learning approaches. Our study is meaningful because as far as we know, it is the first to compare the diagnostic performance of clinicians and CNN to predict malignancy in thyroid nodules with AUS/FLUS cytology. In this study, the AUC of CNN was similar to those of the 8 physicians for diagnosing malignancy. CNN showed higher sensitivity and lower specificity for diagnosing malignancy in AUS/FLUS lesions than the 8 physicians and these results were comparable to those of other recent studies with higher sensitivity and lower specificity for CNN compared to radiologists\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e17\u003c/span\u003e,\u003cspan class=\"CitationRef\"\u003e20\u003c/span\u003e,\u003cspan class=\"CitationRef\"\u003e21\u003c/span\u003e,\u003cspan class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e. However, our results for both CNN and radiologists showed relatively lower sensitivity, higher specificity, and lower AUC values than other studies\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e17\u003c/span\u003e,\u003cspan class=\"CitationRef\"\u003e20\u003c/span\u003e,\u003cspan class=\"CitationRef\"\u003e21\u003c/span\u003e\u003c/sup\u003e. Our study only included nodules with AUS/FLUS confirmed at FNA. Furthermore, the structures of CNNs are varying in each study and used cut-off values to make the decision based on the probability results from CNNs (there are diverse approaches to determine the cut-off value) are different. In comparison, other studies included thyroid nodules without considering their cytologic results of FNA. Thus, the absolute values of the diagnostic performances are affected by these differences. Rather than weighing the absolute values of the diagnostic performances, it would be more appropriate to check and compare trends. Moreover, most of our study population consisted of AUS nodules (78.2%), and CNN also showed similar diagnostic performances with AUS/FLUS.\u003c/p\u003e\n\u003cp\u003eInterobserver variability is a very important issue because US is highly subjective and operator dependent as mentioned above, and diagnosis using captured JPEG images is more subjective\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e5\u003c/span\u003e,\u003cspan class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e. There was a study evaluating the interobserver variability of three radiologists with various experience levels (a resident, a fellow, and a staff), and moderate agreement was observed for each US characteristic (k\u0026thinsp;=\u0026thinsp;0.473\u0026ndash;0.634) except for shape (k\u0026thinsp;=\u0026thinsp;0.034)\u003csup\u003e21\u003c/sup\u003e. Ko et al. reported fair interobserver variability between two radiologists using TI-RADS by Kwak et al., and criteria by Kim et al.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e20\u003c/span\u003e\u003c/sup\u003e. We only analyzed risk levels according to the ACR TI-RADS system for interobserver variability, and did not analyze each US feature. Our results showed moderate interobserver variability among the 8 physicians. Substantial agreement was observed between the 4 radiologists, which is slightly superior to the interobserver variability of all 8 physicians and also the interobserver variability of 4 endocrinologists. Our 4 radiologists had different levels of experience with thyroid US, but their daily work exposed them much more to US images, making them also much more familiar with US images and the ACR TI-RADS system than endocrinologists.\u003c/p\u003e\n\u003cp\u003eOur study has several limitations. First, there was selection bias due to its retrospective study design. Second, the total sample size was not large despite it being a multicenter study, and the number of FLUS cytology nodules was only 44 (21.8%), which is relatively small for generalizing its findings to an entire population. Third, the malignancy rate after surgery was 57.4%, much higher than the rate recommended by the Bethesda system\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e. For AUS/FLUS cytology, excision can be considered when repeated FNA/CNB or molecular tests are not helpful or nodules show suspicious US characteristics. We used the inclusion criteria of surgery-performed lesions only, thus, a higher malignancy rate is expected. Fourth, we only compared the risk levels of the ACR TI-RADS system without considering each US feature, which again was a point of conflict between the 8 physicians (Supplementary Table\u0026nbsp;1).\u003c/p\u003e\n\u003cp\u003eThe diagnostic performance of CNN was comparable to that of physicians with variable experience levels in differentiating malignancy from thyroid nodules with AUS/FLUS cytology on US.\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003eThis multicenter study was based on patient data collected from three tertiary referral institutions in South Korea. The institutional review boards (IRB) of all three institutions approved this retrospective observational study and the need of informed consent was waived for the review of patient images and records by three IRBs (Kangbuk Samsung Hospital Institutional Review Board, 2020-03-020; Yonsei University Health System, Severance Hospital, Institutional Review Board, 4-2020-0106; and Seoul National University College of Medicine/ Seoul National University Hospital Institutional Review Board, 1911-039-1076). This study was performed in accordance with relevant guidelines and regulations.\u003c/p\u003e\n\u003cp\u003eWe collected 3,590 consecutive patients who underwent thyroid surgery at each hospital (Institution A, Jan 2014 to Jun 2019, n\u0026thinsp;=\u0026thinsp;1,938; Institution B, Jan 2019 to Sep 2019, n\u0026thinsp;=\u0026thinsp;1,311; and Institution C, Jan 2017 to Jun 2019, n\u0026thinsp;=\u0026thinsp;341; Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e). In these patients, we searched for nodules\u0026thinsp;\u0026ge;\u0026thinsp;1cm that were confirmed as Bethesda category III on FNA and surgically excised. Finally, 202 nodules in 202 patients were included in this study (A, n\u0026thinsp;=\u0026thinsp;112; B, n\u0026thinsp;=\u0026thinsp;44; and C, n\u0026thinsp;=\u0026thinsp;46; Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eUS Examinations and Imaging Interpretation.\u003c/strong\u003e US examinations were performed using several types of US machines (Supplementary Information 1). One clinician at each hospital reviewed the preoperative thyroid US images, selected the most representative image of each thyroid nodule, and saved them as JPEG files (Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e). A square region-of-interest (ROI) was then drawn to cover each whole nodule using the Microsoft Paint program (version 6.1; Microsoft Corporation, Redmond, WA, USA). The saved images from the 3 hospitals were randomly mixed and numbered by an experienced radiologist (Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e). They were independently reviewed by the following 8 physicians, none who had information on the cytopathologic results of each thyroid nodule: 2 faculty radiologists (7 and 10 years of experience in thyroid imaging), 2 less experienced radiologists (2 and 4 years of experience), 2 faculty endocrinologists (more than 5 years of experience), and 2 less experienced endocrinologists (1 year of experience). Before reviewing the captured images, all of 8 physicians were trained using the user\u0026rsquo;s guide by ACR TI-RADS\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\n\u003cp\u003eThe 8 physicians evaluated the following US features using the TI-RADS system proposed by the ACR \u003csup\u003e\u003cspan class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e: composition (cystic or almost completely cystic, spongiform, mixed cystic and solid, solid or almost completely solid), echogenicity (anechoic, hyperechoic or isoechoic, hypoechoic, very hypoechoic), shape (wider-than-taller, taller-than-wide), margin (smooth, ill-defined, lobulated or irregular, extrathyroidal extension), and echogenic foci (none or large comet-tail artifacts, macrocalcifications, peripheral calcifications, punctate echogenic foci). Eight physicians determined malignancy risk using the ACR TI-RADS system and the assigned risk levels ranged from TI-RADS (TR) 1 (benign, 0 points), TR2 (not suspicious, 2 points), TR3 (mildly suspicious, 3 points), TR4 (moderately suspicious, 4\u0026ndash;6 points), to TR5 (highly suspicious, 7 or more points) (Supplementary Table\u0026nbsp;2)\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDeep Convolutional Neural Network.\u003c/strong\u003e In this study, we used a computer-aided diagnosis (CAD) program to differentiate malignancy from benign lesions, which was recently developed with 13,560 US images of thyroid nodules using a deep convolutional neural network\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e8\u003c/span\u003e\u003c/sup\u003e (Supplementary Information 2 and Supplementary Fig.\u0026nbsp;1).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eStatistical Analysis.\u003c/strong\u003e We collected data on the final diagnosis of each thyroid nodule after surgery that had been recorded in the electronic medical records of each hospital. Cancer probabilities were calculated using CNN, and were presented as percentages (0\u0026thinsp;~\u0026thinsp;100%). Categorical data were summarized as frequencies and percentages, and continuous variables were presented as means\u0026thinsp;\u0026plusmn;\u0026thinsp;standard deviations or median (interquartile range). The Shapiro-Wilk test was performed to assess the normality of continuous variables. We evaluated differences in variables using the independent two-sample t-test, Mann-Whitney U test, Chi-square test, or Fisher\u0026rsquo;s exact test.\u003c/p\u003e\n\u003cp\u003eSensitivities and specificities of the 8 physicians and CNN for predicting malignancy were evaluated and compared by generalized estimating equation (GEE). Of the risk levels of the ACR TI-RADS system, we used a cut-off point of TR 5 for the 8 physicians. The cut-off values of CNN were determined with Youden\u0026rsquo;s index. A receiver operating characteristic (ROC) curve analysis and areas under the curve (AUCs) were compared by DeLong\u0026rsquo;s test. The diagnostic performances of the 8 physicians and CNN were evaluated in each AUS and FLUS group, and also compared using the ROC curve analysis.\u003c/p\u003e\n\u003cp\u003eWe evaluated interobserver variability among all 8 physicians using Fleiss\u0026rsquo; Kappa, and then divided the physicians into 2 groups to also compare interobserver variability among the 4 radiologists and among the 4 endocrinologists separately with Fleiss\u0026rsquo; Kappa. A kappa value (k) of less than 0 indicated no agreement; 0-0.20, slight agreement; 0.21\u0026ndash;0.40, fair agreement; 0.41\u0026ndash;0.60, moderate agreement; 0.61\u0026ndash;0.80, substantial agreement; and 0.81-1.00, almost perfect agreement\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e25\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\n\u003cp\u003eAll P values were calculated using the two-tailed t-test and a P\u0026thinsp;\u0026lt;\u0026thinsp;0.05 was considered to indicate statistical significance. All statistical analyses were performed using commercially available statistical software (SAS, version 9.4, SAS Inc., Cary, NC, USA) and R Statistical Package (Institute for Statistics and Mathematics, Vienna, Austria, ver 4.0.2, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e\u003ca href=\"http://www.R-project.org\" target=\"_blank\"\u003ewww.R-project.org\u003c/a\u003e\u003c/span\u003e\u003c/span\u003e).\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis study was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2019R1A2C1002375 and 2021R1A2C2007492). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.\u003c/p\u003e\n\u003cp\u003eWe collected consecutive patients from three institutions, and the numbers of patients recruited from each hospital was expressed as follows: Institution A, Kangbuk Samsung Hospital; Institution B, Severance Hospital; Institution C, Seoul National University Hospital\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor contributions statement\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eS.W.C. and J.Y.K. designed the study, E.L. developed CNN, I.Y., S.W.C., and J.Y.K. reviewed and captured images, J.Y.K. randomly mixed and numbered the image, I.Y., J.H.Y., M.K., J.M., S.K., S.K., K.J., and S.W.C. reviewed the captured images, H.S.L., Y.J.P., and D.J.P. analyzed the results, I.Y., J.Y.K. wrote the manuscript, S.W.C. and J.Y.K. contributed equally to the work as corresponding authors, all authors reviewed the manuscript.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors declare no competing interests.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eKeh, S. M., El-Shunnar, S. K., Palmer, T. \u0026amp; Ahsan, S. F. Incidence of malignancy in solitary thyroid nodules. \u003cem\u003eJ Laryngol Otol\u003c/em\u003e \u003cstrong\u003e129\u003c/strong\u003e, 677-681, https://doi.org/10.1017/S0022215115000882 (2015).\u003c/li\u003e\n\u003cli\u003eCibas, E. S. \u0026amp; Ali, S. Z. The 2017 Bethesda System for Reporting Thyroid Cytopathology. \u003cem\u003eThyroid\u003c/em\u003e \u003cstrong\u003e27\u003c/strong\u003e, 1341-1346, https://doi.org/10.1089/thy.2017.0500 (2017).\u003c/li\u003e\n\u003cli\u003eGao, L. Y.\u003cem\u003e et al.\u003c/em\u003e Ultrasound is helpful to differentiate Bethesda class III thyroid nodules: A PRISMA-compliant systematic review and meta-analysis. \u003cem\u003eMedicine (Baltimore)\u003c/em\u003e \u003cstrong\u003e96\u003c/strong\u003e, e6564, https://doi.org/10.1097/MD.0000000000006564 (2017).\u003c/li\u003e\n\u003cli\u003eLee, J. H.\u003cem\u003e et al.\u003c/em\u003e Risk Stratification of Thyroid Nodules With Atypia of Undetermined Significance/Follicular Lesion of Undetermined Significance (AUS/FLUS) Cytology Using Ultrasonography Patterns Defined by the 2015 ATA Guidelines. \u003cem\u003eAnn Otol Rhinol Laryngol\u003c/em\u003e \u003cstrong\u003e126\u003c/strong\u003e, 625-633, https://doi.org/10.1177/0003489417719472 (2017).\u003c/li\u003e\n\u003cli\u003eChoi, S. H., Kim, E. K., Kwak, J. Y., Kim, M. J. \u0026amp; Son, E. J. Interobserver and intraobserver variations in ultrasound assessment of thyroid nodules. \u003cem\u003eThyroid\u003c/em\u003e \u003cstrong\u003e20\u003c/strong\u003e, 167-172, https://doi.org/10.1089/thy.2008.0354 (2010).\u003c/li\u003e\n\u003cli\u003eLitjens, G.\u003cem\u003e et al.\u003c/em\u003e A survey on deep learning in medical image analysis. \u003cem\u003eMed Image Anal\u003c/em\u003e \u003cstrong\u003e42\u003c/strong\u003e, 60-88, https://doi.org/10.1016/j.media.2017.07.005 (2017).\u003c/li\u003e\n\u003cli\u003eLi, X.\u003cem\u003e et al.\u003c/em\u003e Diagnosis of thyroid cancer using deep convolutional neural network models applied to sonographic images: a retrospective, multicohort, diagnostic study. \u003cem\u003eLancet Oncol\u003c/em\u003e \u003cstrong\u003e20\u003c/strong\u003e, 193-201, https://doi.org/10.1016/S1470-2045(18)30762-9 (2019).\u003c/li\u003e\n\u003cli\u003eKoh, J.\u003cem\u003e et al.\u003c/em\u003e Diagnosis of thyroid nodules on ultrasonography by a deep convolutional neural network. \u003cem\u003eSci Rep\u003c/em\u003e \u003cstrong\u003e10\u003c/strong\u003e, 15245, https://doi.org/10.1038/s41598-020-72270-6 (2020).\u003c/li\u003e\n\u003cli\u003eHaugen, B. R.\u003cem\u003e et al.\u003c/em\u003e 2015 American Thyroid Association Management Guidelines for Adult Patients with Thyroid Nodules and Differentiated Thyroid Cancer: The American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer. \u003cem\u003eThyroid\u003c/em\u003e \u003cstrong\u003e26\u003c/strong\u003e, 1-133, https://doi.org/10.1089/thy.2015.0020 (2016).\u003c/li\u003e\n\u003cli\u003eYoon, J. H., Kwon, H. J., Kim, E. K., Moon, H. J. \u0026amp; Kwak, J. Y. Subcategorization of atypia of undetermined significance/follicular lesion of undetermined significance (AUS/FLUS): a study applying Thyroid Imaging Reporting and Data System (TIRADS). \u003cem\u003eClin Endocrinol (Oxf)\u003c/em\u003e \u003cstrong\u003e85\u003c/strong\u003e, 275-282, https://doi.org/10.1111/cen.12987 (2016).\u003c/li\u003e\n\u003cli\u003eBaser, H.\u003cem\u003e et al.\u003c/em\u003e Diagnostic accuracy of Thyroid Imaging Reporting and Data System in the prediction of malignancy in nodules with atypia and follicular lesion of undetermined significance cytologies. \u003cem\u003eClin Endocrinol (Oxf)\u003c/em\u003e \u003cstrong\u003e86\u003c/strong\u003e, 584-590, https://doi.org/10.1111/cen.13274 (2017).\u003c/li\u003e\n\u003cli\u003eHa, E. J., Na, D. G., Moon, W.-J., Lee, Y. H. \u0026amp; Choi, N. Diagnostic Performance of Ultrasound-Based Risk-Stratification Systems for Thyroid Nodules: Comparison of the 2015 American Thyroid Association Guidelines with the 2016 Korean Thyroid Association/Korean Society of Thyroid Radiology and 2017 American College of Radiology Guidelines. \u003cem\u003eThyroid\u003c/em\u003e \u003cstrong\u003e28\u003c/strong\u003e, 1532-1537, https://doi.org/10.1089/thy.2018.0094 (2018).\u003c/li\u003e\n\u003cli\u003eKim, S. H.\u003cem\u003e et al.\u003c/em\u003e Observer variability and the performance between faculties and residents: US criteria for benign and malignant thyroid nodules. \u003cem\u003eKorean J Radiol\u003c/em\u003e \u003cstrong\u003e11\u003c/strong\u003e, 149-155, https://doi.org/10.3348/kjr.2010.11.2.149 (2010).\u003c/li\u003e\n\u003cli\u003eMa, J., Wu, F., Zhu, J., Xu, D. \u0026amp; Kong, D. A pre-trained convolutional neural network based method for thyroid nodule diagnosis. \u003cem\u003eUltrasonics\u003c/em\u003e \u003cstrong\u003e73\u003c/strong\u003e, 221-230, https://doi.org/10.1016/j.ultras.2016.09.011 (2017).\u003c/li\u003e\n\u003cli\u003eLee, E.\u003cem\u003e et al.\u003c/em\u003e Differentiation of thyroid nodules on US using features learned and extracted from various convolutional neural networks. \u003cem\u003eSci Rep\u003c/em\u003e \u003cstrong\u003e9\u003c/strong\u003e, 19854, https://doi.org/10.1038/s41598-019-56395-x (2019).\u003c/li\u003e\n\u003cli\u003ePark, V. Y.\u003cem\u003e et al.\u003c/em\u003e Diagnosis of Thyroid Nodules: Performance of a Deep Learning Convolutional Neural Network Model vs. Radiologists. \u003cem\u003eSci Rep\u003c/em\u003e \u003cstrong\u003e9\u003c/strong\u003e, 17843, https://doi.org/10.1038/s41598-019-54434-1 (2019).\u003c/li\u003e\n\u003cli\u003eGao, L.\u003cem\u003e et al.\u003c/em\u003e Computer-aided system for diagnosing thyroid nodules on ultrasound: A comparison with radiologist-based clinical assessments. \u003cem\u003eHead Neck\u003c/em\u003e \u003cstrong\u003e40\u003c/strong\u003e, 778-783, https://doi.org/10.1002/hed.25049 (2018).\u003c/li\u003e\n\u003cli\u003eJeong, E. Y.\u003cem\u003e et al.\u003c/em\u003e Computer-aided diagnosis system for thyroid nodules on ultrasonography: diagnostic performance and reproducibility based on the experience level of operators. \u003cem\u003eEur Radiol\u003c/em\u003e \u003cstrong\u003e29\u003c/strong\u003e, 1978-1985, https://doi.org/10.1007/s00330-018-5772-9 (2019).\u003c/li\u003e\n\u003cli\u003eJin, Z.\u003cem\u003e et al.\u003c/em\u003e Ultrasound Computer-Aided Diagnosis (CAD) Based on the Thyroid Imaging Reporting and Data System (TI-RADS) to Distinguish Benign from Malignant Thyroid Nodules and the Diagnostic Performance of Radiologists with Different Diagnostic Experience. \u003cem\u003eMed Sci Monit\u003c/em\u003e \u003cstrong\u003e26\u003c/strong\u003e, e918452, https://doi.org/10.12659/MSM.918452 (2020).\u003c/li\u003e\n\u003cli\u003eKo, S. Y.\u003cem\u003e et al.\u003c/em\u003e Deep convolutional neural network for the diagnosis of thyroid nodules on ultrasound. \u003cem\u003eHead Neck\u003c/em\u003e \u003cstrong\u003e41\u003c/strong\u003e, 885-891, https://doi.org/10.1002/hed.25415 (2019).\u003c/li\u003e\n\u003cli\u003eChung, S. R.\u003cem\u003e et al.\u003c/em\u003e Computer-Aided Diagnosis System for the Evaluation of Thyroid Nodules on Ultrasonography: Prospective Non-Inferiority Study according to the Experience Level of Radiologists. \u003cem\u003eKorean J Radiol\u003c/em\u003e \u003cstrong\u003e21\u003c/strong\u003e, 369-376, https://doi.org/10.3348/kjr.2019.0581 (2020).\u003c/li\u003e\n\u003cli\u003eZhu, Y., Sang, Q., Jia, S., Wang, Y. \u0026amp; Deyer, T. Deep neural networks could differentiate Bethesda class III versus class IV/V/VI. \u003cem\u003eAnn Transl Med\u003c/em\u003e \u003cstrong\u003e7\u003c/strong\u003e, 231, https://doi.org/10.21037/atm.2018.07.03 (2019).\u003c/li\u003e\n\u003cli\u003eChoi, Y. J.\u003cem\u003e et al.\u003c/em\u003e A Computer-Aided Diagnosis System Using Artificial Intelligence for the Diagnosis and Characterization of Thyroid Nodules on Ultrasound: Initial Clinical Assessment. \u003cem\u003eThyroid\u003c/em\u003e \u003cstrong\u003e27\u003c/strong\u003e, 546-552, https://doi.org/10.1089/thy.2016.0372 (2017).\u003c/li\u003e\n\u003cli\u003eTessler, F. N., Middleton, W. D. \u0026amp; Grant, E. G. Thyroid Imaging Reporting and Data System (TI-RADS): A User's Guide. \u003cem\u003eRadiology\u003c/em\u003e \u003cstrong\u003e287\u003c/strong\u003e, 29-36, https://doi.org/10.1148/radiol.2017171240 (2018).\u003c/li\u003e\n\u003cli\u003eLandis, J. R. \u0026amp; Koch, G. G. The measurement of observer agreement for categorical data. \u003cem\u003eBiometrics\u003c/em\u003e \u003cstrong\u003e33\u003c/strong\u003e, 159-174, https://doi.org/10.2307/2529310 (1977).\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"convolutional neural network (CNN), deep learning, thyroid nodule, atypia of undetermined significance/follicular lesion of undetermined significance (AUS/FLUS), Bethesda system, Biopsy, Fine-Needle","lastPublishedDoi":"10.21203/rs.3.rs-400151/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-400151/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eTo compare the diagnostic performances of physicians and a deep convolutional neural network (CNN) predicting malignancy with ultrasonography images of thyroid nodules with atypia of undetermined significance (AUS)/follicular lesion of undetermined significance (FLUS) results on fine-needle aspiration (FNA).\u003cem\u003e \u003c/em\u003eThis study included 202 patients with 202 nodules ≥ 1cm AUS/FLUS on FNA, and underwent surgery in one of 3 different institutions. Diagnostic performances were compared between 8 physicians (4 radiologists, 4 endocrinologists) with varying experience levels and CNN, and AUS/FLUS subgroups were analyzed. Interobserver variability was assessed among the 8 physicians.\u003cem\u003e \u003c/em\u003eOf the 202 nodules, 158 were AUS, and 44 were FLUS; 86 were benign, and 116 were malignant. The area under the curves (AUCs) of the 8 physicians and CNN were 0.680-0.722 and 0.666, without significant differences (\u003cem\u003eP\u003c/em\u003e \u0026gt; 0.05). In the subgroup analysis, the AUCs for the 8 physicians and CNN were 0.657–0.768 and 0.652 for AUS, 0.469-0.674 and 0.622 for FLUS. Interobserver agreements were moderate (k=0.543), substantial (k=0.652), and moderate (k=0.455) among the 8 physicians, 4 radiologists, and 4 endocrinologists.\u003cem\u003e \u003c/em\u003eFor thyroid nodules with AUS/FLUS cytology, the diagnostic performance of CNN to differentiate malignancy with US images was comparable to that of physicians with variable experience levels.\u003c/p\u003e","manuscriptTitle":"Using the Deep Convolutional Neural Network to Evaluate Thyroid Nodules With Atypia of Undetermined Significance/follicular Lesion of Undetermined Significance Cytology: Multicenter Study","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2021-04-16 22:03:14","doi":"10.21203/rs.3.rs-400151/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Major revision","date":"2021-08-17T09:11:03+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2021-08-03T03:28:49+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"99dea785-2ede-4e27-b71b-59de21f8c441","date":"2021-07-29T07:29:26+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2021-07-19T02:42:58+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2021-07-16T16:19:59+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2021-04-15T11:32:47+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2021-04-15T10:01:07+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2021-04-07T01:39:51+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"9aef3342-6674-4fc9-ab24-5ed7f6af2f96","owner":[],"postedDate":"April 16th, 2021","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":3692894,"name":"Cancer Biology"},{"id":3692895,"name":"Oncology"}],"tags":[],"updatedAt":"2021-09-24T04:29:06+00:00","versionOfRecord":[],"versionCreatedAt":"2021-04-16 22:03:14","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-400151","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-400151","identity":"rs-400151","version":["v1"]},"buildId":"re_ckhLnmML6MCF96OHNJ","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}