An automatic severity grading system using confocal laser endomicroscopy to evaluate inflammatory activity of ulcerative colitis: a prospective study.

This study was conducted at Qilu Hospital of Shandong University (Jinan, China) using offline video material of UC patients from the pCLE database. We developed and validated ASGS with retrospective data and tested the performance of it with a prospectively cohort. There was no risk to the subjects in the retrospective study, so we obtained approval from the research ethics committee of Qilu Hospital of Shandong University to waive informed consent for the retrospective study. In the prospective cohort, we explained the nature, purpose, and potential risks of this study to the patients and acquired their written informed consent. The intravenously injected contrast agent fluorescein sodium is safe and approved for clinical use. The research ethics committee of Qilu Hospital of Shandong University approved the study protocol (ethics approval number: KYLL-202111-228) and completed registration in the Chinese Clinical Trial Registry (Registration number: ChiCTR2200056340). The research was conducted in accordance with the revised Declaration of Helsinki. All consecutive UC patients who completed pCLE examinations were included in the retrospective study. In the prospective study, inclusion criteria were age ≥ 18, previous diagnosis of UC, and completion of pCLE with videos record. Exclusion criteria were allergy to sodium fluorescein, severe cardiopulmonary disease, shock, gastrointestinal perforation, acute gastrointestinal bleeding, significant thoracoabdominal aortic aneurysm and stroke, peritonitis, fulminant colitis, unwillingness to participate, or inability to provide informed consent. In this study, the endoscopic device we used was the pCLE (Cellvizio ® , Paris, France). All patients received bowel preparations with polyethylene glycol and were administered intravenously with 1 ml of 2% sodium fluorescein for allergy testing preoperatively. On successful intubation into the terminal ileum, the endoscopy nurse intravenously injected 5 ml fluorescein sodium (0.1 ml/kg, 10%). Then, the pCLE probe was threaded through the operating channel of the endoscopy and positioned onto the colonic mucosa. Each segment from the cecum to the rectum was sequentially observed by WLE and pCLE, respectively, choosing the most severe part as the targeted mucosa, recorded the result with the Mayo Endoscopic Score (MES) and the UC Endoscopic Index of Severity (UCEIS) completed under WLE. PCLE can dynamically observe the targeted mucosa of six colorectal sites (cecum, ascending, transverse, descending, sigmoid colon, and rectum) in real-time at a scan rate of 12 frames per second, allowing the operator to easily control the confocal imaging plane position. All pCLE videos of each patient were stored on an external hard drive. From each patient, 2 samples for histopathology were obtained at the highest degree of inflammation site (in remission patients biopsied at rectum colon) and suspicious neoplasm by WLE and pCLE observations, respectively. Each biopsy sample had a registration number with the corresponding pCLE video sequence. All pCLE procedures were performed by three experienced endomicroscopists (J, Guo, GQ, Liu and Z, Li) who had completed more than 500 pCLE procedures before this study. All pCLE videos were transformed into single-frame images by the Cellvizio Viewer tool and de-duplicated the identical images by Compare software. pCLE images with typical features of normal colorectal mucosa, inflammation with UC, and unqualified images (images with multiple artifacts, mucosal blur, motion blur, too dark or too bright vision) (examples of unqualified images are shown in Fig.  2 ) were collected and separately documented. The images were labeled according to the CLE four grades classification criteria for UC inflammation, and then sorted into the two grades classification. The retrospective images were assigned (8:1:1) to the training and validation sets for developing the automatic dual grading system and the testing set for assessing grading ability. Notably, images in the training, validation, and testing sets were from different patients. Fig. 2 Examples of unqualified images. Examples of unqualified images. From May 1, 2022, to October 25, 2022, we prospectively collected pCLE videos of UC patients and intercepted meaningful segments. The most severe grade of inflammation as judged by ASGS in each video segment was used as the final diagnosis. Prospectively acquired images and videos were used for prospective testing. All images and videos were selected and marked by two experienced endomicroscopists (J, Guo, and GQ, Liu) and verified by another senior expert endomicroscopist (Z, Li). In the case of any discrepancy, the three endomicroscopists would discuss and attempt to achieve an agreement. Otherwise, the disputed images or videos were discarded. To standardize the image, we used the method of Hough circle detection to cut out the redundant part of each image and retain the region of interest (ROI) (Fig.  3 ). We fixed the input image size of the classification system, and the image was enhanced by automatic enhancement, normalization and geometric transformations (e.g., rotation, flipping). Since multiple images may come from the same video, we divided the training set and testing set by videos to avoid images of the same video being mixed in different sets. Fig. 3 A supporting figure to show how Hough circle detection removes redundant images. To standardize the images, we employed Hough circle detection to locate the endoscopic field of view (FOV). The surrounding redundant regions were then cropped, retaining only the region of interest (ROI). A supporting figure to show how Hough circle detection removes redundant images. To standardize the images, we employed Hough circle detection to locate the endoscopic field of view (FOV). The surrounding redundant regions were then cropped, retaining only the region of interest (ROI). On the NVIDIA Tesla A100 40G development platform, Python 3.7 with Paddle libraries was used for classification system generation, training, and testing. The system of EfficientNet-B4 for the UC severity grading task was pre-trained on the ImageNet dataset to obtain initialization values of weights. EfficientNet has eight structures from B0-B7 according to different widths, depths, and input resolutions of the network and is applied to a wide range of recognition tasks. In this study, EfficientNet-B4 achieved better performance than others (Supplementary Table 1). We trained a convolutional neural network (CNN) of EfficientNet-B4 with RMSProp optimizer and cross-entropy loss function. The best-learned system among 360 epochs was used to extract the weights and biases for UC severity grading. On the NVIDIA Tesla A100 40G development platform, 206 images per second (IPS) are recognized for an image size of 380*380 pixels. In the four grades classification, images were predicted as Grade A or B or C or D or Unqualified; in the two grades classification, images were predicted as Grade A/B or C/D or Unqualified. The final results of UC severity grading for each video depend on the percentage of the number of each predicted grade to the number of all images in the video. The video evaluation rules of ASGS were as follows: firstly, it predicts the grade of each image in the video and eliminates the unqualified images; secondly, it calculates the proportion of the predicted number of Grades D, C, B, A and Grades C/D, A/B in the total images number; thirdly, it determines the grade that exceeded our predefined threshold as the final prediction result of the video. According to the ROC curves of the grading results, the optimal thresholds of Grades B, C, and D are determined as 0.12, 0.26, and 0.30, and the optimal threshold of Grade C/D is determined as 0.44. A sequential computational judgment rule was imbedded in ASGS based on these thresholds. For example, if a video with the proportions of 0.1, 0.05, 0.8, and 0.05 for Grades D, C, B, and A, ASGS will predict it as Grade B because the thresholds of Grades D and C are not reached. The grading threshold is set to avoid the interference of a few wrong predictions in some images to the final diagnostic result of the whole video. In clinical diagnosis, experienced endomicroscopists will also fully consider the comprehensive situation of all image frames in the video to make the final diagnosis. The specific workflow chart is shown in Fig.  4 . Fig. 4 Workflow chart of ASGS to assess the inflammatory activity of UC. Workflow chart of ASGS to assess the inflammatory activity of UC. Biopsy specimens were fixed in 10% formalin, embedded in paraffin, and sections were stained with hematoxylin and eosin for histopathological examination. All samples were scored by an experienced gastrointestinal pathologist (DB, Shi) masked to clinical and endoscopic patient data according to the Geboes Index (GI). The grades are divided into two groups: grades ≤ 3.0 (representing normal or chronic inflammation) and > 3.0 (representing acute inflammation) 27 . Pathology scores were recorded for each prospective UC patient. To further assess the efficacy of ASGS, we validated the CLE classification based on pathology scores at the most inflamed sites on a per UC patient basis. The follow-up period was 12 months. In the prospective study, UC patients with colonoscopic remission (MES ≤ 1 or UCEIS ≤ 1) were followed up every 8 weeks in our department after the end of this endoscopy. Clinical disease activity for patients with UC was determined using the partial Mayo clinical disease activity score (MCS) 28 with clinical remission defined as a partial MCS 1. Additionally, routine laboratory parameters, and current and past medications were assessed. Statistical analysis was performed using MedCalc (version 20.106). Indicators reflecting the performance of ASGS included sensitivity, specificity, accuracy, positive predictive value (PPV), and negative predictive value (NPV). The agreement between ASGS and the endomicroscopist, ASGS and pathology was determined using quadratic weighted kappa (QWK). Meanwhile, the corresponding 95% confidence interval (95% CI) were calculated. The correlation between ASGS and histopathology as well as endoscopic scores (MES and UCEIS) were calculated by the Spearman correlation coefficient, P  < 0.01 was considered statistically significant. Matplotlib (The Matplotlib development team, USA) was used to draw the receiver operating characteristic (ROC) curve and calculate the area under the receiver operating characteristic (AUROC) curve.

Ultimately, 79,443 pCLE images from 215 patients from January 4, 2013 to February 4,2022 were retrospectively collected. After removing duplicate data and images discarded due to discrepancies in annotation, a total of 61,012 images (76.80%) were excluded, leaving 18,431 images to form the retrospective dataset (After removing 60,775 duplicate images, 18,668 images remained. During the image annotation process, 237 images were discarded, with a removal rate of 237/18,668.). The number of images of each class in the four and two grades classification is detailed in Supplementary Table 2. In the prospective test from February 5, 2022 to October 25, 2022, 9,761 images from 19 patients and 263 video clips from another 30 patients (11.49 s per video on average) were included to further evaluate the performance of ASGS (The discard rate for prospective image data during annotation was 192/9953, and none of the videos were discarded during the video annotation process.). Two expert endomicroscopists achieved QWK of 0.9496 in labeling the images (The confusion matrix is presented in Supplementary Table 3). Baseline patient characteristics are shown in Table  1 . Table 1 Baseline characteristics of patients and lesions. Patients, n 215 (retrospective) 49 (prospective) Sex Male, n (%) 125 (58.14%) 25 (51.02%) Age, mean (range), years 46.96 (20–87) 46.47 (18–71) Montreal Classification, n E1 75 14 E2 80 23 E3 60 12 Medical history (years), n 0–5 99 26 5–10 89 11 >10 27 12 MES, n ≤1 76 22 2–3 139 27 UCEIS, n ≤ 1 59 16 2–8 156 33 Baseline characteristics of patients and lesions. The sensitivities of ASGS for predicting Grade A-D were 97.38%, 84.56%, 86.22%, and 97.74%; the specificities were 99.25%, 99.36%, 99.90%, and 97.60%, respectively (Table  2 ). The sensitivities for predicting Grade A/B and Grade C/D were 94.70% and 97.04%; the specificities were 99.32% and 98.82% (Table  3 ). The confusion matrices are presented in Supplementary Tables 4 and 5, respectively. The AUROCs were between 0.9644 and 0.9990 (Fig.  5 ). Table 2 Performance of ASGS in predicting four grades of inflammatory activity with UC (prospective image testing set). Outcome indicators Grading Grade A Grade B Grade C Grade D Unqualified Sensitivity, % (95% CI) 97.38 (96.26–98.24) 84.56 (81.33–87.43) 86.22 (83.75–88.44) 97.74 (96.93–98.38) 97.87 (97.45–98.24) Specificity, % (95% CI) 99.25 (99.04–99.42) 99.36 (99.17–99.51) 99.90 (99.81–99.95) 97.60 (97.24–97.92) 98.42 (98.01–98.78) PPV, % (95% CI) 94.32 (92.87–95.49) 89.09 (86.34–91.35) 98.82 (97.75–99.38) 90.00 (88.66–91.19) 98.74 (98.41–99.00.41.00) NPV, % (95% CI) 99.66 (99.52–99.77) 99.05 (98.85–99.21) 98.67 (98.43–98.87) 99.49 (99.31–99.62) 97.34 (96.83–97.77) Accuracy, % (95% CI) 99.04 (98.82–99.22) 98.49 (98.23–98.73) 98.68 (98.43–98.90) 97.62 (97.30–97.92.30.92) 98.12 (97.83–98.38) Performance of ASGS in predicting four grades of inflammatory activity with UC (prospective image testing set). Sensitivity, % (95% CI) 97.38 (96.26–98.24) 84.56 (81.33–87.43) 86.22 (83.75–88.44) 97.74 (96.93–98.38) 97.87 (97.45–98.24) Specificity, % (95% CI) 99.25 (99.04–99.42) 99.36 (99.17–99.51) 99.90 (99.81–99.95) 97.60 (97.24–97.92) 98.42 (98.01–98.78) PPV, % (95% CI) 94.32 (92.87–95.49) 89.09 (86.34–91.35) 98.82 (97.75–99.38) 90.00 (88.66–91.19) 98.74 (98.41–99.00.41.00) NPV, % (95% CI) 99.66 (99.52–99.77) 99.05 (98.85–99.21) 98.67 (98.43–98.87) 99.49 (99.31–99.62) 97.34 (96.83–97.77) Accuracy, % (95% CI) 99.04 (98.82–99.22) 98.49 (98.23–98.73) 98.68 (98.43–98.90) 97.62 (97.30–97.92.30.92) 98.12 (97.83–98.38) Table 3 Performance of ASGS in predicting two grades of inflammatory activity with UC (prospective image testing set). Outcome indicators Grading Grade A/B Grade C/D Unqualified Sensitivity, % (95% CI) 94.70 (93.51–95.72) 97.04 (96.32–97.66) 98.38 (98.01–98.70) Specificity, % (95% CI) 99.32 (99.12–99.49) 98.82 (98.54–99.06) 97.31 (96.79–97.77) PPV, % (95% CI) 96.66 (95.69–97.41) 96.82 (96.10–97.42.10.42) 97.88 (97.47–98.22) NPV, % (95% CI) 98.90 (98.66–99.10) 98.90 (98.64–99.12) 97.95 (97.49–98.33) Accuracy, % (95% CI) 98.53 (98.27–98.75) 98.34 (98.07–98.58) 97.91 (97.61–98.19) Performance of ASGS in predicting two grades of inflammatory activity with UC (prospective image testing set). Sensitivity, % (95% CI) 94.70 (93.51–95.72) 97.04 (96.32–97.66) 98.38 (98.01–98.70) Specificity, % (95% CI) 99.32 (99.12–99.49) 98.82 (98.54–99.06) 97.31 (96.79–97.77) PPV, % (95% CI) 96.66 (95.69–97.41) 96.82 (96.10–97.42.10.42) 97.88 (97.47–98.22) NPV, % (95% CI) 98.90 (98.66–99.10) 98.90 (98.64–99.12) 97.95 (97.49–98.33) Accuracy, % (95% CI) 98.53 (98.27–98.75) 98.34 (98.07–98.58) 97.91 (97.61–98.19) Fig. 5 ROC Curve of prospective image testing set. ROC Curve of prospective image testing set. The sensitivities predicted as four grades were 75.56%, 94.00%, 92.31%, and 95.24%, respectively. The specificities and accuracies of the four grades classification all reached more than 93% (Table  4 ). The sensitivity and specificity for predicting grade A/B were 100%, 98.31% and for predicting grade C/D were 98.31%, 100% (Table  5 ). The confusion matrices are presented in Supplementary Tables 6 and 7 , respectively. The AUROCs are shown in Fig.  6 . Table 4 Performance of ASGS in predicting four grades of inflammatory activity with UC (prospective video testing set). Outcome indicators Grading Grade A Grade B Grade C Grade D Sensitivity, % (95% CI) 75.56 (60.46–87.12) 94.00 (87.40–97.77.40.77) 92.31 (63.97–99.81) 95.24 (89.24–98.44) Specificity, % (95% CI) 98.17 (95.37–99.50) 93.25 (88.25–96.58) 98.00 (95.40–99.35.40.35) 98.10 (94.55–99.61) PPV, % (95% CI) 89.47 (76.04–95.79) 89.52 (82.82–93.81) 70.59 (49.84–85.29) 97.09 (91.57–99.03) NPV, % (95% CI) 95.11 (92.09–97.02) 96.20 (92.09–98.22) 99.59 (97.39–99.94) 96.88 (92.95–98.65) Accuracy, % (95% CI) 94.30 (90.77–96.77) 93.54 (89.85–96.19) 97.72 (95.10–99.16.10.16) 96.96 (94.10–98.68.10.68) Performance of ASGS in predicting four grades of inflammatory activity with UC (prospective video testing set). Sensitivity, % (95% CI) 75.56 (60.46–87.12) 94.00 (87.40–97.77.40.77) 92.31 (63.97–99.81) 95.24 (89.24–98.44) Specificity, % (95% CI) 98.17 (95.37–99.50) 93.25 (88.25–96.58) 98.00 (95.40–99.35.40.35) 98.10 (94.55–99.61) PPV, % (95% CI) 89.47 (76.04–95.79) 89.52 (82.82–93.81) 70.59 (49.84–85.29) 97.09 (91.57–99.03) NPV, % (95% CI) 95.11 (92.09–97.02) 96.20 (92.09–98.22) 99.59 (97.39–99.94) 96.88 (92.95–98.65) Accuracy, % (95% CI) 94.30 (90.77–96.77) 93.54 (89.85–96.19) 97.72 (95.10–99.16.10.16) 96.96 (94.10–98.68.10.68) Table 5 Performance of ASGS in predicting two grades of inflammatory activity with UC (prospective video testing set). Outcome indicators Grading Grade A/B Grade C/D Sensitivity, % (95% CI) 100.00 (97.49–100.00) 98.31 (94.01–99.79) Specificity, % (95% CI) 98.31 (94.01–99.79) 100.00 (97.49–100.00) PPV, % (95% CI) 98.64 (94.83–99.65) 100.00 (-) NPV, % (95% CI) 100.000 (-) 98.64 (94.83–99.65) Accuracy, % (95% CI) 99.24 (97.28–99.91) 99.24 (97.28–99.91) Performance of ASGS in predicting two grades of inflammatory activity with UC (prospective video testing set). Sensitivity, % (95% CI) 100.00 (97.49–100.00) 98.31 (94.01–99.79) Specificity, % (95% CI) 98.31 (94.01–99.79) 100.00 (97.49–100.00) PPV, % (95% CI) 98.64 (94.83–99.65) 100.00 (-) NPV, % (95% CI) 100.000 (-) 98.64 (94.83–99.65) Accuracy, % (95% CI) 99.24 (97.28–99.91) 99.24 (97.28–99.91) Fig. 6 ROC Curve of prospective video testing set. ROC Curve of prospective video testing set. We performed CLE grading, MES and UCEIS independently for each included UC patient and then compared the results between CLE grading with MES and UCEIS, showing that 33 patients with MES ≤ 1 and 12 patients with UCEIS ≤ 1 were found to have acute inflammation (Grade C/D) during pCLE (Tables  6 and 7 ). Table 6 CLE grading of UC inflammation in relation to MES. MES CLE grading Patients, n Grade A/B Grade C/D ≤1 65 33 98 2–3 0 166 166 Patients, n 65 199 264 CLE grading of UC inflammation in relation to MES. Table 7 CLE grading of UC inflammation in relation to UCEIS. UCEIS CLE grading Patients, n Grade A/B Grade C/D ≤1 63 12 75 2–8 2 187 189 Patients, n 65 199 264 CLE grading of UC inflammation in relation to UCEIS. We recorded the prediction results of ASGS on the prospective testing data, followed by independent evaluation on the CLE grading by three experienced endomicroscopists. Then we compared the agreement between ASGS and endomicroscopists on the UC inflammation grading, and found that the QWKs all reached above 0.950 (Table  8 ). Table 8 The agreement between ASGS and endomicroscopists on UC inflammation grading (QWK). QWK Standard error 95% CI Four grades (prospective image testing set) 0.971 0.00241 0.966–0.976 Two grades (prospective image testing set) 0.953 0.00356 0.946–0.960 Four grades (prospective video testing set) 0.958 0.01027 0.938–0.978 Two grades (prospective video testing set) 0.985 0.01084 0.963–1.000.963.000 The agreement between ASGS and endomicroscopists on UC inflammation grading (QWK). Four grades (prospective image testing set) Two grades (prospective image testing set) Four grades (prospective video testing set) Two grades (prospective video testing set) Ten of the 11 UC patients with histologically confirmed normal or chronic inflammation were diagnosed as Grade A/B by ASGS, and 35 of the 38 patients with histologically confirmed acute inflammation were diagnosed as Grade C/D by ASGS (The confusion matrix is presented in Supplementary Table 8). Pathologic validation of inactive inflammation, the sensitivity and specificity of ASGS were 90.91% and 92.11%; pathologic validation of active inflammation, the sensitivity and specificity were 92.11% and 90.91% (Table  9 ). The agreement between ASGS and conventional histology was excellent (QWK = 0.780). Table 9 Pathologic validation of ASGS for predicting two grades of inflammatory activity with UC. Outcome indicators ASGS Grade A/B Grade C/D Sensitivity, % (95% CI) 90.91 (58.72–99.77) 92.11 (78.62–98.34) Specificity, % (95% CI) 92.11 (78.62–98.34) 90.91 (58.72–99.77) PPV, % (95% CI) 76.92 (52.55–90.94) 97.22 (84.35–99.56) NPV, % (95% CI) 97.22 (84.35–99.56) 76.92 (52.55–90.94) Accuracy, % (95% CI) 91.84 (80.40–97.73.40.73) 91.84 (80.40–97.73.40.73) Pathologic validation of ASGS for predicting two grades of inflammatory activity with UC. Sensitivity, % (95% CI) 90.91 (58.72–99.77) 92.11 (78.62–98.34) Specificity, % (95% CI) 92.11 (78.62–98.34) 90.91 (58.72–99.77) PPV, % (95% CI) 76.92 (52.55–90.94) 97.22 (84.35–99.56) NPV, % (95% CI) 97.22 (84.35–99.56) 76.92 (52.55–90.94) Accuracy, % (95% CI) 91.84 (80.40–97.73.40.73) 91.84 (80.40–97.73.40.73) We recorded the ASGS, histopathology, and endoscopic scores (MES and UCEIS) results of 49 UC patients prospectively included in the study. The confusion matrices are shown in Supplementary Tables 8, 9, and 10. ASGS prediction results were significantly correlated with histopathology and endoscopic scores, with Spearman correlation coefficients of r  = 0.785, 0.694, and 0.804, respectively, and P values all < 0.001. Of the 49 UC patients recruited for the prospective study, one patient failed to return for follow-up due to novel coronavirus infection, and we then followed up patients who were considered to be in remission under the MES, UCEIS, CLE classification, and GI grading, respectively. During the follow-up period, 8 of 21 UC patients with MES ≤ 1 relapsed (relapse rate: 38.1%), 5 of 15 UC patients with UCEIS ≤ 1 score had relapsed (relapse rate: 33.3%). Among the 13 patients with CLE inflammation graded A/B, 2 relapsed (relapse rate: 15.4%) and 2 relapsed in the 11 patients with pathologies GI grades ≤ 3.0 (relapse rate: 18.2%). Demographic and clinical data of patients are shown in Table  10 . Table 10 Demographic and clinical data of patients (MES ≤ 1). Numbers Recruited 49 Excluded 28 (27 for MES of 2–3, 1 failed to be followed up due to novel coronavirus infection and personal reasons) Included 21 Sex 13 males, 8 females Average age (range) 43 (18–69) years Average disease duration (range) 36.5 (3–72) months Maintenance therapy 13 with Vedolizumab (300 mg, Per 8 weeks), 7 with Infliximab (5–10 mg/Kg, Per 8 weeks), 1 with mesalazine (3 g daily). Demographic and clinical data of patients (MES ≤ 1).

In this study, we first developed ASGS, an automatic severity grading system, which achieved a good diagnostic ability to assess the degree of inflammation activity in UC patients and validated the system in real clinical practice. The ASGS showed a good agreement with experienced endoscopists and pathology, thus offered the possibility to solve the difficulty of interpreting endomicroscopic images, improve the efficacy, reduce histological biopsies and might serve as an accurate parameter to predict the prognosis of UC patients. With the recently development of deep learning algorithm, many researchers are working on applying AI systems to various fields of gastrointestinal, including endoscopic diagnosis of colorectal polyps 29 , colorectal tumors 30 , gastrointestinal ulcer 31 , 32 and UC 33 . In a study by Lucille Quénéhervé et al. 34 , they constructed a computer-assisted CLE system for the diagnosis of inflammatory bowel disease and for the differentiation of UC and Crohn’s disease. However, whether AI could help severity grading by using CLE hasn’t been reported. Based on our previously published criteria 4 , 10 , we developed the ASGS for classification of different grade of UC inflammation and prospectively evaluated its efficacy in clinical practice. Our results showed that ASGS had satisfactory diagnostic performance in both image and video testing. The four-grading results of the prospective image test revealed excellent specificity, accuracy, NPV, and AUROC. The specificities were all above 97.60% and the accuracies were all above 97.62%. Of note, the AUROCs of four classifications in prospective image test were all above 0.96, indicating that the overall performance of ASGS is outstanding. To facilitate real-time clinical applications, we further tested the diagnostic performance of ASGS in the prospective video set. Each video contains substandard images caused by multiple factors (e.g., pCLE images move so fast that they produce artifacts, mucosal blur due to insufficient bowel preparation, etc.), which may lead to misdiagnosis if not recognized by ASGS. These artifacts can often occur in real clinical practice, so we kept these images in the video to present the true diagnostic ability of ASGS. The results show that ASGS predicts unqualified images with a high sensitivity and accuracy of over 97%. All the metrics of video testing are satisfactory, especially the specificity, accuracy, NPV, and AUROC, and overall accuracy reaches up to 91.25%. However, PPV of Grade C was low, because a portion of Grade D was misidentified as Grade C. In clinical work, due to the stringency of the diagnostic criteria for Grade C (CLE grading method) of UC inflammation, the number of video samples diagnosed as this grade is small, so a few samples misidentification can have a significant impact on the diagnostic results for Grade C, further improvement would require a larger sample of data. To align with the pathologic diagnostic dichotomy of chronic and acute inflammation, our previous studies also performed a binary classification to assess active inflammation and remission in UC. Therefore, we also trained ASGS on the two grades classification. The results confirmed a superior diagnostic ability for both image and video testings, as compared to the four-classification results. The sensitivity and PPV of the Grade C/D were above 96%, indicating ASGS has a high capacity to detect and correctly identify acute inflammation, allowing for immediate therapeutic intervention or adjustment of treatment regimens for patients with UC. The specificity and NPV were above 98%, showing that ASGS may accurately identify the non-acute inflammatory state of UC. In future studies, the two-classification method may be simpler and easier to learn, and will have a broad application in the assessment of disease activity and prediction of disease prognosis. We know that histological healing is associated with better disease outcomes compared to clinical remission and/or endoscopic remission 35 – 37 . However, only two of the 26 histopathology scores for UC are validated 38 and precise targeted biopsies are difficult to perform, thus limiting determination and histology-based decision-making in clinical practice. More interestingly, a recent study found that the predictive capabilities of the mucosal barrier assessed by CLE for UC inflammation might well exceed that of histology 39 . Therefore, in this study, we used the CLE grading method of UC as the gold standard for establishing ASGS. Meanwhile, in the prospective study, we scored the pathology at the most inflammatory sites per patient to evaluate the efficacy of ASGS, the binary classification ASGS showed a good correlation with pathology, the QWK was 0.780. For UC patients who were in remission under WLE, we validated them under pCLE to determine whether these patients had achieved microscopic healing. We performed two endoscopic scores for UC under WLE before the pCLE examination. Endoscopic remission was defined as Mayo endoscopic score ≤ 1, or Ulcerative Colitis Endoscopic Index of Severity (UCEIS) ≤ 1. Results found that a certain percentage (33/98, 12/75) of UC patients were considered to be in endoscopic remission using only white-light mode colonoscopy, but with evidence of acute inflammation (Grade C/D) when evaluated with pCLE. Consistent with previous studies, our results further proved that pCLE is superior to WLE for UC assessment. As a secondary end point, we monitored the prospective remission uc patients for at least 1 year to assess the potential performance of ASGS in predicting long-term disease behavior. This study used established clinical score MCS rather than biochemical markers or therapeutic regimens to define clinical remission 40 . As expected, patients with confocal endomicroscopic healing were less likely to experience relapse compared with patients with endoscopic remission (MES and UCEIS scores), and with similar results with pathology. A larger number of prospective patients were needed for further verification. It reaffirmed that ASGS has the potential to serve as an accurate parameter to predict prognosis. The QWKs of both the image and video testing sets reached more than 0.950, indicating a solid agreement between ASGS and the experienced endomicroscopists in grading the inflammatory activity of UC. The quadratic weighting ensures that differences of more than 1 step level get penalized more severely. Our study has several advantages. First, the large sample size of the training set enables ASGS to thoroughly learn the features of pCLE images of UC and ensures its generality. Second, the prospective testing set included pCLE images and videos from six colorectal sites of each UC patient and was independent of ASGS training and validation throughout the study, which would increase the variability and heterogeneity of the data. Nonetheless, the excellent results of the prospective testing show the good applicability of our ASGS to UC patients with different states (remission and active) and the whole colorectal segment. This allows for the assessment of not only disease severity but also extent of inflammation. Finally, many pCLE images acquired at the same lesion in the colon are highly similar or even identical, which may lead to overfitting the neural network in the training set. To avoid such bias, we eliminated duplicate image data using Compare software before labelling. Although the number of images was reduced, this processing could decrease the selection bias and generalization error, making the study more objective and standard. This study also has several limitations. First, this is a single-center diagnostic study, and the ASGS needs to be further validated by multi-center randomized controlled trials. Second, probe-based CLE imaging is currently not widely spread and is mainly used at expert centers, which is more costly and time-consuming, limiting application. With the AI development, a reduction in the times needed for CLE image analysis and a better prognostic indicator might facilitate the transferability of this technology to less experienced centers. Third, this is an in vitro study of offline images that were obtained by experienced endoscopists who are capable of obtaining highquality images. However, the AI tool is supposed to be used by non-experts in this theory. If the image quality is poor, the interpretation of the AI may be different, so in the follow-up study, we worked on upgrading to a real-time automatic graded diagnostic system of UC and applying it in real clinical setting. In conclusion, we developed the first AI-assisted pCLE system of ASGS, which could grade the inflammatory activity of UC and has been prospectively validated to have good capability in both the two and four grades-classifications. The system has the potential to reduce the number of biopsies, offer the possibility to predict prognosis and lead to better management of UC patients.

Ulcerative colitis (UC) is a chronic non-specific inflammatory disease of the intestine that is difficult to cure 1 . Several studies have demonstrated that the higher colorectal cancer incidence and recurrence rates of UC are associated with its persistent inflammatory activity 2 – 4 . White-light endoscopy (WLE) is an important tool for establishing the diagnosis, discriminating, and assessing the extent of lesions and inflammatory activity of UC. However, it is not an accurate predictor in some cases, especially for the mucosa that macroscopically appears to be in remission whereas histology can still show different degrees of inflammation 5 , 6 . Advanced endoscopic imaging technologies such as magnifying chromoendoscopy (MC), narrow-band imaging (NBI) and i-Scan endoscopy have been used to evaluate UC inflammation with improved diagnostic performance compared to conventional endoscopy 7 , 8 . However, these are limited to observing the mucosal surface of the gastrointestinal tract. As an “optical biopsy” technology, confocal laser endomicroscopy (CLE) allows real-time in vivo visualization of the cellular and subcellular structure of the mucosa and detection of microscopic inflammatory changes in a macroscopically non-inflamed or normal-appearing mucosa 9 . Considering the significant advantages of CLE, our previous study has successfully proposed a CLE grading system for UC inflammation combining simplified colonic crypt architecture and fluorescein leakage, which consists of a four grades classification (Grade A-D) and a simplified two grades classification (Grade A/B and Grade C/D) systems (Fig. 1 ) 10 . Grade A/B represents normal or chronic inflammation, and Grade C/D indicates acute inflammation. This CLE grading system was proven to correlate well with the histological assessment 10 . Followed by multiple studies also shown that CLE provides a reliable assessment of the inflammation degree of UC 10 – 13 , thus significantly reducing the number of biopsies 14 – 16 . Fig. 1 CLE images representing four types of crypt architecture and fluorescein leakage. A , regular arrangement of crypts surrounded by regular microvessels. Lumen of crypts shows as dark spots without fluorescein leakage. B , normal size and irregular arrangement of crypts. Lumen of crypts are still free of fluorescein, with fluorescein leakage into spaces among epithelial cells but not in the lumen (arrow). Sporadic crypt fusion could be seen. Spaces between some crypts are enlarged. C , some of the cryptal lumens are dilated and bright (arrow), which indicates fluorescein leakage, but the epithelium was still intact. D , most normal crypts were replaced by diffuse necrosis, and the remaining crypts are destroyed, crypt abscess can be seen (triangle). CLE images representing four types of crypt architecture and fluorescein leakage. A , regular arrangement of crypts surrounded by regular microvessels. Lumen of crypts shows as dark spots without fluorescein leakage. B , normal size and irregular arrangement of crypts. Lumen of crypts are still free of fluorescein, with fluorescein leakage into spaces among epithelial cells but not in the lumen (arrow). Sporadic crypt fusion could be seen. Spaces between some crypts are enlarged. C , some of the cryptal lumens are dilated and bright (arrow), which indicates fluorescein leakage, but the epithelium was still intact. D , most normal crypts were replaced by diffuse necrosis, and the remaining crypts are destroyed, crypt abscess can be seen (triangle). CLE systems include the probe-based CLE (pCLE) system and endoscope-based CLE (eCLE) system. Its clinical applicability and generalizability have been impacted due to several issues 17 . For example, the subjective, time-consuming process of pCLE image interpretation. However, advances in artificial intelligence (AI) provide means to address the inherent subjectivity and cumbersomeness in human image interpretation 18 . Since 2017, several reports of AI have achieved comparable performance to human experts on medical image analysis tasks 19 – 22 . There have already been AI systems for assessing UC inflammation activity based on WLE, chromoendoscopy, and endocytoscopy 23 – 26 , but pCLE-based AI system is still yet investigated. Herein, we aimed to develop an automatic severity grading system (ASGS) that can accurately grade the inflammatory activity of UC during pCLE examination.

Below is the link to the electronic supplementary material. Supplementary Material 1 Supplementary Material 1