{"paper_id":"44f45fff-0048-4b80-beb6-fe8d0b2ac64b","body_text":"Development of a Multimodal Deep Learning Model for Predicting Microsatellite Instability in Colorectal Cancer by Integrating Histopathological Images and Clinical Data | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Development of a Multimodal Deep Learning Model for Predicting Microsatellite Instability in Colorectal Cancer by Integrating Histopathological Images and Clinical Data Binsheng He, Wenjing Qiu, Bing Wang, Jingya Yang, Jinyang Mao, and 5 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4200523/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Microsatellite instability (MSI) arises from defective DNA mismatch repair (MMR) systems and is prevalent in various cancer types. MSI is classified as MSI-High (MSI-H), MSI-Low (MSI-L), or Microsatellite Stable (MSS), with the latter two occasionally combined into a single designation called MSI-L/MSS. Identifying the MSI status (i.e., MSI-H vs. MSI-L/MSS) in colorectal cancer (CRC) is critical for guiding immunotherapy and assessing prognosis. Conventional molecular tests for MSI are expensive, time-consuming, and limited by experimental conditions. Advancements in MSI detection have been made using deep learning methods with histopathological images, yet efforts to improve MSI detection's predictive accuracy by integrating histopathological images and clinical data remain limited. This study initially analyzed clinical information variation between the MSI-H and MSI-L/MSS groups, discovering significant differences in cancer stages N and M. Subsequently, texture features were extracted using the Gray-level co-occurrence matrix (GLCM) from both groups, disclosing noteworthy disparities in mean feature information. Finally, a multimodal compact bilinear pool (MCB) was employed to merge histopathological images with clinical data. By applying this analysis framework to the cancer genome atlas (TCGA) CRC data, a prediction area under the curve (AUC) of 0.833 was achieved through 5-fold cross-validation in predicting MSI status. The results demonstrated higher accuracy in determining MSI compared to existing unimodal MSI prediction methods and other contemporary techniques. Additionally, significant regions in whole-slide images (WSI) for determining MSI labels were visualized. To summarize, this study presents an accurate multimodal deep learning model for predicting microsatellite instability in colorectal cancer by integrating histopathological images and clinical data, together with a method to visualize important regions in WSI to determine MSI status. Biological sciences/Cancer Biological sciences/Computational biology and bioinformatics microsatellite instability colorectal cancer deep learning multimodal fusion convolutional neural network Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 1. Introduction Colorectal cancer (CRC), a highly heterogeneous disease driven by a range of genetic and epigenetic events [ 1 – 3 ], is the third-most common cancer and the second-largest cancer-related cause of death worldwide [ 4 , 5 ]. Microsatellite instability (MSI) is the phenomenon of normal microsatellite length change due to the deletion or insertion of repeat bases compared to normal tissue cells [ 6 ]. It is characterized by the generalized instability of short tandem repeat DNA sequences known as the microsatellite [ 7 ]. MSI is one of the three major mechanisms for CRC carcinogenesis, with MSI occurring in about 15% of CRC cases [ 7 ]. MSI has attracted much attention because of its value in diagnosis, treatment reaction, and prognosis for CRC [ 7 – 9 ]. There is some evidence to support the use of MSI testing before making clinical treatment decisions. First, MSI testing is recommended to assist in the diagnosis of Lynch syndrome, the most common inherited colon cancer syndrome associated with germline mutations in the DNA mismatch repair (MMR) gene (MLH1, MSH2, MSH6, or PMS2) [ 10 ]. The MSI status helps to identify families with the syndrome to minimize family members' and relatives’ risk of developing the disease. Second, MSI is one of the key factors affecting the prognosis of CRC, especially in the early stage [ 11 , 12 ]. Patients with stage II CRC with high MSI (MSI-H) / MMR deficiency (d MMR) generally have better outcomes than patients with microsatellite stability (MSS) and low MSI (MSI-L) / MMR (p MMR) [ 11 ]. Third, MSI identifies a unique subset of patients with a better stage-adjusted prognosis, for whom standard fluorouracil chemotherapy is contraindicated, and oxaliplatin and immune checkpoint inhibitors may be particularly beneficial [ 7 , 13 , 14 ]. Fluoropyrimidine (5-FU or capecitabine) is the mainstay of the CRC chemotherapy strategy and plays an important role in both neoadjuvant and translational therapies [ 15 – 18 ]. However, patients with MSI status are generally resistant to 5-FU-based chemotherapeutic agents [ 19 ]. Immunotherapy is an emerging and promising treatment for CRC because MSI tumors have a large number of mutant neoantigens, which makes them sensitive to the immune checkpoint inhibitor [ 20 ]. Therefore, MSI status is critical for selecting CRC treatment and evaluating the response to treatment [ 21 ]. In general, the identification of MSI depends on immunohistochemistry (IHC) staining, polymerase chain reaction (PCR) [ 22 ], but not every patient can avail these options as a result of their high costs and regional limitations. In contrast, histopathological images are routinely available for almost all cancer patients and are inexpensive and non-invasive [ 23 – 25 ]. Given the rapid development of deep learning techniques, researchers have explored the identification of molecular biomarkers through hematoxylin and eosin (H&E) image [ 26 , 27 ]. Studies have shown that the morphological features of H&E histopathological images have an important effect on the prognosis of various malignancies [ 28 – 31 ]. With the continuous development of computer technology and full-slide imaging (Whole Slide Imaging (WSI)), computer-assisted diagnosis and prognostic prediction based on images such as H&E staining tissues have received increasing attention [ 32 , 33 ]. These pathological images not only contain pathological features such as tumor morphology, growth, and distribution, but also have the advantages of radio-mics, such as high speed, non-invasiveness, and low cost. Therefore, more research has been carried out in histopathological-assisted diagnosis by deep learning methods in recent years. Today, it can already help doctors improve the accuracy and speed of diagnostic work and alleviate the problem of insufficient pathological diagnostic resources. Currently, some contributions have been made to MSI prediction of pathological images of CRC [ 34 ]. Kather et al. used ResNet18 to predict histopathological sections of CRC (FFPE) on the Cancer Genome Atlas (TCGA), yielding an AUC of 0.77. The AUC on the DAHCS Colorectal (FFPE) dataset was 0.84 [ 35 ]. Ke et al. used a multistage convolutional neural network (CNN) knowledge distillation model to predict TCGA colorectal cancer (FFPE) with an AUC above 0.802 [ 36 ]. Thus, it has been verified that CNN is a powerful algorithm that can directly process biomedical images. The defect in the subjective bias during the extraction of histological features of H&E staining images has been overcome. However, the above research methods are only based on the qualitative information of histopathological images. In this work, we proposed a predictive framework based on pathological images and clinical information. First, we downloaded the full-slide images (WSIs) and the corresponding clinical features of 360 CRC cases (TCGA-CRC-DX, FFPE tissue) from the TGCA. These WSIs were labeled and then divided into 512 \\(\\times\\) 512 pixels of the patch. Tiles were treated with image preprocessing steps, and preprocessing of the clinical data was done. Second, the data were extracted and evaluated. The features of the image data were extracted using a deep CNN ResNet18 model, and random forest in machine learning was screened for the features of clinical information. A new multimodal fusion classification model was constructed based on the extracted features. Finally, the results of our model predictions were discussed. 2. Materials and Methods 2.1 Dataset TCGA is an open, large-scale cancer genomics database containing a large number of primary cancers and their pathological images. It is used in digital forms to match normal samples of multiple cancer types. It provides researchers with public datasets that they can search, view, and download to help improve diagnostics and standards of treatment, and ultimately prevent cancer. We downloaded WSIs corresponding to clinical data from H&E stained sections of 360 CRC cases (TCGA-CRC-DX, FFPE tissue) from the TCGA database ( https://portal.gdc.cancer.gov/repository/ ). All CRC slide images were stored in SVS format and adjusted to 0.5 um per pixel at the same magnification (40x). Among the pathological tissue images of CRC, the number of MSIs was 65, and the number of MSSs was 295. In this process, we classified MSI-L as the MSS class. Moreover, the histological image of H&E staining for MSI-H was labeled as 1, and that for MSS was labeled as 0. We combined histological images of H&E staining with clinical data for follow-up studies. 2.2 Preprocessing of the image data 2.2.1 The patching and filtering of the images As the WSIs were too large to be used directly, the representative regions of interest (ROI) in each slide were indicated. The black dashed line in Fig. 3 (a) shows the boundaries of the annotated tumor region. Second, it was divided into small blocks of 512 \\(\\times\\) 512 px. As shown in Fig. 3 (b), the less informative slides (e.g., more than 30% of the filtered blanks were covered by the background) were then discarded [ 37 ]. 2.2.2 Color-normalized image data The error of the manual production process and the difference between stains and scanners will produce a color difference between digital sections, which will cause errors in the subsequent analysis work. Color standardization technology is generally used to eliminate color differences between slices. Therefore, we performed color normalization using the Macenko ' s method [ 38 ] in the Tia toolbox software package. The Tia toolbox is a multipurpose name for: 1) a computer program; 2) the Python package for the related program that has been created in the TIA Center to help people start using digital pathology; 3) a repository; and 4) a virtual environment. The main idea of the Macenko ' s method is to map images of the RGB space to the color space of the stain using the color separation method, and then standardize staining using the color deconvolution technique [ 38 , 39 ]. 2.3 Ordering of clinical features based on random forest Feature selection is important for interpretation and prediction, especially for avoiding the high-dimensional curse. The data were first cleaned before feature selection, including removing columns with a null value greater than 25% and filling the data, where the typed variables were numbered, and the continuous data were filled with the average. We selected the data using random forest feature importance. Random forest is a combinatorial classifier model composed of decision tree classifier sets. It belongs to an integrated learning model [ 40 ]. Random forest is not only a representative classifier in machine learning, but also can be used to estimate the importance of variables in the model. The importance of features measured by the mean reduced Gini coefficient is: \\(Gini(t)=1 - \\sum\\limits_{{m=1}}^{M} {\\mathop {^{{p(m/t)}}}\\nolimits^{2} }\\) where \\(M\\) represents the total number of classes of the target variable and \\(P\\left(m∕t\\right)\\) represents the conditional probability that the target variable is of class \\(m\\) at the node \\(t\\) . According to the formula, \\(Gini\\) is calculated. Finally, the larger the value of \\(Gini\\left(t\\right)\\) is, the more important \\(m\\) is. 2.4 Image features extraction based on ResNet18 A residual network is a CNN proposed by four scholars from Microsoft Research, which achieved image classification and object recognition in ImageNet (ILSVRC). Residual networks are characterized by easy optimization and the ability to improve accuracy by increasing depth considerably. Their internal residual block uses jump connections, alleviating the gradient vanishing problem caused by increasing depth in a deep neural network[ 41 ]. Residual networks have had many applications in image feature classification, lesion segmentation, and cell segmentation [ 42 – 44 ]. Figure 1 (a) shows the workflow chart of the ResNet18 network, and (b) is the structural chart of the residual block. The architecture of ResNet18 is divided into four stages. Every Resnet architecture performs the initial convolution and max-pooling using 7 \\(\\times\\) 7 and 3 \\(\\times\\) 3 kernel sizes, respectively. Each stage contains two basic blocks, and one basic block has two convolutions. Note that in stages 2–4, down-sampling is only performed in the first basic block, and in stage 1, down-sampling is not performed Furthermore, the batch normalization (BN) technique is added to the residual network, as the BN network can smooth the landscape of the entire loss function, thus optimizing the predictability and stability of the network. The addition of the residual module and BN technology allows a model to deepen the network hierarchy and training speed while improving the network classification accuracy, general ability, and expression effect. 2.5 Feature fusion The focus in multimodal fusion is on identifying shared information across modalities to obtain robust features of the underlying problem. Accurate quantification of shared information should consider the correlation within and between the various forms of capturing the underlying dependencies. The features of two different patterns must be fused into one eigenvector. There are many operations by which two features can connect or fuse, such as concatenation, element-wise multiplication, and element-wise addition. These simple operations are not as effective as external products and can establish complex relationships between the two modes. However, the complexity of the outer product computation is too high. The n-dimensional vector, the external product, is calculated to obtain a \\({n}^{2}\\) vector, so the multimodal compact bilinear (MCB) [ 45 ] algorithm is adopted. The MCB maps the results of the outer product into a low-dimensional space and does not need to compute the outer product explicitly. Its main idea is: First, the two modal feature vectors get the characteristic Count Sketch through the Count Sketch mapping function. Second, the fused features are obtained by fast Fourier transform (FFT) and inverse fast Fourier transform (IFFT). 2.6 Statistical Analysis All statistical analysis was conducted using R software. All correlation tests used the ‘Pearson’ method, with the statistical significance set at 0.05. Clinical characteristics of patients’ ages, TNM, and tumor stage were analyzed by statistics. Age, TNM, and tumor stage were compared in each group. 2.7 Details of Implementation In this work, we trained a CNN (ResNet18) with residual modules to classify MSI-H and MSS through transfer learning. To better measure the distribution of real markers (p) and the predicted distribution (q) of the trained model, we used a cross-entropy loss function to measure the similarity between p and q. Another advantage of a cross-entropy loss function is that using the sigmoid function can avoid the mean square error loss function learning rate decline during the gradient decline because the learning rate can be controlled by the output error. To avoid the effects of the algorithm falling into local optimal solutions and data noise, we employed the SGD + momentum optimizer, where momentum assigns a value of 0.9. For every seven epochs, the learning rate of the parameters decayed by 0.1 times. Moreover, 25 epochs were trained throughout the process. 3. Results 3.1 Clinical relevance of MSI in colorectal cancer We compared the association of MSI with partial clinical indicators of CRC. As shown in Table 1 , we counted specific columns of different clinical features of CRC patients in TCGA. As seen in Fig. 2 , tumor stages I and II were significantly different from stages III and IV, respectively. M0 and M1 mean no distant metastasis or the presence of distant metastasis (M), and they have no significant differences. There were also no significant differences in gender. These results indicate the potential value of MSI-H in clinical staging. Note: * P ≤ 0.05 and *** P ≤ 0.001 represent a significant difference in the characteristics between the two population groups, and ns P > 0.05 indicates no significant difference between the two data groups. Table 1 Proportion of the different clinical features in TCGA Variables N = 360 MSI (n = 65) MSS (n = 295) Age (years) < 50 43 7(10.77%) 36(12.20%) ≥ 50 316 58(89.23%) 258(87.46%) Sex Female 175 37(56.92%) 138(46.78%) male 184 28(43.08%) 156(52.88%) Tumor status (T) T1 12 2(3.07%%) 10(3.39%) T2 61 11(16.92%) 50(16.95%) T3 246 43(66.15%) 201(68.13%) T4 41 9(13.86%) 32(10.85%) Lymph node status (N) N0 207 53(81.54%) 153(51.86%) N1 95 7(10.77%) 88(29.83%) N2 58 5(7.69%) 53(17.97%) Distant metastasis (M) M0 274 52(80.0%) 217(73.56%) M1 43 2(3.08%) 42(14.24%) Cancer stage I 62 13(20.0%) 49(16.61%) II 132 40(61.54%) 92(31.19%) III 107 9(13.85%) 98(33.22%) IV 45 2(3.08%) 43(14.58%) 3.2 The deep CNN framework predicts the MSI of tumors The complete process of predicting MSI-H in 360 CRC cases is shown in Fig. 3 . First, 360 image data points were downloaded from the TCGA database, annotated, and preprocessed, including denoising and color normalization. Second, the data were divided into training and validation cohorts. Of these, 70% of the data was used for training, and the remaining 30% was used for testing. In the partitioning stage, we performed stratified sampling, and the down-sampling solved the problem of unbalanced positive and negative data samples. Image data were used for feature extraction using ResNet18. Clinical features were selected based on the characteristic importance of random forest. The clinical features of top5, top10, and top15 were separately selected for fusion. Third, the multimodal compact bilinear fusion images and the clinical features were modeled in the training set. Finally, the validation prediction classification was performed on the validation set. The fusion part included images and clinical features. The MCB feature fusion method was first used to fuse the two feature vectors into one feature vector, followed by a BN layer, and finally a multilayer perceptron. In our framework, multilayer refers to a three-layer perceptron including RELU, BN, and a fully connected layer. The whole process achieves the purpose of classification and prediction. To avoid contingency, the above procedure was repeated 10 times. 3.3 Significant differences existed between the image features of the MSI and MSS Texture features of the grayscale symbiosis matrix compared the two MSI-H and MSS samples. Figure 4 (a) shows the texture features of the grayscale symbiosis matrix for two sets of samples, with significant differences in the Mean features. There was no significant effect on the contrast. We also explored the correlation between clinical features and H&E image features, as shown in Fig. 4 (b). The results show that some of the clinical features were related to the H&E image features. For example, the Tumor stage had a strong correlation with the T stage, and Cecum and Ascending colon were at the primary site. Mean in the image features was more associated with tumor and T stages than with other clinical features and the Mean. 3.4 H&E staining images can be used to predict MSI in colorectal cancer and performed well. Based on the data of 360 CRC cases in TCGA, the results of the method presented here are better than those of Jakob. The methods and details of this paper were detailed above. As shown in Fig. 5 (a), we performed the dataset published by Jakob using its published training, tested data, and implemented parameters, with a result of 0.77. In the work of this paper, using the method described above, our mean AUC = 0.79, which was higher than the results of Jakob. In addition, we conducted experiments combining images with clinical features of top5, top10, top15, and the result of clinical information of top5 was higher than that of the other two features. The specific methods were as follows: First, for the pathological images downloaded from the TCGA database, we divided the regions of interest into tiles of 512 × 512 px as the input to the CNN. Then, MCB fusion images and clinical features were used to obtain the training model on the training set. Finally, the validation was performed on the validation set. Ten experiments were repeated and averaged as the validation results. The ROC curves and the AUC are shown in Fig. 5 . Experimental results show that the mean AUC combining clinical information from top5 was 0.833, as shown in Fig. 5 (b), which is higher than the average AUC = 0.793 in Fig. 5 (a) predicted by data separately from images. Moreover, the ResNet18 model performed better than Vgg19 (Fig. 5 (b)). Figure 5 (c) shows the accuracy of images combining the top5 clinical features on the ResNet18 model, accuracy, recall, and higher classification accuracy of F1 scores than individual image prediction, and the performance of images combined with top5 clinical features on the Vgg19 model. The results show that H&E images combined with clinical information contributed to improving predictive power. The ROC curves based on the model ResNet18, the images combining the clinical features of top10, top15, and mean AUC are shown in the Supplementary Materials. 3.5 Visualization of the pathological images To better understand the reliability of the model and the algorithm, they were interpreted visually. In this work, the images were visualized using Gradient-weighted Class Activation Mapping (Grad-CAM). Grad-CAM assigns significant values to each neuron using the last convolutional layer of gradient information flowing into the CNN for specific attention decisions. As shown in Fig. 6 , the proposed algorithm can focus on the prediction of pathological images of cancer cell regions. 4. Discussion We used the H&E images and clinical information of 360 CRC cases (TCGA-CRC-DX) to evaluate the classification accuracy of multimodal MSI-H and MSS fusion prediction based on deep learning. Our results demonstrate the potential of this image feature to integrate clinical features as a tool to assess MSI status in clinical practice. Combining the qualitative characteristics of pathological images and the quantitative characteristics of clinical information could effectively classify MSI-H with MSS, and the model performance was somewhat improved. To the best of our knowledge, of all the reported pathological image studies predicting MSI status in CRC (TCGA-CRC-DX, FFPE diagnosis), our work featured the first predictive model to fuse pathological images and clinical features. Deep learning has altered digital pathology and enabled the detection and typing of tumors [ 46 – 48 ]. In CRC, previous studies automatically predicted MSI directly from WSIs stained with H&E or divided full slides into small blocks to predict MSI and achieved certain results. However, in our experiments, as shown in Fig. 7 , the quality of the pathological images directly affected the predicted results. We found that mucinous adenocarcinoma, as well as the necrotic area, contained most of a small number of cells or even no cells, which would directly affect the prediction results. In this respect, predicting pathological organization based on deep learning is challenging. There were limitations to this study. First, the retrospective nature of single-center studies may lead to the inevitable case selection bias, as well as the limited generality. Second, considering the high incidence of CRC, the cohort size was still small, which affected the general adaptation of the results of this study. A large-scale, prospective, multicenter study is needed to validate our results. Third, we did not perform a quality check on the image data. The quality of the H&E images, including the folding, thickness, and necrotic areas, affected our predicted classification results. Therefore, it is important to perform a quality inspection of the images. 5. Conclusion This study introduces a novel approach for multimodal data fusion to identify the MSI status of colorectal cancer by combining pathological image features and clinical data. The results of this research may substantially improve the clinical decision-making process for CRC treatment. Integrating diverse data sources shows promise for the advancement of personalized medicine in the context of colorectal cancer. Declarations Authors’contributions Jiasheng Yang, Peizhen Wang, and Jianjun He designed the study; Binsheng He, Wenjing Qiu, Bing Wang, Jingya Yang, Jinyang Mao and Geng Tian performed the study, analyzed the data and interpreted data; Wenjing Qiu wrote the manuscript; Jiasheng Yang, Bing Wang, Jingya Yang, Geng Tian and Peizhen Wang reviewed the manuscript. Competing interests Wenjing Qiu, Jingya Yang, Jinyang Mao and Geng Tian are employed in Geneis Beijing Co., Ltd., Beijing; other authors declare that they have no competing interests. Funding The study was partially supported by the Foundation of Hunan Educational Committee (Grant No. 19A060) and the provincial key R&D projects of Hunan Provincial Science and Technology Department (No. 2022SK2074).And it was supported by the National Natural Science Foundation of China (No. 62172004), and Educational Commission of Anhui Province (No. KJ2019ZD05). And it was funded by the National Natural Science Foundation of China (number NO. 51574004，No. 62172004); Natural Science Foundation of the Higher Education Institutions of Anhui Province, China (number KJ2019A0085); Academic Foundation for Top Talents of the Higher Education Institutions of Anhui Province (number gxbjZD2016041) and Educational Commission of Anhui Province (No. KJ2019ZD05). Acknowledgements We thank LetPub (www.letpub.com) for its linguistic assistance during the preparation of this manuscript. Data availability statement Publicly available datasets were analyzed in this study. These data can be found here: https://portal.gdc.cancer.gov/repository/. References Dekker, E., et al., Colorectal cancer . Lancet, 2019. 394(10207): p. 1467–1480. Peng, P., et al., Prognostic Factors in Stage IV Colorectal Cancer Patients With Resection of Liver and/or Pulmonary Metastases: A Population-Based Cohort Study . Front Oncol, 2022. 12: p. 850937. Xiao, Y., et al., Interaction between linc01615 and miR-491-5p regulates the survival and metastasis of colorectal cancer cells . Transl Cancer Res, 2020. 9(4): p. 2638–2647. Ferlay, J., et al., Estimating the global cancer incidence and mortality in 2018: GLOBOCAN sources and methods . Int J Cancer, 2019. 144(8): p. 1941–1953. Yang, M., et al., A multi-omics machine learning framework in predicting the survival of colorectal cancer patients . Comput Biol Med, 2022. 146: p. 105516. Ionov, Y., et al., Ubiquitous somatic mutations in simple repeated sequences reveal a new mechanism for colonic carcinogenesis . Nature, 1993. 363(6429): p. 558–61. Kawakami, H., A. Zaanan, and F.A. Sinicrope, Microsatellite instability testing and its role in the management of colorectal cancer . Curr Treat Options Oncol, 2015. 16(7): p. 30. Evrard, C., et al., Microsatellite Instability: Diagnosis, Heterogeneity, Discordance, and Clinical Impact in Colorectal Cancer . Cancers (Basel), 2019. 11(10). Liu, W., et al., Sigmoid Colon Adenocarcinoma with Isolated Loss of PMS2 Presenting in a Patient with Synchronous Prostate Cancer with Intact MMR: Diagnosis and Analysis of the Family Pedigree . Anticancer Res, 2018. 38(8): p. 4847–4852. Vasen, H.F., Clinical description of the Lynch syndrome [hereditary nonpolyposis colorectal cancer (HNPCC)] . Fam Cancer, 2005. 4(3): p. 219–25. Merok, M.A., et al., Microsatellite instability has a positive prognostic impact on stage II colorectal cancer after complete resection: results from a large, consecutive Norwegian series . Ann Oncol, 2013. 24(5): p. 1274–82. Popat, S., R. Hubner, and R.S. Houlston, Systematic review of microsatellite instability and colorectal cancer prognosis . J Clin Oncol, 2005. 23(3): p. 609–18. Le, D.T., et al., PD-1 Blockade in Tumors with Mismatch-Repair Deficiency . N Engl J Med, 2015. 372(26): p. 2509–20. Overman, M.J., et al., Nivolumab in patients with metastatic DNA mismatch repair-deficient or microsatellite instability-high colorectal cancer (CheckMate 142): an open-label, multicentre, phase 2 study . Lancet Oncol, 2017. 18(9): p. 1182–1191. Ludmir, E.B., et al., Total neoadjuvant therapy for rectal cancer: An emerging option . Cancer, 2017. 123(9): p. 1497–1506. Tomasello, G., et al., FOLFOXIRI Plus Bevacizumab as Conversion Therapy for Patients With Initially Unresectable Metastatic Colorectal Cancer: A Systematic Review and Pooled Analysis . JAMA Oncol, 2017. 3(7): p. e170278. Tong, G., et al., Effects of GLP-1 Receptor Agonists on Biological Behavior of Colorectal Cancer Cells by Regulating PI3K/AKT/mTOR Signaling Pathway . Front Pharmacol, 2022. 13: p. 901559. Liang, W., et al., KRT17 Promotes T-lymphocyte Infiltration Through the YTHDF2–CXCL10 Axis in Colorectal Cancer . Cancer Immunology Research, 2023. 11(7): p. 875–894. Fischer, F., K. Baerenfaller, and J. Jiricny, 5-Fluorouracil is efficiently removed from DNA by the base excision and mismatch repair systems. Gastroenterology, 2007. 133(6): p. 1858-68. Chalabi, M., et al., Neoadjuvant immunotherapy leads to pathological responses in MMR-proficient and MMR-deficient early-stage colon cancers . Nat Med, 2020. 26(4): p. 566–576. Diagnosis and C. Treatment Guidelines For Colorectal Cancer Working Group, Chinese Society of Clinical Oncology (CSCO) diagnosis and treatment guidelines for colorectal cancer 2018 (English version) . Chin J Cancer Res, 2019. 31(1): p. 117–134. Yuan, L., et al., Immunohistochemistry and microsatellite instability analysis in molecular subtyping of colorectal carcinoma based on mismatch repair competency . Int J Clin Exp Med, 2015. 8(11): p. 20988–1000. He, B., et al., A New Method for CTC Images Recognition Based on Machine Learning . Front Bioeng Biotechnol, 2020. 8: p. 897. Xie, X., et al., Evaluating Cancer-Related Biomarkers Based on Pathological Images: A Systematic Review . Front Oncol, 2021. 11: p. 763527. Yi, X., et al., Adrenal incidentaloma: machine learning-based quantitative texture analysis of unenhanced CT can effectively differentiate sPHEO from lipid-poor adrenal adenoma . J Cancer, 2018. 9(19): p. 3577–3582. Li, J., et al., Predicting gastric cancer tumor mutational burden from histopathological images using multimodal deep learning . Brief Funct Genomics, 2023. Huang, K., et al., Predicting colorectal cancer tumor mutational burden from histopathological images and clinical information using multi-modal deep learning . Bioinformatics, 2022: p. btac641. Ji, M.Y., et al., Nuclear shape, architecture and orientation features from H&E images are able to predict recurrence in node-negative gastric adenocarcinoma . J Transl Med, 2019. 17(1): p. 92. Luo, X., et al., Comprehensive Computational Pathological Image Analysis Predicts Lung Cancer Prognosis . J Thorac Oncol, 2017. 12(3): p. 501–509. Yang, J., et al., Prediction of HER2-positive breast cancer recurrence and metastasis risk from histopathological images and clinical information via multimodal deep learning . Comput Struct Biotechnol J, 2022. 20: p. 333–342. Yao, Y., et al., ICSDA: a multi-modal deep learning model to predict breast cancer recurrence and metastasis risk by integrating pathological, clinical and gene expression data . Briefings in Bioinformatics, 2022: p. bbac448c. Ye, Z., et al., Cervical Cancer Metastasis and Recurrence Risk Prediction Based on Deep Convolutional Neural Network . Current Bioinformatics, 2022. 17(2): p. 164–173. Liu, X., et al., Predicting breast cancer recurrence and metastasis risk by integrating color and texture features of histopathological images and machine learning technologies . Comput Biol Med, 2022. 146: p. 105569. Qiu, W., et al., Evaluating the Microsatellite Instability of Colorectal Cancer Based on Multimodal Deep Learning Integrating Histopathological and Molecular Data . Front Oncol, 2022. 12: p. 925079. Kather, J.N., et al., Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer . Nat Med, 2019. 25(7): p. 1054–1056. Ke, J., et al. Identifying patch-level MSI from histological images of Colorectal Cancer by a Knowledge Distillation Model . in 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) . 2020. Wu, Z., et al., DeepLRHE: A Deep Convolutional Neural Network Framework to Evaluate the Risk of Lung Cancer Recurrence and Metastasis From Histopathology Images . Front Genet, 2020. 11: p. 768. Macenko, M., et al. A method for normalizing histology slides for quantitative analysis . in Proceedings of the 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, Boston, MA, USA, June 28 - July 1, 2009 . 2009. Vahadane, A., et al. Structure-preserved color normalization for histological images . in Biomedical Imaging (ISBI), 2015 IEEE 12th International Symposium on . 2015. Prusa, J., et al. Using Random Undersampling to Alleviate Class Imbalance on Tweet Sentiment Data . in IEEE International Conference on Information Reuse & Integration . 2015. He, K., et al., Deep Residual Learning for Image Recognition . IEEE, 2016. Russakovsky, O., et al., ImageNet Large Scale Visual Recognition Challenge . International Journal of Computer Vision, 2015. 115(3): p. 211–252. Bel, T.D., et al. Automatic segmentation of histopathological slides of renal tissue using deep learning . in Digital Pathology . 2018. Wang, D., et al., Deep Learning for Identifying Metastatic Breast Cancer. 2016. Yang, G., et al. Compact Bilinear Pooling . in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . 2016. Kather, J.N., et al., Pan-cancer image-based detection of clinically actionable genetic alterations . Nat Cancer, 2020. 1(8): p. 789–799. Schmauch, B., et al., A deep learning model to predict RNA-Seq expression of tumours from whole slide images . Nat Commun, 2020. 11(1): p. 3877. Yu, F., et al., Pan-cancer computational histopathology reveals mutations, tumor composition and prognosis . Nature Cancer, 2020. 1(8): p. 1–11. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {\"props\":{\"pageProps\":{\"initialData\":{\"identity\":\"rs-4200523\",\"acceptedTermsAndConditions\":true,\"allowDirectSubmit\":true,\"archivedVersions\":[],\"articleType\":\"Article\",\"associatedPublications\":[],\"authors\":[{\"id\":298110378,\"identity\":\"28dad31c-fdb5-4211-a2f7-fd8d08dfefd0\",\"order_by\":0,\"name\":\"Binsheng He\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Changsha Medical University\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Binsheng\",\"middleName\":\"\",\"lastName\":\"He\",\"suffix\":\"\"},{\"id\":298110379,\"identity\":\"904de683-5a97-47bd-a8da-380d2b2d3c53\",\"order_by\":1,\"name\":\"Wenjing Qiu\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Geneis Beijing Co., Ltd\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Wenjing\",\"middleName\":\"\",\"lastName\":\"Qiu\",\"suffix\":\"\"},{\"id\":298110380,\"identity\":\"8b683b8c-a13d-4c46-bdb5-ea7ba376e4ee\",\"order_by\":2,\"name\":\"Bing Wang\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Anhui University of Technology\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Bing\",\"middleName\":\"\",\"lastName\":\"Wang\",\"suffix\":\"\"},{\"id\":298110381,\"identity\":\"2bec55b1-8592-4f89-a729-e44f4fa51b9b\",\"order_by\":3,\"name\":\"Jingya Yang\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Geneis Beijing Co., Ltd\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Jingya\",\"middleName\":\"\",\"lastName\":\"Yang\",\"suffix\":\"\"},{\"id\":298110382,\"identity\":\"20c2dc98-9e0e-474e-971f-c8ae90794452\",\"order_by\":4,\"name\":\"Jinyang Mao\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Geneis Beijing Co., Ltd\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Jinyang\",\"middleName\":\"\",\"lastName\":\"Mao\",\"suffix\":\"\"},{\"id\":298110383,\"identity\":\"303aacec-024f-430a-997f-2011c28bc893\",\"order_by\":5,\"name\":\"Geng Tian\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Geneis Beijing Co., Ltd\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Geng\",\"middleName\":\"\",\"lastName\":\"Tian\",\"suffix\":\"\"},{\"id\":298110384,\"identity\":\"487dd637-a438-4ea3-8f6c-7f949ec9132e\",\"order_by\":6,\"name\":\"Tao Xu\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Changsha Medical University\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Tao\",\"middleName\":\"\",\"lastName\":\"Xu\",\"suffix\":\"\"},{\"id\":298110385,\"identity\":\"aab0685a-75ed-4f5f-9237-d9a0a09f686c\",\"order_by\":7,\"name\":\"Jianjun He\",\"email\":\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA70lEQVRIiWNgGAWjYBACAwaGhAMMDGwMDOwNUKEDRGvhgSklQgsUSCQQqcWcveHhgZ87+BK3Sz5+/OlmG4Mc340Exs8FeLRY9hxIONh7hi1x5+w0M+ncNgZjyRsJzNIz8DnsRkLCAd42tsQNt3PYmIFaEjfcSGBj5sGn5f6DhIN/QVpunmH+DNRST1jLDYaEw2BbbvAwgByWYEBQy5mEhMOybWzGG84A/ZJzTsJw5pmHzdJ4tRw/k/zxbdsx2Q3HDz/+nFNmI893PPngZ3xagFGYACSOwXgSQMzYgFcDMKEcABI1BBSNglEwCkbBiAYAIwtToEKUDGgAAAAASUVORK5CYII=\",\"orcid\":\"\",\"institution\":\"Changsha Medical University\",\"correspondingAuthor\":true,\"prefix\":\"\",\"firstName\":\"Jianjun\",\"middleName\":\"\",\"lastName\":\"He\",\"suffix\":\"\"},{\"id\":298110389,\"identity\":\"7e1a9f27-756c-45c3-a4bf-b3d69dc9014e\",\"order_by\":8,\"name\":\"Peizhen Wang\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Anhui University of Technology\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Peizhen\",\"middleName\":\"\",\"lastName\":\"Wang\",\"suffix\":\"\"},{\"id\":298110395,\"identity\":\"94a03a23-c633-45f7-94b2-286df1c21386\",\"order_by\":9,\"name\":\"Jiasheng Yang\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Anhui University of Technology\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Jiasheng\",\"middleName\":\"\",\"lastName\":\"Yang\",\"suffix\":\"\"}],\"badges\":[],\"createdAt\":\"2024-04-01 12:09:50\",\"currentVersionCode\":1,\"declarations\":\"\",\"doi\":\"10.21203/rs.3.rs-4200523/v1\",\"doiUrl\":\"https://doi.org/10.21203/rs.3.rs-4200523/v1\",\"draftVersion\":[],\"editorialEvents\":[],\"editorialNote\":\"\",\"failedWorkflow\":false,\"files\":[{\"id\":55819592,\"identity\":\"d6a8acbd-7c16-4ccf-a10e-f9d063be5f0e\",\"added_by\":\"auto\",\"created_at\":\"2024-05-03 21:24:14\",\"extension\":\"png\",\"order_by\":1,\"title\":\"Figure 1\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":170524,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003e\\u003cstrong\\u003eThe Network architecture of ResNet18.\\u003c/strong\\u003e (a) ResNet18 structure diagram. (b) The basic structure of the ResNet18 residual network.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"Figure1.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4200523/v1/5d53fc8000a834a60e520f7d.png\"},{\"id\":55819595,\"identity\":\"9b2caca0-4ed5-4683-ba5a-abb0275edab1\",\"added_by\":\"auto\",\"created_at\":\"2024-05-03 21:24:15\",\"extension\":\"png\",\"order_by\":2,\"title\":\"Figure 2\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":262601,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003e\\u003cstrong\\u003eViolin chart analysis of the features of clinical data.\\u003c/strong\\u003e (a) Sex had no significant effect on the MSI-H in CRC. (b) The cancer stage showed a significant correlation with MSI-H, especially between phase II and III–IV. (c) The N stage showed a significant correlation with the MSI-H, except for the N1 and N2. (d) The M stage was also significant when compared with MSI-H.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"Figure2.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4200523/v1/c2aa42026b8982ef59c3aa31.png\"},{\"id\":55819598,\"identity\":\"55086f18-f5ed-46b5-b857-0a311f84c513\",\"added_by\":\"auto\",\"created_at\":\"2024-05-03 21:24:15\",\"extension\":\"png\",\"order_by\":3,\"title\":\"Figure 3\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":2657025,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003e\\u003cstrong\\u003eThe whole process of the MSI forecast.\\u003c/strong\\u003e (a) H\\u0026amp;E-stained histological images of the tumor and annotated cancer areas. (b) Color normalization was performed on areas where the filtering blank ratio was \\u0026lt; 30%. (c) Predicted MSS (left) and MSI-H (right) sample images. (d) The prediction was performed based on the MCB fusion method.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"Figure3.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4200523/v1/eda465f2dcf071fdffee0dbf.png\"},{\"id\":55819597,\"identity\":\"12e0d13d-3c3e-4791-b8dc-d4888f8bb997\",\"added_by\":\"auto\",\"created_at\":\"2024-05-03 21:24:15\",\"extension\":\"png\",\"order_by\":4,\"title\":\"Figure 4\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":156986,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003e\\u003cstrong\\u003eFeatures of image data.\\u003c/strong\\u003e (a) The two groups showed significant differences in the image characteristics; (b) Correlation of image features and clinical features.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"4.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4200523/v1/fbe4f44ca8a7c538aea4c2aa.png\"},{\"id\":55819599,\"identity\":\"b37610ca-22b6-4f40-9cec-8ae265a49e70\",\"added_by\":\"auto\",\"created_at\":\"2024-05-03 21:24:15\",\"extension\":\"png\",\"order_by\":5,\"title\":\"Figure 5\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":308474,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003e\\u003cstrong\\u003ePerformance of the model on the TCGA-CRC-DX dataset.\\u003c/strong\\u003e (a) Comparison of our method and Jakob’s methods. (b) The three curves represent the average AUC based on image data, the average AUC curve based on ResNet18 image data, and clinical data (top5). Average AUC curves based on Vgg19 image data and clinical data (top5). (c) Performance on the ACC, Precision, Recall, and F1_score metrics.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"Figure5.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4200523/v1/f050c9f3cfa98be744e71b9c.png\"},{\"id\":55819593,\"identity\":\"e741b22e-5289-4abe-b9ec-98c870a5a38e\",\"added_by\":\"auto\",\"created_at\":\"2024-05-03 21:24:14\",\"extension\":\"png\",\"order_by\":6,\"title\":\"Figure 6\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":1523030,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003e\\u003cstrong\\u003eVisualization interpretation.\\u003c/strong\\u003e (a) MSS, (b) MSI.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"Figure6.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4200523/v1/6442b1585b9b396d247436ae.png\"},{\"id\":55819801,\"identity\":\"923832c0-3dae-4fb0-b4b2-96ce234414c1\",\"added_by\":\"auto\",\"created_at\":\"2024-05-03 21:32:15\",\"extension\":\"png\",\"order_by\":7,\"title\":\"Figure 7\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":733392,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003e\\u003cstrong\\u003eImages affecting the predicted results.\\u003c/strong\\u003e (a\\u003cstrong\\u003e) \\u003c/strong\\u003eFold, (b) Interstitial, (c) Mucus, (d) Necrotic region.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"Figure7.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4200523/v1/f5c4cfeb270deb8325bef2c8.png\"},{\"id\":59105052,\"identity\":\"aae37c34-896c-4393-91a9-3c08e46bbd80\",\"added_by\":\"auto\",\"created_at\":\"2024-06-26 11:57:45\",\"extension\":\"pdf\",\"order_by\":0,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"manuscript-pdf\",\"size\":9116066,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"manuscript.pdf\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4200523/v1/d727a5af-3533-4a98-87fa-fddba2729ba0.pdf\"}],\"financialInterests\":\"No competing interests reported.\",\"formattedTitle\":\"Development of a Multimodal Deep Learning Model for Predicting Microsatellite Instability in Colorectal Cancer by Integrating Histopathological Images and Clinical Data\",\"fulltext\":[{\"header\":\"1. Introduction\",\"content\":\"\\u003cp\\u003eColorectal cancer (CRC), a highly heterogeneous disease driven by a range of genetic and epigenetic events [\\u003cspan additionalcitationids=\\\"CR2\\\" citationid=\\\"CR1\\\" class=\\\"CitationRef\\\"\\u003e1\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR3\\\" class=\\\"CitationRef\\\"\\u003e3\\u003c/span\\u003e], is the third-most common cancer and the second-largest cancer-related cause of death worldwide [\\u003cspan citationid=\\\"CR4\\\" class=\\\"CitationRef\\\"\\u003e4\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR5\\\" class=\\\"CitationRef\\\"\\u003e5\\u003c/span\\u003e]. Microsatellite instability (MSI) is the phenomenon of normal microsatellite length change due to the deletion or insertion of repeat bases compared to normal tissue cells [\\u003cspan citationid=\\\"CR6\\\" class=\\\"CitationRef\\\"\\u003e6\\u003c/span\\u003e]. It is characterized by the generalized instability of short tandem repeat DNA sequences known as the microsatellite [\\u003cspan citationid=\\\"CR7\\\" class=\\\"CitationRef\\\"\\u003e7\\u003c/span\\u003e]. MSI is one of the three major mechanisms for CRC carcinogenesis, with MSI occurring in about 15% of CRC cases [\\u003cspan citationid=\\\"CR7\\\" class=\\\"CitationRef\\\"\\u003e7\\u003c/span\\u003e]. MSI has attracted much attention because of its value in diagnosis, treatment reaction, and prognosis for CRC [\\u003cspan additionalcitationids=\\\"CR8\\\" citationid=\\\"CR7\\\" class=\\\"CitationRef\\\"\\u003e7\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR9\\\" class=\\\"CitationRef\\\"\\u003e9\\u003c/span\\u003e].\\u003c/p\\u003e \\u003cp\\u003eThere is some evidence to support the use of MSI testing before making clinical treatment decisions. First, MSI testing is recommended to assist in the diagnosis of Lynch syndrome, the most common inherited colon cancer syndrome associated with germline mutations in the DNA mismatch repair (MMR) gene (MLH1, MSH2, MSH6, or PMS2) [\\u003cspan citationid=\\\"CR10\\\" class=\\\"CitationRef\\\"\\u003e10\\u003c/span\\u003e]. The MSI status helps to identify families with the syndrome to minimize family members' and relatives\\u0026rsquo; risk of developing the disease. Second, MSI is one of the key factors affecting the prognosis of CRC, especially in the early stage [\\u003cspan citationid=\\\"CR11\\\" class=\\\"CitationRef\\\"\\u003e11\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR12\\\" class=\\\"CitationRef\\\"\\u003e12\\u003c/span\\u003e]. Patients with stage II CRC with high MSI (MSI-H) / MMR deficiency (d MMR) generally have better outcomes than patients with microsatellite stability (MSS) and low MSI (MSI-L) / MMR (p MMR) [\\u003cspan citationid=\\\"CR11\\\" class=\\\"CitationRef\\\"\\u003e11\\u003c/span\\u003e]. Third, MSI identifies a unique subset of patients with a better stage-adjusted prognosis, for whom standard fluorouracil chemotherapy is contraindicated, and oxaliplatin and immune checkpoint inhibitors may be particularly beneficial [\\u003cspan citationid=\\\"CR7\\\" class=\\\"CitationRef\\\"\\u003e7\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR13\\\" class=\\\"CitationRef\\\"\\u003e13\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR14\\\" class=\\\"CitationRef\\\"\\u003e14\\u003c/span\\u003e]. Fluoropyrimidine (5-FU or capecitabine) is the mainstay of the CRC chemotherapy strategy and plays an important role in both neoadjuvant and translational therapies [\\u003cspan additionalcitationids=\\\"CR16 CR17\\\" citationid=\\\"CR15\\\" class=\\\"CitationRef\\\"\\u003e15\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR18\\\" class=\\\"CitationRef\\\"\\u003e18\\u003c/span\\u003e]. However, patients with MSI status are generally resistant to 5-FU-based chemotherapeutic agents [\\u003cspan citationid=\\\"CR19\\\" class=\\\"CitationRef\\\"\\u003e19\\u003c/span\\u003e]. Immunotherapy is an emerging and promising treatment for CRC because MSI tumors have a large number of mutant neoantigens, which makes them sensitive to the immune checkpoint inhibitor [\\u003cspan citationid=\\\"CR20\\\" class=\\\"CitationRef\\\"\\u003e20\\u003c/span\\u003e]. Therefore, MSI status is critical for selecting CRC treatment and evaluating the response to treatment [\\u003cspan citationid=\\\"CR21\\\" class=\\\"CitationRef\\\"\\u003e21\\u003c/span\\u003e].\\u003c/p\\u003e \\u003cp\\u003eIn general, the identification of MSI depends on immunohistochemistry (IHC) staining, polymerase chain reaction (PCR) [\\u003cspan citationid=\\\"CR22\\\" class=\\\"CitationRef\\\"\\u003e22\\u003c/span\\u003e], but not every patient can avail these options as a result of their high costs and regional limitations. In contrast, histopathological images are routinely available for almost all cancer patients and are inexpensive and non-invasive [\\u003cspan additionalcitationids=\\\"CR24\\\" citationid=\\\"CR23\\\" class=\\\"CitationRef\\\"\\u003e23\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR25\\\" class=\\\"CitationRef\\\"\\u003e25\\u003c/span\\u003e]. Given the rapid development of deep learning techniques, researchers have explored the identification of molecular biomarkers through hematoxylin and eosin (H\\u0026amp;E) image [\\u003cspan citationid=\\\"CR26\\\" class=\\\"CitationRef\\\"\\u003e26\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR27\\\" class=\\\"CitationRef\\\"\\u003e27\\u003c/span\\u003e]. Studies have shown that the morphological features of H\\u0026amp;E histopathological images have an important effect on the prognosis of various malignancies [\\u003cspan additionalcitationids=\\\"CR29 CR30\\\" citationid=\\\"CR28\\\" class=\\\"CitationRef\\\"\\u003e28\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR31\\\" class=\\\"CitationRef\\\"\\u003e31\\u003c/span\\u003e]. With the continuous development of computer technology and full-slide imaging (Whole Slide Imaging (WSI)), computer-assisted diagnosis and prognostic prediction based on images such as H\\u0026amp;E staining tissues have received increasing attention [\\u003cspan citationid=\\\"CR32\\\" class=\\\"CitationRef\\\"\\u003e32\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR33\\\" class=\\\"CitationRef\\\"\\u003e33\\u003c/span\\u003e]. These pathological images not only contain pathological features such as tumor morphology, growth, and distribution, but also have the advantages of radio-mics, such as high speed, non-invasiveness, and low cost. Therefore, more research has been carried out in histopathological-assisted diagnosis by deep learning methods in recent years. Today, it can already help doctors improve the accuracy and speed of diagnostic work and alleviate the problem of insufficient pathological diagnostic resources.\\u003c/p\\u003e \\u003cp\\u003eCurrently, some contributions have been made to MSI prediction of pathological images of CRC [\\u003cspan citationid=\\\"CR34\\\" class=\\\"CitationRef\\\"\\u003e34\\u003c/span\\u003e]. Kather et al. used ResNet18 to predict histopathological sections of CRC (FFPE) on the Cancer Genome Atlas (TCGA), yielding an AUC of 0.77. The AUC on the DAHCS Colorectal (FFPE) dataset was 0.84 [\\u003cspan citationid=\\\"CR35\\\" class=\\\"CitationRef\\\"\\u003e35\\u003c/span\\u003e]. Ke et al. used a multistage convolutional neural network (CNN) knowledge distillation model to predict TCGA colorectal cancer (FFPE) with an AUC above 0.802 [\\u003cspan citationid=\\\"CR36\\\" class=\\\"CitationRef\\\"\\u003e36\\u003c/span\\u003e]. Thus, it has been verified that CNN is a powerful algorithm that can directly process biomedical images. The defect in the subjective bias during the extraction of histological features of H\\u0026amp;E staining images has been overcome. However, the above research methods are only based on the qualitative information of histopathological images.\\u003c/p\\u003e \\u003cp\\u003eIn this work, we proposed a predictive framework based on pathological images and clinical information. First, we downloaded the full-slide images (WSIs) and the corresponding clinical features of 360 CRC cases (TCGA-CRC-DX, FFPE tissue) from the TGCA. These WSIs were labeled and then divided into 512\\u003cspan class=\\\"InlineEquation\\\"\\u003e\\u003cspan class=\\\"mathinline\\\"\\u003e\\\\(\\\\times\\\\)\\u003c/span\\u003e\\u003c/span\\u003e512 pixels of the patch. Tiles were treated with image preprocessing steps, and preprocessing of the clinical data was done. Second, the data were extracted and evaluated. The features of the image data were extracted using a deep CNN ResNet18 model, and random forest in machine learning was screened for the features of clinical information. A new multimodal fusion classification model was constructed based on the extracted features. Finally, the results of our model predictions were discussed.\\u003c/p\\u003e\"},{\"header\":\"2. Materials and Methods\",\"content\":\"\\u003cdiv id=\\\"Sec3\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e2.1 Dataset\\u003c/h2\\u003e \\u003cp\\u003eTCGA is an open, large-scale cancer genomics database containing a large number of primary cancers and their pathological images. It is used in digital forms to match normal samples of multiple cancer types. It provides researchers with public datasets that they can search, view, and download to help improve diagnostics and standards of treatment, and ultimately prevent cancer. We downloaded WSIs corresponding to clinical data from H\\u0026amp;E stained sections of 360 CRC cases (TCGA-CRC-DX, FFPE tissue) from the TCGA database (\\u003cspan class=\\\"ExternalRef\\\"\\u003e\\u003cspan class=\\\"RefSource\\\"\\u003ehttps://portal.gdc.cancer.gov/repository/\\u003c/span\\u003e\\u003cspan address=\\\"https://portal.gdc.cancer.gov/repository/\\\" targettype=\\\"URL\\\" class=\\\"RefTarget\\\"\\u003e\\u003c/span\\u003e\\u003c/span\\u003e). All CRC slide images were stored in SVS format and adjusted to 0.5 um per pixel at the same magnification (40x).\\u003c/p\\u003e \\u003cp\\u003eAmong the pathological tissue images of CRC, the number of MSIs was 65, and the number of MSSs was 295. In this process, we classified MSI-L as the MSS class. Moreover, the histological image of H\\u0026amp;E staining for MSI-H was labeled as 1, and that for MSS was labeled as 0. We combined histological images of H\\u0026amp;E staining with clinical data for follow-up studies.\\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec4\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e2.2 Preprocessing of the image data\\u003c/h2\\u003e \\u003cdiv id=\\\"Sec5\\\" class=\\\"Section3\\\"\\u003e \\u003ch2\\u003e2.2.1 The patching and filtering of the images\\u003c/h2\\u003e \\u003cp\\u003eAs the WSIs were too large to be used directly, the representative regions of interest (ROI) in each slide were indicated. The black dashed line in Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig3\\\" class=\\\"InternalRef\\\"\\u003e3\\u003c/span\\u003e(a) shows the boundaries of the annotated tumor region. Second, it was divided into small blocks of 512\\u003cspan class=\\\"InlineEquation\\\"\\u003e\\u003cspan class=\\\"mathinline\\\"\\u003e\\\\(\\\\times\\\\)\\u003c/span\\u003e\\u003c/span\\u003e512 px. As shown in Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig3\\\" class=\\\"InternalRef\\\"\\u003e3\\u003c/span\\u003e(b), the less informative slides (e.g., more than 30% of the filtered blanks were covered by the background) were then discarded [\\u003cspan citationid=\\\"CR37\\\" class=\\\"CitationRef\\\"\\u003e37\\u003c/span\\u003e].\\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec6\\\" class=\\\"Section3\\\"\\u003e \\u003ch2\\u003e2.2.2 Color-normalized image data\\u003c/h2\\u003e \\u003cp\\u003eThe error of the manual production process and the difference between stains and scanners will produce a color difference between digital sections, which will cause errors in the subsequent analysis work. Color standardization technology is generally used to eliminate color differences between slices. Therefore, we performed color normalization using the Macenko\\u003csup\\u003e'\\u003c/sup\\u003es method [\\u003cspan citationid=\\\"CR38\\\" class=\\\"CitationRef\\\"\\u003e38\\u003c/span\\u003e] in the Tia toolbox software package. The Tia toolbox is a multipurpose name for: 1) a computer program; 2) the Python package for the related program that has been created in the TIA Center to help people start using digital pathology; 3) a repository; and 4) a virtual environment. The main idea of the Macenko\\u003csup\\u003e'\\u003c/sup\\u003e s method is to map images of the RGB space to the color space of the stain using the color separation method, and then standardize staining using the color deconvolution technique [\\u003cspan citationid=\\\"CR38\\\" class=\\\"CitationRef\\\"\\u003e38\\u003c/span\\u003e, \\u003cspan citationid=\\\"CR39\\\" class=\\\"CitationRef\\\"\\u003e39\\u003c/span\\u003e].\\u003c/p\\u003e \\u003c/div\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec7\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e2.3 Ordering of clinical features based on random forest\\u003c/h2\\u003e \\u003cp\\u003eFeature selection is important for interpretation and prediction, especially for avoiding the high-dimensional curse. The data were first cleaned before feature selection, including removing columns with a null value greater than 25% and filling the data, where the typed variables were numbered, and the continuous data were filled with the average. We selected the data using random forest feature importance. Random forest is a combinatorial classifier model composed of decision tree classifier sets. It belongs to an integrated learning model [\\u003cspan citationid=\\\"CR40\\\" class=\\\"CitationRef\\\"\\u003e40\\u003c/span\\u003e]. Random forest is not only a representative classifier in machine learning, but also can be used to estimate the importance of variables in the model. The importance of features measured by the mean reduced Gini coefficient is:\\u003c/p\\u003e \\u003cp\\u003e \\u003cspan class=\\\"InlineEquation\\\"\\u003e \\u003cspan class=\\\"mathinline\\\"\\u003e\\\\(Gini(t)=1 - \\\\sum\\\\limits_{{m=1}}^{M} {\\\\mathop {^{{p(m/t)}}}\\\\nolimits^{2} }\\\\)\\u003c/span\\u003e \\u003c/span\\u003e \\u003c/p\\u003e \\u003cp\\u003ewhere \\u003cspan class=\\\"InlineEquation\\\"\\u003e\\u003cspan class=\\\"mathinline\\\"\\u003e\\\\(M\\\\)\\u003c/span\\u003e\\u003c/span\\u003erepresents the total number of classes of the target variable and \\u003cspan class=\\\"InlineEquation\\\"\\u003e\\u003cspan class=\\\"mathinline\\\"\\u003e\\\\(P\\\\left(m∕t\\\\right)\\\\)\\u003c/span\\u003e\\u003c/span\\u003e represents the conditional probability that the target variable is of class \\u003cspan class=\\\"InlineEquation\\\"\\u003e\\u003cspan class=\\\"mathinline\\\"\\u003e\\\\(m\\\\)\\u003c/span\\u003e\\u003c/span\\u003e at the node \\u003cspan class=\\\"InlineEquation\\\"\\u003e\\u003cspan class=\\\"mathinline\\\"\\u003e\\\\(t\\\\)\\u003c/span\\u003e\\u003c/span\\u003e. According to the formula, \\u003cspan class=\\\"InlineEquation\\\"\\u003e\\u003cspan class=\\\"mathinline\\\"\\u003e\\\\(Gini\\\\)\\u003c/span\\u003e\\u003c/span\\u003eis calculated. Finally, the larger the value of \\u003cspan class=\\\"InlineEquation\\\"\\u003e\\u003cspan class=\\\"mathinline\\\"\\u003e\\\\(Gini\\\\left(t\\\\right)\\\\)\\u003c/span\\u003e\\u003c/span\\u003e is, the more important \\u003cspan class=\\\"InlineEquation\\\"\\u003e\\u003cspan class=\\\"mathinline\\\"\\u003e\\\\(m\\\\)\\u003c/span\\u003e\\u003c/span\\u003e is.\\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec8\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e2.4 Image features extraction based on ResNet18\\u003c/h2\\u003e \\u003cp\\u003eA residual network is a CNN proposed by four scholars from Microsoft Research, which achieved image classification and object recognition in ImageNet (ILSVRC). Residual networks are characterized by easy optimization and the ability to improve accuracy by increasing depth considerably. Their internal residual block uses jump connections, alleviating the gradient vanishing problem caused by increasing depth in a deep neural network[\\u003cspan citationid=\\\"CR41\\\" class=\\\"CitationRef\\\"\\u003e41\\u003c/span\\u003e]. Residual networks have had many applications in image feature classification, lesion segmentation, and cell segmentation [\\u003cspan additionalcitationids=\\\"CR43\\\" citationid=\\\"CR42\\\" class=\\\"CitationRef\\\"\\u003e42\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR44\\\" class=\\\"CitationRef\\\"\\u003e44\\u003c/span\\u003e]. Figure\\u0026nbsp;\\u003cspan refid=\\\"Fig1\\\" class=\\\"InternalRef\\\"\\u003e1\\u003c/span\\u003e (a) shows the workflow chart of the ResNet18 network, and (b) is the structural chart of the residual block. The architecture of ResNet18 is divided into four stages. Every Resnet architecture performs the initial convolution and max-pooling using 7\\u003cspan class=\\\"InlineEquation\\\"\\u003e\\u003cspan class=\\\"mathinline\\\"\\u003e\\\\(\\\\times\\\\)\\u003c/span\\u003e\\u003c/span\\u003e7 and 3\\u003cspan class=\\\"InlineEquation\\\"\\u003e\\u003cspan class=\\\"mathinline\\\"\\u003e\\\\(\\\\times\\\\)\\u003c/span\\u003e\\u003c/span\\u003e3 kernel sizes, respectively. Each stage contains two basic blocks, and one basic block has two convolutions. Note that in stages 2\\u0026ndash;4, down-sampling is only performed in the first basic block, and in stage 1, down-sampling is not performed Furthermore, the batch normalization (BN) technique is added to the residual network, as the BN network can smooth the landscape of the entire loss function, thus optimizing the predictability and stability of the network. The addition of the residual module and BN technology allows a model to deepen the network hierarchy and training speed while improving the network classification accuracy, general ability, and expression effect.\\u003c/p\\u003e \\u003cp\\u003e \\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec9\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e2.5 Feature fusion\\u003c/h2\\u003e \\u003cp\\u003eThe focus in multimodal fusion is on identifying shared information across modalities to obtain robust features of the underlying problem. Accurate quantification of shared information should consider the correlation within and between the various forms of capturing the underlying dependencies.\\u003c/p\\u003e \\u003cp\\u003eThe features of two different patterns must be fused into one eigenvector. There are many operations by which two features can connect or fuse, such as concatenation, element-wise multiplication, and element-wise addition. These simple operations are not as effective as external products and can establish complex relationships between the two modes. However, the complexity of the outer product computation is too high. The n-dimensional vector, the external product, is calculated to obtain a \\u003cspan class=\\\"InlineEquation\\\"\\u003e\\u003cspan class=\\\"mathinline\\\"\\u003e\\\\({n}^{2}\\\\)\\u003c/span\\u003e\\u003c/span\\u003e vector, so the multimodal compact bilinear (MCB) [\\u003cspan citationid=\\\"CR45\\\" class=\\\"CitationRef\\\"\\u003e45\\u003c/span\\u003e] algorithm is adopted. The MCB maps the results of the outer product into a low-dimensional space and does not need to compute the outer product explicitly. Its main idea is: First, the two modal feature vectors get the characteristic Count Sketch through the Count Sketch mapping function. Second, the fused features are obtained by fast Fourier transform (FFT) and inverse fast Fourier transform (IFFT).\\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec10\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e2.6 Statistical Analysis\\u003c/h2\\u003e \\u003cp\\u003eAll statistical analysis was conducted using R software. All correlation tests used the \\u0026lsquo;Pearson\\u0026rsquo; method, with the statistical significance set at 0.05. Clinical characteristics of patients\\u0026rsquo; ages, TNM, and tumor stage were analyzed by statistics. Age, TNM, and tumor stage were compared in each group.\\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec11\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e2.7 Details of Implementation\\u003c/h2\\u003e \\u003cp\\u003eIn this work, we trained a CNN (ResNet18) with residual modules to classify MSI-H and MSS through transfer learning. To better measure the distribution of real markers (p) and the predicted distribution (q) of the trained model, we used a cross-entropy loss function to measure the similarity between p and q. Another advantage of a cross-entropy loss function is that using the sigmoid function can avoid the mean square error loss function learning rate decline during the gradient decline because the learning rate can be controlled by the output error. To avoid the effects of the algorithm falling into local optimal solutions and data noise, we employed the SGD\\u0026thinsp;+\\u0026thinsp;momentum optimizer, where momentum assigns a value of 0.9. For every seven epochs, the learning rate of the parameters decayed by 0.1 times. Moreover, 25 epochs were trained throughout the process.\\u003c/p\\u003e \\u003c/div\\u003e\"},{\"header\":\"3. Results\",\"content\":\"\\u003cdiv id=\\\"Sec13\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e3.1 Clinical relevance of MSI in colorectal cancer\\u003c/h2\\u003e \\u003cp\\u003eWe compared the association of MSI with partial clinical indicators of CRC. As shown in Table\\u0026nbsp;\\u003cspan refid=\\\"Tab1\\\" class=\\\"InternalRef\\\"\\u003e1\\u003c/span\\u003e, we counted specific columns of different clinical features of CRC patients in TCGA. As seen in Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig2\\\" class=\\\"InternalRef\\\"\\u003e2\\u003c/span\\u003e, tumor stages I and II were significantly different from stages III and IV, respectively. M0 and M1 mean no distant metastasis or the presence of distant metastasis (M), and they have no significant differences. There were also no significant differences in gender. These results indicate the potential value of MSI-H in clinical staging. Note: * \\u003cem\\u003eP\\u003c/em\\u003e\\u0026thinsp;\\u0026le;\\u0026thinsp;0.05 and ***\\u003cem\\u003eP\\u003c/em\\u003e\\u0026thinsp;\\u0026le;\\u0026thinsp;0.001 represent a significant difference in the characteristics between the two population groups, and ns \\u003cem\\u003eP\\u003c/em\\u003e\\u0026thinsp;\\u0026gt;\\u0026thinsp;0.05 indicates no significant difference between the two data groups.\\u003c/p\\u003e \\u003cp\\u003e \\u003cdiv class=\\\"gridtable\\\"\\u003e\\u003ctable float=\\\"Yes\\\" id=\\\"Tab1\\\" border=\\\"1\\\"\\u003e \\u003ccaption language=\\\"En\\\"\\u003e \\u003cdiv class=\\\"CaptionNumber\\\"\\u003eTable 1\\u003c/div\\u003e \\u003cdiv class=\\\"CaptionContent\\\"\\u003e \\u003cp\\u003eProportion of the different clinical features in TCGA\\u003c/p\\u003e \\u003c/div\\u003e \\u003c/caption\\u003e \\u003ccolgroup cols=\\\"4\\\"\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c1\\\" colnum=\\\"1\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"char\\\" char=\\\".\\\" class=\\\"colspec\\\" colname=\\\"c2\\\" colnum=\\\"2\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c3\\\" colnum=\\\"3\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c4\\\" colnum=\\\"4\\\"\\u003e\\u003c/div\\u003e \\u003cthead\\u003e \\u003ctr\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eVariables\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eN\\u0026thinsp;=\\u0026thinsp;360\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eMSI (n\\u0026thinsp;=\\u0026thinsp;65)\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003eMSS (n\\u0026thinsp;=\\u0026thinsp;295)\\u003c/p\\u003e \\u003c/th\\u003e \\u003c/tr\\u003e \\u003c/thead\\u003e \\u003ctbody\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eAge (years)\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e\\u0026lt;\\u0026thinsp;50\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e43\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e7(10.77%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e36(12.20%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e\\u0026ge;\\u0026thinsp;50\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e316\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e58(89.23%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e258(87.46%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eSex\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eFemale\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e175\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e37(56.92%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e138(46.78%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003emale\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e184\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e28(43.08%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e156(52.88%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eTumor status (T)\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eT1\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e12\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e2(3.07%%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e10(3.39%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eT2\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e61\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e11(16.92%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e50(16.95%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eT3\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e246\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e43(66.15%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e201(68.13%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eT4\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e41\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e9(13.86%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e32(10.85%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eLymph node status (N)\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colspan=\\\"3\\\" nameend=\\\"c4\\\" namest=\\\"c2\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eN0\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e207\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e53(81.54%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e153(51.86%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eN1\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e95\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e7(10.77%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e88(29.83%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eN2\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e58\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e5(7.69%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e53(17.97%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eDistant metastasis (M)\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eM0\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e274\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e52(80.0%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e217(73.56%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eM1\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e43\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e2(3.08%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e42(14.24%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eCancer stage\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eI\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e62\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e13(20.0%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e49(16.61%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eII\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e132\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e40(61.54%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e92(31.19%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eIII\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e107\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e9(13.85%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e98(33.22%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eIV\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e45\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e2(3.08%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e43(14.58%)\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003c/tbody\\u003e \\u003c/colgroup\\u003e \\u003c/table\\u003e\\u003c/div\\u003e \\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec14\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e3.2 The deep CNN framework predicts the MSI of tumors\\u003c/h2\\u003e \\u003cp\\u003eThe complete process of predicting MSI-H in 360 CRC cases is shown in Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig3\\\" class=\\\"InternalRef\\\"\\u003e3\\u003c/span\\u003e. First, 360 image data points were downloaded from the TCGA database, annotated, and preprocessed, including denoising and color normalization. Second, the data were divided into training and validation cohorts. Of these, 70% of the data was used for training, and the remaining 30% was used for testing. In the partitioning stage, we performed stratified sampling, and the down-sampling solved the problem of unbalanced positive and negative data samples. Image data were used for feature extraction using ResNet18. Clinical features were selected based on the characteristic importance of random forest. The clinical features of top5, top10, and top15 were separately selected for fusion. Third, the multimodal compact bilinear fusion images and the clinical features were modeled in the training set. Finally, the validation prediction classification was performed on the validation set. The fusion part included images and clinical features. The MCB feature fusion method was first used to fuse the two feature vectors into one feature vector, followed by a BN layer, and finally a multilayer perceptron. In our framework, multilayer refers to a three-layer perceptron including RELU, BN, and a fully connected layer. The whole process achieves the purpose of classification and prediction. To avoid contingency, the above procedure was repeated 10 times.\\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec15\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e3.3 Significant differences existed between the image features of the MSI and MSS\\u003c/h2\\u003e \\u003cp\\u003eTexture features of the grayscale symbiosis matrix compared the two MSI-H and MSS samples. Figure\\u0026nbsp;\\u003cspan refid=\\\"Fig4\\\" class=\\\"InternalRef\\\"\\u003e4\\u003c/span\\u003e (a) shows the texture features of the grayscale symbiosis matrix for two sets of samples, with significant differences in the Mean features. There was no significant effect on the contrast. We also explored the correlation between clinical features and H\\u0026amp;E image features, as shown in Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig4\\\" class=\\\"InternalRef\\\"\\u003e4\\u003c/span\\u003e (b). The results show that some of the clinical features were related to the H\\u0026amp;E image features. For example, the Tumor stage had a strong correlation with the T stage, and Cecum and Ascending colon were at the primary site. Mean in the image features was more associated with tumor and T stages than with other clinical features and the Mean.\\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec16\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e3.4 H\\u0026amp;E staining images can be used to predict MSI in colorectal cancer and performed well.\\u003c/h2\\u003e \\u003cp\\u003eBased on the data of 360 CRC cases in TCGA, the results of the method presented here are better than those of Jakob. The methods and details of this paper were detailed above.\\u003c/p\\u003e \\u003cp\\u003eAs shown in Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig5\\\" class=\\\"InternalRef\\\"\\u003e5\\u003c/span\\u003e (a), we performed the dataset published by Jakob using its published training, tested data, and implemented parameters, with a result of 0.77. In the work of this paper, using the method described above, our mean AUC\\u0026thinsp;=\\u0026thinsp;0.79, which was higher than the results of Jakob.\\u003c/p\\u003e \\u003cp\\u003eIn addition, we conducted experiments combining images with clinical features of top5, top10, top15, and the result of clinical information of top5 was higher than that of the other two features. The specific methods were as follows: First, for the pathological images downloaded from the TCGA database, we divided the regions of interest into tiles of 512 \\u0026times; 512 px as the input to the CNN. Then, MCB fusion images and clinical features were used to obtain the training model on the training set. Finally, the validation was performed on the validation set. Ten experiments were repeated and averaged as the validation results. The ROC curves and the AUC are shown in Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig5\\\" class=\\\"InternalRef\\\"\\u003e5\\u003c/span\\u003e. Experimental results show that the mean AUC combining clinical information from top5 was 0.833, as shown in Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig5\\\" class=\\\"InternalRef\\\"\\u003e5\\u003c/span\\u003e(b), which is higher than the average AUC\\u0026thinsp;=\\u0026thinsp;0.793 in Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig5\\\" class=\\\"InternalRef\\\"\\u003e5\\u003c/span\\u003e(a) predicted by data separately from images. Moreover, the ResNet18 model performed better than Vgg19 (Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig5\\\" class=\\\"InternalRef\\\"\\u003e5\\u003c/span\\u003e(b)). Figure\\u0026nbsp;\\u003cspan refid=\\\"Fig5\\\" class=\\\"InternalRef\\\"\\u003e5\\u003c/span\\u003e(c) shows the accuracy of images combining the top5 clinical features on the ResNet18 model, accuracy, recall, and higher classification accuracy of F1 scores than individual image prediction, and the performance of images combined with top5 clinical features on the Vgg19 model. The results show that H\\u0026amp;E images combined with clinical information contributed to improving predictive power. The ROC curves based on the model ResNet18, the images combining the clinical features of top10, top15, and mean AUC are shown in the Supplementary Materials.\\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec17\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e3.5 Visualization of the pathological images\\u003c/h2\\u003e \\u003cp\\u003eTo better understand the reliability of the model and the algorithm, they were interpreted visually. In this work, the images were visualized using Gradient-weighted Class Activation Mapping (Grad-CAM). Grad-CAM assigns significant values to each neuron using the last convolutional layer of gradient information flowing into the CNN for specific attention decisions. As shown in Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig6\\\" class=\\\"InternalRef\\\"\\u003e6\\u003c/span\\u003e, the proposed algorithm can focus on the prediction of pathological images of cancer cell regions.\\u003c/p\\u003e \\u003c/div\\u003e\"},{\"header\":\"4. Discussion\",\"content\":\"\\u003cp\\u003eWe used the H\\u0026amp;E images and clinical information of 360 CRC cases (TCGA-CRC-DX) to evaluate the classification accuracy of multimodal MSI-H and MSS fusion prediction based on deep learning. Our results demonstrate the potential of this image feature to integrate clinical features as a tool to assess MSI status in clinical practice. Combining the qualitative characteristics of pathological images and the quantitative characteristics of clinical information could effectively classify MSI-H with MSS, and the model performance was somewhat improved. To the best of our knowledge, of all the reported pathological image studies predicting MSI status in CRC (TCGA-CRC-DX, FFPE diagnosis), our work featured the first predictive model to fuse pathological images and clinical features.\\u003c/p\\u003e \\u003cp\\u003eDeep learning has altered digital pathology and enabled the detection and typing of tumors [\\u003cspan additionalcitationids=\\\"CR47\\\" citationid=\\\"CR46\\\" class=\\\"CitationRef\\\"\\u003e46\\u003c/span\\u003e\\u0026ndash;\\u003cspan citationid=\\\"CR48\\\" class=\\\"CitationRef\\\"\\u003e48\\u003c/span\\u003e]. In CRC, previous studies automatically predicted MSI directly from WSIs stained with H\\u0026amp;E or divided full slides into small blocks to predict MSI and achieved certain results. However, in our experiments, as shown in Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig7\\\" class=\\\"InternalRef\\\"\\u003e7\\u003c/span\\u003e, the quality of the pathological images directly affected the predicted results. We found that mucinous adenocarcinoma, as well as the necrotic area, contained most of a small number of cells or even no cells, which would directly affect the prediction results. In this respect, predicting pathological organization based on deep learning is challenging.\\u003c/p\\u003e \\u003cp\\u003e \\u003c/p\\u003e \\u003cp\\u003eThere were limitations to this study. First, the retrospective nature of single-center studies may lead to the inevitable case selection bias, as well as the limited generality. Second, considering the high incidence of CRC, the cohort size was still small, which affected the general adaptation of the results of this study. A large-scale, prospective, multicenter study is needed to validate our results. Third, we did not perform a quality check on the image data. The quality of the H\\u0026amp;E images, including the folding, thickness, and necrotic areas, affected our predicted classification results. Therefore, it is important to perform a quality inspection of the images.\\u003c/p\\u003e\"},{\"header\":\"5. Conclusion\",\"content\":\"\\u003cp\\u003eThis study introduces a novel approach for multimodal data fusion to identify the MSI status of colorectal cancer by combining pathological image features and clinical data. The results of this research may substantially improve the clinical decision-making process for CRC treatment. Integrating diverse data sources shows promise for the advancement of personalized medicine in the context of colorectal cancer.\\u003c/p\\u003e\"},{\"header\":\"Declarations\",\"content\":\"\\u003cp\\u003e\\u003cstrong\\u003eAuthors\\u0026rsquo;contributions\\u003c/strong\\u003e\\u003c/p\\u003e\\n\\u003cp\\u003eJiasheng Yang, Peizhen Wang, and Jianjun He designed the study; Binsheng He, Wenjing Qiu, Bing Wang, Jingya Yang, Jinyang Mao and Geng Tian performed the study, analyzed the data and interpreted data; Wenjing Qiu wrote the manuscript; Jiasheng Yang, Bing Wang, Jingya Yang, Geng Tian and Peizhen Wang reviewed the manuscript.\\u0026nbsp;\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eCompeting interests\\u003c/strong\\u003e\\u003c/p\\u003e\\n\\u003cp\\u003eWenjing Qiu, Jingya Yang, Jinyang Mao and Geng Tian are employed in Geneis Beijing Co., Ltd., Beijing; other authors declare that they have no competing interests.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eFunding\\u003c/strong\\u003e\\u003c/p\\u003e\\n\\u003cp\\u003eThe study was partially supported by the Foundation of Hunan Educational Committee (Grant No. 19A060) and the provincial key R\\u0026amp;D projects of Hunan Provincial Science and Technology Department (No. 2022SK2074).And it was supported by the National Natural Science Foundation of China (No. 62172004), and Educational Commission of Anhui Province (No. KJ2019ZD05). And it was funded by the National Natural Science Foundation of China (number NO. 51574004，No. 62172004); Natural Science Foundation of the Higher Education Institutions of Anhui Province, China (number KJ2019A0085); Academic Foundation for Top Talents of the Higher Education Institutions of Anhui Province (number gxbjZD2016041) and Educational Commission of Anhui Province (No. KJ2019ZD05).\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eAcknowledgements\\u003c/strong\\u003e\\u003c/p\\u003e\\n\\u003cp\\u003eWe thank LetPub (www.letpub.com) for its linguistic assistance during the preparation of this manuscript.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eData availability statement\\u003c/strong\\u003e\\u003c/p\\u003e\\n\\u003cp\\u003ePublicly available datasets were analyzed in this study. These data can be found here: https://portal.gdc.cancer.gov/repository/.\\u0026nbsp;\\u003c/p\\u003e\"},{\"header\":\"References\",\"content\":\"\\u003col\\u003e\\u003cli\\u003e\\u003cspan\\u003eDekker, E., et al., \\u003cem\\u003eColorectal cancer\\u003c/em\\u003e. Lancet, 2019. 394(10207): p. 1467\\u0026ndash;1480.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003ePeng, P., et al., \\u003cem\\u003ePrognostic Factors in Stage IV Colorectal Cancer Patients With Resection of Liver and/or Pulmonary Metastases: A Population-Based Cohort Study\\u003c/em\\u003e. Front Oncol, 2022. 12: p. 850937.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eXiao, Y., et al., \\u003cem\\u003eInteraction between linc01615 and miR-491-5p regulates the survival and metastasis of colorectal cancer cells\\u003c/em\\u003e. Transl Cancer Res, 2020. 9(4): p. 2638\\u0026ndash;2647.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eFerlay, J., et al., \\u003cem\\u003eEstimating the global cancer incidence and mortality in 2018: GLOBOCAN sources and methods\\u003c/em\\u003e. Int J Cancer, 2019. 144(8): p. 1941\\u0026ndash;1953.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eYang, M., et al., \\u003cem\\u003eA multi-omics machine learning framework in predicting the survival of colorectal cancer patients\\u003c/em\\u003e. Comput Biol Med, 2022. 146: p. 105516.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eIonov, Y., et al., \\u003cem\\u003eUbiquitous somatic mutations in simple repeated sequences reveal a new mechanism for colonic carcinogenesis\\u003c/em\\u003e. Nature, 1993. 363(6429): p. 558\\u0026ndash;61.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eKawakami, H., A. Zaanan, and F.A. Sinicrope, \\u003cem\\u003eMicrosatellite instability testing and its role in the management of colorectal cancer\\u003c/em\\u003e. Curr Treat Options Oncol, 2015. 16(7): p. 30.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eEvrard, C., et al., \\u003cem\\u003eMicrosatellite Instability: Diagnosis, Heterogeneity, Discordance, and Clinical Impact in Colorectal Cancer\\u003c/em\\u003e. Cancers (Basel), 2019. 11(10).\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eLiu, W., et al., \\u003cem\\u003eSigmoid Colon Adenocarcinoma with Isolated Loss of PMS2 Presenting in a Patient with Synchronous Prostate Cancer with Intact MMR: Diagnosis and Analysis of the Family Pedigree\\u003c/em\\u003e. Anticancer Res, 2018. 38(8): p. 4847\\u0026ndash;4852.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eVasen, H.F., \\u003cem\\u003eClinical description of the Lynch syndrome [hereditary nonpolyposis colorectal cancer (HNPCC)]\\u003c/em\\u003e. Fam Cancer, 2005. 4(3): p. 219\\u0026ndash;25.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eMerok, M.A., et al., \\u003cem\\u003eMicrosatellite instability has a positive prognostic impact on stage II colorectal cancer after complete resection: results from a large, consecutive Norwegian series\\u003c/em\\u003e. Ann Oncol, 2013. 24(5): p. 1274\\u0026ndash;82.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003ePopat, S., R. Hubner, and R.S. Houlston, \\u003cem\\u003eSystematic review of microsatellite instability and colorectal cancer prognosis\\u003c/em\\u003e. J Clin Oncol, 2005. 23(3): p. 609\\u0026ndash;18.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eLe, D.T., et al., \\u003cem\\u003ePD-1 Blockade in Tumors with Mismatch-Repair Deficiency\\u003c/em\\u003e. N Engl J Med, 2015. 372(26): p. 2509\\u0026ndash;20.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eOverman, M.J., et al., \\u003cem\\u003eNivolumab in patients with metastatic DNA mismatch repair-deficient or microsatellite instability-high colorectal cancer (CheckMate 142): an open-label, multicentre, phase 2 study\\u003c/em\\u003e. Lancet Oncol, 2017. 18(9): p. 1182\\u0026ndash;1191.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eLudmir, E.B., et al., \\u003cem\\u003eTotal neoadjuvant therapy for rectal cancer: An emerging option\\u003c/em\\u003e. Cancer, 2017. 123(9): p. 1497\\u0026ndash;1506.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eTomasello, G., et al., \\u003cem\\u003eFOLFOXIRI Plus Bevacizumab as Conversion Therapy for Patients With Initially Unresectable Metastatic Colorectal Cancer: A Systematic Review and Pooled Analysis\\u003c/em\\u003e. JAMA Oncol, 2017. 3(7): p. e170278.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eTong, G., et al., \\u003cem\\u003eEffects of GLP-1 Receptor Agonists on Biological Behavior of Colorectal Cancer Cells by Regulating PI3K/AKT/mTOR Signaling Pathway\\u003c/em\\u003e. Front Pharmacol, 2022. 13: p. 901559.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eLiang, W., et al., \\u003cem\\u003eKRT17 Promotes T-lymphocyte Infiltration Through the YTHDF2\\u0026ndash;CXCL10 Axis in Colorectal Cancer\\u003c/em\\u003e. Cancer Immunology Research, 2023. 11(7): p. 875\\u0026ndash;894.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eFischer, F., K. Baerenfaller, and J. Jiricny, \\u003cem\\u003e5-Fluorouracil is efficiently removed from DNA by the base excision and mismatch repair systems.\\u003c/em\\u003e Gastroenterology, 2007. 133(6): p. 1858-68.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eChalabi, M., et al., \\u003cem\\u003eNeoadjuvant immunotherapy leads to pathological responses in MMR-proficient and MMR-deficient early-stage colon cancers\\u003c/em\\u003e. Nat Med, 2020. 26(4): p. 566\\u0026ndash;576.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eDiagnosis and C. Treatment Guidelines For Colorectal Cancer Working Group, \\u003cem\\u003eChinese Society of Clinical Oncology (CSCO) diagnosis and treatment guidelines for colorectal cancer 2018 (English version)\\u003c/em\\u003e. Chin J Cancer Res, 2019. 31(1): p. 117\\u0026ndash;134.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eYuan, L., et al., \\u003cem\\u003eImmunohistochemistry and microsatellite instability analysis in molecular subtyping of colorectal carcinoma based on mismatch repair competency\\u003c/em\\u003e. Int J Clin Exp Med, 2015. 8(11): p. 20988\\u0026ndash;1000.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eHe, B., et al., \\u003cem\\u003eA New Method for CTC Images Recognition Based on Machine Learning\\u003c/em\\u003e. Front Bioeng Biotechnol, 2020. 8: p. 897.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eXie, X., et al., \\u003cem\\u003eEvaluating Cancer-Related Biomarkers Based on Pathological Images: A Systematic Review\\u003c/em\\u003e. Front Oncol, 2021. 11: p. 763527.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eYi, X., et al., \\u003cem\\u003eAdrenal incidentaloma: machine learning-based quantitative texture analysis of unenhanced CT can effectively differentiate sPHEO from lipid-poor adrenal adenoma\\u003c/em\\u003e. J Cancer, 2018. 9(19): p. 3577\\u0026ndash;3582.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eLi, J., et al., \\u003cem\\u003ePredicting gastric cancer tumor mutational burden from histopathological images using multimodal deep learning\\u003c/em\\u003e. Brief Funct Genomics, 2023.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eHuang, K., et al., \\u003cem\\u003ePredicting colorectal cancer tumor mutational burden from histopathological images and clinical information using multi-modal deep learning\\u003c/em\\u003e. Bioinformatics, 2022: p. btac641.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eJi, M.Y., et al., \\u003cem\\u003eNuclear shape, architecture and orientation features from H\\u0026amp;E images are able to predict recurrence in node-negative gastric adenocarcinoma\\u003c/em\\u003e. J Transl Med, 2019. 17(1): p. 92.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eLuo, X., et al., \\u003cem\\u003eComprehensive Computational Pathological Image Analysis Predicts Lung Cancer Prognosis\\u003c/em\\u003e. J Thorac Oncol, 2017. 12(3): p. 501\\u0026ndash;509.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eYang, J., et al., \\u003cem\\u003ePrediction of HER2-positive breast cancer recurrence and metastasis risk from histopathological images and clinical information via multimodal deep learning\\u003c/em\\u003e. Comput Struct Biotechnol J, 2022. 20: p. 333\\u0026ndash;342.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eYao, Y., et al., \\u003cem\\u003eICSDA: a multi-modal deep learning model to predict breast cancer recurrence and metastasis risk by integrating pathological, clinical and gene expression data\\u003c/em\\u003e. Briefings in Bioinformatics, 2022: p. bbac448c.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eYe, Z., et al., \\u003cem\\u003eCervical Cancer Metastasis and Recurrence Risk Prediction Based on Deep Convolutional Neural Network\\u003c/em\\u003e. Current Bioinformatics, 2022. 17(2): p. 164\\u0026ndash;173.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eLiu, X., et al., \\u003cem\\u003ePredicting breast cancer recurrence and metastasis risk by integrating color and texture features of histopathological images and machine learning technologies\\u003c/em\\u003e. Comput Biol Med, 2022. 146: p. 105569.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eQiu, W., et al., \\u003cem\\u003eEvaluating the Microsatellite Instability of Colorectal Cancer Based on Multimodal Deep Learning Integrating Histopathological and Molecular Data\\u003c/em\\u003e. Front Oncol, 2022. 12: p. 925079.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eKather, J.N., et al., \\u003cem\\u003eDeep learning can predict microsatellite instability directly from histology in gastrointestinal cancer\\u003c/em\\u003e. Nat Med, 2019. 25(7): p. 1054\\u0026ndash;1056.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eKe, J., et al. \\u003cem\\u003eIdentifying patch-level MSI from histological images of Colorectal Cancer by a Knowledge Distillation Model\\u003c/em\\u003e. in 2020 \\u003cem\\u003eIEEE International Conference on Bioinformatics and Biomedicine (BIBM)\\u003c/em\\u003e. 2020.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eWu, Z., et al., \\u003cem\\u003eDeepLRHE: A Deep Convolutional Neural Network Framework to Evaluate the Risk of Lung Cancer Recurrence and Metastasis From Histopathology Images\\u003c/em\\u003e. Front Genet, 2020. 11: p. 768.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eMacenko, M., et al. \\u003cem\\u003eA method for normalizing histology slides for quantitative analysis\\u003c/em\\u003e. in \\u003cem\\u003eProceedings of the 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, Boston, MA, USA, June 28 - July 1, 2009\\u003c/em\\u003e. 2009.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eVahadane, A., et al. \\u003cem\\u003eStructure-preserved color normalization for histological images\\u003c/em\\u003e. in \\u003cem\\u003eBiomedical Imaging (ISBI), 2015 IEEE 12th International Symposium on\\u003c/em\\u003e. 2015.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003ePrusa, J., et al. \\u003cem\\u003eUsing Random Undersampling to Alleviate Class Imbalance on Tweet Sentiment Data\\u003c/em\\u003e. in \\u003cem\\u003eIEEE International Conference on Information Reuse \\u0026amp; Integration\\u003c/em\\u003e. 2015.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eHe, K., et al., \\u003cem\\u003eDeep Residual Learning for Image Recognition\\u003c/em\\u003e. IEEE, 2016.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eRussakovsky, O., et al., \\u003cem\\u003eImageNet Large Scale Visual Recognition Challenge\\u003c/em\\u003e. International Journal of Computer Vision, 2015. 115(3): p. 211\\u0026ndash;252.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eBel, T.D., et al. \\u003cem\\u003eAutomatic segmentation of histopathological slides of renal tissue using deep learning\\u003c/em\\u003e. in \\u003cem\\u003eDigital Pathology\\u003c/em\\u003e. 2018.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eWang, D., et al., Deep Learning for Identifying Metastatic Breast Cancer. 2016.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eYang, G., et al. \\u003cem\\u003eCompact Bilinear Pooling\\u003c/em\\u003e. in \\u003cem\\u003e2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)\\u003c/em\\u003e. 2016.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eKather, J.N., et al., \\u003cem\\u003ePan-cancer image-based detection of clinically actionable genetic alterations\\u003c/em\\u003e. Nat Cancer, 2020. 1(8): p. 789\\u0026ndash;799.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eSchmauch, B., et al., \\u003cem\\u003eA deep learning model to predict RNA-Seq expression of tumours from whole slide images\\u003c/em\\u003e. Nat Commun, 2020. 11(1): p. 3877.\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eYu, F., et al., \\u003cem\\u003ePan-cancer computational histopathology reveals mutations, tumor composition and prognosis\\u003c/em\\u003e. Nature Cancer, 2020. 1(8): p. 1\\u0026ndash;11.\\u003c/span\\u003e\\u003c/li\\u003e\\u003c/ol\\u003e\"}],\"fulltextSource\":\"\",\"fullText\":\"\",\"funders\":[],\"hasAdminPriorityOnWorkflow\":false,\"hasManuscriptDocX\":true,\"hasOptedInToPreprint\":true,\"hasPassedJournalQc\":\"\",\"hasAnyPriority\":false,\"hideJournal\":true,\"highlight\":\"\",\"institution\":\"\",\"isAcceptedByJournal\":false,\"isAuthorSuppliedPdf\":false,\"isDeskRejected\":\"\",\"isHiddenFromSearch\":false,\"isInQc\":false,\"isInWorkflow\":false,\"isPdf\":false,\"isPdfUpToDate\":true,\"isWithdrawnOrRetracted\":false,\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"researchsquare\",\"isNatureJournal\":false,\"hasQc\":true,\"allowDirectSubmit\":true,\"externalIdentity\":\"\",\"sideBox\":\"\",\"snPcode\":\"\",\"submissionUrl\":\"/submission\",\"title\":\"Research Square\",\"twitterHandle\":\"researchsquare\",\"acdcEnabled\":true,\"dfaEnabled\":false,\"editorialSystem\":\"\",\"reportingPortfolio\":\"\",\"inReviewEnabled\":false,\"inReviewRevisionsEnabled\":true},\"keywords\":\"microsatellite instability, colorectal cancer, deep learning, multimodal fusion, convolutional neural network\",\"lastPublishedDoi\":\"10.21203/rs.3.rs-4200523/v1\",\"lastPublishedDoiUrl\":\"https://doi.org/10.21203/rs.3.rs-4200523/v1\",\"license\":{\"name\":\"CC BY 4.0\",\"url\":\"https://creativecommons.org/licenses/by/4.0/\"},\"manuscriptAbstract\":\"\\u003cp\\u003eMicrosatellite instability (MSI) arises from defective DNA mismatch repair (MMR) systems and is prevalent in various cancer types. MSI is classified as MSI-High (MSI-H), MSI-Low (MSI-L), or Microsatellite Stable (MSS), with the latter two occasionally combined into a single designation called MSI-L/MSS. Identifying the MSI status (i.e., MSI-H vs. MSI-L/MSS) in colorectal cancer (CRC) is critical for guiding immunotherapy and assessing prognosis. Conventional molecular tests for MSI are expensive, time-consuming, and limited by experimental conditions. Advancements in MSI detection have been made using deep learning methods with histopathological images, yet efforts to improve MSI detection's predictive accuracy by integrating histopathological images and clinical data remain limited. This study initially analyzed clinical information variation between the MSI-H and MSI-L/MSS groups, discovering significant differences in cancer stages N and M. Subsequently, texture features were extracted using the Gray-level co-occurrence matrix (GLCM) from both groups, disclosing noteworthy disparities in mean feature information. Finally, a multimodal compact bilinear pool (MCB) was employed to merge histopathological images with clinical data. By applying this analysis framework to the cancer genome atlas (TCGA) CRC data, a prediction area under the curve (AUC) of 0.833 was achieved through 5-fold cross-validation in predicting MSI status. The results demonstrated higher accuracy in determining MSI compared to existing unimodal MSI prediction methods and other contemporary techniques. Additionally, significant regions in whole-slide images (WSI) for determining MSI labels were visualized. To summarize, this study presents an accurate multimodal deep learning model for predicting microsatellite instability in colorectal cancer by integrating histopathological images and clinical data, together with a method to visualize important regions in WSI to determine MSI status.\\u003c/p\\u003e\",\"manuscriptTitle\":\"Development of a Multimodal Deep Learning Model for Predicting Microsatellite Instability in Colorectal Cancer by Integrating Histopathological Images and Clinical Data\",\"msid\":\"\",\"msnumber\":\"\",\"nonDraftVersions\":[{\"code\":1,\"date\":\"2024-05-03 21:24:09\",\"doi\":\"10.21203/rs.3.rs-4200523/v1\",\"editorialEvents\":[{\"type\":\"communityComments\",\"content\":0}],\"status\":\"published\",\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"researchsquare\",\"isNatureJournal\":false,\"hasQc\":true,\"allowDirectSubmit\":true,\"externalIdentity\":\"\",\"sideBox\":\"\",\"snPcode\":\"\",\"submissionUrl\":\"/submission\",\"title\":\"Research Square\",\"twitterHandle\":\"researchsquare\",\"acdcEnabled\":true,\"dfaEnabled\":false,\"editorialSystem\":\"\",\"reportingPortfolio\":\"\",\"inReviewEnabled\":false,\"inReviewRevisionsEnabled\":true}}],\"origin\":\"\",\"ownerIdentity\":\"a9759a70-34f2-444c-a6cc-8b78333a7079\",\"owner\":[],\"postedDate\":\"May 3rd, 2024\",\"published\":true,\"recentEditorialEvents\":[],\"rejectedJournal\":[],\"revision\":\"\",\"amendment\":\"\",\"status\":\"posted\",\"subjectAreas\":[{\"id\":31449518,\"name\":\"Biological sciences/Cancer\"},{\"id\":31449519,\"name\":\"Biological sciences/Computational biology and bioinformatics\"}],\"tags\":[],\"updatedAt\":\"2024-06-26T11:49:32+00:00\",\"versionOfRecord\":[],\"versionCreatedAt\":\"2024-05-03 21:24:09\",\"video\":\"\",\"vorDoi\":\"\",\"vorDoiUrl\":\"\",\"workflowStages\":[]},\"version\":\"v1\",\"identity\":\"rs-4200523\",\"journalConfig\":\"researchsquare\"},\"__N_SSP\":true},\"page\":\"/article/[identity]/[[...version]]\",\"query\":{\"redirect\":\"/article/rs-4200523\",\"identity\":\"rs-4200523\",\"version\":[\"v1\"]},\"buildId\":\"8U1c8b4HqxoKbykW_rLl7\",\"isFallback\":false,\"isExperimentalCompile\":false,\"dynamicIds\":[84888],\"gssp\":true,\"scriptLoader\":[]}","source_license":"CC-BY-4.0","license_restricted":false}