ResSAT: Enhancing Spatial Transcriptomics Prediction from H&E- Stained Histology Images with Interactive Spot Transformer

ResSAT: Enhancing Spatial Transcriptomics Prediction from H&E- Stained Histology Images with Interactive Spot Transformer | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article ResSAT: Enhancing Spatial Transcriptomics Prediction from H&E- Stained Histology Images with Interactive Spot Transformer Anqi Liu, Yue Zhao, Hui Shen, Zhengming Ding, Hong-Wen Deng This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4707959/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 9 You are reading this latest preprint version Abstract Spatial transcriptomics (ST) revolutionizes RNA quantification with high spatial resolution. Hematoxylin and eosin (H&E) images, the gold standard in medical diagnosis, offer insights into tissue structure, correlating with gene expression patterns. Current methods for predicting spatial gene expression from H&E images often overlook spatial relationships. We introduce ResSAT (Residual networks - Self-Attention Transformer), a framework generating spatially resolved gene expression profiles from H&E images by capturing tissue structures and using a self-attention transformer to enhance prediction. Benchmarking on 10x Visium datasets, ResSAT significantly outperformed existing methods, promising reduced ST profiling costs and rapid acquisition of numerous profiles. Spatial transcriptomics Hematoxylin and eosin images transformer Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Background The rapid advancement of spatial transcriptomics (ST) technology has revolutionized the field of RNA abundance quantification by offering remarkable spatial resolution for concurrent gene expression profiling and precise spatial spot localization 1 . This breakthrough allows researchers to generate detailed maps of gene expression within tissues, providing unprecedented insights into cellular function and tissue organization 2 . However, currently, the resource-intensive and time-consuming nature of ST profiling has limited its widespread application, creating a demand for more accessible methods. In contrast, Hematoxylin and eosin (H&E) staining is a widely used histological technique that provides detailed insights into tissue structure and composition at a microscopic level 3 , 4 . H&E images are instrumental in medical diagnostics, offering clear visualizations of cellular morphology and tissue architecture 5 . As the gold standard in many diagnostic procedures, H&E imaging is cost-effective, readily available, and extensively utilized in clinical settings. Moreover, there is a close relationship between H&E staining and ST, as H&E images capture detailed cellular and tissue morphology that correlates with gene expression patterns 6 . The staining highlights different cellular components: hematoxylin stains the cell nuclei blue, indicating areas of high DNA concentration, while eosin stains the cytoplasm and extracellular matrix pink, showing the structural context 7 . These visual cues from H&E images reflect the underlying biological processes and molecular activities within the tissue 3 , 4 , providing a spatial map that can be linked to gene expression data. Previous studies have also demonstrated that changes in gene expression or genetic mutations often influence cell morphology, structure, and distribution, resulting in alterations in histological features 8 , 9 ​. This correlation enables the use of H&E images to predict spatial gene expression profiles, leveraging the morphological context provided by the staining to infer molecular states. Recognizing the potential to integrate these two powerful tools, recent studies 10 – 12 have focused on developing computational approaches to predict ST data from H&E images. This innovative synergy aims to leverage the comprehensive tissue insights provided by H&E images to infer spatial gene expression profiles. By doing so, these approaches can potentially circumvent the limitations of ST technology, making high-resolution gene expression mapping more accessible and practical. Several existing approaches, such as ST-Net 6 , HisToGene 11 , and BLEEP 12 , have shown promising results in predicting expression from histology images. Both ST-Net and HisToGene treat expression prediction as regression tasks and train them in a feed-forward manner. ST-Net utilizes a ResNet50 image encoder, while HisToGene utilizes a vision transformer backbone. BLEEP, on the other hand, draws inspiration from contrastive language-image pretraining to establish a comparable joint embedding between spot expression profiles and their spatially paired image patches. Although HisToGene incorporates spatial location information, it does not explicitly consider spatial relationships between different locations. It's worth highlighting existing methods capable of generating spatially resolved expression predictions, either have limitations regarding the predicted panel (ST-Net, HisToGene, and BLEEP) or are prone to overfitting (HisToGene). Additionally, existing approaches in spatial expression prediction from H&E images often overlook the crucial relationships between different spatial locations, which provide essential context about tissue architecture, including the organization and interaction of cells, and spatial heterogeneity, and thus reflect variations in cellular composition and functions across different tissue regions. These factors may significantly impact biological interpretation and predictive accuracy 13 . To address these challenges, we proposed a novel approach called ResSAT (Residual networks - Self-Attention Transformer) for predicting spatial transcriptomics profiles using H&E-stained histology images. We utilized a ResNet50 architecture to extract comprehensive image features from the H&E images, enabling our model to capture diverse characteristics of tissue structures and cellular compositions depicted in the images. Additionally, we introduced a self-attention transformer mechanism to identify and cluster spots with high correlation, allowing the model to focus on interactions between spots and enhance spatial gene expression prediction performance. We validated the effectiveness of ResSAT by benchmarking its performance on two different mice brain datasets obtained from the 10x Visium platform. Our results demonstrated significant improvements over existing methods such as BLEEP 12 , HisToGene 11 , and ST-Net 10 in terms of mean correlations across all genes and mean correlations among the top 50 highly expressed genes. This innovative approach significantly enhances the performance of spatial gene expression prediction from histology images. The proposed framework has the potential to substantially reduce the time and cost associated with spatial transcriptomics profiling, opening up new possibilities for acquiring numerous spatial transcriptomics profiles rapidly and reconstructing comprehensive 3D spatial transcriptomics from adjacent 2D spatial transcriptomics profiles. Results ResSAT enables gene expression prediction and consistently performs well. To assess the predictive efficacy of ResSAT in quantifying gene expression, we applied our method to both the sagittal anterior (SA) and sagittal posterior (SP) datasets. We compared ResSAT to three other methods for spatial gene expression prediction, including BLEEP, HisToGene, and ST-Net. The predicted expression profiles from ResSAT showed the highest mean correlation with ground truth, achieving an increase in PCCs for both mean correlations of all genes (Table 1 ) and top 50 most highly expressed genes (HEGs) (Table 2 ) in the two different datasets. To ensure robustness and reliability, the evaluations were repeated five times, and the resulting mean and standard deviation values were computed for further analysis and validation. Table 1 Mean correlations of all genes in predicted expression compared to ground truth expressions on held out datasets (Section 1). Methods SA SP ResSAT 0.2218 ± 0.0055 0.2437 ± 0.0026 BLEEP 0.0115 ± 0.0038 0.0091 ± 0.0054 HistoGene 0.1216 ± 0.0020 0.1215 ± 0.0016 ST-Net 0.0239 ± 0.0032 0.0340 ± 0.0061 Note: mean ± standard deviation Table 2 Mean correlations of top 50 most highly expressed genes (HEGs) in predicted expression compared to ground truth expressions on held out datasets (Section 1). Methods SA SP ResSAT 0.3485 ± 0.0039 0.4003 ± 0.0252 BLEEP 0.0297 ± 0.0249 -0.0306 ± 0.0193 HistoGene -0.0506 ± 0.0351 -0.0174 ± 0.04469 ST-Net 0.0774 ± 0.0043 0.0611 ± 0.0083 Note: mean ± standard deviation Respectively, for each section within the SA and SP datasets, we trained ResSAT using one section and evaluated the correlation between predicted gene expression and actual gene expression on the other section. Specifically, after training on Section 2 and testing on Section 1 as shown in the Tables 1 and 2 , we also trained on Section 1 and tested on Section 2 to validate the results. As illustrated in Fig. 2 , ResSAT consistently yielded the highest PCCs between spatially resolved gene expression and actual gene expression across all sections in both datasets. Examining the effect of each module within ResSAT on the predicted gene expression results. In order to better understand why ResSAT performs better than other methods, we conducted an analysis to see how each component contributes to its performance. We did this by removing certain modules of ResSAT and observing the impact on its ability to predict gene expression, as outlined in Figs. 3 and 4 . We found that keeping all modules intact resulted in the strongest correlation between predicted and observed gene expression. Specifically, when we excluded the ResNet module and SAT module respectively, we noticed an average PCC decrease in performance in both the SA and SP datasets. This indicates that the ResNet module and SAT module are both crucial for ResSAT to effectively uncover the relationships between different spots. In summary, our ablation experiment highlights the importance of maintaining all modules within ResSAT to achieve optimal performance in predicting gene expression. ResSAT accurately predicts brain-related genes. We examined whether the predicted gene expression by ResSAT accurately mirrors the actual expression of brain-related genes. Within the SA dataset, we assessed the correlation between observed and predicted gene expression, computing the PCC for each gene. We then ranked these genes in descending order of their PCCs and selected the predicted genes with the top 5 highest correlations obtained from our method for visualization ( CALB2, GNG4, CDHR1 , DOC2G , and SHISA8 ), as shown in Fig. 5 . CALB2 (Calretinin) expression in the mouse olfactory bulb is associated with inhibitory interneurons 32 . These interneurons play essential roles in regulating neural circuits involved in processing sensory information related to odors. GNG4 (G Protein Subunit Gamma 4) is a gene encoding a subunit of heterotrimeric G proteins, which are involved in signal transduction pathways in neurons 33 . In the mouse olfactory bulb, G proteins play a crucial role in mediating signaling pathways involved in odorant detection and processing. Heterotrimeric G proteins, consisting of alpha, beta, and gamma subunits, are involved in transducing signals from odorant receptors to downstream effector molecules, leading to neuronal activation and olfactory perception 33 . CDHR1 (Cadherin-Related Family Member 1) is a gene expressed in the olfactory bulb of mice, where it likely contributes to the organization and maintenance of the olfactory sensory epithelium 34 , 35 . DOC2G , also known as Double C2 Domain Gamma, is a gene encoding a calcium-binding protein involved in vesicle exocytosis and neurotransmitter release. In the mammalian olfactory system, complex information processing starts in the olfactory bulb, whose output is conveyed by mitral cells (MCs) and tufted cells (TCs) 36 . DOC2G was identified to be differentially expressed between MCs and TCs of the mouse 36 . SHISA8 , a member of the Shisa family of transmembrane proteins, has a broad role in synaptic function and neuronal development, suggesting its potential involvement in olfactory processing 37 . To further validate the robustness of ResSAT in predicting brain-related genes, we extended our analysis to the SP dataset, achieving similarly accurate predictions, as shown in Fig. 6 . ResSAT enabled accurate prediction of key genes associated with mouse brain. Figure 6 shows the top 5 genes ( NRGN, CTXN1, PCP2, NNAT , and CAMK2A ) in the SA dataset predicted by ResSAT. NRGN (Neurogranin) is a gene encoding a protein found primarily in the brain, specifically in dendritic spines of neurons. It is involved in regulating synaptic plasticity and learning processes by modulating the function of calmodulin, a calcium-binding protein. NRGN has been implicated in various neurological disorders, including Alzheimer's disease and schizophrenia 38 . CTXN1 (Cortexin-1) is a gene encoding a protein involved in neuronal development and synaptic function, especially highly expressed in cerebral cortex 39 . PCP2 (Purkinje Cell Protein 2) is a gene expressed primarily in Purkinje cells of the cerebellum 40 . It encodes a protein involved in dendritic development, synaptic transmission, and calcium signaling within Purkinje cells. Mutations in PCP2 have been linked to certain neurodevelopmental disorders. NNAT (Neuronatin) is a gene encoding a protein expressed in the brain, particularly in neurons, where it regulates neuronal development and function. It is involved in processes such as neuronal differentiation, synaptogenesis, and neurotransmitter release 41 . NNAT has been implicated in neurological disorders and metabolic regulation 42 . CAMK2A (Calcium/Calmodulin-Dependent Protein Kinase II Alpha) is a gene encoding a protein kinase involved in calcium signaling and synaptic plasticity. It plays a crucial role in neuronal excitability, synaptic transmission, and learning and memory processes. Dysregulation of CAMK2A has been implicated in various neurological disorders, including Alzheimer's disease and epilepsy 43 . These genes are known for their critical roles in neuronal function and brain development, reinforcing the model's capability across different brain regions and datasets. This consistency underscores ResSAT's reliability in predicting key genetic markers relevant to mouse brains. Discussion To address the challenges inherent in spatial transcriptomics prediction, we devised a novel approach called ResSAT. Leveraging a ResNet50 architecture, we extracted comprehensive image features from provided H&E images. This method enabled our model to capture a wide range of tissue structures and cellular compositions depicted in the images. We then introduced a self-attention transformer mechanism to cluster spots exhibiting high correlation. This empowered the model to focus on interactions between spots, thereby enhancing spatial gene expression prediction performance. In our experimental evaluations on benchmark ST datasets, our proposed method demonstrated its effectiveness in accurately predicting spatial gene expression patterns in two different mice brain datasets. We achieved significantly higher mean correlations across all genes and the top 50 most highly expressed genes (HEGs), representing substantial improvements compared to existing methods. Additionally, ResSAT exhibited similar expression patterns for the top 5 predicted genes compared to observed expression profiles. The results accurately predicted spatial correlations of genes, with the locations of predicted genes closely matching the spatial locations of observed genes. This underscores the efficacy of our approach in spatial transcriptomics prediction from H&E images, indicating its potential to generate numerous spatial transcriptomics profiles efficiently. Our focus in this paper primarily centers on mouse brain datasets, laying the groundwork for our subsequent 3D reconstructed map of brain regions in spatial transcriptomics. Despite the successful prediction of specific genes showcased in Figs. 5 and 6 , we acknowledge that the overall absolute correlations in Tables 1 and 2 remain low, indicating the challenging nature of the prediction task for the majority of genes. The low scores may stem from various factors, including the weak correlation between the expression of certain genes and morphological features, limitations in the detection of certain genes by the Visium platform resulting in less predictable expression, and the presence of experimental artifacts introducing non-biological variability into the data, independent of the image. Due to the relatively small number of tissue sections available, neither ResSAT nor other existing methods can reliably predict gene expression with high accuracy. However, ResSAT still demonstrates superior prediction accuracy compared to other methods. While the reliance on a relatively large training set poses a potential limitation for deep learning-based models, we anticipate that as more training ST data become available in the near future, ResSAT's performance and robustness can be further enhanced. Conclusion We introduced ResSAT, a novel framework for predicting spatial gene expression profiles from H&E-stained histology images using a ResNet50 architecture and a self-attention transformer mechanism. ResSAT effectively captures tissue structures and clusters correlated spots to enhance prediction performance. Our evaluations on mouse brain datasets demonstrated significant improvements over existing methods, with higher mean correlations and accurate spatial predictions. Despite challenges like low absolute correlations for certain genes and limited tissue sections, ResSAT outperformed current methods, showing potential for efficient and cost-effective ST profiling. Future availability of more training data is expected to further enhance ResSAT's performance and robustness, advancing the field of spatial transcriptomics. Methods Datasets: Two mouse brain datasets (SA and SP). ST profiling of the anterior part and posterior part of the mouse brain tissue sagittal sections was generated with the Visium technology from 10x Genomics 14 – 17 . Both datasets consist of serial H&E histology images and paired gene expressions at the spatial spots and their coordinates. The analyzed gene expression count matrices are outputs of the SpaceRanger pipeline 18 . The anterior sagittal dataset comprises two sets of H&E-stained histology images, alongside corresponding spatial gene expression data 14 , 15 . In slice 1, the spatial transcriptomics (ST) data encapsulates the expression profiles of 31,053 genes across 2,825 spots, given by read counts. Slice 2 features ST data for 31,053 genes across 2,696 spots, also detailed through read counts. The posterior sagittal dataset includes two collections of H&E-stained histology images and their associated spatial gene expression data 16 , 17 . For the first slice, spatial transcriptomics (ST) analysis reveals the expression levels of 32,285 genes across 3,355 spots, given by read count data. The second slice presents ST information for the same number of genes, but across 3,289 spots, with expression quantified similarly through read counts. Data preprocessing For the H&E histology images, we extracted patches based on the size and location of each spot. Each patch was a 224x224 image centered around a spot, approximately 55µm on each side, and paired with the spot's corresponding gene expression profile. For the spatial gene expression profiles of each tissue section, each spot was normalized to the total count and log normalized. The union of the top 1,000 most highly variable genes from each of the slices was used for training and prediction. Finally, the expression data of these samples were batch corrected using Harmony 19 before one of the slices was randomly selected to be held out for testing. For these two datasets, the slice 2 was selected to be held out for training, while the slice 1 was selected to be held out for testing. Learning image embedding for expression prediction. Residual networks (ResNets) 20 are widely recognized architectures in image classification, initially acclaimed for their significant advancement upon introduction. They continue to be a benchmark in various image analyses 21 – 23 and serve as baselines in image studies introducing novel architectures 24 , 25 . Our focus in this paper was to extract highly representative features from H&E images using ResNet50. During the training phase, we utilized a dataset comprising \(\:N\) pairs of H&E images and their corresponding gene expressions. Each image patch was represented as a 3D tensor \(\:{\mathbf{X}}_{i}\in\:{\mathbb{R}}^{3\times\:W\times\:H}\) , where \(\:N\) and \(\:H\) denoted the width and height of the patch, respectively. The associated gene expression \(\:{\mathbf{y}}_{i}\) was represented as a \(\:d\) -dimensional vector in \(\:{\mathbb{R}}^{d}\) . Our objective was to develop a deep learning framework capable of accurately predicting gene expression from a given image patch. We conceptualized our model as comprising two main components: a backbone network, denoted as \(\:\mathcal{F}\left(\bullet\:\right)\) , for initial feature extraction module, followed by a feature refinement module, \(\:\mathcal{G}\left(\bullet\:\right)\) , to enhance the predictive capability of the extracted features. To optimize the performance of both modules, we employed the Mean Squared Error (MSE) loss function during the training process as follows: $$\:{\mathcal{L}}_{mse}=\sum\:_{i=1}^{N}{‖{\mathbf{y}}_{i}-\mathcal{G}\left(\mathcal{F}\left({\mathbf{X}}_{i}\right)\right)‖}_{{\mathcal{l}}_{2}}$$ , where \(\:{\mathcal{l}}_{2}\) denotes the L -2 norm to measure the difference between predicted gene expression and ground-truth one. Spot-Interaction Discovery. The emergence of transformers has led to their widespread application in image and omics data analysis, as demonstrated by numerous studies 26 – 29 . Particularly, recent transformer-based architectures have drawn attention to self-attention and cross-attention mechanisms, offering means to capture interdependencies between different input modalities 30 , 31 . In the context of our spatially resolved gene expression prediction approach, the Self-Attention Transformer (SAT) plays a crucial role in exploring spot-spot interactions. By clustering spots that exhibit high correlation, SAT enhances our model's ability to represent gene expression accurately. This approach facilitates a deeper understanding of spatial relationships within the data, contributing to improved predictive performance. In our endeavor to analyze H&E images for spot interaction, we proposed the development of a spot-interaction module, as shown in Fig. 1 . This module aimed to refine predictions of gene expression. To streamline the model and minimize the number of parameters, we introduced a non-parametric attentive module. The operation of this attention module was defined by the following Equation: $$\:\mathcal{A}\left(Q,\:K,\:V\right)=\mathbf{s}\mathbf{o}\mathbf{f}\mathbf{t}\mathbf{m}\mathbf{a}\mathbf{x}\left(\frac{Q{K}^{\text{T}}}{\sqrt{d}}\right)V$$ , where \(\:K\) , \(\:Q\) , and \(\:V\) represent the key, query, and value matrices of a Transformer module. To model the spot interaction, we aimed to learn a compact gene expression correlation between the spots in the training set. Here, \(\:{\mathbf{X}}_{i}\) represented the input features, and \(\:\mathcal{F}\left(\bullet\:\right)\) and \(\:\mathcal{G}\left(\bullet\:\right)\) denoted transformation functions. The aim of this formulation was to explore the matrix of spot-spot interactions. By doing so, it clustered spots that exhibit high correlation, thereby enhancing the robustness of the gene expression representation. Moreover, this approach elucidated the correlation among spots, improving the model’s inference capabilities. To enhance the model’s learning process by integrating knowledge of ground-truth gene expressions, we introduced a secondary MSE loss function. This additional MSE loss was formulated as follows: $$\:{\mathcal{L}}_{mse}^{{\prime\:}}=\sum\:_{i=1}^{N}{‖{\mathbf{y}}_{i}-\mathcal{A}\left(\mathcal{G}\left(\mathcal{F}\left({\mathbf{X}}_{i}\right)\right)\right)‖}_{{\mathcal{l}}_{2}}$$ . Integrating dual objectives into a unified framework, we defined the overall loss function as follows: $$\:\mathcal{L}={{\mathcal{L}}_{mse}+\mathcal{L}}_{mse}^{{\prime\:}}$$ . This combined loss framework was designed to enhance the model’s predictive performance by balancing the feature extraction and interactive feature refinement. By carefully summing up the contribution of the primary and secondary MSE losses, the model was steered to pay detailed attention to the subtleties and complexities of gene expression data. This, in turn, was expected to improve the model’s capacity for capturing the intricate biological relationships that were represented within H&E images. Evaluated Metrics In this study, we employed the Pearson correlation coefficient (PCC) to measure the spatial gene expression predicted by ResSAT with the observed gene expression, in order to assess their level of correlation. The PCC had a range of values between − 1 and 1. It was determined by dividing the covariance of two variables by the product of their individual standard deviations: $$\:\text{P}\text{C}\text{C}=\:\frac{\text{C}\text{o}\text{v}({\text{y}}_{\text{t}\text{r}\text{u}\text{e}},\:{\text{y}}_{\text{p}\text{r}\text{e}\text{d}})}{{\sigma\:}\left({\text{y}}_{\text{t}\text{r}\text{u}\text{e}}\right)\bullet\:{\sigma\:}\left({\text{y}}_{\text{p}\text{r}\text{e}\text{d}}\right)}$$ , where \(\:\text{C}\text{o}\text{v}(\bullet\:,\:\bullet\:)\) denotes covariance; \(\:{\text{y}}_{\text{t}\text{r}\text{u}\text{e}}\) and \(\:{\text{y}}_{\text{p}\text{r}\text{e}\text{d}}\) represents the original gene expression and the gene expression obtained by prediction, respectively. \(\:{\sigma\:}\left(\bullet\:\right)\) means standard deviation. We computed the mean correlation of all genes as follows: $$\:\stackrel{-}{R}=\frac{1}{k}{\sum\:}_{i=1}^{k}\mathbf{C}\mathbf{o}\mathbf{r}\mathbf{r}\left({\mathbf{y}}_{\text{true}}^{i},{\mathbf{y}}_{\text{pred}}^{i}\right)$$ , where \(\:\mathbf{C}\mathbf{o}\mathbf{r}\mathbf{r}\left({\mathbf{y}}_{\text{true}}^{i},{\mathbf{y}}_{\text{pred}}^{i}\right)\) represented the Pearson correlation coefficient between the true and predicted expression values of the \(\:i\) -th gene, \(\:{\mathbf{y}}_{\text{true}}^{i}\) and \(\:{\mathbf{y}}_{\text{pred}}^{i}\) , respectively. \(\:k\) is the number of all genes. In addition to the mean correlation of all genes, we computed the mean correlation of the top 50 most highly expressed genes (HEGs) in predicted gene expression, compared with the observed gene expression. This metric provided insight into the performance of the prediction method specifically for genes that are highly expressed in the spatial context. Technically, the mean correlation of the top \(\:k{\prime\:}\) HEGs was calculated as follows: $$\:{\stackrel{-}{R}}_{50}=\frac{1}{k}{\sum\:}_{i=1}^{k}\mathbf{C}\mathbf{o}\mathbf{r}\mathbf{r}\left({\mathbf{y}}_{\text{true}}^{i},{\mathbf{y}}_{\text{pred}}^{i}\right)$$ , where \(\:\mathbf{C}\mathbf{o}\mathbf{r}\mathbf{r}\left({\mathbf{y}}_{\text{true}}^{i},{\mathbf{y}}_{\text{pred}}^{i}\right)\) represented the Pearson correlation coefficient between the true and predicted expression values of the \(\:i\) -th gene, \(\:{\mathbf{y}}_{\text{true}}^{i}\) and \(\:{\mathbf{y}}_{\text{pred}}^{i}\) , respectively. We set \(\:k{\prime\:}=50\) in our experiments by default. Algorithm 1 Spatial transcriptomics prediction using H&E-stained images. Require : Image patch \(\:\left(\mathbf{X}\in\:{\mathbb{R}}^{N\times\:\left(3\times\:W\times\:H\right)}\right)\) Paired gene expression profile \(\:\left({\mathbf{y}}_{true}\in\:{\mathbb{R}}^{N\times\:d}\right)\) Ensure: Expression prediction \(\:\left({\mathbf{y}}_{\text{p}\text{r}\text{e}\text{d}}\in\:{\mathbb{R}}^{N\times\:d}\right)\) 1: function ResSAT \(\:\left(\mathbf{X}\right)\) 2: \(\:{f}_{x}\leftarrow\:\mathcal{G}\left(\mathcal{F}\left(\mathbf{X}\right)\right)\) \(\:N\) image embeddings 3: \(\:Q,K,V,d\) \(\:\leftarrow\:\) \(\:{f}_{x}\) dimension of \(\:K\:\) as \(\:d\) 4: \(\:{\mathbf{y}}_{pred}\:\leftarrow\:\text{s}\text{o}\text{f}\text{t}\text{m}\text{a}\text{x}\left(Q{K}^{T}/\sqrt{d}\right)\bullet\:V\) self-attention transformer 5: \(\:\mathcal{L}={{\mathcal{L}}_{mse}+\mathcal{L}}_{mse}^{{\prime\:}}\) 6: return \(\:{\mathbf{y}}_{true}\) , \(\:{\mathbf{y}}_{pred}\) , \(\:\mathcal{L}\) Declarations Ethics approval and consent to participate Not applicable Consent for publication Not applicable Availability of data and materials The source code of this work can be accessed at: https://github.com/Wonderangela/ResSAT under the GitHub. The raw public datasets analyzed during the current study are available in the 10X Genomics, https://www.10xgenomics.com/. Competing interests The authors declare that they have no competing interests. Funding None Authors' contributions AL contributed to the conception, design of the work, analysis, drafting of the work, and substantive revision. YZ contributed to the analysis. HS contributed to the design of the work and substantive revision. ZD contributed to the design of the work, analysis, drafting of the work, and substantive revision. HD contributed to the initiation, conception and substantive revision. The authors have approved the submitted version. Acknowledgements This work is made possible with partial supported by grants from the NIH (U19AG055373, R01AG061917, P20GM109036 and P20GM103629). References Jin Y, Zuo Y, Li G, Liu W, Pan Y, Fan T, Fu X, Yao X, Peng Y. Advances in spatial transcriptomics and its applications in cancer research. Mol Cancer. 2024;23(1):129. Epub 20240620. doi: 10.1186/s12943-024-02040-9. PubMed PMID: 38902727; PMCID: PMC11188176. Du J, Yang YC, An ZJ, Zhang MH, Fu XH, Huang ZF, Yuan Y, Hou J. Advances in spatial transcriptomics and related data analysis strategies. J Transl Med. 2023;21(1):330. Epub 20230518. doi: 10.1186/s12967-023-04150-2. PubMed PMID: 37202762; PMCID: PMC10193345. Titford M. The long history of hematoxylin. Biotech Histochem. 2005;80(2):73-8. doi: 10.1080/10520290500138372. PubMed PMID: 16195172. Wittekind D. Traditional staining for routine diagnostic pathology including the role of tannic acid. 1. Value and limitations of the hematoxylin-eosin stain. Biotech Histochem. 2003;78(5):261-70. doi: 10.1080/10520290310001633725. PubMed PMID: 14989644. Rosai J. Why microscopy will remain a cornerstone of surgical pathology. Lab Invest. 2007;87(5):403-8. Epub 20070402. doi: 10.1038/labinvest.3700551. PubMed PMID: 17401434. Chen X, Lin J, Wang Y, Zhang W, Xie W, Zheng Z, Wong KC. HE2Gene: image-to-RNA translation via multi-task learning for spatial transcriptomics data. Bioinformatics. 2024;40(6). doi: 10.1093/bioinformatics/btae343. PubMed PMID: 38837395; PMCID: PMC11164830. Chan JK. The wonderful colors of the hematoxylin-eosin stain in diagnostic surgical pathology. Int J Surg Pathol. 2014;22(1):12-32. Epub 20140109. doi: 10.1177/1066896913517939. PubMed PMID: 24406626. McGranahan N, Swanton C. Clonal Heterogeneity and Tumor Evolution: Past, Present, and the Future. Cell. 2017;168(4):613-28. doi: 10.1016/j.cell.2017.01.018. PubMed PMID: 28187284. Rizvi NA, Hellmann MD, Snyder A, Kvistborg P, Makarov V, Havel JJ, Lee W, Yuan J, Wong P, Ho TS, Miller ML, Rekhtman N, Moreira AL, Ibrahim F, Bruggeman C, Gasmi B, Zappasodi R, Maeda Y, Sander C, Garon EB, Merghoub T, Wolchok JD, Schumacher TN, Chan TA. Cancer immunology. Mutational landscape determines sensitivity to PD-1 blockade in non-small cell lung cancer. Science. 2015;348(6230):124-8. Epub 20150312. doi: 10.1126/science.aaa1348. PubMed PMID: 25765070; PMCID: PMC4993154. He B, Bergenstrahle L, Stenbeck L, Abid A, Andersson A, Borg A, Maaskola J, Lundeberg J, Zou J. Integrating spatial gene expression and breast tumour morphology via deep learning. Nat Biomed Eng. 2020;4(8):827-34. Epub 20200622. doi: 10.1038/s41551-020-0578-x. PubMed PMID: 32572199. Pang M, Su K, Li M. Leveraging information in spatial transcriptomics to predict super-resolution gene expression from histology images in tumors. BioRxiv2021. Xie R, Pang K, Chung S, Perciani C, MacParland S, Wang B, Bader G. Spatially resolved gene expression prediction from histology images via bi-modal contrastive learning. Advances in Neural Information Processing Systems2024. Tanevski J, Flores ROR, Gabor A, Schapiro D, Saez-Rodriguez J. Explainable multiview framework for dissecting spatial relationships from highly multiplexed data. Genome Biol. 2022;23(1):97. Epub 20220414. doi: 10.1186/s13059-022-02663-5. PubMed PMID: 35422018; PMCID: PMC9011939. Mouse brain serial section 1 (sagittal-anterior). Available from: https://www.10xgenomics.com/ datasets/mouse-brain-serial-section-1-sagittal-anterior-1-standard-1-1-0. Mouse brain serial section 2 (sagittal-anterior). Available from: https://www.10xgenomics.com/ datasets/mouse-brain-serial-section-2-sagittal-anterior-1-standard-1-1-0. Mouse brain serial section 1 (sagittal-posterior). Available from: https://www.10xgenomics.com/ datasets/mouse-brain-serial-section-1-sagittal-posterior-1-standard-1-0-0. Mouse brain serial section 2 (sagittal-posterior). Available from: https://www.10xgenomics.com/ datasets/mouse-brain-serial-section-2-sagittal-posterior-1-standard-1-0-0. What is space ranger? Available from: https://www.10xgenomics.com/support/software/ space-ranger/2.1/getting-started/what-is-space-ranger. Korsunsky I, Millard N, Fan J, Slowikowski K, Zhang F, Wei K, Baglaenko Y, Brenner M, Loh PR, Raychaudhuri S. Fast, sensitive and accurate integration of single-cell data with Harmony. Nat Methods. 2019;16(12):1289-96. Epub 20191118. doi: 10.1038/s41592-019-0619-0. PubMed PMID: 31740819; PMCID: PMC6884693. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2016:770-8. Cubuk ED, Zoph B, Shlens J, Le QV. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. 2020:702-3. Yun S, Han D, Oh SJ, Chun S, Choe J, Yoo Y. Cutmix: Regularization strategy to train strong classifiers with localizable features. Proceedings of the IEEE/CVF international conference on computer vision. 2019:6023-32. Wightman R, Touvron H, Jégou H. Resnet strikes back: An improved training procedure in timm. arXiv preprint2021. Xie S, Girshick R, Dollár P, Tu Z, He K. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2017:1492-500. Zhang H, Wu C, Zhang Z, et al. Resnest: Split-attention networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022:2736-46. Gong P, Cheng L, Zhang Z, Meng A, Li E, Chen J, Zhang L. Multi-omics integration method based on attention deep learning network for biomedical data classification. Comput Methods Programs Biomed. 2023;231:107377. Epub 20230127. doi: 10.1016/j.cmpb.2023.107377. PubMed PMID: 36739624. Wang S, Wang S, Yang D, Li M, Qing Z, Su L, Zhang L. Handgcat: Occlusion-robust 3d hand mesh reconstruction from monocular images. In 2023 IEEE International Conference on Multimedia and Expo (ICME). 2023:2495-500. Hu R, Singh A. Unit: Multimodal multitask learning with a unified transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021:1439–49. Nguyen S, Ng B, Kaplan AD, Ray P. Attend and decode: 4d fmri task state decoding using attention models. In Machine Learning for Health. 2020:267-79. Long NHB. Step catformer: Spatial-temporal effective body-part cross attention transformer for skeleton-based action recognition. arXiv preprint 2023. Wang J, Liao N, Du X, Chen Q, Wei B. A semi-supervised approach for the integration of multi-omics data based on transformer multi-head self-attention mechanism and graph convolutional networks. BMC Genomics. 2024;25(1):86. Epub 20240122. doi: 10.1186/s12864-024-09985-7. PubMed PMID: 38254021; PMCID: PMC10802018. Maksimova MA, Cansler HL, Zuk KE, Torres JM, Roberts DJ, Meeks JP. Interneuron Functional Diversity in the Mouse Accessory Olfactory Bulb. eNeuro. 2019;6(4). Epub 20190813. doi: 10.1523/ENEURO.0058-19.2019. PubMed PMID: 31358509; PMCID: PMC6712203. Bakalyar HA, Reed RR. Identification of a specialized adenylyl cyclase that may mediate odorant detection. Science. 1990;250(4986):1403-6. doi: 10.1126/science.2255909. PubMed PMID: 2255909. Nakajima D, Nakayama M, Kikuno R, Hirosawa M, Nagase T, Ohara O. Identification of three novel non-classical cadherin genes through comprehensive analysis of large cDNAs. Brain Res Mol Brain Res. 2001;94(1-2):85-95. doi: 10.1016/s0169-328x(01)00218-2. PubMed PMID: 11597768. Sharon D, Blackshaw S, Cepko CL, Dryja TP. Profile of the genes expressed in the human peripheral retina, macula, and retinal pigment epithelium determined through serial analysis of gene expression (SAGE). Proc Natl Acad Sci U S A. 2002;99(1):315-20. Epub 20011226. doi: 10.1073/pnas.012582799. PubMed PMID: 11756676; PMCID: PMC117558. Koldaeva A, Zhang C, Huang YP, Reinert JK, Mizuno S, Sugiyama F, Takahashi S, Soliman T, Matsunami H, Fukunaga I. Generation and Characterization of a Cell Type-Specific, Inducible Cre-Driver Line to Study Olfactory Processing. J Neurosci. 2021;41(30):6449-67. Epub 20210607. doi: 10.1523/JNEUROSCI.3076-20.2021. PubMed PMID: 34099512; PMCID: PMC8318078. Abdollahi Nejat M, Klaassen RV, Spijker S, Smit AB. Auxiliary subunits of the AMPA receptor: The Shisa family of proteins. Curr Opin Pharmacol. 2021;58:52-61. Epub 20210421. doi: 10.1016/j.coph.2021.03.001. PubMed PMID: 33892364. Pak JH, Huang FL, Li J, Balschun D, Reymann KG, Chiang C, Westphal H, Huang KP. Involvement of neurogranin in the modulation of calcium/calmodulin-dependent protein kinase II, synaptic plasticity, and spatial learning: a study with knockout mice. Proc Natl Acad Sci U S A. 2000;97(21):11232-7. doi: 10.1073/pnas.210184697. PubMed PMID: 11016969; PMCID: PMC17183. Coulter PM, 2nd, Bautista EA, Margulies JE, Watson JB. Identification of cortexin: a novel, neuron-specific, 82-residue membrane protein enriched in rodent cerebral cortex. J Neurochem. 1993;61(2):756-9. doi: 10.1111/j.1471-4159.1993.tb02183.x. PubMed PMID: 8336151. Guan J, Luo Y, Denker BM. Purkinje cell protein-2 (Pcp2) stimulates differentiation in PC12 cells by Gbetagamma-mediated activation of Ras and p38 MAPK. Biochem J. 2005;392(Pt 2):389-97. doi: 10.1042/BJ20042102. PubMed PMID: 15948714; PMCID: PMC1316275. Oyang EL, Davidson BC, Lee W, Poon MM. Functional characterization of the dendritically localized mRNA neuronatin in hippocampal neurons. PLoS One. 2011;6(9):e24879. Epub 20110914. doi: 10.1371/journal.pone.0024879. PubMed PMID: 21935485; PMCID: PMC3173491. Braun JL, Geromella MS, Hamstra SI, Fajardo VA. Neuronatin regulates whole-body metabolism: is thermogenesis involved? FASEB Bioadv. 2020;2(10):579-86. Epub 20200902. doi: 10.1096/fba.2020-00052. PubMed PMID: 33089074; PMCID: PMC7566048. Hudmon A, Schulman H. Structure-function of the multifunctional Ca2+/calmodulin-dependent protein kinase II. Biochem J. 2002;364(Pt 3):593-611. doi: 10.1042/BJ20020228. PubMed PMID: 11931644; PMCID: PMC1222606. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 02 Jun, 2025 Reviews received at journal 02 Jun, 2025 Reviewers agreed at journal 27 May, 2025 Reviews received at journal 30 Jan, 2025 Reviewers agreed at journal 23 Jan, 2025 Reviewers invited by journal 22 Jan, 2025 Editor assigned by journal 17 Jul, 2024 Submission checks completed at journal 09 Jul, 2024 First submitted to journal 08 Jul, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4707959","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":331710587,"identity":"b4416823-5d98-44e8-900d-68f78b6a4891","order_by":0,"name":"Anqi Liu","email":"","orcid":"","institution":"Tulane University","correspondingAuthor":false,"prefix":"","firstName":"Anqi","middleName":"","lastName":"Liu","suffix":""},{"id":331710588,"identity":"46505ec4-71d7-4e74-bbff-e7febb7893cb","order_by":1,"name":"Yue Zhao","email":"","orcid":"","institution":"Tulane University","correspondingAuthor":false,"prefix":"","firstName":"Yue","middleName":"","lastName":"Zhao","suffix":""},{"id":331710589,"identity":"c94ab150-98e1-495e-a9c2-f8656552666f","order_by":2,"name":"Hui Shen","email":"","orcid":"","institution":"Tulane University","correspondingAuthor":false,"prefix":"","firstName":"Hui","middleName":"","lastName":"Shen","suffix":""},{"id":331710590,"identity":"d505000a-14c5-40ba-ad96-cd9c8706d26d","order_by":3,"name":"Zhengming Ding","email":"","orcid":"","institution":"Tulane University","correspondingAuthor":false,"prefix":"","firstName":"Zhengming","middleName":"","lastName":"Ding","suffix":""},{"id":331710591,"identity":"67444bc3-4df9-4879-b645-2a1b816a3b37","order_by":4,"name":"Hong-Wen Deng","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAzUlEQVRIiWNgGAWjYBACA4YEIFlhw8DADhFgbCBOy5k0CQZmkrQwth0mQYs5e/Kxh1/YztcZHGZ+9uADg43shgMEtFj2PEs3luG5LWFwmM3ccAZDmjFBLQY3csykJSRAWhjMpHkYDicSoSX/m7SEwTmgFvZv0n8Y/hOjJYdN8kPCAaAWHjNpBoYDRGg58wyo8kCy5MzDPGWSPQbJxjMJajme/Ezy5z87fr7j7dskflTYyfYR0gICzDwIE4hQDgKMP4hUOApGwSgYBSMUAAC01kMlVONpGQAAAABJRU5ErkJggg==","orcid":"","institution":"Tulane University","correspondingAuthor":true,"prefix":"","firstName":"Hong-Wen","middleName":"","lastName":"Deng","suffix":""}],"badges":[],"createdAt":"2024-07-08 21:37:29","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4707959/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4707959/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":62158066,"identity":"4e7a322e-2a97-4b4a-8275-218a003bb4b2","added_by":"auto","created_at":"2024-08-09 21:29:44","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":89241,"visible":true,"origin":"","legend":"\u003cp\u003eOverview of workflow.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-4707959/v1/05406690ac0a30032a8bf6c1.png"},{"id":62159023,"identity":"cb745301-207b-4c84-8efd-9c65ed3236d9","added_by":"auto","created_at":"2024-08-09 21:37:44","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":27023,"visible":true,"origin":"","legend":"\u003cp\u003eComparative evaluation of spatial transcriptome prediction methods. SA_Section 1 represented Section 2 in the SA dataset was selected to be held out for training, while the Section 1 was selected to be held out for testing. SA_Section 2 represented Section 1 in the SA dataset was selected to be held out for training, while the Section 2 was selected to be held out for testing. SP_Section 1 represented Section 2 in the SP dataset was selected to be held out for training, while the Section 1 was selected to be held out for testing. SP_Section 2 represented Section 1 in the SP dataset was selected to be held out for training, while the Section 2 was selected to be held out for testing.\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-4707959/v1/83dd742b443981d8038e8bd5.png"},{"id":62158071,"identity":"86e5c011-eedb-4558-bb13-642a07bfa088","added_by":"auto","created_at":"2024-08-09 21:29:44","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":15659,"visible":true,"origin":"","legend":"\u003cp\u003eAblation experiments on the SA dataset.\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-4707959/v1/433c81a66a1d068708ae3064.png"},{"id":62158067,"identity":"ce5e8f97-949b-436b-a30a-89324713c1a2","added_by":"auto","created_at":"2024-08-09 21:29:44","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":17565,"visible":true,"origin":"","legend":"\u003cp\u003eAblation experiments on the SP dataset.\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-4707959/v1/d23e14bc82799fcceccbf805.png"},{"id":62158070,"identity":"67409790-184d-4243-be60-b351caacf0d9","added_by":"auto","created_at":"2024-08-09 21:29:44","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":1661931,"visible":true,"origin":"","legend":"\u003cp\u003eVisualization of the mouse brain (SA) dataset by the top five predicted genes with the highest correlations: \u003cem\u003eCALB2, GNG4, CDHR1\u003c/em\u003e, \u003cem\u003eDOC2G,\u003c/em\u003eand \u003cem\u003eSHISA8\u003c/em\u003e.\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-4707959/v1/b5aa1c161b4dc16579ce5b8b.png"},{"id":62158069,"identity":"1d4e18bf-b092-4778-b1df-ad69fbb857c6","added_by":"auto","created_at":"2024-08-09 21:29:44","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":1755907,"visible":true,"origin":"","legend":"\u003cp\u003eVisualization of the mouse brain (SP) dataset by the top five predicted genes with the highest correlations: \u003cem\u003eNRGN, CTXN1, PCP2, NNAT,\u003c/em\u003eand \u003cem\u003eCAMK2A\u003c/em\u003e.\u003c/p\u003e","description":"","filename":"6.png","url":"https://assets-eu.researchsquare.com/files/rs-4707959/v1/90b541b75d992a7463cb79cc.png"},{"id":62159348,"identity":"a16a065e-cc2e-4862-9c8e-5776ad3c330d","added_by":"auto","created_at":"2024-08-09 21:45:47","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":4367620,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4707959/v1/3b99e86e-c1d9-4e5b-80e8-e8eac5c50a01.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"ResSAT: Enhancing Spatial Transcriptomics Prediction from H\u0026E- Stained Histology Images with Interactive Spot Transformer","fulltext":[{"header":"Background","content":"\u003cp\u003eThe rapid advancement of spatial transcriptomics (ST) technology has revolutionized the field of RNA abundance quantification by offering remarkable spatial resolution for concurrent gene expression profiling and precise spatial spot localization \u003csup\u003e\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u003c/sup\u003e. This breakthrough allows researchers to generate detailed maps of gene expression within tissues, providing unprecedented insights into cellular function and tissue organization \u003csup\u003e\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e. However, currently, the resource-intensive and time-consuming nature of ST profiling has limited its widespread application, creating a demand for more accessible methods.\u003c/p\u003e \u003cp\u003eIn contrast, Hematoxylin and eosin (H\u0026amp;E) staining is a widely used histological technique that provides detailed insights into tissue structure and composition at a microscopic level \u003csup\u003e\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u003c/sup\u003e. H\u0026amp;E images are instrumental in medical diagnostics, offering clear visualizations of cellular morphology and tissue architecture \u003csup\u003e\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u003c/sup\u003e. As the gold standard in many diagnostic procedures, H\u0026amp;E imaging is cost-effective, readily available, and extensively utilized in clinical settings.\u003c/p\u003e \u003cp\u003eMoreover, there is a close relationship between H\u0026amp;E staining and ST, as H\u0026amp;E images capture detailed cellular and tissue morphology that correlates with gene expression patterns \u003csup\u003e\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u003c/sup\u003e. The staining highlights different cellular components: hematoxylin stains the cell nuclei blue, indicating areas of high DNA concentration, while eosin stains the cytoplasm and extracellular matrix pink, showing the structural context \u003csup\u003e\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e\u003c/sup\u003e. These visual cues from H\u0026amp;E images reflect the underlying biological processes and molecular activities within the tissue \u003csup\u003e\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u003c/sup\u003e, providing a spatial map that can be linked to gene expression data. Previous studies have also demonstrated that changes in gene expression or genetic mutations often influence cell morphology, structure, and distribution, resulting in alterations in histological features \u003csup\u003e\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e, \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e\u003c/sup\u003e​. This correlation enables the use of H\u0026amp;E images to predict spatial gene expression profiles, leveraging the morphological context provided by the staining to infer molecular states. Recognizing the potential to integrate these two powerful tools, recent studies \u003csup\u003e\u003cspan additionalcitationids=\"CR11\" citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u003c/sup\u003e have focused on developing computational approaches to predict ST data from H\u0026amp;E images. This innovative synergy aims to leverage the comprehensive tissue insights provided by H\u0026amp;E images to infer spatial gene expression profiles. By doing so, these approaches can potentially circumvent the limitations of ST technology, making high-resolution gene expression mapping more accessible and practical.\u003c/p\u003e \u003cp\u003eSeveral existing approaches, such as ST-Net \u003csup\u003e\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u003c/sup\u003e, HisToGene \u003csup\u003e\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e, and BLEEP \u003csup\u003e\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u003c/sup\u003e, have shown promising results in predicting expression from histology images. Both ST-Net and HisToGene treat expression prediction as regression tasks and train them in a feed-forward manner. ST-Net utilizes a ResNet50 image encoder, while HisToGene utilizes a vision transformer backbone. BLEEP, on the other hand, draws inspiration from contrastive language-image pretraining to establish a comparable joint embedding between spot expression profiles and their spatially paired image patches. Although HisToGene incorporates spatial location information, it does not explicitly consider spatial relationships between different locations.\u003c/p\u003e \u003cp\u003eIt's worth highlighting existing methods capable of generating spatially resolved expression predictions, either have limitations regarding the predicted panel (ST-Net, HisToGene, and BLEEP) or are prone to overfitting (HisToGene). Additionally, existing approaches in spatial expression prediction from H\u0026amp;E images often overlook the crucial relationships between different spatial locations, which provide essential context about tissue architecture, including the organization and interaction of cells, and spatial heterogeneity, and thus reflect variations in cellular composition and functions across different tissue regions. These factors may significantly impact biological interpretation and predictive accuracy \u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e. To address these challenges, we proposed a novel approach called ResSAT (Residual networks - Self-Attention Transformer) for predicting spatial transcriptomics profiles using H\u0026amp;E-stained histology images. We utilized a ResNet50 architecture to extract comprehensive image features from the H\u0026amp;E images, enabling our model to capture diverse characteristics of tissue structures and cellular compositions depicted in the images. Additionally, we introduced a self-attention transformer mechanism to identify and cluster spots with high correlation, allowing the model to focus on interactions between spots and enhance spatial gene expression prediction performance.\u003c/p\u003e \u003cp\u003eWe validated the effectiveness of ResSAT by benchmarking its performance on two different mice brain datasets obtained from the 10x Visium platform. Our results demonstrated significant improvements over existing methods such as BLEEP \u003csup\u003e\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u003c/sup\u003e, HisToGene \u003csup\u003e\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e, and ST-Net \u003csup\u003e\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u003c/sup\u003e in terms of mean correlations across all genes and mean correlations among the top 50 highly expressed genes.\u003c/p\u003e \u003cp\u003eThis innovative approach significantly enhances the performance of spatial gene expression prediction from histology images. The proposed framework has the potential to substantially reduce the time and cost associated with spatial transcriptomics profiling, opening up new possibilities for acquiring numerous spatial transcriptomics profiles rapidly and reconstructing comprehensive 3D spatial transcriptomics from adjacent 2D spatial transcriptomics profiles.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003e \u003cb\u003eResSAT enables gene expression prediction and consistently performs well.\u003c/b\u003e \u003c/p\u003e \u003cp\u003eTo assess the predictive efficacy of ResSAT in quantifying gene expression, we applied our method to both the sagittal anterior (SA) and sagittal posterior (SP) datasets. We compared ResSAT to three other methods for spatial gene expression prediction, including BLEEP, HisToGene, and ST-Net. The predicted expression profiles from ResSAT showed the highest mean correlation with ground truth, achieving an increase in PCCs for both mean correlations of all genes (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e) and top 50 most highly expressed genes (HEGs) (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e) in the two different datasets. To ensure robustness and reliability, the evaluations were repeated five times, and the resulting mean and standard deviation values were computed for further analysis and validation.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eMean correlations of all genes in predicted expression compared to ground truth expressions on held out datasets (Section 1).\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\"\u0026plusmn;\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\"\u0026plusmn;\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMethods\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSA\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSP\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResSAT\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e0.2218\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0055\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e0.2437\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0026\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBLEEP\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e0.0115\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0038\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e0.0091\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0054\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHistoGene\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e0.1216\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0020\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e0.1215\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0016\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eST-Net\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e0.0239\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0032\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e0.0340\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0061\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"3\"\u003eNote: mean\u0026thinsp;\u0026plusmn;\u0026thinsp;standard deviation\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eMean correlations of top 50 most highly expressed genes (HEGs) in predicted expression compared to ground truth expressions on held out datasets (Section 1).\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\"\u0026plusmn;\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\"\u0026plusmn;\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMethods\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSA\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSP\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResSAT\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e0.3485\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0039\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e0.4003\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0252\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBLEEP\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e0.0297\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0249\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e-0.0306\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0193\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHistoGene\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e-0.0506\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0351\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e-0.0174\u0026thinsp;\u0026plusmn;\u0026thinsp;0.04469\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eST-Net\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e \u003cp\u003e0.0774\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0043\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c3\"\u003e \u003cp\u003e0.0611\u0026thinsp;\u0026plusmn;\u0026thinsp;0.0083\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"3\"\u003eNote: mean\u0026thinsp;\u0026plusmn;\u0026thinsp;standard deviation\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eRespectively, for each section within the SA and SP datasets, we trained ResSAT using one section and evaluated the correlation between predicted gene expression and actual gene expression on the other section. Specifically, after training on Section 2 and testing on Section 1 as shown in the Tables\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e and \u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e, we also trained on Section 1 and tested on Section 2 to validate the results. As illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e2\u003c/span\u003e, ResSAT consistently yielded the highest PCCs between spatially resolved gene expression and actual gene expression across all sections in both datasets.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003eExamining the effect of each module within ResSAT on the predicted gene expression results.\u003c/b\u003e \u003c/p\u003e \u003cp\u003eIn order to better understand why ResSAT performs better than other methods, we conducted an analysis to see how each component contributes to its performance. We did this by removing certain modules of ResSAT and observing the impact on its ability to predict gene expression, as outlined in Figs.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e3\u003c/span\u003e and \u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e4\u003c/span\u003e. We found that keeping all modules intact resulted in the strongest correlation between predicted and observed gene expression. Specifically, when we excluded the ResNet module and SAT module respectively, we noticed an average PCC decrease in performance in both the SA and SP datasets. This indicates that the ResNet module and SAT module are both crucial for ResSAT to effectively uncover the relationships between different spots. In summary, our ablation experiment highlights the importance of maintaining all modules within ResSAT to achieve optimal performance in predicting gene expression.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003eResSAT accurately predicts brain-related genes.\u003c/b\u003e \u003c/p\u003e \u003cp\u003eWe examined whether the predicted gene expression by ResSAT accurately mirrors the actual expression of brain-related genes.\u003c/p\u003e \u003cp\u003eWithin the SA dataset, we assessed the correlation between observed and predicted gene expression, computing the PCC for each gene. We then ranked these genes in descending order of their PCCs and selected the predicted genes with the top 5 highest correlations obtained from our method for visualization (\u003cem\u003eCALB2, GNG4, CDHR1\u003c/em\u003e, \u003cem\u003eDOC2G\u003c/em\u003e, and \u003cem\u003eSHISA8\u003c/em\u003e), as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e5\u003c/span\u003e. \u003cem\u003eCALB2\u003c/em\u003e (Calretinin) expression in the mouse olfactory bulb is associated with inhibitory interneurons \u003csup\u003e\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e\u003c/sup\u003e. These interneurons play essential roles in regulating neural circuits involved in processing sensory information related to odors. \u003cem\u003eGNG4\u003c/em\u003e (G Protein Subunit Gamma 4) is a gene encoding a subunit of heterotrimeric G proteins, which are involved in signal transduction pathways in neurons \u003csup\u003e\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e\u003c/sup\u003e. In the mouse olfactory bulb, G proteins play a crucial role in mediating signaling pathways involved in odorant detection and processing. Heterotrimeric G proteins, consisting of alpha, beta, and gamma subunits, are involved in transducing signals from odorant receptors to downstream effector molecules, leading to neuronal activation and olfactory perception \u003csup\u003e\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e\u003c/sup\u003e. \u003cem\u003eCDHR1\u003c/em\u003e (Cadherin-Related Family Member 1) is a gene expressed in the olfactory bulb of mice, where it likely contributes to the organization and maintenance of the olfactory sensory epithelium \u003csup\u003e\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e, \u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e\u003c/sup\u003e. \u003cem\u003eDOC2G\u003c/em\u003e, also known as Double C2 Domain Gamma, is a gene encoding a calcium-binding protein involved in vesicle exocytosis and neurotransmitter release. In the mammalian olfactory system, complex information processing starts in the olfactory bulb, whose output is conveyed by mitral cells (MCs) and tufted cells (TCs) \u003csup\u003e\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e\u003c/sup\u003e. \u003cem\u003eDOC2G\u003c/em\u003e was identified to be differentially expressed between MCs and TCs of the mouse \u003csup\u003e\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e\u003c/sup\u003e. \u003cem\u003eSHISA8\u003c/em\u003e, a member of the Shisa family of transmembrane proteins, has a broad role in synaptic function and neuronal development, suggesting its potential involvement in olfactory processing \u003csup\u003e\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eTo further validate the robustness of ResSAT in predicting brain-related genes, we extended our analysis to the SP dataset, achieving similarly accurate predictions, as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e6\u003c/span\u003e. ResSAT enabled accurate prediction of key genes associated with mouse brain. Figure\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e6\u003c/span\u003e shows the top 5 genes (\u003cem\u003eNRGN, CTXN1, PCP2, NNAT\u003c/em\u003e, and \u003cem\u003eCAMK2A\u003c/em\u003e) in the SA dataset predicted by ResSAT. \u003cem\u003eNRGN\u003c/em\u003e (Neurogranin) is a gene encoding a protein found primarily in the brain, specifically in dendritic spines of neurons. It is involved in regulating synaptic plasticity and learning processes by modulating the function of calmodulin, a calcium-binding protein. \u003cem\u003eNRGN\u003c/em\u003e has been implicated in various neurological disorders, including Alzheimer's disease and schizophrenia \u003csup\u003e\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e\u003c/sup\u003e. \u003cem\u003eCTXN1\u003c/em\u003e (Cortexin-1) is a gene encoding a protein involved in neuronal development and synaptic function, especially highly expressed in cerebral cortex \u003csup\u003e\u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e39\u003c/span\u003e\u003c/sup\u003e. \u003cem\u003ePCP2\u003c/em\u003e (Purkinje Cell Protein 2) is a gene expressed primarily in Purkinje cells of the cerebellum \u003csup\u003e\u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e40\u003c/span\u003e\u003c/sup\u003e. It encodes a protein involved in dendritic development, synaptic transmission, and calcium signaling within Purkinje cells. Mutations in \u003cem\u003ePCP2\u003c/em\u003e have been linked to certain neurodevelopmental disorders. \u003cem\u003eNNAT\u003c/em\u003e (Neuronatin) is a gene encoding a protein expressed in the brain, particularly in neurons, where it regulates neuronal development and function. It is involved in processes such as neuronal differentiation, synaptogenesis, and neurotransmitter release \u003csup\u003e\u003cspan citationid=\"CR41\" class=\"CitationRef\"\u003e41\u003c/span\u003e\u003c/sup\u003e. \u003cem\u003eNNAT\u003c/em\u003e has been implicated in neurological disorders and metabolic regulation \u003csup\u003e\u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e42\u003c/span\u003e\u003c/sup\u003e. \u003cem\u003eCAMK2A\u003c/em\u003e (Calcium/Calmodulin-Dependent Protein Kinase II Alpha) is a gene encoding a protein kinase involved in calcium signaling and synaptic plasticity. It plays a crucial role in neuronal excitability, synaptic transmission, and learning and memory processes. Dysregulation of \u003cem\u003eCAMK2A\u003c/em\u003e has been implicated in various neurological disorders, including Alzheimer's disease and epilepsy \u003csup\u003e\u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e43\u003c/span\u003e\u003c/sup\u003e. These genes are known for their critical roles in neuronal function and brain development, reinforcing the model's capability across different brain regions and datasets. This consistency underscores ResSAT's reliability in predicting key genetic markers relevant to mouse brains.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eTo address the challenges inherent in spatial transcriptomics prediction, we devised a novel approach called ResSAT. Leveraging a ResNet50 architecture, we extracted comprehensive image features from provided H\u0026amp;E images. This method enabled our model to capture a wide range of tissue structures and cellular compositions depicted in the images. We then introduced a self-attention transformer mechanism to cluster spots exhibiting high correlation. This empowered the model to focus on interactions between spots, thereby enhancing spatial gene expression prediction performance.\u003c/p\u003e \u003cp\u003eIn our experimental evaluations on benchmark ST datasets, our proposed method demonstrated its effectiveness in accurately predicting spatial gene expression patterns in two different mice brain datasets. We achieved significantly higher mean correlations across all genes and the top 50 most highly expressed genes (HEGs), representing substantial improvements compared to existing methods. Additionally, ResSAT exhibited similar expression patterns for the top 5 predicted genes compared to observed expression profiles. The results accurately predicted spatial correlations of genes, with the locations of predicted genes closely matching the spatial locations of observed genes. This underscores the efficacy of our approach in spatial transcriptomics prediction from H\u0026amp;E images, indicating its potential to generate numerous spatial transcriptomics profiles efficiently.\u003c/p\u003e \u003cp\u003eOur focus in this paper primarily centers on mouse brain datasets, laying the groundwork for our subsequent 3D reconstructed map of brain regions in spatial transcriptomics. Despite the successful prediction of specific genes showcased in Figs.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e5\u003c/span\u003e and \u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e6\u003c/span\u003e, we acknowledge that the overall absolute correlations in Tables\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e and \u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e remain low, indicating the challenging nature of the prediction task for the majority of genes. The low scores may stem from various factors, including the weak correlation between the expression of certain genes and morphological features, limitations in the detection of certain genes by the Visium platform resulting in less predictable expression, and the presence of experimental artifacts introducing non-biological variability into the data, independent of the image. Due to the relatively small number of tissue sections available, neither ResSAT nor other existing methods can reliably predict gene expression with high accuracy. However, ResSAT still demonstrates superior prediction accuracy compared to other methods. While the reliance on a relatively large training set poses a potential limitation for deep learning-based models, we anticipate that as more training ST data become available in the near future, ResSAT's performance and robustness can be further enhanced.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eWe introduced ResSAT, a novel framework for predicting spatial gene expression profiles from H\u0026amp;E-stained histology images using a ResNet50 architecture and a self-attention transformer mechanism. ResSAT effectively captures tissue structures and clusters correlated spots to enhance prediction performance. Our evaluations on mouse brain datasets demonstrated significant improvements over existing methods, with higher mean correlations and accurate spatial predictions.\u003c/p\u003e \u003cp\u003eDespite challenges like low absolute correlations for certain genes and limited tissue sections, ResSAT outperformed current methods, showing potential for efficient and cost-effective ST profiling. Future availability of more training data is expected to further enhance ResSAT's performance and robustness, advancing the field of spatial transcriptomics.\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003e \u003cb\u003eDatasets: Two mouse brain datasets (SA and SP).\u003c/b\u003e \u003c/p\u003e \u003cp\u003eST profiling of the anterior part and posterior part of the mouse brain tissue sagittal sections was generated with the Visium technology from 10x Genomics \u003csup\u003e\u003cspan additionalcitationids=\"CR15 CR16\" citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e\u003c/sup\u003e. Both datasets consist of serial H\u0026amp;E histology images and paired gene expressions at the spatial spots and their coordinates. The analyzed gene expression count matrices are outputs of the SpaceRanger pipeline \u003csup\u003e\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eThe anterior sagittal dataset comprises two sets of H\u0026amp;E-stained histology images, alongside corresponding spatial gene expression data \u003csup\u003e\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u003c/sup\u003e. In slice 1, the spatial transcriptomics (ST) data encapsulates the expression profiles of 31,053 genes across 2,825 spots, given by read counts. Slice 2 features ST data for 31,053 genes across 2,696 spots, also detailed through read counts.\u003c/p\u003e \u003cp\u003eThe posterior sagittal dataset includes two collections of H\u0026amp;E-stained histology images and their associated spatial gene expression data \u003csup\u003e\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e\u003c/sup\u003e. For the first slice, spatial transcriptomics (ST) analysis reveals the expression levels of 32,285 genes across 3,355 spots, given by read count data. The second slice presents ST information for the same number of genes, but across 3,289 spots, with expression quantified similarly through read counts.\u003c/p\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003eData preprocessing\u003c/h2\u003e \u003cp\u003eFor the H\u0026amp;E histology images, we extracted patches based on the size and location of each spot. Each patch was a 224x224 image centered around a spot, approximately 55\u0026micro;m on each side, and paired with the spot's corresponding gene expression profile. For the spatial gene expression profiles of each tissue section, each spot was normalized to the total count and log normalized. The union of the top 1,000 most highly variable genes from each of the slices was used for training and prediction. Finally, the expression data of these samples were batch corrected using Harmony \u003csup\u003e\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e before one of the slices was randomly selected to be held out for testing. For these two datasets, the slice 2 was selected to be held out for training, while the slice 1 was selected to be held out for testing.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003eLearning image embedding for expression prediction.\u003c/b\u003e \u003c/p\u003e \u003cp\u003eResidual networks (ResNets) \u003csup\u003e\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e\u003c/sup\u003e are widely recognized architectures in image classification, initially acclaimed for their significant advancement upon introduction. They continue to be a benchmark in various image analyses \u003csup\u003e\u003cspan additionalcitationids=\"CR22\" citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e and serve as baselines in image studies introducing novel architectures \u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e, \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e\u003c/sup\u003e. Our focus in this paper was to extract highly representative features from H\u0026amp;E images using ResNet50. During the training phase, we utilized a dataset comprising \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:N\\)\u003c/span\u003e\u003c/span\u003e pairs of H\u0026amp;E images and their corresponding gene expressions. Each image patch was represented as a 3D tensor \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\mathbf{X}}_{i}\\in\\:{\\mathbb{R}}^{3\\times\\:W\\times\\:H}\\)\u003c/span\u003e\u003c/span\u003e, where \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:N\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:H\\)\u003c/span\u003e\u003c/span\u003e denoted the width and height of the patch, respectively. The associated gene expression \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\mathbf{y}}_{i}\\)\u003c/span\u003e\u003c/span\u003e was represented as a \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:d\\)\u003c/span\u003e\u003c/span\u003e-dimensional vector in \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\mathbb{R}}^{d}\\)\u003c/span\u003e\u003c/span\u003e.\u003c/p\u003e \u003cp\u003eOur objective was to develop a deep learning framework capable of accurately predicting gene expression from a given image patch. We conceptualized our model as comprising two main components: a backbone network, denoted as \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\mathcal{F}\\left(\\bullet\\:\\right)\\)\u003c/span\u003e\u003c/span\u003e, for initial feature extraction module, followed by a feature refinement module, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\mathcal{G}\\left(\\bullet\\:\\right)\\)\u003c/span\u003e\u003c/span\u003e, to enhance the predictive capability of the extracted features. To optimize the performance of both modules, we employed the Mean Squared Error (MSE) loss function during the training process as follows:\u003cdiv id=\"Equa\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equa\" name=\"EquationSource\"\u003e\n$$\\:{\\mathcal{L}}_{mse}=\\sum\\:_{i=1}^{N}{‖{\\mathbf{y}}_{i}-\\mathcal{G}\\left(\\mathcal{F}\\left({\\mathbf{X}}_{i}\\right)\\right)‖}_{{\\mathcal{l}}_{2}}$$\u003c/div\u003e\u003c/div\u003e,\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\mathcal{l}}_{2}\\)\u003c/span\u003e\u003c/span\u003e denotes the \u003cem\u003eL\u003c/em\u003e-2 norm to measure the difference between predicted gene expression and ground-truth one.\u003c/p\u003e \u003cp\u003e \u003cb\u003eSpot-Interaction Discovery.\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThe emergence of transformers has led to their widespread application in image and omics data analysis, as demonstrated by numerous studies \u003csup\u003e\u003cspan additionalcitationids=\"CR27 CR28\" citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e\u003c/sup\u003e. Particularly, recent transformer-based architectures have drawn attention to self-attention and cross-attention mechanisms, offering means to capture interdependencies between different input modalities \u003csup\u003e\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e, \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eIn the context of our spatially resolved gene expression prediction approach, the Self-Attention Transformer (SAT) plays a crucial role in exploring spot-spot interactions. By clustering spots that exhibit high correlation, SAT enhances our model's ability to represent gene expression accurately. This approach facilitates a deeper understanding of spatial relationships within the data, contributing to improved predictive performance.\u003c/p\u003e \u003cp\u003eIn our endeavor to analyze H\u0026amp;E images for spot interaction, we proposed the development of a spot-interaction module, as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e1\u003c/span\u003e. This module aimed to refine predictions of gene expression. To streamline the model and minimize the number of parameters, we introduced a non-parametric attentive module. The operation of this attention module was defined by the following Equation:\u003cdiv id=\"Equb\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equb\" name=\"EquationSource\"\u003e\n$$\\:\\mathcal{A}\\left(Q,\\:K,\\:V\\right)=\\mathbf{s}\\mathbf{o}\\mathbf{f}\\mathbf{t}\\mathbf{m}\\mathbf{a}\\mathbf{x}\\left(\\frac{Q{K}^{\\text{T}}}{\\sqrt{d}}\\right)V$$\u003c/div\u003e\u003c/div\u003e,\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:K\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:Q\\)\u003c/span\u003e\u003c/span\u003e, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:V\\)\u003c/span\u003e\u003c/span\u003e represent the key, query, and value matrices of a Transformer module. To model the spot interaction, we aimed to learn a compact gene expression correlation between the spots in the training set. Here, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\mathbf{X}}_{i}\\)\u003c/span\u003e\u003c/span\u003e represented the input features, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\mathcal{F}\\left(\\bullet\\:\\right)\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\mathcal{G}\\left(\\bullet\\:\\right)\\)\u003c/span\u003e\u003c/span\u003e denoted transformation functions. The aim of this formulation was to explore the matrix of spot-spot interactions. By doing so, it clustered spots that exhibit high correlation, thereby enhancing the robustness of the gene expression representation. Moreover, this approach elucidated the correlation among spots, improving the model\u0026rsquo;s inference capabilities. To enhance the model\u0026rsquo;s learning process by integrating knowledge of ground-truth gene expressions, we introduced a secondary MSE loss function. This additional MSE loss was formulated as follows:\u003cdiv id=\"Equc\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equc\" name=\"EquationSource\"\u003e\n$$\\:{\\mathcal{L}}_{mse}^{{\\prime\\:}}=\\sum\\:_{i=1}^{N}{‖{\\mathbf{y}}_{i}-\\mathcal{A}\\left(\\mathcal{G}\\left(\\mathcal{F}\\left({\\mathbf{X}}_{i}\\right)\\right)\\right)‖}_{{\\mathcal{l}}_{2}}$$\u003c/div\u003e\u003c/div\u003e.\u003c/p\u003e \u003cp\u003eIntegrating dual objectives into a unified framework, we defined the overall loss function as follows:\u003cdiv id=\"Equd\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equd\" name=\"EquationSource\"\u003e\n$$\\:\\mathcal{L}={{\\mathcal{L}}_{mse}+\\mathcal{L}}_{mse}^{{\\prime\\:}}$$\u003c/div\u003e\u003c/div\u003e.\u003c/p\u003e \u003cp\u003eThis combined loss framework was designed to enhance the model\u0026rsquo;s predictive performance by balancing the feature extraction and interactive feature refinement. By carefully summing up the contribution of the primary and secondary MSE losses, the model was steered to pay detailed attention to the subtleties and complexities of gene expression data. This, in turn, was expected to improve the model\u0026rsquo;s capacity for capturing the intricate biological relationships that were represented within H\u0026amp;E images.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003eEvaluated Metrics\u003c/h2\u003e \u003cp\u003eIn this study, we employed the Pearson correlation coefficient (PCC) to measure the spatial gene expression predicted by ResSAT with the observed gene expression, in order to assess their level of correlation. The PCC had a range of values between \u0026minus;\u0026thinsp;1 and 1. It was determined by dividing the covariance of two variables by the product of their individual standard deviations:\u003cdiv id=\"Eque\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Eque\" name=\"EquationSource\"\u003e\n$$\\:\\text{P}\\text{C}\\text{C}=\\:\\frac{\\text{C}\\text{o}\\text{v}({\\text{y}}_{\\text{t}\\text{r}\\text{u}\\text{e}},\\:{\\text{y}}_{\\text{p}\\text{r}\\text{e}\\text{d}})}{{\\sigma\\:}\\left({\\text{y}}_{\\text{t}\\text{r}\\text{u}\\text{e}}\\right)\\bullet\\:{\\sigma\\:}\\left({\\text{y}}_{\\text{p}\\text{r}\\text{e}\\text{d}}\\right)}$$\u003c/div\u003e\u003c/div\u003e,\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\text{C}\\text{o}\\text{v}(\\bullet\\:,\\:\\bullet\\:)\\)\u003c/span\u003e\u003c/span\u003e denotes covariance; \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\text{y}}_{\\text{t}\\text{r}\\text{u}\\text{e}}\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\text{y}}_{\\text{p}\\text{r}\\text{e}\\text{d}}\\)\u003c/span\u003e\u003c/span\u003e represents the original gene expression and the gene expression obtained by prediction, respectively. \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\sigma\\:}\\left(\\bullet\\:\\right)\\)\u003c/span\u003e\u003c/span\u003e means standard deviation.\u003c/p\u003e \u003cp\u003eWe computed the mean correlation of all genes as follows:\u003cdiv id=\"Equf\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equf\" name=\"EquationSource\"\u003e\n$$\\:\\stackrel{-}{R}=\\frac{1}{k}{\\sum\\:}_{i=1}^{k}\\mathbf{C}\\mathbf{o}\\mathbf{r}\\mathbf{r}\\left({\\mathbf{y}}_{\\text{true}}^{i},{\\mathbf{y}}_{\\text{pred}}^{i}\\right)$$\u003c/div\u003e\u003c/div\u003e,\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\mathbf{C}\\mathbf{o}\\mathbf{r}\\mathbf{r}\\left({\\mathbf{y}}_{\\text{true}}^{i},{\\mathbf{y}}_{\\text{pred}}^{i}\\right)\\)\u003c/span\u003e\u003c/span\u003e represented the Pearson correlation coefficient between the true and predicted expression values of the \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:i\\)\u003c/span\u003e\u003c/span\u003e-th gene, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\mathbf{y}}_{\\text{true}}^{i}\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\mathbf{y}}_{\\text{pred}}^{i}\\)\u003c/span\u003e\u003c/span\u003e, respectively. \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:k\\)\u003c/span\u003e\u003c/span\u003e is the number of all genes.\u003c/p\u003e \u003cp\u003eIn addition to the mean correlation of all genes, we computed the mean correlation of the top 50 most highly expressed genes (HEGs) in predicted gene expression, compared with the observed gene expression. This metric provided insight into the performance of the prediction method specifically for genes that are highly expressed in the spatial context. Technically, the mean correlation of the top \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:k{\\prime\\:}\\)\u003c/span\u003e\u003c/span\u003e HEGs was calculated as follows:\u003cdiv id=\"Equg\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equg\" name=\"EquationSource\"\u003e\n$$\\:{\\stackrel{-}{R}}_{50}=\\frac{1}{k}{\\sum\\:}_{i=1}^{k}\\mathbf{C}\\mathbf{o}\\mathbf{r}\\mathbf{r}\\left({\\mathbf{y}}_{\\text{true}}^{i},{\\mathbf{y}}_{\\text{pred}}^{i}\\right)$$\u003c/div\u003e\u003c/div\u003e,\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\mathbf{C}\\mathbf{o}\\mathbf{r}\\mathbf{r}\\left({\\mathbf{y}}_{\\text{true}}^{i},{\\mathbf{y}}_{\\text{pred}}^{i}\\right)\\)\u003c/span\u003e\u003c/span\u003e represented the Pearson correlation coefficient between the true and predicted expression values of the \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:i\\)\u003c/span\u003e\u003c/span\u003e-th gene, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\mathbf{y}}_{\\text{true}}^{i}\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\mathbf{y}}_{\\text{pred}}^{i}\\)\u003c/span\u003e\u003c/span\u003e, respectively. We set \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:k{\\prime\\:}=50\\)\u003c/span\u003e\u003c/span\u003e in our experiments by default.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"No\" id=\"Taba\" border=\"1\"\u003e \u003ccolgroup cols=\"1\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAlgorithm 1 Spatial transcriptomics prediction using H\u0026amp;E-stained images.\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eRequire\u003c/b\u003e: Image patch \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\left(\\mathbf{X}\\in\\:{\\mathbb{R}}^{N\\times\\:\\left(3\\times\\:W\\times\\:H\\right)}\\right)\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003cp\u003ePaired gene expression profile \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\left({\\mathbf{y}}_{true}\\in\\:{\\mathbb{R}}^{N\\times\\:d}\\right)\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eEnsure: Expression prediction\u003c/b\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\left({\\mathbf{y}}_{\\text{p}\\text{r}\\text{e}\\text{d}}\\in\\:{\\mathbb{R}}^{N\\times\\:d}\\right)\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e1: \u003cb\u003efunction\u003c/b\u003e ResSAT\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\left(\\mathbf{X}\\right)\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003cp\u003e2: \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{f}_{x}\\leftarrow\\:\\mathcal{G}\\left(\\mathcal{F}\\left(\\mathbf{X}\\right)\\right)\\)\u003c/span\u003e\u003c/span\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:N\\)\u003c/span\u003e\u003c/span\u003e image embeddings\u003c/p\u003e \u003cp\u003e3: \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:Q,K,V,d\\)\u003c/span\u003e\u003c/span\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\leftarrow\\:\\)\u003c/span\u003e\u003c/span\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{f}_{x}\\)\u003c/span\u003e\u003c/span\u003e dimension of \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:K\\:\\)\u003c/span\u003e\u003c/span\u003eas \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:d\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003cp\u003e4: \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\mathbf{y}}_{pred}\\:\\leftarrow\\:\\text{s}\\text{o}\\text{f}\\text{t}\\text{m}\\text{a}\\text{x}\\left(Q{K}^{T}/\\sqrt{d}\\right)\\bullet\\:V\\)\u003c/span\u003e\u003c/span\u003e self-attention transformer\u003c/p\u003e \u003cp\u003e5: \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\mathcal{L}={{\\mathcal{L}}_{mse}+\\mathcal{L}}_{mse}^{{\\prime\\:}}\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003cp\u003e6: \u003cb\u003ereturn\u003c/b\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\mathbf{y}}_{true}\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\mathbf{y}}_{pred}\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\mathcal{L}\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"Declarations","content":"\u003ch3\u003eEthics approval and consent to participate\u003c/h3\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e\n\u003ch3\u003eConsent for publication\u003c/h3\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e\n\u003ch3\u003eAvailability of data and materials\u003c/h3\u003e\n\u003cp\u003eThe source code of this work can be accessed at: https://github.com/Wonderangela/ResSAT under the GitHub. The raw public datasets analyzed during the current study are available in the 10X Genomics, https://www.10xgenomics.com/.\u003c/p\u003e\n\u003ch3\u003eCompeting interests\u003c/h3\u003e\n\u003cp\u003eThe authors declare that they have no competing interests.\u003c/p\u003e\n\u003ch3\u003eFunding\u003c/h3\u003e\n\u003cp\u003eNone\u003c/p\u003e\n\u003ch3\u003eAuthors\u0026apos; contributions\u003c/h3\u003e\n\u003cp\u003eAL contributed to the conception, design of the work, analysis, drafting of the work, and substantive revision. YZ contributed to the analysis. HS contributed to the design of the work and substantive revision. ZD contributed to the design of the work, analysis, drafting of the work, and substantive revision. HD contributed to the initiation, conception and substantive revision. The authors have approved the submitted version.\u003c/p\u003e\n\u003ch3\u003eAcknowledgements\u003c/h3\u003e\n\u003cp\u003eThis work is made possible with partial supported by grants from the NIH (U19AG055373, R01AG061917, P20GM109036 and P20GM103629).\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eJin Y, Zuo Y, Li G, Liu W, Pan Y, Fan T, Fu X, Yao X, Peng Y. Advances in spatial transcriptomics and its applications in cancer research. Mol Cancer. 2024;23(1):129. Epub 20240620. doi: 10.1186/s12943-024-02040-9. PubMed PMID: 38902727; PMCID: PMC11188176.\u003c/li\u003e\n\u003cli\u003eDu J, Yang YC, An ZJ, Zhang MH, Fu XH, Huang ZF, Yuan Y, Hou J. Advances in spatial transcriptomics and related data analysis strategies. J Transl Med. 2023;21(1):330. Epub 20230518. doi: 10.1186/s12967-023-04150-2. PubMed PMID: 37202762; PMCID: PMC10193345.\u003c/li\u003e\n\u003cli\u003eTitford M. The long history of hematoxylin. Biotech Histochem. 2005;80(2):73-8. doi: 10.1080/10520290500138372. PubMed PMID: 16195172.\u003c/li\u003e\n\u003cli\u003eWittekind D. Traditional staining for routine diagnostic pathology including the role of tannic acid. 1. Value and limitations of the hematoxylin-eosin stain. Biotech Histochem. 2003;78(5):261-70. doi: 10.1080/10520290310001633725. PubMed PMID: 14989644.\u003c/li\u003e\n\u003cli\u003eRosai J. Why microscopy will remain a cornerstone of surgical pathology. Lab Invest. 2007;87(5):403-8. Epub 20070402. doi: 10.1038/labinvest.3700551. PubMed PMID: 17401434.\u003c/li\u003e\n\u003cli\u003eChen X, Lin J, Wang Y, Zhang W, Xie W, Zheng Z, Wong KC. HE2Gene: image-to-RNA translation via multi-task learning for spatial transcriptomics data. Bioinformatics. 2024;40(6). doi: 10.1093/bioinformatics/btae343. PubMed PMID: 38837395; PMCID: PMC11164830.\u003c/li\u003e\n\u003cli\u003eChan JK. The wonderful colors of the hematoxylin-eosin stain in diagnostic surgical pathology. Int J Surg Pathol. 2014;22(1):12-32. Epub 20140109. doi: 10.1177/1066896913517939. PubMed PMID: 24406626.\u003c/li\u003e\n\u003cli\u003eMcGranahan N, Swanton C. Clonal Heterogeneity and Tumor Evolution: Past, Present, and the Future. Cell. 2017;168(4):613-28. doi: 10.1016/j.cell.2017.01.018. PubMed PMID: 28187284.\u003c/li\u003e\n\u003cli\u003eRizvi NA, Hellmann MD, Snyder A, Kvistborg P, Makarov V, Havel JJ, Lee W, Yuan J, Wong P, Ho TS, Miller ML, Rekhtman N, Moreira AL, Ibrahim F, Bruggeman C, Gasmi B, Zappasodi R, Maeda Y, Sander C, Garon EB, Merghoub T, Wolchok JD, Schumacher TN, Chan TA. Cancer immunology. Mutational landscape determines sensitivity to PD-1 blockade in non-small cell lung cancer. Science. 2015;348(6230):124-8. Epub 20150312. doi: 10.1126/science.aaa1348. PubMed PMID: 25765070; PMCID: PMC4993154.\u003c/li\u003e\n\u003cli\u003eHe B, Bergenstrahle L, Stenbeck L, Abid A, Andersson A, Borg A, Maaskola J, Lundeberg J, Zou J. Integrating spatial gene expression and breast tumour morphology via deep learning. Nat Biomed Eng. 2020;4(8):827-34. Epub 20200622. doi: 10.1038/s41551-020-0578-x. PubMed PMID: 32572199.\u003c/li\u003e\n\u003cli\u003ePang M, Su K, Li M. Leveraging information in spatial transcriptomics to predict super-resolution gene expression from histology images in tumors. BioRxiv2021.\u003c/li\u003e\n\u003cli\u003eXie R, Pang K, Chung S, Perciani C, MacParland S, Wang B, Bader G. Spatially resolved gene expression prediction from histology images via bi-modal contrastive learning. Advances in Neural Information Processing Systems2024.\u003c/li\u003e\n\u003cli\u003eTanevski J, Flores ROR, Gabor A, Schapiro D, Saez-Rodriguez J. Explainable multiview framework for dissecting spatial relationships from highly multiplexed data. Genome Biol. 2022;23(1):97. Epub 20220414. doi: 10.1186/s13059-022-02663-5. PubMed PMID: 35422018; PMCID: PMC9011939.\u003c/li\u003e\n\u003cli\u003eMouse brain serial section 1 (sagittal-anterior). Available from: https://www.10xgenomics.com/ datasets/mouse-brain-serial-section-1-sagittal-anterior-1-standard-1-1-0.\u003c/li\u003e\n\u003cli\u003eMouse brain serial section 2 (sagittal-anterior). Available from: https://www.10xgenomics.com/ datasets/mouse-brain-serial-section-2-sagittal-anterior-1-standard-1-1-0.\u003c/li\u003e\n\u003cli\u003eMouse brain serial section 1 (sagittal-posterior). Available from: https://www.10xgenomics.com/ datasets/mouse-brain-serial-section-1-sagittal-posterior-1-standard-1-0-0.\u003c/li\u003e\n\u003cli\u003eMouse brain serial section 2 (sagittal-posterior). Available from: https://www.10xgenomics.com/ datasets/mouse-brain-serial-section-2-sagittal-posterior-1-standard-1-0-0.\u003c/li\u003e\n\u003cli\u003eWhat is space ranger? Available from: https://www.10xgenomics.com/support/software/ space-ranger/2.1/getting-started/what-is-space-ranger.\u003c/li\u003e\n\u003cli\u003eKorsunsky I, Millard N, Fan J, Slowikowski K, Zhang F, Wei K, Baglaenko Y, Brenner M, Loh PR, Raychaudhuri S. Fast, sensitive and accurate integration of single-cell data with Harmony. Nat Methods. 2019;16(12):1289-96. Epub 20191118. doi: 10.1038/s41592-019-0619-0. PubMed PMID: 31740819; PMCID: PMC6884693.\u003c/li\u003e\n\u003cli\u003eHe K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2016:770-8.\u003c/li\u003e\n\u003cli\u003eCubuk ED, Zoph B, Shlens J, Le QV. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. 2020:702-3.\u003c/li\u003e\n\u003cli\u003eYun S, Han D, Oh SJ, Chun S, Choe J, Yoo Y. Cutmix: Regularization strategy to train strong classifiers with localizable features. Proceedings of the IEEE/CVF international conference on computer vision. 2019:6023-32.\u003c/li\u003e\n\u003cli\u003eWightman R, Touvron H, J\u0026eacute;gou H. Resnet strikes back: An improved training procedure in timm. arXiv preprint2021.\u003c/li\u003e\n\u003cli\u003eXie S, Girshick R, Doll\u0026aacute;r P, Tu Z, He K. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2017:1492-500.\u003c/li\u003e\n\u003cli\u003eZhang H, Wu C, Zhang Z, et al. Resnest: Split-attention networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022:2736-46.\u003c/li\u003e\n\u003cli\u003eGong P, Cheng L, Zhang Z, Meng A, Li E, Chen J, Zhang L. Multi-omics integration method based on attention deep learning network for biomedical data classification. Comput Methods Programs Biomed. 2023;231:107377. Epub 20230127. doi: 10.1016/j.cmpb.2023.107377. PubMed PMID: 36739624.\u003c/li\u003e\n\u003cli\u003eWang S, Wang S, Yang D, Li M, Qing Z, Su L, Zhang L. Handgcat: Occlusion-robust 3d hand mesh reconstruction from monocular images. In 2023 IEEE International Conference on Multimedia and Expo (ICME). 2023:2495-500.\u003c/li\u003e\n\u003cli\u003eHu R, Singh A. Unit: Multimodal multitask learning with a unified transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021:1439\u0026ndash;49.\u003c/li\u003e\n\u003cli\u003eNguyen S, Ng B, Kaplan AD, Ray P. Attend and decode: 4d fmri task state decoding using attention models. In Machine Learning for Health. 2020:267-79.\u003c/li\u003e\n\u003cli\u003eLong NHB. Step catformer: Spatial-temporal effective body-part cross attention transformer for skeleton-based action recognition. arXiv preprint 2023.\u003c/li\u003e\n\u003cli\u003eWang J, Liao N, Du X, Chen Q, Wei B. A semi-supervised approach for the integration of multi-omics data based on transformer multi-head self-attention mechanism and graph convolutional networks. BMC Genomics. 2024;25(1):86. Epub 20240122. doi: 10.1186/s12864-024-09985-7. PubMed PMID: 38254021; PMCID: PMC10802018.\u003c/li\u003e\n\u003cli\u003eMaksimova MA, Cansler HL, Zuk KE, Torres JM, Roberts DJ, Meeks JP. Interneuron Functional Diversity in the Mouse Accessory Olfactory Bulb. eNeuro. 2019;6(4). Epub 20190813. doi: 10.1523/ENEURO.0058-19.2019. PubMed PMID: 31358509; PMCID: PMC6712203.\u003c/li\u003e\n\u003cli\u003eBakalyar HA, Reed RR. Identification of a specialized adenylyl cyclase that may mediate odorant detection. Science. 1990;250(4986):1403-6. doi: 10.1126/science.2255909. PubMed PMID: 2255909.\u003c/li\u003e\n\u003cli\u003eNakajima D, Nakayama M, Kikuno R, Hirosawa M, Nagase T, Ohara O. Identification of three novel non-classical cadherin genes through comprehensive analysis of large cDNAs. Brain Res Mol Brain Res. 2001;94(1-2):85-95. doi: 10.1016/s0169-328x(01)00218-2. PubMed PMID: 11597768.\u003c/li\u003e\n\u003cli\u003eSharon D, Blackshaw S, Cepko CL, Dryja TP. Profile of the genes expressed in the human peripheral retina, macula, and retinal pigment epithelium determined through serial analysis of gene expression (SAGE). Proc Natl Acad Sci U S A. 2002;99(1):315-20. Epub 20011226. doi: 10.1073/pnas.012582799. PubMed PMID: 11756676; PMCID: PMC117558.\u003c/li\u003e\n\u003cli\u003eKoldaeva A, Zhang C, Huang YP, Reinert JK, Mizuno S, Sugiyama F, Takahashi S, Soliman T, Matsunami H, Fukunaga I. Generation and Characterization of a Cell Type-Specific, Inducible Cre-Driver Line to Study Olfactory Processing. J Neurosci. 2021;41(30):6449-67. Epub 20210607. doi: 10.1523/JNEUROSCI.3076-20.2021. PubMed PMID: 34099512; PMCID: PMC8318078.\u003c/li\u003e\n\u003cli\u003eAbdollahi Nejat M, Klaassen RV, Spijker S, Smit AB. Auxiliary subunits of the AMPA receptor: The Shisa family of proteins. Curr Opin Pharmacol. 2021;58:52-61. Epub 20210421. doi: 10.1016/j.coph.2021.03.001. PubMed PMID: 33892364.\u003c/li\u003e\n\u003cli\u003ePak JH, Huang FL, Li J, Balschun D, Reymann KG, Chiang C, Westphal H, Huang KP. Involvement of neurogranin in the modulation of calcium/calmodulin-dependent protein kinase II, synaptic plasticity, and spatial learning: a study with knockout mice. Proc Natl Acad Sci U S A. 2000;97(21):11232-7. doi: 10.1073/pnas.210184697. PubMed PMID: 11016969; PMCID: PMC17183.\u003c/li\u003e\n\u003cli\u003eCoulter PM, 2nd, Bautista EA, Margulies JE, Watson JB. Identification of cortexin: a novel, neuron-specific, 82-residue membrane protein enriched in rodent cerebral cortex. J Neurochem. 1993;61(2):756-9. doi: 10.1111/j.1471-4159.1993.tb02183.x. PubMed PMID: 8336151.\u003c/li\u003e\n\u003cli\u003eGuan J, Luo Y, Denker BM. Purkinje cell protein-2 (Pcp2) stimulates differentiation in PC12 cells by Gbetagamma-mediated activation of Ras and p38 MAPK. Biochem J. 2005;392(Pt 2):389-97. doi: 10.1042/BJ20042102. PubMed PMID: 15948714; PMCID: PMC1316275.\u003c/li\u003e\n\u003cli\u003eOyang EL, Davidson BC, Lee W, Poon MM. Functional characterization of the dendritically localized mRNA neuronatin in hippocampal neurons. PLoS One. 2011;6(9):e24879. Epub 20110914. doi: 10.1371/journal.pone.0024879. PubMed PMID: 21935485; PMCID: PMC3173491.\u003c/li\u003e\n\u003cli\u003eBraun JL, Geromella MS, Hamstra SI, Fajardo VA. Neuronatin regulates whole-body metabolism: is thermogenesis involved? FASEB Bioadv. 2020;2(10):579-86. Epub 20200902. doi: 10.1096/fba.2020-00052. PubMed PMID: 33089074; PMCID: PMC7566048.\u003c/li\u003e\n\u003cli\u003eHudmon A, Schulman H. Structure-function of the multifunctional Ca2+/calmodulin-dependent protein kinase II. Biochem J. 2002;364(Pt 3):593-611. doi: 10.1042/BJ20020228. PubMed PMID: 11931644; PMCID: PMC1222606.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"genome-biology","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"gbio","sideBox":"Learn more about [Genome Biology](https://genomebiology.biomedcentral.com/)","snPcode":"13059","submissionUrl":"https://submission.springernature.com/new-submission/13059/3","title":"Genome Biology","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Spatial transcriptomics, Hematoxylin and eosin images, transformer","lastPublishedDoi":"10.21203/rs.3.rs-4707959/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4707959/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eSpatial transcriptomics (ST) revolutionizes RNA quantification with high spatial resolution. Hematoxylin and eosin (H\u0026amp;E) images, the gold standard in medical diagnosis, offer insights into tissue structure, correlating with gene expression patterns. Current methods for predicting spatial gene expression from H\u0026amp;E images often overlook spatial relationships. We introduce ResSAT (Residual networks - Self-Attention Transformer), a framework generating spatially resolved gene expression profiles from H\u0026amp;E images by capturing tissue structures and using a self-attention transformer to enhance prediction. Benchmarking on 10x Visium datasets, ResSAT significantly outperformed existing methods, promising reduced ST profiling costs and rapid acquisition of numerous profiles.\u003c/p\u003e","manuscriptTitle":"ResSAT: Enhancing Spatial Transcriptomics Prediction from H\u0026amp;E- Stained Histology Images with Interactive Spot Transformer","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-08-09 21:29:40","doi":"10.21203/rs.3.rs-4707959/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-06-02T11:02:06+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-06-02T05:08:16+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"101916586300949715259565290780456314404","date":"2025-05-27T14:32:45+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-01-30T06:32:42+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"243741177286497411141624404468759403303","date":"2025-01-23T20:16:18+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-01-22T15:27:14+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-07-17T12:30:34+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-07-09T08:29:34+00:00","index":"","fulltext":""},{"type":"submitted","content":"Genome Biology","date":"2024-07-08T21:36:12+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"genome-biology","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"gbio","sideBox":"Learn more about [Genome Biology](https://genomebiology.biomedcentral.com/)","snPcode":"13059","submissionUrl":"https://submission.springernature.com/new-submission/13059/3","title":"Genome Biology","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"6fe4a477-b166-4d76-bd43-68fac28b921e","owner":[],"postedDate":"August 9th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-05-07T13:54:55+00:00","versionOfRecord":[],"versionCreatedAt":"2024-08-09 21:29:40","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4707959","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4707959","identity":"rs-4707959","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}