STASCAN deciphers fine-resolution cell-distribution maps in spatial transcriptomics by deep learning

Background The spatial transcriptomics (ST) technologies have been widely applied to decode the spatial distribution of cells by resolving gene expression profiles in tissues. However, a fine-resolved spatial cell map is still limited by algorithmic tools and sequencing techniques. Results Here we develop a novel deep learning approach, STASCAN, which could define the spatial cellular distribution of both captured and uncharted areas by cell feature learning that combines gene expression profiles and histology images. STASCAN additionally adopts optional transfer learning and pseudo-labeling methods to improve the accuracy of the cell-type prediction from images. We have successfully applied STASCAN to enhance cell resolution, and revealed finer organizational structures across diverse datasets from various species and tissues generated from 10× Visium technology. STASCAN improves cell resolution of Schmidtea mediterranea datasets by six times and reconstructs more detailed 3D cell-type models. Furthermore, STASCAN could accurately pinpoint the boundaries of distinct cell layers in human intestinal tissue, specifically identify a micrometer-scale smooth muscle bundle structure in consistent with anatomic insights in human lung tissue, and redraw the spatial structural variation with enhanced cell patterns in human myocardial infarction tissue. Additionally, through STASCAN on embryonic mouse brain datasets generated by DBiT-derived MISAR-seq technology, the increased cellular resolution and distinct anatomical tissue domains with cell-type niches are revealed. Collectively, STASCAN is compatible with different ST technologies and has notable advantages in generating cell maps solely from histology images, thereby enhancing the spatial cellular resolution. Conclusions In short, STASCAN displays significant advantages in deciphering higher-resolution cellular distribution, resolving enhanced organizational structures and demonstrating its potential applications in exploring cell-cell interactions within the tissue microenvironment.