CoSTA: Unsupervised Convolutional Neural Network Learning for Spatial Transcriptomics Analysis

The rise of spatial transcriptomics technologies is leading to new insights about how gene regulation happens in a spatial context. Here, we present CoSTA: a novel approach to learn spatial similarities between gene expression matrices via convolutional neural network (ConvNet) clustering. By analyzing simulated and previously published spatial transcriptomics data, we demonstrate that CoSTA learns spatial relationships between genes in a way that emphasizes whole patterns rather than pixel-level correlation. CoSTA provides a quantitative measure of how similar each pair of genes are by their spatial pattern rather than only classifying genes into categories. We find that CoSTA identifies narrower, but biologically relevant, sets of significantly related genes as compared to other approaches.