FloraBERT: cross-species transfer learning withattention-based neural networks for geneexpression prediction

Abstract Recent work in applying deep learning models has demonstrated that endophenotypes, such as RNA transcript abundance, can be predicted from an organism's regulatory DNA. However, due to the vast amount of labelled data required to train previous types of deep learning models, this work has been constrained to species with large amounts of data labelled for a particular task. Here, we present FloraBERT, a transfer-learning based deep learning model that is able to improve predictions of gene expression in a single target species, and it does so by exploiting cross-species genomic information in the form of genome assemblies from all of plantae. FloraBERT significantly outperforms simple bag-of-k-mers baseline models and achieves comparable performance to prior work that concerns less complex species. Furthermore, investigation of the learned parameters of FloraBERT reveals that the training process encodes biologically salient information, such as taxonomic similarity between species and positional relevance of nucleotides within a promoter. To facilitate future research, we have made the source code and model weights publicly available on https://github.com/benlevyx/florabert.