Comprehensive benchmark and architectural analysis of deep learning models for Nanopore sequencing basecalling

A bstract Nanopore-based DNA sequencing relies on basecalling the electric current signal. Basecalling requires neural networks to achieve competitive accuracies. To improve sequencing accuracy further, new models are continuously proposed. However, benchmarking is currently not standardized, and evaluation metrics and datasets used are defined on a per publication basis, impeding progress in the field. To standardize the process of benchmarking, we unified existing benchmarking datasets and defined a rigorous set of evaluation metrics. We benchmarked the latest seven basecaller models and analyzed their deep learning architectures. Our results show that overall Bonito has the best architecture for basecalling. We find, however, that species bias in training can have a large impact on performance. Our comprehensive evaluation of 90 novel architecture demonstrates that different models excel at reducing different types of errors and using RNNs (LSTM) and a CRF decoder are the main drivers of high performing models.