When Less Is Enough: Low-Rank Structure in DNA Sequence-to-Function Models

Motivation The rapid success of deep learning sequence-to-function (S2F) models has driven a trend toward ever larger architectures for regulatory genomics. While these models achieve meaningful predictive performance, their growing size and computational cost have made them increasingly opaque, difficult to deploy, and impractical for widespread use. As S2F models mature, an open question is how much model capacity is truly necessary to represent regulatory sequence function. Results We show that, within linear layers, much of the predictive power of state-of-the-art S2F models is concentrated in low-rank structure. Using post hoc singular value decomposition, we construct low-rank approximations of the Sei model that reduce model size by up to 90% while preserving over 90% correlation with full-model predictions. Strikingly, extremely low-rank models — including rank 1 — can match or exceed full-model performance on several variant-effect benchmarks, indicating that dominant regulatory signals lie in a surprisingly low-dimensional subspace. Combining low-rank approximation with static model quantization enables practical CPU inference, yielding over 5× speedups relative to the full Sei model and more than 100× faster inference than larger S2F models on comparable tasks. Applying the same approach to Borzoi and Enformer reveals consistent behavior, demonstrating that low-rank structure is a general property of modern S2F architectures rather than a model-specific artifact. Availability and Implementation Low-rank Sei is available at https://github.com/kostkalab/seillra . Contact [email protected] , [email protected]