Stated vs. Revealled Robustness: How Biological LMs Behave Under Batch Effects and Lab Protocols

Biological language models (LMs) are increasingly applied to decode cellular and molecular sequences, yet their robustness to technical artifacts—such as batch effects and variations in lab protocols—remains poorly characterized. In this study, we systematically distinguish stated robustness (a model’s claimed or designed invariance to non-biological variation) from revealed robustness (empirically measured performance under controlled perturbations). Using three distinct single-cell RNA-seq and proteomic datasets with known batch structures and protocol variants, we evaluate a range of state-of-the-art biological LMs (including gene-specific and pan-species architectures). Our findings reveal a substantial gap between stated and revealed robustness: while models often incorporate batch-normalization or adversarial components in principle, their predictions degrade significantly when tested on unseen batches (up to 34% drop in cell-type classification F1) and across common protocol changes (e.g., 10x vs. Drop-seq). Notably, fine-tuning on multi-batch data improves revealed robustness only modestly, and no single architectural choice confers universal resistance. We further identify that representation entropy and attention entropy over technical covariates predict fragility better than model size or pretraining data volume. Finally, we provide a benchmark suite and actionable recommendations for reporting biological LM robustness, emphasizing that revealed , not stated, robustness should guide deployment in translational settings. Our results caution against overreliance on static validation splits and argue for explicit stress testing against batch and protocol shifts.