Negative Binomial Additive Model for RNA-Seq Data Analysis

SUMMARY High-throughput sequencing experiments followed by differential expression analysis is a widely used approach for detecting genomic biomarkers. A fundamental step in differential expression analysis is to model the association between gene counts and co-variates of interest. Existing models assume linear effect of covariates, which is restrictive and may not be sufficient for some phenotypes. In this paper, we introduce NBAMSeq, a flexible statistical model based on the generalized additive model and allows for information sharing across genes in variance estimation. Specifically, we model the logarithm of mean gene counts as sums of smooth functions with the smoothing parameters and coefficients estimated simultaneously within a nested iterative method. The variance is estimated by the Bayesian shrinkage approach to fully exploit the information across all genes. Based on extensive simulation and case studies of RNA-Seq data, we show that NBAMSeq offers improved performance in detecting nonlinear effect and maintains equivalent performance in detecting linear effect compared to existing methods. Our proposed NBAMSeq is available for download at https://github.com/reese3928/NBAMSeq and in submission to Bioconductor repository.