Sources of Bias in General Linear Models: Evaluating the Analytic Practice in Psychological Research

Unbiased estimation and hypothesis testing in General Linear Models requires several conditions to be met, such as adherence to statistical assumptions and absence of influential data points. This study examined the extent to which researchers in psychology attend to these sources of bias when presented with analytic tasks. We recruited 188 psychology researchers who completed two data analytic scenarios. Researchers' analytic scripts and descriptions were subsequently coded based on their level of attentiveness to issues with non-normal model errors, heteroscedasticity, outliers, and influential cases. We found that for all categories, researchers either performed no checks for potential issues or performed checks that were insufficient to detect the issue at hand. Researchers who teach research methods were more likely to perform the correct checks than non-teaching faculty, post-doctoral researchers and Ph.D. students, however this difference was small and the highest posterior density intervals overlapped substantially. We discuss the implications of routinely neglecting violated assumptions and influential cases, and present the case for more frequent application of robust statistical methods to supplement or replace ordinary least squares general linear models.