Improving statistical inferences: Can a MOOC reduce statistical misconceptions about p-values, confidence intervals, and Bayes factors?

Statistical reforms in psychology and surrounding domains have been largely motivated by doubts surrounding the methodological and statistical rigor of researchers. In light of high rates of p-value misunderstandings, against the backdrop systematically raised criticisms of NHST, some authors have deemed misconceptions about p-value “impervious to correction” (Haller & Krauss, 2002, p. 1), while others have advocated for the entire abandonment of statistical significance in place of alternative methods of inference (e.g., Amrhein, Greenland, & McShane, 2019). Surprisingly little work, however, has empirically investigated the extent to which statistical misconceptions can be improved reliably within individuals, nor whether these improvements are transient or maintained. The current study (N = 2,320) evaluated baseline misconception rates of p-value, confidence interval, and Bayes factor interpretations among online learners, as well as rates of improvement in accuracy across an 8-week massive open online course (MOOC). Results demonstrated statistically significant improvements in accuracy rates, across all three concepts, for immediate learning (first post-test), as well as support for retained learning until week 8 (second post-test). The current work challenges the idea that statistical misconceptions are impervious to correction, and highlights the importance of pinpointing those more problematic misunderstandings that may warrant more efforts to support improvement. Before we abandon one of the most widely used approaches to statistical inferences in place of an alternative, it is worth exploring whether it is first possible to improve existing misunderstandings amid current methods used.