Hidden multiplicity in the analysis of variance (ANOVA): multiple contrast tests as an alternative

In bio-medical studies, the p-values of the F-tests in ANOVA are usually interpreted independently as measures of the significance of the associated factors. This ’hidden multiplicity’ effect increases the false positive rate. Therefore, Cramer et al. (2016) proposed the Bonferroni adjustment of the p-values to control for familywise error rate for the experiment. Here, instead of using F-tests, it is alternatively suggested to use multiple contrast tests vs. total mean and to perform multiplicity adjustment by object merging in the interplay between the R-packages emmeans and multcomp . This new approach, denotes as multipleANOM , allows not only to interpret global factor effects but also local effects between factor levels as adjusted p-values or simultaneous confidence intervals for selected effect measures in generalized linear models. R-code is provided by means of selected data examples.