Exploration of suboptimal modeling choices - Ordinal modeling as a way to better understand effect size heterogeneity?

Heterogeneity in population effect sizes has often been suggested as impairing replication success. The validity of this line of argumentation rests on the assumption that reported heterogeneity estimates provide an accurate description of effect size heterogeneity. However, efforts to precisely measure between-study heterogeneity may be affected by inadequate model specification. Most primary analyses in psychology treat Likert-scaled data as continuous and not as ordered-categorical therefore applying models that strictly speaking are misspecified. This can lead to various statistical issues like inflated error rates and distorted estimates which introduce biases in the heterogeneity estimation. To assess the impact such model misspecifications have, 16 effects from the multi-lab replication projects Many Labs and Psychological Science Accelerator with a total of 94.863 participants clustered in 644 single replication sites were re-analysed with Bayesian ordinal regression models. The reanalysis allows for a comparison between heterogeneity estimates obtained using either the ordinal or standard linear model which treats Likert scales as continuous measures. The comparison of heterogeneity estimates between the two modeling approaches showed only small differences. However, generalisability is limited as some of the assessed effect sizes are close to zero and show little heterogeneity. Based on these results, further implications on the role of model choice for better assessment of between-study heterogeneity are discussed.