Utilizing Primary Study Quality in Meta-Analyses in Psychology: A Step-by-Step Tutorial

Evaluating the quality of primary studies is a key step in meta-analyses in psychology. This step is aimed at reducing the risk of bias and establishing the validity of the inferences drawn from the meta-analytic findings. However, the extant body of research offers little guidance on how to represent and incorporate primary study quality (PSQ) in meta-analyses, and some common procedures, such as creating sum scores from a set of quality indicators, often lack the backing from measurement models. Addressing these issues, we present a tutorial that guides researchers in their analytic decisions and approaches to represent and incorporate PSQ in their meta-analyses. Specifically, we describe, review, and illustrate approaches to (a) represent PSQ by single or multiple quality indicators or aggregated scores; (b) examine the moderator effects of PSQ; and (c) test the sensitivity of moderator effects to PSQ. We illustrate these approaches and present a step-by-step tutorial with analytic code for researchers’ guidance. We also encourage meta-analysts to take a measurement perspective on representing PSQ if multiple quality indicators are aggregated into a quality score. Moreover, we argue for conducting moderator sensitivity analyses to obtain more evidence on the impact of PSQ in a meta-analysis.