Meta-analysis suggests that the effects of demand characteristics can be consequential, unreliable, and difficult to explain

Demand characteristics are a fundamental methodological concern in experimental psychology. Yet, little is known about the direction, magnitude, consistency, and mechanisms underlying their effects. We conducted a three-level meta-analysis of 252 effect sizes from 52 studies that provided experimental tests of demand effects by explicitly manipulating cues about the study hypothesis. These manipulations tended to produce small overall increases in hypothesis-consistent responding (g = 0.21, 95% CI [0.12, 0.31]). However, effects were extremely heterogeneous (between-study τ = 0.28; within-study σ = 0.18), with the prediction interval ranging from g = 0.89 (a large increase in hypothesis-consistent responding) to g = -0.46 (a moderate decrease in hypothesis-consistent responding). Both the observed and estimated distribution of these effects suggested that demand characteristics can create false positives, false negatives, upward bias, and downward bias. This heterogeneity is currently difficult to explain. New participants who reviewed key study details were neither able to predict nor provide insights into psychological mechanisms theorized to underlie demand effects. Participants’ ratings of three theorized moderators – motivation to adjust responses, opportunity to adjust responses, and belief in the researcher’s hypothesis – failed to predict observed demand effects. Coded methodological features (e.g., whether participants were paid) also often failed to predict observed effects – explaining approximately 15% of in-sample variability. Although the meta-analysis did not capture the full depth of the demand characteristics construct, the synthesis of even a narrow subset of the literature suggests that their effects can be inferentially consequential, unreliable, and difficult to explain.