The Trouble with Deviation Scores: Misinterpreting Mean Shifts and Variability

Deviation-based metrics (e.g., mean absolute deviations-MAD) are commonly used toassess anchoring effects, judgment accuracy, and related phenomena. However, thismanuscript demonstrates that these metrics inherently conflate meaningful shifts in meanjudgment positions with incidental changes in variance, potentially misleading researchersabout treatment effects. Using intuitive examples and rigorous simulations, we show thatdeviation scores frequently produce results that contradict the true underlying effects. Wefurther reanalyze previous anchoring and wisdom-of-the-inner-crowd studies to illustratethese problems in practice. Ultimately, we argue for a conceptual and analytical separationof position from variance.