Measurement Invariance Testing Works

Psychometricians have argued that measurement invariance (MI) testing is necessary to ascertain whether the same psychological traits are measured in different groups of people. Data from five experiments recently became available for testing whether this is an estimand of MI testing.In the first experiment, participants were administered a questionnaire on belief in free will and were given one of either a questionnaire on the meaning of life or the meaning of a nonsense concept called “gavagai”. Because the meaning of life and the meaning of gavagai conceptually differ, MI should have been violated when the groups were treated as if their questionnaires were identical. In fact, MI was severely violated, indicating the groups interpreted the questionnaires differently, as they should have if they measured different things.In the second and third experiments, participants were randomized to watch either treatment videos explaining the rules of figural matrices or task-irrelevant control videos. Participants were then administered intelligence tests and figural matrices tests. Since the intervention worked, because the experimental group had an additional influence on figural matrix performance in the form of knowing matrix rules, their performance on the matrices tests should have violated MI and been anomalously high for their ability levels. In both trials, MI was severely violated, indicating that the intervention worked, and the meaning of matrices tests differed by group.In the fourth and fifth experiments, individuals were subjected to a growth mindset intervention that a twin study revealed changed the amount of additive genetic variance in the mindset measure without affecting other indicators of self-determination. When comparing the treatment and passive control groups, MI was attainable before treatment, but after treatment, the mindset indicator became biased. Moreover, the control group’s self-determination measurement was longitudinally invariant, but the same was untrue for the treatment group, due to the mindset measure.Experimental evidence implies MI testing is a valid means of assessing whether the same things are measured in different groups.