Item Construction Rules Revisited: Learnings from Measurement of Latent Variables with Gold-Standard Items

This article investigates whether gold-standard items are helpful for questionnaire construction.Based on an initial item pool (138 items) derived by a deductive item construction (𝑁 = 124) andprototype approach (𝑁 = 24), followed by cognitive interviews (𝑁 = 8) and a pilot study (𝑁 =390) of a preliminary item set (61 items), three 12-item scales were constructed to measure thephysical traits of body height, body weight, and age. We collected data on these scales usingresponse formats with either two (𝑁 = 921) or six (𝑁 = 933) categories. We also collectednumeric self-reports of body weight, body height, and age as gold-standard items. Confirmatoryfactor analyses revealed that the gold-standard items did not consistently exhibit the highestloading on their corresponding latent variable. Furthermore, when controlling for the self-reported physical body height, body weight, and age as gold-standard items, as well as gender,we did not always find an interpretable, systematic residual variance. Finally, the pattern ofcorrelations between the latent variables did not reflect the correlations between the self-reportedgold-standard items, suggesting that the item scales and the gold-standard items do not have thesame validity. While these results are consistent with previous studies, our analyses also showedthat items with two response categories were at least as valid as those with six categories,contradicting past recommendations. When constructing a questionnaire, we would argue that theitems intended to measure the latent variable most directly should have the highest loading onthat variable. If this is not the case, the content validity is questionable at best. The implicationsare that intensive cognitive pretesting is necessary. Questionnaires with different responseformats should be compared empirically and hypotheses about which items best represent thelatent variable should be tested.