Can’t make it better nor worse: An empirical study about the effectiveness of general rules of item construction on psychometric properties

Some of the most popular psychological questionnaires violate general rules of item construction: precise, positively keyed items without negations, multiple aspects of content, absolute statements or vague quantifiers. To investigate if following these rules results in more desirable psychometric properties, 1733 participants completed online either the original NEO Five-Factor Inventory, an “improved” version whose items follow the rules of item construction, or a “deteriorated” version whose items strongly violate these rules. We compared reliability estimates, item-total correlations, CFA model fit, and fit to the partial credit model between the three versions. Neither of the manipulations resulted in considerable or consistent effects on any of the psychometric indices. Our results question the ability of standard analyses in test construction to distinguish good items from bad ones, as well as the effectiveness of general rules of item construction. To increase the reproducibility of psychological science, more focus should be laid on improving psychological measures.