Can High-Dimensional Questionnaires Resolve the Ipsativity Issue of Forced-Choice Response Formats?

Forced-choice questionnaires can prevent faking and other response biasestypically associated with rating scales. However, the derived trait scoresare often unreliable and ipsative, making inter-individual comparisons inhigh-stakes situations impossible. Several studies suggest that these problemsvanish if the number of measured traits is high. To determine the necessarynumber of traits under varying sample sizes, factor loadings, and intertrait-correlations, simulations were performed for the two most widely usedscoring methods, namely the classical (ipsative) approach and ThurstonianIRT models. Results demonstrate that while especially Thurstonian IRTmodels perform well under ideal conditions, both methods yield insufficientreliabilities in most conditions resembling applied contexts. Moreover, notonly the classical estimates but also the Thurstonian IRT estimates forquestionnaires with equally keyed items remain (partially) ipsative, evenwhen the number of traits is very high (i.e., 30). This result not only questionsearlier assumptions regarding the use of classical scores in high dimensionalquestionnaires, but it also raises doubts about many validation studies onThurstonian IRT models because correlations of (partially) ipsative scoreswith external criteria cannot be interpreted in a usual way.