How to Use Item Desirability Ratings in Constructing Forced-Choice Tests

The forced-choice response format has been proposed as a method for preventing applicant faking on self-report non-cognitive measures. This potential benefit of the format depends on how closely the items comprising each forced-choice block are matched in terms of desirability for the job. Current desirability matching procedures rely on differences in items’ mean desirability ratings to quantify similarity in items’ desirability. We argue that relying on means, while ignoring individual differences in desirability ratings, may yield inaccurate similarity values and result in inferior item matches. As an alternative, we propose a distance-based measure that considers differences in desirability ratings at the individual level and may thus yield accurate similarity values and optimal matches. We support our arguments on a set of desirability ratings obtained with an explicit instruction to rate desirability of items.