Balancing Between Categorical and Dimensional Assessment in Short-Scale Construction Using Ant Colony Optimization

Language proficiency assessment poses particular challenges for test developers in selecting items that allow for a clear assignment of individuals to language proficiency levels (categorical assessment), while at the same time providing a reliable and comprehensive dimensional assessment of language proficiency. We show how Ant Colony Optimization (ACO) can be used to achieve a balance between these measurement goals, using a German entry-level language assessment as a working example. We tailored competing ACO algorithms to develop short scales of different lengths that met several pre-specified criteria, including model fit, composite reliability, and criterion validity. In optimizing the short scales, we favored either accurate dimensional assessment (model fit and composite reliability), between-category classification accuracy (a high polychoric correlation between model-predicted and independently assessed proficiency levels), or a balance of both. We argue that scale optimization strategies such as ACO are essential for balancing conflicting measurement goals such as optimizing between categorical and dimensional assessment.