A Machine-Learning-Based Approach for Detecting Item Preknowledge in Computerized Adaptive Testing

Item compromise and preknowledge have become common concerns in educational testing (Wollack & Schoenig, 2018). We propose a machine learning approach to simultaneously detect compromised items and examinees with item preknowledge in computerized adaptive testing. The suggested approach provides a confidence score that represents the confidence that the detection result truly corresponds to item preknowledge and draws on ideas in ensemble learning (Sagi & Rokach, 2018), conducting multiple detections independently on subsets of the data and then combining the results. Each detection first classifies a set of responses as aberrant using a self- training algorithm (Zhu & Goldberg, 2009) and support vector machine (Suykens & Vandewalle, 1999), and identifies suspicious examinees and items based on the classification result. The confidence score is adapted, using the autoencoder algorithm (Goodfellow, Bengio, & Courville, 2016), from the confidence score that Pan and Wollack (2022) suggested for non-adaptive tests. Simulation studies demonstrate that the proposed approach performs well in item preknowledge detection and the confidence score can provide helpful information for practitioners.