Analyzing Test-Taking Behaviors Through Process Data Using the IRT Explanatory Model for Guessing

Guessing behavior is a known problem in international large-scale assessments (ILSAs), and it can impact the assessment's reliability and validity by introducing random error and bias. Testing and accounting for such biases is essential in ensuring the accuracy and comparability of test scores and secondary analysis results. This chapter proposes a flexible item response theory (IRT) modeling approach for guessing behavior: the IRT Explanatory Model for Guessing (IRT-EMG). The IRT-EMG combines mixture models with response time and other process data information (number of actions) from computer-based ILSAs. It presents an extension of the grade of membership response time model for detecting guessing behaviors proposed by Pokropek (2016) and is based on the grade of membership (GoM) model (Erosheva, 2005) and the HY-BRID model (Yamamoto, 1989; Yamamoto & Everson, 1997). We show that IRT-EMG is a generalization of the classical 3PL-IRT model and that the 3PL-IRT model could be presented as a restricted case of the IRT-EMG. This chapter describes the features and advantages of the IRT-EMG model and illustrates its application on data from the Programme for the International Assessment of Adult Competencies (PIAAC). Results show how information about different characteristics of items, respondents, and process data (response time, number of actions) might help detect different types of guessing behaviors and identify items that are particularly prone to guessing.