Reliability Coefficient for Bayesian Knowledge Tracing Models

Knowledge tracing (KT) refers to the process of efficiently tracking student achievement in online learning systems. Bayesian knowledge tracing (BKT) is a representative statistical model used for this process. Despite the widespread application of BKT in predicting student performance and modeling knowledge acquisition, methods to evaluate the consistency of measurement by the BKT models remain unestablished. In psychometrics, reliability is a fundamental concept that gauges the degree to which an assessment tool generates stable and consistent results. Evaluation of reliability is crucial because without reliable measurement, the adequacy of educational assessments and the decisions based on them would be compromised. To address the lack of a method to measure reliability, we propose a novel approach for estimating the reliability coefficient by extending the existing method for diagnostic classification models to time-series data. By applying the proposed method to actual response data, we demonstrate its capability to assess the reliability of BKT models, highlighting the importance of evaluating reliability and its potential for improving individualized learning experiences.