Balancing stability and flexibility: Investigating a dynamic K value approach for the Elo rating system in adaptive learning environments

In adaptive digital learning environments, it is essential to track learning trajectories. The Elo rating system, known for its computational simplicity, is frequently employed for this purpose. Current Elo-based systems cannot handle rapid changes in ability or are unable to balance accuracy and speed when updating player and item ratings. Changes in Elo ratings depend on the sensitivity parameter $K$. Using fixed $K$ values necessitates a trade-off: Larger values facilitate the tracking of evolving ability levels but introduce greater rating volatility. Smaller values yield more stable estimates, but are slower to reflect actual ability levels. Existing modifications of the Elo system, which diminish $K$ as the number of responses increases, are inadequate in scenarios characterized by considerable ability fluctuation, a common occurrence in digital learning environments. To address this challenge, we introduce a novel approach for dynamically adjusting $K$ values in response to observed trends in rating changes. This method increases $K$ during noticeable upward or downward shifts in ratings and reduces it otherwise. We present a computationally efficient implementation of this idea and validate its superiority over existing $K$ adjustment strategies through simulation studies. Additionally, we describe the implementation of this adaptive $K$ model in a widely-used digital learning platform, Math Garden, which leverages both accuracy and response time in its assessments. By successfully integrating speed and precision, this innovative implementation enhances the effectiveness of digital adaptive learning environments