Evidence-Based Adaptivity Decisions for Educational Interventions

Digital technologies enable adaptive support by tailoring assistance to students’ behavior. However, determining optimal support for each student is methodologically challenging: If later adaptations depend on prior behavior, adaptivity decisions are not fully randomized, complicating causal inference. We introduce a methodological approach based on sequential experimental designs and directed acyclic graphs (DAGs) to evaluate adaptivity decisions and apply it to a mobile intervention designed to promote frequent vocabulary study. Students received planning prompts during a 20-day first-stage intervention, followed by an adapted prompting schedule in the second stage. Using a DAG, we formalized causal assumptions, identified confounding, and derived appropriate adjustment strategies. The number of prompts scheduled for students showed no significant effects, likely due to low engagement. In contrast, the number of prompts students actually received had a significant positive effect. Furthermore, stage-2 effects depended on prior study activity and prompt exposure, supporting the value of adapting support.