Key Concepts in Online Learning and Decision Making for Just-in-Time Adaptive Interventions

Background: Just-in-time adaptive interventions (JITAIs) provide digital health support that is timely and adaptive: intervention delivery is tailored to internal and external factors that may change rapidly, known as “tailoring variables.” JITAIs make decisions on how to tailor support using “decision rules.” Decision rules specify which intervention option should be selected (e.g., whether to send a text-based intervention message) based on tailoring variables. JITAI decision rules are typically designed via expert opinion – informed by domain knowledge and statistical analyses of past data – and kept the same throughout a JITAI’s deployment. A challenge for JITAIs is that decision rules informed by past data can become stale or outdated for several reasons, including differences between recipients of a JITAI from one deployment to the next and broader societal shifts (e.g., post-pandemic behavior shifts). As a result, JITAIs may suffer from reduced effectiveness in deployment. Purpose: Herein, we describe how online learning and decision-making algorithms create “personalizing JITAIs” (pJITAIs) to address this gap, enabling pre-specified, within-deployment modifications to decision rules based on streaming data collected during deployment. This streaming data provides up-to-date insights on how relationships between tailoring variables, intervention options, and outcomes may differ. Methods: We detail key concepts for online learning and decision-making algorithms, unifying two dominant perspectives in the pJITAI literature: reinforcement learning (RL) and control engineering. Conclusions: Although RL and control engineering approaches have generally progressed in parallel, we elucidate their substantial overlap and unify terminology to streamline the collaborative design of online learning and decision-making algorithms for pJITAIs.