Testing delayed, gradual, and temporary treatment effects in single-case experiments: A general response function framework

Randomization tests represent a class of significance tests to assess the statistical significance of treatment effects in randomized single-case experiments. Most applications of single-case randomization tests concern simple treatment effects: immediate, abrupt, and permanent changes in the level of the outcome variable. However, in educational practice, these simple effects are the exception rather than the rule. In many educational applications, researchers are confronted with delayed, gradual, and temporary treatment effects; in general with “response functions” that are markedly different from single-step functions. In the current text we introduce a general framework that allows specifying a test statistic for a randomization test based on predicted response functions that is sensitive for a wide variety of data patterns. For instance, the response functions allow for different latencies (degrees of delay) of effect, for abrupt versus gradual effects, and for different durations of the effect (permanent or temporary). There may be reasonable expectations regarding the kind of effect (abrupt or gradual), entailing a different focal data feature (e.g., level or slope). However, the exact amount of latency and the exact duration of a temporary effect may not be known a priori. Therefore, a multiverse approach can be followed exploring the effect of specifying different latencies or delayed effects and different durations for temporary effects. In the paper, we illustrate the randomization tests analyses, using response functions as a test statistic and following a multiverse approach, with real data. We also provide a user-friendly and freely available web application implementing several response functions from a multiverse perspective.