Navigating complexity in real-world intervention evaluation: Insights from three case studies

In Intervention Mapping (IM) Step 6, evaluation takes place. In earlier IM steps, a planning group will have developed logic models to understand the health problem, and logic models of change on which the intervention is based. These logic models provide input for evaluators to develop evaluation questions. In terms of evaluation designs, there is a general distinction between effect evaluation and process evaluation. In effect evaluation, researchers determine whether the intervention is associated with changes in health, behavioral, or environmental outcomes, determinants, or performance objectives. Effect evaluation can be described as either efficacy or effectiveness. Efficacy refers to a program that is evaluated under controlled conditions, such as with motivated volunteers who participate in the program regardless of the time and effort required. Effectiveness refers to a program evaluated under real-world conditions, for example, with representatives of the at-risk group. Process evaluation seeks to describe how a program is implemented. Relevant questions related to process evaluation include whether an intervention was implemented as intended, for example in terms of adherence, whether contextual factors affected implementation, what hindered or facilitated implementation, and the extent to which the intended audience was satisfied with the intervention. In addition, process evaluation allows for critical reflection on intervention design and helps determine whether the success or failure of an intervention is due to the intervention itself or the way it was implemented or applied (Bartholomew Eldredge et al., 2016; Stutterheim et al., 2023).The evaluation design determines how confidently an evaluator can assess indicator changes and attribute them to the program. Randomized controlled trials (RCTs) are referred to as the gold standard of effect evaluation due to its rigor and ability to make causal inferences. RCTs require the selection of primary outcomes, careful randomization, and no changes in the content of the intervention or how it is delivered (Houle, 2015). This unfortunately misaligns with interventions in real-life settings that are already widely available, which limit randomization and control (Koelen et al., 2001). Moreover, the lengthy process of the RCT hinders evaluation of interventions that need to be developed and delivered quickly, for example because of a pandemic. Over the years, alternative research methods have become increasingly viable options (Bonten et al., 2020; Skivington et al., 2021). For example, in the Dutch and British frameworks for evaluation and recognition of interventions, qualitative and mixed research methods are now deemed valuable besides RCTs in ascertaining intervention effectiveness (Dutch Partnership for Recognition of Interventions, 2018; Skivington et al., 2021).The aim of this paper is not to discredit the RCT as a means of evaluating interventions. Rather, it aims to draw attention to the challenges and complexity of evaluating public health interventions in a real-world context. By presenting three cases of studies that, in their own way, addressed the challenges of the use of the RCT, we hope to contribute to the search for meaningful ways of evaluation.