Do model-based and model-free reinforcement learning correspond to goal-directed and habitual actions, respectively? A systematic review.

The idea that model-based reinforcement learning (RL) corresponds to goal-directed actions and model-free RL corresponds to habitual actions is deeply entrenched in psychology. To test this hypothesis, we will first review and evaluate the evidence base from behavioral and neural studies. Behaviorally, positive correlations between behavioral parameters that index model-based/model-free RL and goal-directed/habitual actions would support the hypothesis. Neurally, overlapping neural substrates that underlie the model-based/goal-directed constructs and between the model-free/habitual constructs, as well as a dissociation between the neural substrates underlying both groups of constructs, would support the hypothesis. We will then discuss alternative classes of computational theories beyond the model-based/model-free framework that purport to describe goal-directed and habitual behaviour, and compare these theories against the model-based/model-free framework as well as against each other. Some of the alternative theories covered in this review include dichotomy-based frameworks (e.g., value-based vs. value-free), hierarchical frameworks (e.g., action sequences or active inference), and biological frameworks (e.g., actor-critic models). We then outline potential approaches to synthesize the findings and future avenues of research. Overall, we find that model-based RL maps onto certain facets of goal-directed actions but not necessarily onto other facets. On the other hand, there is much evidence suggesting that model-free RL does not track habitual actions.