Consistency within change: Evaluating the psychometric properties of a widely-used predictive-inference task

Rapid adaptation to sudden changes in the environment is a hallmark of human behaviour. Many computational, neuroimaging, and even clinical investigations, which capture this ability have relied on a behavioural paradigm known as the predictive-inference task. However, the psychometric quality of this task has never been examined, leaving unanswered whether it is indeed suited to capture behavioural variation on a within- and between-subject level. Using a large-scale test-retest design (N=330), we assessed the internal (internal consistency) and temporal (test-retest reliability) stability of the task’s relevant measures. We show that while the main measures capturing flexible adaptation yield good internal consistency and overall satisfying test-retest reliability, more complex markers of flexible behaviour lack convincing psychometric quality. Our findings have implications for the large corpus of previous studies using this task and provide clear guidance as to which measures should and should not be used in future studies.