The complexity of measuring reliability in learning tasks: An illustration using the Alternating Serial Reaction Time task

Despite the fact that reliability estimation is crucial for robust inference, it is underutilized in neuroscience and cognitive psychology. Appreciating reliability can help researchers increase statistical power, effect sizes and reproducibility, decrease the impact of measurement error and inform methodological choices. However, accurately calculating reliability for many experimental learning tasks is challenging. In this study, we highlight a number of these issues, and estimate multiple metrics of internal consistency and split-half reliability of a widely used learning task, on a large sample of 180 subjects. We show how pre-processing choices, task length and sample size can affect reliability and its estimation. Our results show that the Alternating Serial Reaction Time Task has respectable reliability, especially when learning scores are calculated based on reaction times, and two-stage averaging. We also show that a task length of 25 blocks can be sufficient to meet usual thresholds for minimally acceptable reliability. We further illustrate how relying on a single point estimate of reliability can be misleading, and the calculation of multiple metrics, along with their uncertainties can lead to a more complete characterization of the psychometric properties of tasks.