Assessing effect sizes, variability, and power in the web-based study of language production

Abstract With the pandemic, many experimental psychologists/linguists have started to collect data over the internet (hereafter “on-line data”). The feasibility of such experiments and the sample sizes required to achieve sufficient statistical power in future experiments have to be assessed. This in turn requires information on effect sizes and variability. We present a series of analyses to compare repeated measurement data across settings. We apply these analyses to response time from the same word production experiment conducted in the lab and on-line. These analyses allow determining whether the two settings differ in effect sizes, in the consistency of responses over the course of the experiment, in the variability of average response times across participants, in the magnitude of effect sizes across participants, or in the amount of unexplained variability. We assess the impact of these differences on the power of the design in a series of simulations. Our findings temper the enthusiasm raised by previous studies and suggest that on-line production studies might be feasible but at a non-negligible cost. The sample sizes required to achieve sufficient power in on-line language production studies come with a non-negligible increase in the amount of manual labour. Findings further show that researchers cannot simply assume that effect sizes are similar across settings.