The psychology of experimental psychologists: Overcoming cognitive constraints to improve research

Experimental psychology is affected by a "replication crisis" that is causing concern in many areas of science. Approaches to tackling this crisis include better training in statistical methods, greater transparency and openness, and changes to the incentives created by funding agencies, journals and institutions. Here I argue that if proposed solutions are to be effective, we need also to take into account people's cognitive constraints that can distort all stages of the research process: designing and executing experiments, analysing data, and writing up findings for publication. I focus specifically on cognitive schemata in perception and memory, confirmation bias, systematic misunderstanding of statistics, and asymmetry in moral judgements of errors of commission and omission. Finally, I consider methods that may help mitigate the effects of cognitive constraints: better training, including use of simulations to overcome statistical misunderstanding, specific programs directed at inoculating against cognitive biases, adoption of Registered Reports to encourage more critical reflection in planning studies, and using methods such as triangulation and "pre mortem" evaluation of study design to make a culture of criticism more acceptable.