Think Indifferent: On the Perils Of Scientific Deliberation, Without Attention For Critical Evaluation

Replication attempts of empirical psychological phenomena lack guidance by a larger effort to evaluate thepsychological theory that predicted the phenomena. In this paper we present a template for theory evaluationsuited for the class of theories produced by psychological science: Theories of construction. As a case study,we perform a rigorous post-publication peer-review of the theoretical core of Unconscious Thought Theory(UTT). We present several uncomplicated indices that quantify the quality of evaluation of the results andconclusions by experts who reviewed the article and the amount of interpretation bias on behalf of theauthors. The results reveal a failure of expert peer-review to detect empirical reports of sub-standard quality.The analyses reveal there is in fact hardly any empirical evidence for the predictions of UTT in importantpapers that claim its support. Our advice is to evaluate before you replicate.NOTE: This manuscript is presented here for reference only see e.g.Nieuwenstein, M. R., Wierenga, T., Morey, R. D., Wicherts, J. M., Blom, T. N., Wagenmakers, E. J., & van Rijn, H. (2015). On making the right choice: a meta-analysis and large-scale replication attempt of the unconscious thought advantage. Judgment and Decision Making, 10(1), 1. http://journal.sjdm.org/14/14321/jdm14321.html. It will not be submitted for publication in its present form.