Using model-based predictions to inform the mathematical aggregation of human-based predictions of replicability

Crowd-sourced human judgments about the trustworthiness of research claims may be used to generate forecasts of the probability of those claims being successfully replicated . Predictive models are less time and resource-intensive methods of generating forecasts about the likely replicability of research claims , however, are likely to be less accurate than human judgments. In this paper we propose and demonstrate a method that combines both crowd-sourced human judgments and model-based predictions of the likely replicability of research claims. We employ a Bayesian model that takes the model-based predictions as the prior data, and updates these values using crowd-sourced human judgments to generate predictions of replication outcomes using an annotated database of eight large-scale replication studies.