Irrelevant Events and Voting Behavior: Replications Using Principles from Open Science

Replication and transparency are increasingly important in bolstering the credibility of political science research, yet open science tools are typically designed for experiments. For observational studies, current replication practice suffers from an important pathology: just as researchers can often "p-hack" their way to initial findings, it is often possible to "null hack" findings away through specification and case search. We propose an observational open science framework that consists of extending the original time series, independent data collection, pre-registration, multiple simultaneous replications, and collaborators with mixed incentives. We apply the approach to three studies on "irrelevant" events and voting behavior. Each study replicates well in some areas and poorly in others. Had we sought to debunk any of the three with ex post specification search, we could have done so. However, our approach required us to see the full, complicated picture. We conclude with suggestions for future refinements to our approach.