Many Pipelines, One Dataset: Analytic Flexibility and Its Impact in EEG Research

The EEGManyPipelines initiative included 396 scientists (168 teams) who independently analyzed the same dataset to test the same hypotheses, mapping the full landscape of analytical decisions. This captured an underexamined source of variability in research results: data processing choices needed to arrive at an analyzable dataset. Considering full analysis pipelines, ERP variability across teams was equivalent to variability across participants, suggesting that who analyses the data is as important as who provides it. In contrast to prior many-analyst projects, typically explaining less than 10% of variance in estimates, our granular codings of researcher choices explained 53%, 75%, and 51% of the variance in difference waves across three hypotheses, with unusual results partly attributable to deviations from the prototypical pipeline. Despite the heterogeneity in approaches and estimates, teams reached a substantial consensus on effect presence. By being transparent across all analysis stages, researchers can identify uncertainty and still draw meaningful inferences.