The Bots Ruining Social Science Aren’t Bots At All

Online data collection from human subjects currently faces a conundrum: it is both essential to how behavioral science functions and threatened by low-quality data. It is often assumed that random, inconsistent, and otherwise incomprehensible data in online surveys comes mainly from “bots.” Despite this assumption, few studies have directly examined where problematic data comes from, even though identifying the source has important implications for creating the right solutions. We examined this issue on several popular participant recruitment platforms, including Mechanical Turk and Lucid. Across four studies spanning five years using multiple methods, we provide evidence that most of the data quality problems affecting online research using online panels can be tied to fraudulent users from outside of the US—not bots. We identify many of the telltale signs that humans leave behind and describe the most effective ways of blocking problematic human responses to address the online data quality problem.