Debugging the Internet Survey: A Tutorial for Detecting Fraudulent Responses

Online research is increasingly popular to reach under-represented populations and expand research samples; however, it is also vulnerable to fraud in the form of “bots” (automated respondents, e.g., Large Language Models) and humans, such as completing a survey multiple times or when they are not eligible. Often, “infiltration” by such fraudulent responses is only noted and addressed after data collection has started. Here we present a classification process consisting of automated and manual detection techniques (“flags”) with practical suggestions for customization and implementation in other research contexts, grounded in the literature and our own experiences in a large online survey of video gaming behaviours. Using this protocol, we screened out 7638 likely fraudulent responses out of 8474 (90.13%). Among the flags implemented, those related to participant address, overall survey speed, and pattern of responses in “batches” provided the best performance. Ultimately, we recommend using multiple routes and layers to detect possibly fraudulent responses rather than reliance on any one or two indicators. By proactively considering how to detect and respond to fraud, researchers can have greater confidence in the integrity of their data, preserve limited resources, and minimize burden from post hoc fraud identification.