Creating Carelessness: A comparative analysis of common techniques for the simulation of careless responder data

It is an accepted fact in survey research that not all participants will respond to items with the thoughtful introspection required to produce a valid response. When participants respond without sufficient effort their responses are considered to be careless, and these responses represent error. Many methods exist for the detection of these individuals (Huang, Curran, Keeney, Poposki, & Deshon, 2012; Johnson, 2005; Meade & Craig, 2012), and several techniques exist for testing their effectiveness. These techniques often involve generating careless responses through some process, then attempting to detect those known cases in otherwise normal data. One method to produce these data is through the simulation of data with varying degrees of randomness. Despite the common use of this technique, we know little about how it actually maps onto real world data. The purpose of this paper is to compare simulated data with real world data on commonly used careless response metrics. Results suggest that care should be applied when simulating data, and that decisions researchers make when generating this data can have large effects on the apparent effectiveness of these metrics. Despite these potential limitations, it appears that with proper use and continued research simulation techniques can still be quite valuable.