Using mixture modeling to examine differences in perceptual decision-making as a function of the time and method of participant recruitment

We examine whether perceptual decision-making differs as a function of the time in the academic term and whether the participant is an undergraduate participating for course credit, a paid in-lab participant, or a paid online participant recruited via Amazon Mechanical Turk. To accomplish this, we demonstrate a mixture modeling approach that separates stimulus-driven responses from contaminant responses, allowing us to distinguish between the degree of focus demonstrated when engaged in the task and the consistency in this task focus. We report a survey of cognitive psychologists' expectations regarding the quality of data obtained from these different participant groups. Cognitive psychologists expect performance and response caution to be lowest among undergraduate participants who enroll at the end of the academic term, and highest among paid in-lab participants. We then present two experiments that test these expectations using two common paradigms. We found little evidence for time-of-term effects. The participant groups responded to standard discrimination difficulty and speed/accuracy emphasis manipulations in similar ways. However, participants recruited via Mechanical Turk had higher rates of guessing and, consequently, poorer motion discrimination performance (but not brightness discrimination performance). Comparison of the mixture versus standard modeling approaches suggested diverging conclusions about how speed/accuracy emphasis affected performance, suggesting that some of this relationship may be due to differences in the rate of guessing. We conclude that online crowdsourcing platforms can provide quality perceptual decision-making data, but recommend that mixture modeling be used to adequately account for data generated by processes other than the psychological phenomena under investigation.