Reflection-Philosophy Order Effects and Correlations: Aggregating and comparing results from mTurk, CloudResearch, Prolific, and undergraduate samples

How does reflective thinking impact decisions about ethics, mind, politics, or other philosophical domains? Reflective reasoning often correlates with better decision-making performance and certain philosophical preferences (e.g., utilitarian moral decisions). However, experiments suggest that reflection is not always the cause of these outcomes. Further, some evidence casts doubt on the trustworthiness of data from certain online crowd work platforms such as Amazon Mechanical Turk (mTurk). This paper reports results of a pre-registered experiment on participants from multiple sources (mTurk, CloudResearch, Prolific, and a university). The experiment investigated how reflective thinking relates to judgments about philosophical thought experiments (concerning knowledge, morality, personal identity, and more). First- and third-party data quality measures found up to 18 times as much low-quality data from mTurk as other sources. Analyzing across participant sources, some prior correlations between reflection and certain philosophical tendencies replicated (e.g., denying that accidentally true beliefs count as knowledge). However, a common reflection test prime had no detectable effect on the corresponding philosophical preferences. Rather, a philosophical reflection effect emerged: thinking through philosophical vignettes before taking a reflection test improved test performance—but only among those that passed the data quality checks. Together, these data suggest that the direction of causation between reflective and philosophical thinking may be the opposite of what dual process theories have suggested, and that researchers’ ability to detect such relationships can depend on participant source and data quality.