Reflection-Philosophy Order Effects and Correlations: Aggregating and comparing results from mTurk, CloudResearch, Prolific, and undergraduate samples

Reflective reasoning often correlates with certain philosophical decisions, but it is often unclear whether reflection causes those decisions. So a pre-registered experiment assessed how reflective thinking relates to decisions about 10 thought experiments from epistemology, ethics, and philosophy of mind. Participants from the United States were recruited from Amazon Mechanical Turk, CloudResearch, Prolific, and a university. One participant source yielded up to 18 times as many low-quality respondents as the other three. Among remaining respondents, some prior correlations between reflective and philosophical thinking replicated. For example, reflection predicted denying that accidentally justified true beliefs count as knowledge. However, no reflection test priming effects were detected. Rather, an unexpected philosophical reflection effect emerged: making the philosophical decisions before the reflection test improved test performance. These and other data suggest causal paths between reflection and philosophy can go both directions, but detecting such results can depend on factors such as data quality.