Cognitive reflection and Normality Identities: two new benchmarks for models of probability judgments

We propose two novel benchmarks for assessing models of probability judgments: the impact of Cognitive Reflection Test (CRT) on probability judgment expressions and 16 "normality identities" expected to sum to 1 under classical probability theory. We compared three models on these benchmarks – the Probability Plus Noise Model (PPN), the Bayesian Sampler (BS), and the Quantum Sequential Sampler (QSS) – using the largest dataset to date for probability judgments. Our results reveal that higher CRT scores are associated with fewer probabilistic fallacies and identity violations, a trend most accurately captured by the QSS, although we also identified QSS limitations. Regarding the normality identities, the QSS outperformed the PPN and the BS, which had difficulty with both the average values of the normality identities and their dependence on CRT scores. Additionally, we uncovered a unique "1 crossing" effect for normality identities N8 and N11, an effect PPN and BS cannot capture.