Evaluating false positive rates of standard and hierarchical measures of metacognitive accuracy

A key aspect of metacognition is metacognitive accuracy, i.e., the degree to which confidence judgments differentiate between correct and incorrect trials. To measure metacognitive accuracy, researchers are faced with a choice between a plethora of established and novel options. The present study investigated false positive rates associated with various measures of metacognitive accuracy by hierarchical resampling from datasets of the confidence database to accurately represent the statistical properties underlying confidence judgements. Many measures that require the computation of summary-statistics separately for each participant and a subsequent group-level analysis using these summary-statistics performed adequately in terms of false positive rate, including gamma correlations, confidence slopes, meta-d’, and the area under the type 2 ROC curves. Meta-d’/d’ was associated with a false positive rate below the nominal alpha level, but log-transformed meta-d’/d’ performed adequately. Measures based on a hierarchical analysis did not perform equally well: HMeta-d was associated with a false positive rate slightly above the nominal alpha level. Logistic mixed model regression analysis was associated with dramatically inflated false positive rates when random slopes were omitted from model specification. In general, we argue that measures of metacognitive accuracy should not be trusted unless there is a rigorous examination of associated false positive rate.