Can individual subjective confidence in prior questions predict group performance in future questions?

When people have to solve many tasks, they can aggregate diverse individuals’ judgments using the majority rule, which often improves the accuracy of judgments (wisdom of crowds). When aggregating individuals’ judgments, people sometimes decide whose judgments they should accept based on their subjective confidence. However, can confidence in one task set predict performance not only in the same (prior) task set, but also in another (future) task set? We examined this issue through computer simulations using behavioral data obtained from binary-choice experimental tasks. In our simulations, we developed a “training-test” approach: we split the questions used in behavioral experiments into “training questions” (as prior questions) and “test questions” (as future questions), similar to the cross validation method in machine learning. We found that (i) through analyses of behavioral data, confidence in one question could predict accuracy for the same question, but not always well in another question. (ii) Through a computer simulation of the accordance of two individuals’ judgments, individuals rated with high confidence in one training question tended to make less diverse judgments between each other in test questions. (iii) Through a computer simulation of group judgments, the accuracy of groups constructed by individuals with high confidence in the training question(s) sometimes largely decreased in the test questions, especially when only one training question was available. These results suggest that if people aggregate diverse individuals in terms of subjective confidence in prior questions, they will likely avoid decreasing performance in future questions. We believe that our simulations following a “training-test” approach provide practical implications for keeping groups’ ability to solve future tasks and emphasize the importance of a diversity of confidence in a group.