People optimally and flexibly process emotional information across multiple modalities

People infer others’ emotions based on an immense array of information, including facial expressions, prosody, and content of speech. How do people perform this complex inferential feat using naturalistic, dynamic, and multimodal input? We propose that such affective cognition is not only structured and rational, but also optimal and flexible. We tested this hypothesis across four behavioral experiments, in two different cultures and with machine learning modeling, to investigate how accurately people are able to identify a target’s affect as they describe emotional life events, when given different combinations of modalities. Comparisons with state-of-the-art deep learning models suggest that human emotional reasoning is optimal given the available perceptual information; and it is flexible in that people tend to rely on linguistic information, but adapt to rely more on facial expressions when the former is unavailable. Our results support a complex view of human affective cognition in everyday social interactions.