Unimodal and cross-modal identity judgements using an audiovisual sorting task: Evidence for independent processing of faces and voices

Unimodal and cross-modal information provided by faces and voices can contribute to identity percepts. To examine how these unimodal and cross-modal sources of information interact, we devised a novel audiovisual identity sorting task in which participants were required to group video-only and audio-only clips into two identities. In a series of three experiments, we show that unimodal face and voice sorting were more accurate than cross-modal sorting accuracy: While face sorting was consistently most accurate followed by voice sorting, cross-modal sorting was at chancel-level or below. In Experiment 1, we contextualised performance in our novel audiovisual sorting task by comparing it to a traditional identity matching task. Here we found that unimodal and cross-modal identity perception were more accurate in the matching task. In Experiment 2, we separated unimodal from cross-modal sorting, which led to small improvements in accuracy for unimodal sorting, but no change in cross-modal sorting performance. Finally, in Experiment 3 we explored the effect of minimal audiovisual training: Participants were shown an audiovisual clip of the two identities in conversation prior to completing the sorting task. This minimal training led to small but non-significant improvements in accuracy for both unimodal and cross-modal sorting. Our results indicate that, for unfamiliar people, face and voice perception operate relatively independently with no evidence of mutual benefit. We also show that extracting reliable redundant cross-modal information for identity judgements is challenging.