Bidirectional effects of speech and gesture: Evidence from a novel levels-of-congruence paradigm

Previous research has shown that speech and gesture are integrated during comprehension, with differences in response times and electrophysiological activity observed between incongruent and congruent gesture–speech pairings. Beyond these incongruency effects, it remains unclear whether comprehenders use gestures to adapt or extend conventional interpretations for novel communicative purposes, and whether gestures are equally influenced by speech cues. The current study examined the bidirectional influence of speech and gesture by employing a novel levels-of-congruence paradigm, in which the same metaphor (e.g., Men are oaks) was paired with a gesture matching its preferred interpretation (men are robust) or a less preferred but possible interpretation (men are tall). In Experiment 1, participants rated the meaningfulness of the preferred and possible interpretations under different gesture conditions (gesture-of-preferred, gesture-of-possible, non-gestured). In Experiment 2, participants chose a single interpretation under the same gesture conditions. In Experiment 3, video clips of an actor producing the same gesture and utterance (e.g., flexing both arms to convey robustness, while saying Men are oaks) was presented with no additional cues (muted, blurred lips), with only lip movements (muted, visible lips), and with audible speech. Participants rated how well each gesture matched the gestured interpretation (e.g., robust), an alternative interpretation related to the speech but not the gesture (e.g., tall), and an unrelated interpretation (e.g., hikers). Overall, our findings are consistent with the integrated-systems hypothesis, showing that speech and gesture mutually and obligatorily interact in ways that enhance comprehension.