Speech Cues Influence the Interpretation of Metaphoric Gestures

Language and gesture are argued to form an integrated system where each modality contributes to the meaning of utterances. While gestures are known to influence language comprehension, less is known about how language affects the interpretation of gestures. We investigated whether the interpretation of metaphoric gestures is modulated by multimodal cues. Participants viewed hand gestures depicting the figurative interpretation of metaphors such as Men are oaks, where the actor flexed both arms to convey robustness while speaking. Gestures were presented with no additional cues, only lip movements, or audible speech. Participants then rated how well each gesture matched the gestured interpretation (e.g., robust), an alternative interpretation related to the metaphor but not to the gesture (e.g., tall), and an unrelated distractor (e.g., hikers). We found that ratings for alternative interpretations increased when gestures were paired with speech cues, compared to conditions without speech cues, suggesting that gesture comprehension is influenced by verbal content.