A kinematic-acoustic analysis of gesture-speech coupling in persons with aphasia

Aphasia is a profound speech pathology hampering speech production and/or comprehension. People with Aphasia (PWA) use more manual gestures than neurotypicals. This intuitively invokes the idea of some kind of compensatory mechanism, but there is variable evidence of this gesture-boosting effect on speech processes. The status quo in gesture research with PWA is an emphasis on categorical analysis of gesture types, how often they are recruited, and whether they aid communication or speaking. However, theAphasia is a profound language pathology hampering speech production and/or comprehension. People With Aphasia (PWA) use more manual gestures than neurotypicals. This intuitively invokes the idea that gesture is compensatory in some way, but there is variable evidence of a gesture-boosting effect on speech processes. The status quo in gesture research with PWA is an emphasis on categorical analysis of gesture types, focusing on how often they are recruited, and whether more or less gesturing aids communication or speaking. However, there are increasingly louder calls for the investigation of gesture and speech as continuous entangled modes of expression. In neurotypical adults, expressive moments of gesture and speech are synchronized on the prosodic level. It has been neglected how this multimodal prosody is instantiated in PWA. In the current study we perform the first acoustic-kinematic gesture-speech analysis in persons of Aphasia (i.e., Wernicke’s, Broca’s, Anomic) relative to age-matched controls, where we apply several multimodal signal analysis methods. Specifically, we related the speech peaks (smoothed amplitude envelope change) with that of the nearest peaks in the gesture acceleration profile. We obtained that the magnitude of gesture vs. speech peaks are positively related across the groups, though more variably for PWA, and such coupling was related to less severe Aphasia-related symptoms. No differences were found between controls and PWA in terms of temporal ordering of speech envelope versus acceleration peaks. Finally, we show that both gesture and speech have slower quasi-rhythmic structure, indicating that next to speech, gesture is slowed down too. The current results indicate that there is a basic gesture-speech coupling mechanism that is not fully reliant on core linguistic competences, as it is found relatively intact in PWA. This resonates with a recent biomechanical theory of gesture, which renders gesture-vocal coupling as fundamental and a priori to the (evolutionary) development of core linguistic competences (Pouw & Fuchs, 2022). re are increasingly louder calls for the investigation of gesture and speech as continuous entangled modes of expression. In neurotypical adults, expressive moments of gesture and speech are synchronized. It is unknown how this multimodal prosody is instantiated in PWA. In the current study we perform the first acoustic-kinematic gesture-speech analysis in persons of aphasia (Wernicke, Broca, Anomic) relative to age-matched controls, where we apply several multimodal signal processing methods (e.g., motion tracking). Specifically, we related the speech peaks (smoothed amplitude envelope change) with that of the nearest peaks in the gesture acceleration profile. We obtained that the magnitude of gesture vs. speech peaks are positively related across the groups, though more variably for PWA, and such coupling was related to less severe Aphasia-related symptoms in PWA. When comparing the temporal ordering of speech envelope versus acceleration peaks, no statistically reliable differences between controls and PWA were observed. We did however obtain when gesture kinematic peaks tended to anticipate speech peaks more, this was related to less severe aphasia-related symptoms. Finally, we show that both gesture and speech have slower quasi-rhythmic structure, indicating that next to speech, gesture is slowed down too. We conclude that future aphasia research will benefit from the study of gesture-speech interconnections.