Multimodal Signals of Multiparty Audience Design in Moderate-Severe Traumatic Brain Injury: Divergent Patterns in Spoken Language, Gesture, and Eye Gaze

Background: Multiparty communication challenges speakers to adapt their messages for multiple addressees with varying knowledge levels. Such studies of multiparty audience design in both typical and clinical populations have narrowly focused on spoken language only. In the current study, we extend this analysis to examine how speakers with and without TBI adapt their gesture and eye gaze behaviors for multiparty audience design. Method: 30 adults with moderate-severe TBI and 30 non-injured participants played a referential communication game with two Matchers. In the Sorting phase, they described abstract images repeatedly across four rounds. In a subsequent Test phase, participants described the images again to groups of two Matchers who were either both knowledgeable, both naïve, or mixed. We used video-based motion tracking to quantify adaptations in gesture form kinematics for groups of listeners with different levels of knowledge and manually annotated gaze events toward Matchers. Results: Non-injured participants demonstrated a coordinated effort for multiparty audience design across modalities, maximizing the informativeness of their spoken and gestural descriptions and using gaze to monitor understanding in naïve and mixed group conditions. In contrast, despite showing reduced adaptation in spoken language, participants with TBI showed preserved adaptation in kinematic features of gesture form and appropriately allocated gaze to naïve Matchers, signaling awareness of partner-specific knowledge representations. Participants with TBI produced gestures at significantly higher rates than non-injured peers. Conclusions: The results highlight the importance of a multimodal approach to studying cognitive-communication disorders and new insights into cognitive mechanisms underlying disruptions to multiparty audience design.