Neural Integration of Affective Prosodic and Semantic Cues in Non-literal Forms of Speech Understanding

Abstract Emotions in speech are conveyed through both semantics and prosody—voice ‘melody’, yet how listeners integrate these cues in the brain remains unclear. We investigated non-literal forms of speech, such as irony and sarcasm, where understanding beyond literal meaning relies on the dynamic interplay between affective prosodic and semantic cues, alongside theory of mind (ToM). We used functional magnetic resonance imaging while participants listened to short dialogues between two characters, which varied in prosody and semantics to convey either literal or non-literal meanings. Behaviorally, semantics and prosody interacted in shaping participants’ evaluations, although the data suggest a prosody dominance effect. Non-literal speech engaged a distributed network including the bilateral inferior frontal gyrus, temporal speech regions, and ToM areas, with ROI analyses revealing heterogeneous prosody-semantics integration profiles across regions and tasks. Taken together, our data clarify the behavioral and neural underpinnings of the integration of prosody and semantics in non-literal speech and open new venues in this field. Competing Interest Statement The authors have declared no competing interest. Footnotes Author note The authors made the following contributions. Adrien Wittmann: Conceptualization, Methodology, Data curation, Formal analysis, Software, Investigation, Validation, Visualisation, Writing – Original Draft Preparation, Writing – Review & Editing; Leonardo Ceravolo: Methodology, Software, Writing – Review & Editing; Audrey Mayr: Conceptualization, Software, Methodology, Investigation, Validation; Didier Grandjean: Funding acquisition, Supervision, Conceptualization, Writing – Review & Editing. We substantially revised our neuroimaging analysis strategy to better align with the study's central objective: characterizing how prosodic and semantic cues are integrated during non-literal speech processing. The original approach relied in part on whole-brain contrasts comparing integrative (irony, sarcasm, ToM) versus non-integrative (prosody, semantics) tasks. This contrast was limited in its interpretability, as it conflated integration with general task demands and did not directly test the mechanism of interest. To address this, we replaced the integrative vs. non-integrative whole-brain contrast with a targeted, hypothesis-driven ROI analysis centered on the prosody-by-semantics interaction. This revision brings the neural analyses into direct conceptual and methodological alignment with our behavioral analyses, which operationalize integration explicitly as the interaction between these two cue types across tasks. Our revised approach follows a two-step logic. First, we retained the Non-literal > Literal contrast as a principled and theory-neutral method to identify the broader network engaged in non-literal speech processing across tasks. Second, within the regions identified by this contrast, we extracted beta values and directly tested for sensitivity to the prosody-by-semantics interaction across all five tasks. This allows us to move beyond identifying regions that are merely more active, and instead determine where and under which task demands prosodic and semantic information are jointly represented. We further refined our hypotheses to reflect functional differentiation within this network. Specifically, we formulated a confirmatory prediction for the left IFG as a key integration hub, expecting a robust prosody-by-semantics interaction across all tasks, with stronger effects in tasks requiring explicit integration. In contrast, analyses in other ROIs (e.g., temporal and mentalizing regions) were treated as exploratory, with the goal of characterizing task-dependent variation in integration sensitivity and functional specialization. Overall, this revision substantially improves the theoretical coherence, interpretability, and specificity of our neuroimaging results.