Metamodal Coupling of Vibrotactile and Auditory Speech Processing Systems Through Matched Stimulus Representations

Summary It has been postulated that the brain is organized by “metamodal”, sensory-independent cortical modules implementing particular computations, leading to the intriguing hypothesis that brain areas can perform tasks (such as word recognition) not just in “standard” sensory modalities but also in novel sensory modalities. Yet, evidence for this theory, especially in neurotypical subjects, has been variable. We hypothesized that effective metamodal engagement of a brain area requires congruence between the novel and standard sensory modalities not only at the task level (e.g., “word recognition”) but critically also a match at the algorithmic level (in Marr’s terminology), i.e., at the level of neural representation of the information of interest. To test this hypothesis, we trained participants to recognize vibrotactile versions of auditory words using two encoding schemes. The vocoded approach preserved the dynamics and representational similarities of auditory speech while the token-based approach used an abstract phoneme-based code. Although both groups learned the vibrotactile word recognition task, only in the vocoded group did trained vibrotactile stimuli recruit the auditory speech network and lead to increased coupling between somatosensory and auditory speech areas. In contrast, the token-based encoding appeared to rely on paired-associate learning. Thus, matching neural input representations is a critical factor for assessing and leveraging the metamodal potential of cortical modules.