Semantic-Aware Decoding of Covert Inner Speech: A Multimodal EEG–EMG–Audio Framework

Semantic-Aware Decoding of Covert Inner Speech: A Multimodal EEG–EMG–Audio Framework | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Semantic-Aware Decoding of Covert Inner Speech: A Multimodal EEG–EMG–Audio Framework Hossein Ahmadi, Eduardo Santamaría-Vázquez, Luca Mesin, Roberto Hornero This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9365678/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Non-invasive brain–computer interfaces (BCIs) aim to restore communication by decoding intended messages directly from neural activity, even when no audible speech is produced. However, evidence remains limited that non-invasive signals can support semantic-level decoding of covert (inner) speech under subject-held-out evaluation. We investigate whether a semanticaware framework can generalise from overt spoken commands to covert inner speech using scalp electroencephalography (EEG). Ten healthy participants produced four everyday commands (water, toilet, light, pain) in overt and covert phases. Overt trials included synchronized EEG, bilateral electromyography (EMG), and audio, whereas covert trials included EEG and EMG without audio. The proposed model learns a multimodal latent representation from overt data using supervised contrastive learning, ArcFace-based classification, and semantic alignment to sentenceembedding prototypes. Evaluation is performed in a subjectheld-out covert protocol with target-subject overt calibration: covert trials from the held-out participant are never used for training or model selection. Across subjects, the model achieves a mean overt-validation accuracy of 0.54 and a mean covert-test accuracy of 0.42 on the four-class task, above the 0.25 chance level. Covert predictions are moderately calibrated, and latentspace and semantic-retrieval analyses indicate that the learned representation preserves class structure and aligns with textbased semantic prototypes. These results show that multimodal overt supervision and semantic regularisation can support noninvasive decoding of everyday inner-speech commands, while highlighting subject variability and calibration as key challenges for future BCIs. Biochemical Research Methods Inner speech imagined speech EEG semantic decoding multimodal BCI EMG Full Text Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9365678","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":620168646,"identity":"4de17b60-0832-4f4b-b4b1-009fcaf614aa","order_by":0,"name":"Hossein Ahmadi","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA+UlEQVRIiWNgGAWjYBACAzgpAcQ8BjYMDOyMDVAucVrSGBiYGRuBeiRw6jGAs8BaGA4DtTDgt8acvf3xhw8Fdxj4Z3cnfnhTcD6Pn5m5/cEHBos6XFose86YSc4weMYgcefsZsk5BreLJZsZGxtn4HPYjRw2Zh4DoHtu5G6Q5jG4nbjhMGNjMw9eLemPP/8BapG/kbv5N4/BOWK0JBhIMwC1GNzI3Qa05QARWs4A/dJjcJjHEKjFco5BMtgvM2cYSEg24NJyHBhiP/4clpMDOuzGmz92efzswAD7UFHHj8sWGOCBMRKgRhHSgAQSSFA7CkbBKBgFIwQAAMIjVz2BvutuAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0002-3650-0280","institution":"Politecnico di Torino","correspondingAuthor":true,"prefix":"","firstName":"Hossein","middleName":"","lastName":"Ahmadi","suffix":""},{"id":620168647,"identity":"c212282f-0600-4b2c-9320-679429af9c03","order_by":1,"name":"Eduardo Santamaría-Vázquez","email":"","orcid":"https://orcid.org/0000-0002-7688-4258","institution":"University of Valladolid","correspondingAuthor":false,"prefix":"","firstName":"Eduardo","middleName":"","lastName":"Santamaría-Vázquez","suffix":""},{"id":620168648,"identity":"5b04f8f9-818a-4987-a528-c88009893bd0","order_by":2,"name":"Luca Mesin","email":"","orcid":"https://orcid.org/0000-0002-8239-2348","institution":"Politecnico di Torino","correspondingAuthor":false,"prefix":"","firstName":"Luca","middleName":"","lastName":"Mesin","suffix":""},{"id":620168649,"identity":"9d1c510a-3d27-420e-8db2-bd86a567f0a7","order_by":3,"name":"Roberto Hornero","email":"","orcid":"https://orcid.org/0000-0001-9915-2570","institution":"University of Valladolid","correspondingAuthor":false,"prefix":"","firstName":"Roberto","middleName":"","lastName":"Hornero","suffix":""}],"badges":[],"createdAt":"2026-04-09 08:46:48","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-9365678/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9365678/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":106595018,"identity":"64fc078e-ebd4-4c71-aaa7-e396cffde800","added_by":"auto","created_at":"2026-04-10 09:17:12","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2368987,"visible":true,"origin":"","legend":"","description":"","filename":"InnerSpeechDecoding.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9365678/v1_covered_74c7bc79-c1b6-4a1c-86a5-9c9c44fe585b.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003e\u003cem\u003eSemantic-Aware Decoding of Covert Inner Speech: A Multimodal EEG–EMG–Audio Framework\u003c/em\u003e\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Inner speech, imagined speech, EEG, semantic decoding, multimodal BCI, EMG","lastPublishedDoi":"10.21203/rs.3.rs-9365678/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9365678/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eNon-invasive brain–computer interfaces (BCIs) aim to restore communication by decoding intended messages directly from neural activity, even when no audible speech is produced. However, evidence remains limited that non-invasive signals can support semantic-level decoding of covert (inner) speech under subject-held-out evaluation. We investigate whether a semanticaware framework can generalise from overt spoken commands to covert inner speech using scalp electroencephalography (EEG). Ten healthy participants produced four everyday commands (water, toilet, light, pain) in overt and covert phases. Overt trials included synchronized EEG, bilateral electromyography (EMG), and audio, whereas covert trials included EEG and EMG without audio. The proposed model learns a multimodal latent representation from overt data using supervised contrastive learning, ArcFace-based classification, and semantic alignment to sentenceembedding prototypes. Evaluation is performed in a subjectheld-out covert protocol with target-subject overt calibration: covert trials from the held-out participant are never used for training or model selection. Across subjects, the model achieves a mean overt-validation accuracy of 0.54 and a mean covert-test accuracy of 0.42 on the four-class task, above the 0.25 chance level. Covert predictions are moderately calibrated, and latentspace and semantic-retrieval analyses indicate that the learned representation preserves class structure and aligns with textbased semantic prototypes. These results show that multimodal overt supervision and semantic regularisation can support noninvasive decoding of everyday inner-speech commands, while highlighting subject variability and calibration as key challenges for future BCIs.\u003c/p\u003e","manuscriptTitle":"Semantic-Aware Decoding of Covert Inner Speech: A Multimodal EEG–EMG–Audio Framework","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-10 09:16:51","doi":"10.21203/rs.3.rs-9365678/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"df2d4486-6bb0-4d28-b631-2389079939ec","owner":[],"postedDate":"April 10th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":65995600,"name":"Biochemical Research Methods"}],"tags":[],"updatedAt":"2026-04-10T09:16:51+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-10 09:16:51","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9365678","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9365678","identity":"rs-9365678","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}