A fully-memristive trimodal fusion perception system integrating multisensory neuron with hybrid neural network accelerator

A fully-memristive trimodal fusion perception system integrating multisensory neuron with hybrid neural network accelerator | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article A fully-memristive trimodal fusion perception system integrating multisensory neuron with hybrid neural network accelerator Xiaodong Huang, Mingyang Wang, Ruiqing Xie, Linbo Shan, Dong Wang, and 8 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8459031/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted You are reading this latest preprint version Abstract Intelligent perception systems require efficient sensing and processing of heterogeneous multimodal information. Memristive sensory neurons and hybrid neural networks (HNNs) respectively enable multimodal sensing and flexible processing. However, existing implementations suffer from cross-modal interference caused by memristive sensory neurons restricted to two-dimensional feature spaces and excessive hardware overhead resulting from complex CMOS-based HNN architectures. Critically, a fully-memristive system integrating both sensory and HNN functions has not been realized. Here, we report a fully-memristive perception system comprising a memristive multisensory neuron (MMN) and memristive HNN accelerator (MHA) for efficient trimodal sensory processing. The MMN, realized via a NbOx memristor-sensor-integrated oscillator, expands encoding dimensions to achieve trimodal fusion without cross-modal interference. The MHA, constructed with a 40-nm 1-Mb RRAM chip and NbOx-based memristive hybrid neurons (MHNs), flexibly processes fused signals. Each MHN provides reconfigurable Tanh and leaky-integrate-and-fire functions, enabled respectively by the nonlinear and threshold-switching properties of NbOx memristor, thereby reducing the hardware overhead. For battery state-of-charge estimation, the complete MMN-MHA system demonstrates 1.71% mean absolute error in artificial-neural-network mode and 92.1% classification accuracy in spiking-neural-network mode. Effective trimodal fusion yields 2.01× and 1.78× precision improvements over unimodal and bimodal schemes under high-noise conditions, demonstrating great potential of this fully-memristive system for scalable, energy-efficient multimodal perception. Physical sciences/Nanoscience and technology/Nanoscale devices/Electronic devices Physical sciences/Engineering/Electrical and electronic engineering Full Text Additional Declarations There is NO Competing Interest. Supplementary Files manuscriptSI.pdf Supplementary Information Cite Share Download PDF Status: Under Review Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8459031","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":571909926,"identity":"07f7988c-a19c-4952-bcc5-931de469ade0","order_by":0,"name":"Xiaodong Huang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAz0lEQVRIiWNgGAWjYFCCHDApB+GwkaDFmHQtiQ1EazE4nntM4ueO2vT57T0GDB/KDjPwz27Ar0Wy512aZO+Z47kbzpwxYJxx7jCDxJ0D+LXwS+SYSfC2HcvdIJFjwMzbdpjBQCIBvxY2oBbJv23H0uXnvzFg/kuMFpAt0rxtNQkMN3gMmBmJ0SLZ88bYWrbtgOGGM2kFB3vOpfNI3CCgxeB4juHNt2118vLthzc++FFmLcc/g4AWKDgMJg8AMQ9R6oGgjliFo2AUjIJRMBIBANZvQVvX8w+HAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0003-0303-7821","institution":"Southeast University","correspondingAuthor":true,"prefix":"","firstName":"Xiaodong","middleName":"","lastName":"Huang","suffix":""},{"id":571909927,"identity":"6e144fda-a780-40e1-a786-56479833b524","order_by":1,"name":"Mingyang Wang","email":"","orcid":"","institution":"Southeast University","correspondingAuthor":false,"prefix":"","firstName":"Mingyang","middleName":"","lastName":"Wang","suffix":""},{"id":571909928,"identity":"7fbc23b9-c2e3-4343-a6b6-1bd7c26fa498","order_by":2,"name":"Ruiqing Xie","email":"","orcid":"","institution":"Peking University","correspondingAuthor":false,"prefix":"","firstName":"Ruiqing","middleName":"","lastName":"Xie","suffix":""},{"id":571909929,"identity":"9f90f16a-1c30-47ff-9324-81a69be102bb","order_by":3,"name":"Linbo Shan","email":"","orcid":"","institution":"Peking University","correspondingAuthor":false,"prefix":"","firstName":"Linbo","middleName":"","lastName":"Shan","suffix":""},{"id":571909930,"identity":"111736ff-e570-4c1a-940d-ceaef6068166","order_by":4,"name":"Dong Wang","email":"","orcid":"","institution":"Southeast University","correspondingAuthor":false,"prefix":"","firstName":"Dong","middleName":"","lastName":"Wang","suffix":""},{"id":571909931,"identity":"39497ff7-16fc-4eaa-b5f6-0ecb9c5a41bf","order_by":5,"name":"Fan Yue","email":"","orcid":"","institution":"Southeast University","correspondingAuthor":false,"prefix":"","firstName":"Fan","middleName":"","lastName":"Yue","suffix":""},{"id":571909932,"identity":"05835647-f5e3-45fa-8765-6edc0fcbf1d7","order_by":6,"name":"Xinyi He","email":"","orcid":"","institution":"Southeast University","correspondingAuthor":false,"prefix":"","firstName":"Xinyi","middleName":"","lastName":"He","suffix":""},{"id":571909933,"identity":"6caf6cbd-b576-4a30-9a6e-84e92612d3df","order_by":7,"name":"Kai Liu","email":"","orcid":"","institution":"Southeast University","correspondingAuthor":false,"prefix":"","firstName":"Kai","middleName":"","lastName":"Liu","suffix":""},{"id":571909934,"identity":"408ce7da-4acf-453a-a6dd-772caa1a59cc","order_by":8,"name":"Ruizhe Zhang","email":"","orcid":"","institution":"Southeast University","correspondingAuthor":false,"prefix":"","firstName":"Ruizhe","middleName":"","lastName":"Zhang","suffix":""},{"id":571909935,"identity":"64facfe6-ee32-4c0f-9a46-5cc66fdafea0","order_by":9,"name":"Zongwei Wang","email":"","orcid":"https://orcid.org/0000-0001-6297-2700","institution":"Peking University","correspondingAuthor":false,"prefix":"","firstName":"Zongwei","middleName":"","lastName":"Wang","suffix":""},{"id":571909936,"identity":"385a9c02-1492-44c0-b49c-93ad0bd405d4","order_by":10,"name":"Chuan Yu Han","email":"","orcid":"https://orcid.org/0000-0002-4732-0429","institution":"Xi'an Jiaotong University","correspondingAuthor":false,"prefix":"","firstName":"Chuan","middleName":"Yu","lastName":"Han","suffix":""},{"id":571909937,"identity":"de07e4e8-f9c0-4c91-ba54-64be6abe0133","order_by":11,"name":"Lindong Wu","email":"","orcid":"","institution":"Southeast University","correspondingAuthor":false,"prefix":"","firstName":"Lindong","middleName":"","lastName":"Wu","suffix":""},{"id":571909938,"identity":"1628a1fa-850c-4548-a82f-0e612c7c34b5","order_by":12,"name":"Yimao Cai","email":"","orcid":"","institution":"Peking University","correspondingAuthor":false,"prefix":"","firstName":"Yimao","middleName":"","lastName":"Cai","suffix":""}],"badges":[],"createdAt":"2025-12-27 06:30:10","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8459031/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8459031/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":100025127,"identity":"8d6dff8a-fcc2-418c-8ce5-6a547e1db066","added_by":"auto","created_at":"2026-01-12 08:26:09","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5995645,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8459031/v1/3d5a6dd10bc3dc43321234c5.pdf"},{"id":100025125,"identity":"832f38b4-4672-4dd2-a1e7-92cc23c8f263","added_by":"auto","created_at":"2026-01-12 08:26:09","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":12815,"visible":true,"origin":"","legend":"","description":"","filename":"NCOMMS25104983.json","url":"https://assets-eu.researchsquare.com/files/rs-8459031/v1/03972d206afe3c32b67998eb.json"},{"id":100025126,"identity":"39b9dd8c-22f3-4c1f-8f76-435204371d42","added_by":"auto","created_at":"2026-01-12 08:26:09","extension":"pdf","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":10159132,"visible":true,"origin":"","legend":"","description":"","filename":"manuscriptSI.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8459031/v1/cf0ba1cbaa229b70bea44fde.pdf"},{"id":100362278,"identity":"ab8e6643-fcba-47f7-9993-d0a25004d6c4","added_by":"auto","created_at":"2026-01-16 07:46:29","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2820773,"visible":true,"origin":"","legend":"Article File","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8459031/v1_covered_76c6c08a-a7b1-469a-9b58-431de962cbb7.pdf"},{"id":100025128,"identity":"18707227-0794-4dd5-bce2-731752e22ac2","added_by":"auto","created_at":"2026-01-12 08:26:10","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":10159132,"visible":true,"origin":"","legend":"Supplementary Information","description":"","filename":"manuscriptSI.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8459031/v1/88b4b5510444e86822102939.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"A fully-memristive trimodal fusion perception system integrating multisensory neuron with hybrid neural network accelerator","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"nature-portfolio","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"","title":"Nature Portfolio","twitterHandle":"","acdcEnabled":false,"dfaEnabled":false,"editorialSystem":"ejp","reportingPortfolio":"","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-8459031/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8459031/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Intelligent perception systems require efficient sensing and processing of heterogeneous multimodal information. Memristive sensory neurons and hybrid neural networks (HNNs) respectively enable multimodal sensing and flexible processing. However, existing implementations suffer from cross-modal interference caused by memristive sensory neurons restricted to two-dimensional feature spaces and excessive hardware overhead resulting from complex CMOS-based HNN architectures. Critically, a fully-memristive system integrating both sensory and HNN functions has not been realized. Here, we report a fully-memristive perception system comprising a memristive multisensory neuron (MMN) and memristive HNN accelerator (MHA) for efficient trimodal sensory processing. The MMN, realized via a NbOx memristor-sensor-integrated oscillator, expands encoding dimensions to achieve trimodal fusion without cross-modal interference. The MHA, constructed with a 40-nm 1-Mb RRAM chip and NbOx-based memristive hybrid neurons (MHNs), flexibly processes fused signals. Each MHN provides reconfigurable Tanh and leaky-integrate-and-fire functions, enabled respectively by the nonlinear and threshold-switching properties of NbOx memristor, thereby reducing the hardware overhead. For battery state-of-charge estimation, the complete MMN-MHA system demonstrates 1.71% mean absolute error in artificial-neural-network mode and 92.1% classification accuracy in spiking-neural-network mode. Effective trimodal fusion yields 2.01× and 1.78× precision improvements over unimodal and bimodal schemes under high-noise conditions, demonstrating great potential of this fully-memristive system for scalable, energy-efficient multimodal perception.","manuscriptTitle":"A fully-memristive trimodal fusion perception system integrating multisensory neuron with hybrid neural network accelerator","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-01-12 08:26:00","doi":"10.21203/rs.3.rs-8459031/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"nature-communications","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"NCOMMS","sideBox":"Learn more about [Nature Communications](http://www.nature.com/ncomms/)","snPcode":"","submissionUrl":"https://mts-ncomms.nature.com/","title":"Nature Communications","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Nature Communications","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"68700dce-48ec-4f18-b5cd-c8940815a798","owner":[],"postedDate":"January 12th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":60862575,"name":"Physical sciences/Nanoscience and technology/Nanoscale devices/Electronic devices"},{"id":60862576,"name":"Physical sciences/Engineering/Electrical and electronic engineering"}],"tags":[],"updatedAt":"2026-04-30T10:22:13+00:00","versionOfRecord":[],"versionCreatedAt":"2026-01-12 08:26:00","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8459031","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8459031","identity":"rs-8459031","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}