Actor-Critic Networks with Analogue Memristors Mimicking Reward-Based Learning

Actor-Critic Networks with Analogue Memristors Mimicking Reward-Based Learning | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Actor-Critic Networks with Analogue Memristors Mimicking Reward-Based Learning Kevin Portner*, Till Zellweger*, Flavio Martinelli, Laura Bégon-Lours, and 14 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3993700/v2 This work is licensed under a CC BY 4.0 License Status: Posted Version 2 posted You are reading this latest preprint version Show more versions Abstract Advancements in memristive devices have given rise to a new generation of specialized hardware for bio-inspired computing. However, most of these implementations draw only partial inspiration from the architecture and functionalities of the mammalian brain. Moreover, the use of memristive hardware is typically restricted to specific elements within the learning algorithm, leaving computationally expensive operations to be executed in software. Here, we demonstrate reinforcement learning through an actor-critic temporal difference (TD) algorithm implemented on analogue memristors, mirroring the principles of reward-based learning in a neural network architecture similar to the one found in biology. Memristors are used as multi-purpose elements within the learning algorithm: They act as synaptic weights that are trained online, they calculate the weight updates associated with the TD-error directly in hardware, and they determine the actions to navigate the environment. Thanks to these features, weight training can take place entirely in-memory, eliminating data movement. We test our framework on two navigation tasks - the T-maze and the Morris water-maze - using analogue memristors based on the valence change memory (VCM) effect. Our approach represents a first step towards fully in-memory and online neuromorphic computing engines based on bio-inspired learning schemes. Physical sciences/Nanoscience and technology/Nanoscale devices/Electronic devices Physical sciences/Mathematics and computing/Computational science Biological sciences/Computational biology and bioinformatics/Computational neuroscience/Network models reinforcement learning neuromorphic computing bio-inspired computing three-factor learning navigation tasks analogue memristors Full Text Additional Declarations The authors declare no competing interests. Supplementary Files SupplementaryMaterial.pdf Supplementary Information: Actor-Critic Networks with Analogue Memristors Mimicking Reward-Based Learning Cite Share Download PDF Status: Posted Version 2 posted You are reading this latest preprint version Show more versions Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3993700","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":282076492,"identity":"b01d718d-3000-4a34-8655-bc05932cd1f7","order_by":0,"name":"Kevin Portner*","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA7UlEQVRIie3RvQrCMBDA8QuCU7FriqKvcFIQBNFXCRScBFcFh0Igm7gK+hziWAnEpT6BboJzxF1N8GuxraND/kM5aH+F5ABcrj+Mvga/ROLn6NlHkk/s64C/SPlXgu9vikiw4OpyXneG4bbENYw64DemUsPkkEmqNRXNN2m/vZJEUEj7EIgKQ1CnTFKngxA2QmLLECBCAioPGYllEblhyAnX5HqDniFJHqk+SIJoLpmSOAEse804jwRLFcFOREjtWZiKPKrMT5jKJnTPJYxFF/3Z9qj1pFv3eRqaIZt81m9jz9Xb4Uficrlcrm/dAbJlUJ1oZpnzAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0002-1175-9122","institution":"ETH Zurich","correspondingAuthor":true,"prefix":"","firstName":"Kevin","middleName":"","lastName":"Portner*","suffix":""},{"id":282076493,"identity":"657c39e5-73e8-49e8-9f4d-5473da8c1646","order_by":1,"name":"Till Zellweger*","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABG0lEQVRIie2RsWrDMBCGzwjkRcSrgkPzCiqBlEBwXkXF4KnQjp2KS9ZAVz9GH6DDGUGn0jxAl0AgdHQJlEAN6ckmgYKVroXqW+6XTh86IQCP5y/C2D1SGfTa5VTwQ0fZbochWJBb5XAyOyrSqcBPxRw7TmUWWqWmeWRq3j+eloNehMGqgundBYXtjWswQfPILJsUmzcKmp0XkMlJjiwuXIo0gsdX45FAqwCPBRipypwz4VLUnpTrz1GNr4JHGH7VsJfKwAlFo72FrQEpgOb0apTq2aEYUjSmgg8342Bhg7yc9xcq7T++BPO4QwkfTLmq6uQsEmZd7TCZDWmn2t0mkVqactuhNOi20MMbgrz9+iachlW/nfB4PJ7/yTd+sViarJ+owQAAAABJRU5ErkJggg==","orcid":"https://orcid.org/0000-0001-9402-8917","institution":"ETH Zurich","correspondingAuthor":true,"prefix":"","firstName":"Till","middleName":"","lastName":"Zellweger*","suffix":""},{"id":282076494,"identity":"19500eb4-ec0c-427d-8e7b-04caf8e22cad","order_by":2,"name":"Flavio Martinelli","email":"","orcid":"https://orcid.org/0009-0007-1514-0718","institution":"EPFL","correspondingAuthor":false,"prefix":"","firstName":"Flavio","middleName":"","lastName":"Martinelli","suffix":""},{"id":282076495,"identity":"029a6605-e3b3-4ee7-a75d-c21d17519298","order_by":3,"name":"Laura Bégon-Lours","email":"","orcid":"","institution":"ETH Zurich","correspondingAuthor":false,"prefix":"","firstName":"Laura","middleName":"","lastName":"Bégon-Lours","suffix":""},{"id":282076496,"identity":"f16ec3c8-7ef8-4438-a84a-170d4a8814ef","order_by":4,"name":"Valeria Bragaglia","email":"","orcid":"https://orcid.org/0000-0003-0636-4211","institution":"IBM Research Europe - Zurich","correspondingAuthor":false,"prefix":"","firstName":"Valeria","middleName":"","lastName":"Bragaglia","suffix":""},{"id":282076497,"identity":"493e3a5f-acab-412f-82a0-fe4e2c58aaa0","order_by":5,"name":"Christoph Weilenmann","email":"","orcid":"","institution":"ETH Zurich","correspondingAuthor":false,"prefix":"","firstName":"Christoph","middleName":"","lastName":"Weilenmann","suffix":""},{"id":282076498,"identity":"f603d4fb-f785-46a9-a4c3-b73d01ab6ce0","order_by":6,"name":"Daniel Jubin","email":"","orcid":"","institution":"IBM Research Europe - Zurich","correspondingAuthor":false,"prefix":"","firstName":"Daniel","middleName":"","lastName":"Jubin","suffix":""},{"id":282076499,"identity":"f4f54920-df19-451c-8b3d-af13f2b8b6e4","order_by":7,"name":"Donato Falcone","email":"","orcid":"","institution":"IBM Research Europe - Zurich","correspondingAuthor":false,"prefix":"","firstName":"Donato","middleName":"","lastName":"Falcone","suffix":""},{"id":282076500,"identity":"c2d1b769-8015-40b7-ac37-30a3c647b50b","order_by":8,"name":"Felix Hermann","email":"","orcid":"","institution":"IBM Research Europe - Zurich","correspondingAuthor":false,"prefix":"","firstName":"Felix","middleName":"","lastName":"Hermann","suffix":""},{"id":282076501,"identity":"f0f3d967-17d5-4b78-8a20-ece153225f05","order_by":9,"name":"Oscar Hrynkevych","email":"","orcid":"","institution":"ETH Zurich","correspondingAuthor":false,"prefix":"","firstName":"Oscar","middleName":"","lastName":"Hrynkevych","suffix":""},{"id":282076502,"identity":"56e96b17-573b-42dc-bf33-1adbcf329745","order_by":10,"name":"Tommaso Stecconi","email":"","orcid":"","institution":"IBM Research Europe - Zurich","correspondingAuthor":false,"prefix":"","firstName":"Tommaso","middleName":"","lastName":"Stecconi","suffix":""},{"id":282076503,"identity":"33f84c03-f3e2-4e4a-99d8-ec81284ac64f","order_by":11,"name":"Antonio La Porta","email":"","orcid":"https://orcid.org/0000-0003-3734-1354","institution":"IBM Research Europe - Zurich","correspondingAuthor":false,"prefix":"","firstName":"Antonio","middleName":"La","lastName":"Porta","suffix":""},{"id":282076504,"identity":"80d04f60-4f61-4aab-a856-c7ba71cbc79f","order_by":12,"name":"Ute Drechsler","email":"","orcid":"","institution":"IBM Research Europe - Zurich","correspondingAuthor":false,"prefix":"","firstName":"Ute","middleName":"","lastName":"Drechsler","suffix":""},{"id":282076505,"identity":"07a33e27-19e4-4525-825d-7531e5c5c4a1","order_by":13,"name":"Antonis Olziersky","email":"","orcid":"","institution":"IBM Research Europe - Zurich","correspondingAuthor":false,"prefix":"","firstName":"Antonis","middleName":"","lastName":"Olziersky","suffix":""},{"id":282076506,"identity":"10296452-ae33-4d23-85c5-2c228853ad42","order_by":14,"name":"Bert Offrein","email":"","orcid":"https://orcid.org/0000-0001-6082-0068","institution":"IBM Research Europe - Zurich","correspondingAuthor":false,"prefix":"","firstName":"Bert","middleName":"","lastName":"Offrein","suffix":""},{"id":282076507,"identity":"6b55ca47-0b44-4c81-8d73-a040e41c75ec","order_by":15,"name":"Wulfram Gerstner","email":"","orcid":"","institution":"EPFL","correspondingAuthor":false,"prefix":"","firstName":"Wulfram","middleName":"","lastName":"Gerstner","suffix":""},{"id":282076508,"identity":"bb59e760-40d2-43b8-8d5f-1b74bea83ccd","order_by":16,"name":"Mathieu Luisier","email":"","orcid":"https://orcid.org/0000-0002-2212-7972","institution":"ETH Zurich","correspondingAuthor":false,"prefix":"","firstName":"Mathieu","middleName":"","lastName":"Luisier","suffix":""},{"id":282076509,"identity":"e935e5e4-27e2-4e4e-8f03-d4fdbd8262ca","order_by":17,"name":"Alexandros Emboras","email":"","orcid":"","institution":"ETH Zurich","correspondingAuthor":false,"prefix":"","firstName":"Alexandros","middleName":"","lastName":"Emboras","suffix":""}],"badges":[],"createdAt":"2024-02-27 11:11:04","currentVersionCode":2,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-3993700/v2","doiUrl":"https://doi.org/10.21203/rs.3.rs-3993700/v2","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":93258212,"identity":"876d5549-980c-4f2c-8dc5-f1b57e2c65ab","added_by":"auto","created_at":"2025-10-10 17:19:59","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":3989813,"visible":true,"origin":"","legend":"","description":"","filename":"ManuscriptWithLineNumbers.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3993700/v2_covered_66dbeb11-6dc0-4430-b39f-2531dc93407a.pdf"},{"id":93257048,"identity":"1e0c5e22-aece-409c-9290-adcc1de71936","added_by":"auto","created_at":"2025-10-10 17:03:52","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":36774434,"visible":true,"origin":"","legend":"\u003cp\u003eSupplementary Information: Actor-Critic Networks with Analogue Memristors Mimicking Reward-Based Learning\u003c/p\u003e","description":"","filename":"SupplementaryMaterial.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3993700/v2/e85f34a44976af094188850c.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"Actor-Critic Networks with Analogue Memristors Mimicking Reward-Based Learning","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"reinforcement learning, neuromorphic computing, bio-inspired computing, three-factor learning, navigation tasks, analogue memristors","lastPublishedDoi":"10.21203/rs.3.rs-3993700/v2","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3993700/v2","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eAdvancements in memristive devices have given rise to a new generation of specialized hardware for bio-inspired computing. However, most of these implementations draw only partial inspiration from the architecture and functionalities of the mammalian brain. Moreover, the use of memristive hardware is typically restricted to specific elements within the learning algorithm, leaving computationally expensive operations to be executed in software. Here, we demonstrate reinforcement learning through an actor-critic temporal difference (TD) algorithm implemented on analogue memristors, mirroring the principles of reward-based learning in a neural network architecture similar to the one found in biology. Memristors are used as multi-purpose elements within the learning algorithm: They act as synaptic weights that are trained online, they calculate the weight updates associated with the TD-error directly in hardware, and they determine the actions to navigate the environment. Thanks to these features, weight training can take place entirely in-memory, eliminating data movement. We test our framework on two navigation tasks - the T-maze and the Morris water-maze - using analogue memristors based on the valence change memory (VCM) effect. Our approach represents a first step towards fully in-memory and online neuromorphic computing engines based on bio-inspired learning schemes.\u003c/p\u003e","manuscriptTitle":"Actor-Critic Networks with Analogue Memristors Mimicking Reward-Based Learning","msid":"","msnumber":"","nonDraftVersions":[{"code":2,"date":"2025-10-10 17:03:47","doi":"10.21203/rs.3.rs-3993700/v2","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}},{"code":1,"date":"2024-03-21 04:41:15","doi":"10.21203/rs.3.rs-3993700/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"7cad4364-6510-433d-b551-fe7099b649fd","owner":[],"postedDate":"October 10th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":29700604,"name":"Physical sciences/Nanoscience and technology/Nanoscale devices/Electronic devices"},{"id":29700605,"name":"Physical sciences/Mathematics and computing/Computational science"},{"id":29700606,"name":"Biological sciences/Computational biology and bioinformatics/Computational neuroscience/Network models"}],"tags":[],"updatedAt":"2025-04-09T18:25:14+00:00","versionOfRecord":[],"versionCreatedAt":"2025-10-10 17:03:47","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v2","identity":"rs-3993700","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3993700","identity":"rs-3993700","version":["v2"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}