Simulating closed-loop transcranial brain stimulation for reinforcement learning-based treatment discovery

Simulating closed-loop transcranial brain stimulation for reinforcement learning-based treatment discovery | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Simulating closed-loop transcranial brain stimulation for reinforcement learning-based treatment discovery Chirath Hettiarachchi, Neil Bailey, Paul Fitzgerald, Hanna Suominen This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7958165/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted You are reading this latest preprint version Abstract The human brain is a complex system, where closed-loop mechanisms are vital in maintaining stability, synchronization, and adaptability of brain function. An important component of these mechanisms are neural oscillations (brain waves). Abnormal neural oscillation patterns are associated with psychiatric and neurological disorders. Brain stimulation techniques are being explored as a treatment option that can alter these abnormal oscillatory patterns. In closed-loop brain stimulation, an electric current or magnetic field stimulates brain activity, while being dynamically adjusted using feedback from ongoing brain activity captured through real-time electroencephalogram (EEG) measurements, with the aim to enhance the effectiveness of stimulation at modulating the target neural activity. Developing these treatment strategies is challenging due to the complex nonlinear dynamics of brain activity, so treatment development benefits from computer simulations. However, existing simulation tools do not integrate brain dynamics with transcranial stimulation to conduct closed-loop simulations, restricting the application of advanced methods such as reinforcement learning (RL), which refer to machine and deep learning algorithms capable of identifying complex patterns in data to discover novel decision-making strategies. Therefore, in this study, we first introduce a framework named NeuroStimEnv for simulating closed-loop brain stimulation. The tool provides the capability for researchers to integrate different models of neural circuits (exhibiting characteristics observed in Alzheimer's disease or depression), configure different electrode montage setups for stimulation and EEG measurement, and simulate treatment strategies. We have released the code as open source under the MIT license. Next, we simulate a depression microcircuit to demonstrate the feasibility of using RL to discover a transcranial alternating current stimulation treatment strategy. We demonstrate promising initial results on using RL-based algorithms for treatment selection, to successfully shift neural circuits representative of depression towards circuits representative of healthy individuals. The proposed simulation framework provides valuable insights into transcranial stimulation and enables the application of RL methods as shown in the case study. Physical sciences/Mathematics and computing/Computer science Physical sciences/Mathematics and computing/Software Biological sciences/Computational biology and bioinformatics/Computational neuroscience Health sciences/Medical research/Preclinical research Physical sciences/Engineering/Biomedical engineering Computational Neuroscience Computer Simulation Evaluation Study Neurosciences Reinforcement Learning Transcranial Stimulation Full Text Additional Declarations There is NO Competing Interest. Cite Share Download PDF Status: Under Review Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7958165","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":583172954,"identity":"21e04e17-4151-4c81-95fb-d873931d9881","order_by":0,"name":"Chirath Hettiarachchi","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABOElEQVRIie3PPWvCQBjA8ScE0uUga0rfvsKJoEhb+1XuOLBL6eIiKBgIZBJdI5V+BifnhINk8QMEGtqIkMmhXSSFUnpJlEIanTvcfwjkSX73AiCT/ccMxRRPF3D2QvIJaEB6u68HCcmIqpi/ZHmM5KvvCOyJYh8m+pNlv3+mETRPJkEcD16vxCSJV8/tR+yq3guCNjVLm0SeNR2RBFojLg7md2tO5DcxXbAudjV2jYCVCTaoBYhwwCETRCPK3HjQDLpQ6dxFjTMEahVRvjLythbkm9wVZDYURN8KMqwiarGLuD61CS2IybNdNEH4n7uEgpx3OMJLVnPomDAn7DQM4gd0yrV6a4aDeonozv1a2dzwSxx48Ue6JbcThyWn6aBPx4G1Cje9/kWJ7EMVMzU/efX/MplMJjvaD+vMdz4dFBW1AAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0002-7702-0718","institution":"The Australian National University","correspondingAuthor":true,"prefix":"","firstName":"Chirath","middleName":"","lastName":"Hettiarachchi","suffix":""},{"id":583172955,"identity":"fc7e8803-45e1-4897-94c0-0f5b3d23fc71","order_by":1,"name":"Neil Bailey","email":"","orcid":"","institution":"The Australian National University","correspondingAuthor":false,"prefix":"","firstName":"Neil","middleName":"","lastName":"Bailey","suffix":""},{"id":583172956,"identity":"035bfb5c-8d5a-4a75-946e-f2b3833d1224","order_by":2,"name":"Paul Fitzgerald","email":"","orcid":"","institution":"The Australian National University","correspondingAuthor":false,"prefix":"","firstName":"Paul","middleName":"","lastName":"Fitzgerald","suffix":""},{"id":583172957,"identity":"cd64400c-50fb-4f1c-8d84-7e83ccfe7af1","order_by":3,"name":"Hanna Suominen","email":"","orcid":"","institution":"Australian National University","correspondingAuthor":false,"prefix":"","firstName":"Hanna","middleName":"","lastName":"Suominen","suffix":""}],"badges":[],"createdAt":"2025-10-27 12:29:03","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7958165/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7958165/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":103050634,"identity":"c7f9bee1-aed1-4c96-ac99-2dc2b5420dd6","added_by":"auto","created_at":"2026-02-20 07:51:27","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":9697309,"visible":true,"origin":"","legend":"Article File","description":"","filename":"NeuroStimEnvOct27.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7958165/v1_covered_f28a4a0e-d7b4-4023-8dac-59382fb5798b.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"Simulating closed-loop transcranial brain stimulation for reinforcement learning-based treatment discovery","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"nature-portfolio","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"","title":"Nature Portfolio","twitterHandle":"","acdcEnabled":false,"dfaEnabled":false,"editorialSystem":"ejp","reportingPortfolio":"","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Computational Neuroscience, Computer Simulation, Evaluation Study, Neurosciences, Reinforcement Learning, Transcranial Stimulation","lastPublishedDoi":"10.21203/rs.3.rs-7958165/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7958165/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"The human brain is a complex system, where closed-loop mechanisms are vital in maintaining stability, synchronization, and adaptability of brain function. An important component of these mechanisms are neural oscillations (brain waves). Abnormal neural oscillation patterns are associated with psychiatric and neurological disorders. Brain stimulation techniques are being explored as a treatment option that can alter these abnormal oscillatory patterns. In closed-loop brain stimulation, an electric current or magnetic field stimulates brain activity, while being dynamically adjusted using feedback from ongoing brain activity captured through real-time electroencephalogram (EEG) measurements, with the aim to enhance the effectiveness of stimulation at modulating the target neural activity. Developing these treatment strategies is challenging due to the complex nonlinear dynamics of brain activity, so treatment development benefits from computer simulations. However, existing simulation tools do not integrate brain dynamics with transcranial stimulation to conduct closed-loop simulations, restricting the application of advanced methods such as reinforcement learning (RL), which refer to machine and deep learning algorithms capable of identifying complex patterns in data to discover novel decision-making strategies. Therefore, in this study, we first introduce a framework named NeuroStimEnv for simulating closed-loop brain stimulation. The tool provides the capability for researchers to integrate different models of neural circuits (exhibiting characteristics observed in Alzheimer's disease or depression), configure different electrode montage setups for stimulation and EEG measurement, and simulate treatment strategies. We have released the code as open source under the MIT license. Next, we simulate a depression microcircuit to demonstrate the feasibility of using RL to discover a transcranial alternating current stimulation treatment strategy. We demonstrate promising initial results on using RL-based algorithms for treatment selection, to successfully shift neural circuits representative of depression towards circuits representative of healthy individuals. The proposed simulation framework provides valuable insights into transcranial stimulation and enables the application of RL methods as shown in the case study.","manuscriptTitle":"Simulating closed-loop transcranial brain stimulation for reinforcement learning-based treatment discovery","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-02-20 06:14:12","doi":"10.21203/rs.3.rs-7958165/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"communications-biology","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"commsbio","sideBox":"Learn more about [Communications Biology](http://www.nature.com/commsbio/)","snPcode":"","submissionUrl":"","title":"Communications Biology","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Communications Series","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"b66788b4-0159-4ef1-9338-327a1a07e08d","owner":[],"postedDate":"February 20th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":62048879,"name":"Physical sciences/Mathematics and computing/Computer science"},{"id":62048880,"name":"Physical sciences/Mathematics and computing/Software"},{"id":62048881,"name":"Biological sciences/Computational biology and bioinformatics/Computational neuroscience"},{"id":62048882,"name":"Health sciences/Medical research/Preclinical research"},{"id":62048883,"name":"Physical sciences/Engineering/Biomedical engineering"}],"tags":[],"updatedAt":"2026-02-20T06:14:12+00:00","versionOfRecord":[],"versionCreatedAt":"2026-02-20 06:14:12","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7958165","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7958165","identity":"rs-7958165","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}