How the brain can be trained to achieve an intermittent control strategy for stabilizing quiet stance by means of reinforcement learning?

How the brain can be trained to achieve an intermittent control strategy for stabilizing quiet stance by means of reinforcement learning? | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article How the brain can be trained to achieve an intermittent control strategy for stabilizing quiet stance by means of reinforcement learning? Tomoki Takazawa, Yasuyuki Suzuki, Akihiro Nakamura, Risa Matsuo, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4221173/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 12 Jul, 2024 Read the published version in Biological Cybernetics → Version 1 posted You are reading this latest preprint version Abstract The stabilization of human quiet stance is achieved by a combination of the intrinsic elastic properties of ankle muscles and an active closed-loop activation of the ankle muscles, driven by the delayed feedback of the ongoing sway angle and the corresponding angular velocity in a way of a delayed proportional (P) and derivative (D) feedback controller. It has been shown that the active component of the stabilization process is likely to operate in an intermittent manner rather than as a continuous controller: the switching policy is defined in the phase-plane, which is divided in dangerous and safe regions, separated by appropriate switching boundaries. When the state enters a dangerous region, the delayed PD control is activated, and it is switched off when it enters a safe region, leaving the system to evolve freely. In comparison with continuous feedback control, the intermittent mechanism is more robust and capable to better reproduce postural sway patterns in healthy people. However, the superior performance of the intermittent control paradigm as well as its biological plausibility, suggested by experimental evidence of the intermittent activation of the ankle muscles, leaves open the quest of a feasible learning process, by which the brain can identify the appropriate state-dependent switching policy and tune accordingly the P and D parameters. In this work, it is shown how such a goal can be achieved with a reinforcement motor learning paradigm, building upon the evidence that, in general, the basal ganglia are known to play a central role in reinforcement learning for action selection and, in particular, were found to be specifically involved in postural stabilization. Computational Neuroscience postural control postural sway intermittent control reinforcement learning Full Text Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Published Journal Publication published 12 Jul, 2024 Read the published version in Biological Cybernetics → Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4221173","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":287794375,"identity":"22561828-8e45-4a10-9be0-4c94187abbe3","order_by":0,"name":"Tomoki Takazawa","email":"","orcid":"","institution":"Osaka University","correspondingAuthor":false,"prefix":"","firstName":"Tomoki","middleName":"","lastName":"Takazawa","suffix":""},{"id":287794376,"identity":"d370ea15-2816-48c4-befb-c84be309340e","order_by":1,"name":"Yasuyuki Suzuki","email":"","orcid":"https://orcid.org/0000-0001-9681-7856","institution":"Osaka University","correspondingAuthor":false,"prefix":"","firstName":"Yasuyuki","middleName":"","lastName":"Suzuki","suffix":""},{"id":287794377,"identity":"57580798-11cd-4aa4-8484-6a619b56ca68","order_by":2,"name":"Akihiro Nakamura","email":"","orcid":"https://orcid.org/0000-0002-9571-0392","institution":"Osaka University","correspondingAuthor":false,"prefix":"","firstName":"Akihiro","middleName":"","lastName":"Nakamura","suffix":""},{"id":287794378,"identity":"6ef42a83-b104-4a49-ab63-9932bb7152f3","order_by":3,"name":"Risa Matsuo","email":"","orcid":"","institution":"Osaka University","correspondingAuthor":false,"prefix":"","firstName":"Risa","middleName":"","lastName":"Matsuo","suffix":""},{"id":287794379,"identity":"8f554ace-d5d1-49e6-8f60-a5c178874585","order_by":4,"name":"Pietro Morasso","email":"","orcid":"https://orcid.org/0000-0002-3837-8004","institution":"Istituto Italiano di Tecnologia","correspondingAuthor":false,"prefix":"","firstName":"Pietro","middleName":"","lastName":"Morasso","suffix":""},{"id":287794380,"identity":"2146867c-dcd3-41d6-8831-7a365d1a5538","order_by":5,"name":"Taishin Nomura","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABIUlEQVRIie2QsWrDMBCGzwjixSWrQWC/goKHdkjJq1gUpCWhmToLBMrS4tWFQl8hW+nmYvBUOmvoEFNwV4OXDC5ULiFZlEC3QvWBxEnwcf8dgMPxJ/EEAgIROfyM9lVqE4KdkvxGAUDm0LVNsTLzS9ktl+/8CcsC2h6ieFXR1lMQjwU0G1uXgCqck2bx/FCl3r0yCV9ZGRplkhfAiU0BqlBAysVazwk6E0NCLvCXAs9EZaFNGdeyMwon+rpFfQ/0MfuUW9NldlQJqcBGSYmemz2MgArNqiEYParoWg3KxMxCXu5UmBDdsAt4C6/y0j6Ln/GPLujL+BzLerPtp1GcsUTDzfQyW90y28YOmAzFz71/ooCdNMAS269OKw6Hw/FP+AbkUV9B/7K+vwAAAABJRU5ErkJggg==","orcid":"https://orcid.org/0000-0002-6545-2097","institution":"Osaka University","correspondingAuthor":true,"prefix":"","firstName":"Taishin","middleName":"","lastName":"Nomura","suffix":""}],"badges":[],"createdAt":"2024-04-05 06:54:55","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-4221173/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4221173/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s00422-024-00993-0","type":"published","date":"2024-07-12T08:02:36+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":61390922,"identity":"62648b2d-0816-4251-a89d-c96a801125eb","added_by":"auto","created_at":"2024-07-30 08:02:47","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":3171725,"visible":true,"origin":"","legend":"","description":"","filename":"takazawa2024final.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4221173/v1_covered_24f73377-a55e-48f8-bca2-16e30898d408.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003eHow the brain can be trained to achieve an intermittent control strategy for stabilizing quiet stance by means of reinforcement learning?\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"Osaka University","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":true,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"postural control, postural sway, intermittent control, reinforcement learning","lastPublishedDoi":"10.21203/rs.3.rs-4221173/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4221173/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe stabilization of human quiet stance is achieved by a combination of the intrinsic elastic properties of ankle muscles and an active closed-loop activation of the ankle muscles, driven by the delayed feedback of the ongoing sway angle and the corresponding angular velocity in a way of a delayed proportional (P) and derivative (D) feedback controller. It has been shown that the active component of the stabilization process is likely to operate in an intermittent manner rather than as a continuous controller: the switching policy is defined in the phase-plane, which is divided in dangerous and safe regions, separated by appropriate switching boundaries. When the state enters a dangerous region, the delayed PD control is activated, and it is switched off when it enters a safe region, leaving the system to evolve freely. In comparison with continuous feedback control, the intermittent mechanism is more robust and capable to better reproduce postural sway patterns in healthy people. However, the superior performance of the intermittent control paradigm as well as its biological plausibility, suggested by experimental evidence of the intermittent activation of the ankle muscles, leaves open the quest of a feasible learning process, by which the brain can identify the appropriate state-dependent switching policy and tune accordingly the P and D parameters. In this work, it is shown how such a goal can be achieved with a reinforcement motor learning paradigm, building upon the evidence that, in general, the basal ganglia are known to play a central role in reinforcement learning for action selection and, in particular, were found to be specifically involved in postural stabilization.\u0026nbsp;\u003c/p\u003e","manuscriptTitle":"How the brain can be trained to achieve an intermittent control strategy for stabilizing quiet stance by means of reinforcement learning?","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-04-08 03:07:06","doi":"10.21203/rs.3.rs-4221173/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"fbf866d4-0cd0-4626-9792-6badc2ff7c43","owner":[],"postedDate":"April 8th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":30299357,"name":"Computational Neuroscience"}],"tags":[],"updatedAt":"2024-08-02T04:22:20+00:00","versionOfRecord":{"articleIdentity":"rs-4221173","link":"https://doi.org/10.1007/s00422-024-00993-0","journal":{"identity":"biological-cybernetics","isVorOnly":false,"title":"Biological Cybernetics"},"publishedOn":"2024-07-12 08:02:36","publishedOnDateReadable":"July 12th, 2024"},"versionCreatedAt":"2024-04-08 03:07:06","video":"","vorDoi":"10.1007/s00422-024-00993-0","vorDoiUrl":"https://doi.org/10.1007/s00422-024-00993-0","workflowStages":[]},"version":"v1","identity":"rs-4221173","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4221173","identity":"rs-4221173","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}