Safe Model-Free Q-Learning for Discrete-Time Fully Cooperative Multi-Input Systems with State and Control Constraints via Control Barrier Functions

Safe Model-Free Q-Learning for Discrete-Time Fully Cooperative Multi-Input Systems with State and Control Constraints via Control Barrier Functions | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Safe Model-Free Q-Learning for Discrete-Time Fully Cooperative Multi-Input Systems with State and Control Constraints via Control Barrier Functions Md Nur-A-Adam Dony, Bernard Arhin This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9108014/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract This paper proposes a safe model-free Q-learning algorithm for fully cooperative multi-input discrete-time nonlinear systems subject to both state and control constraints. In the fully cooperative setting, all control inputs share a common performance index and cooperate to stabilize the system while satisfying prescribed safety constraints. Unlike existing approaches that require system dynamics knowledge or neural network identification, the proposed method employs tabular Q-learning to directly learn the optimal cooperative control policies from measured state transitions without any model information. Discrete-time exponential control barrier functions are integrated as a safety filter, ensuring forward invariance of the safe set at every time step during both learning and deployment. The constrained value iteration framework guarantees convergence to the optimal safe policies without requiring initial admissible control policies. Theoretical analysis establishes both the safety guarantee via barrier function conditions and convergence of the iterative scheme. Two numerical examples are presented: a two-input nonlinear system with linear state constraints and a three-input nonlinear system with an elliptical state constraint. Simulation results demonstrate that the proposed algorithm achieves a 100% safety rate across all tested initial conditions, while unconstrained Q-learning violates safety in 40--60% of cases. The model-free nature and guaranteed safety make the approach attractive for safety-critical applications where system dynamics are unknown. Reinforcement learning Q-learning Control barrier functions Cooperative control Discrete-time systems Safe control Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9108014","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":608289223,"identity":"c2cf741d-0131-49df-b3eb-faa30f1b749a","order_by":0,"name":"Md Nur-A-Adam Dony","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAzklEQVRIiWNgGAWjYFCCBCDmsZEDsxkbgMQBorTIpBnzMDCTpMXmcGIP0Vp025OPPebJYU7fL91/8MHPHQxyfDcS8GsxO/Ms3ZjnDFtuj8xhZsPeMwzGkgS13Mgxk+bt4cntkUhmk2ZsY0jcQJyWfxLpPBLJ7L+BWuqJ08LDY5AA1MLGDNSSYECEX9Ik5/AkGPbcSDaW7G2TMJx55gEBLceTj0m84fkvzz4j8eGHn2028nzHCdgCAkw8CLYEYeUgwPiDOHWjYBSMglEwUgEAlfhCobNKwuEAAAAASUVORK5CYII=","orcid":"","institution":"Pennsylvania State University","correspondingAuthor":true,"prefix":"","firstName":"Md","middleName":"Nur-A-Adam","lastName":"Dony","suffix":""},{"id":608289226,"identity":"273ac89e-e421-4b4e-9c75-aeca92ab9773","order_by":1,"name":"Bernard Arhin","email":"","orcid":"","institution":"Chungnam National University","correspondingAuthor":false,"prefix":"","firstName":"Bernard","middleName":"","lastName":"Arhin","suffix":""}],"badges":[],"createdAt":"2026-03-12 19:53:32","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9108014/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9108014/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":108468025,"identity":"7eef64d5-8428-4308-a513-0dcb17af11b5","added_by":"auto","created_at":"2026-05-05 04:24:51","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2487279,"visible":true,"origin":"","legend":"","description":"","filename":"IJDCjournal.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9108014/v1_covered_fb13aa73-137b-4371-aab6-0f965ba21905.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Safe Model-Free Q-Learning for Discrete-Time Fully Cooperative Multi-Input Systems with State and Control Constraints via Control Barrier Functions","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Reinforcement learning, Q-learning, Control barrier functions, Cooperative control, Discrete-time systems, Safe control","lastPublishedDoi":"10.21203/rs.3.rs-9108014/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9108014/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"This paper proposes a safe model-free Q-learning algorithm for fully cooperative multi-input discrete-time nonlinear systems subject to both state and control constraints. In the fully cooperative setting, all control inputs share a common performance index and cooperate to stabilize the system while satisfying prescribed safety constraints. Unlike existing approaches that require system dynamics knowledge or neural network identification, the proposed method employs tabular Q-learning to directly learn the optimal cooperative control policies from measured state transitions without any model information. Discrete-time exponential control barrier functions are integrated as a safety filter, ensuring forward invariance of the safe set at every time step during both learning and deployment. The constrained value iteration framework guarantees convergence to the optimal safe policies without requiring initial admissible control policies. Theoretical analysis establishes both the safety guarantee via barrier function conditions and convergence of the iterative scheme. Two numerical examples are presented: a two-input nonlinear system with linear state constraints and a three-input nonlinear system with an elliptical state constraint. Simulation results demonstrate that the proposed algorithm achieves a 100\\% safety rate across all tested initial conditions, while unconstrained Q-learning violates safety in 40--60\\% of cases. The model-free nature and guaranteed safety make the approach attractive for safety-critical applications where system dynamics are unknown.","manuscriptTitle":"Safe Model-Free Q-Learning for Discrete-Time Fully Cooperative Multi-Input Systems with State and Control Constraints via Control Barrier Functions","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-03-23 11:35:38","doi":"10.21203/rs.3.rs-9108014/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"26cbb3f1-c7ae-4ce5-88d6-a2f53d05a56e","owner":[],"postedDate":"March 23rd, 2026","published":true,"recentEditorialEvents":[{"type":"decision","content":"Rejected","date":"2026-05-05T04:07:43+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-04T14:46:35+00:00","index":20,"fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-05-05T04:23:30+00:00","versionOfRecord":[],"versionCreatedAt":"2026-03-23 11:35:38","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9108014","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9108014","identity":"rs-9108014","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}