Safe Reinforcement Learning for Vision-Based Robotic Manipulation in Human-Centered Environments

Safe Reinforcement Learning for Vision-Based Robotic Manipulation in Human-Centered Environments | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Safe Reinforcement Learning for Vision-Based Robotic Manipulation in Human-Centered Environments Fawad Khan, Wei Feng, Zhiyong Wang, Tianlun Huang, Xiao Liu, Yunduan Cui, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6736564/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 05 Dec, 2025 Read the published version in International Journal of Intelligent Robotics and Applications → Version 1 posted 9 You are reading this latest preprint version Abstract Autonomous systems performing object manipulation in human-robot collaboration scenarios face fundamental challenges in balancing adaptability with safety constraints. We present a RL framework that addresses these challenges through safety-aware policy learning. Building upon OpenAI Safety Gym, we extend its capabilities by implementing a robotic arm model for object manipulation tasks. Our approach employs end-to-end policy learning, comparing a constrained Lagrangian variant of Proximal Policy Optimization (cPPO) against standard PPO and Soft Actor-Critic (SAC) baselines. To handle high-dimensional 1 visual inputs, we develop a structured representation learning method that effectively captures multiple skills, objects, and their interactions. The framework enables goal-conditioned manipulation across object configurations, demonstrating strong compositional generalization: the agent trained on simple two-cubic object scenarios successfully generalized to tasks with three distinct objects in more cluttered settings. Due to computational constraints of high-mass objects in the simulation environment, testing was limited to scenarios with up to three objects. Experimental results show that cPPO achieves superior safety performance with an average episode cost of 15.26 compared to 18.03 for PPO and 19.48 for SAC. While cPPO’s task performance (average episode reward ∼30) is slightly lower than PPO’s (∼35), it significantly outperforms SAC (∼12). The algorithms demonstrate convergence by 200,000 environment steps, with cPPO achieving rapid safety compliance while exhibiting steady convergence in learning performance. These findings demonstrate the effectiveness of integrating safety constraints within RL for autonomous manipulation, advancing the practical deployment of collaborative robotic systems. Safe RL Robotic grasping Autonomous learning safety-critical coordination Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 05 Dec, 2025 Read the published version in International Journal of Intelligent Robotics and Applications → Version 1 posted Editorial decision: Revision requested 08 Sep, 2025 Reviews received at journal 07 Jul, 2025 Reviews received at journal 17 Jun, 2025 Reviewers agreed at journal 17 Jun, 2025 Reviewers agreed at journal 14 Jun, 2025 Reviewers invited by journal 12 Jun, 2025 Editor assigned by journal 28 May, 2025 Submission checks completed at journal 24 May, 2025 First submitted to journal 23 May, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6736564","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":471277451,"identity":"cad2ba04-1bc3-4828-8996-1751a9d70a6d","order_by":0,"name":"Fawad Khan","email":"","orcid":"","institution":"Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences","correspondingAuthor":false,"prefix":"","firstName":"Fawad","middleName":"","lastName":"Khan","suffix":""},{"id":471277452,"identity":"b9b95b5e-0d46-4df6-89eb-e755d97a56c0","order_by":1,"name":"Wei Feng","email":"","orcid":"","institution":"Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences","correspondingAuthor":false,"prefix":"","firstName":"Wei","middleName":"","lastName":"Feng","suffix":""},{"id":471277453,"identity":"5267ac37-27fc-4d54-b2f1-66969ac10190","order_by":2,"name":"Zhiyong Wang","email":"","orcid":"","institution":"Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences","correspondingAuthor":false,"prefix":"","firstName":"Zhiyong","middleName":"","lastName":"Wang","suffix":""},{"id":471277454,"identity":"a2c91233-f3aa-469d-9f2a-009791386265","order_by":3,"name":"Tianlun Huang","email":"","orcid":"","institution":"Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences","correspondingAuthor":false,"prefix":"","firstName":"Tianlun","middleName":"","lastName":"Huang","suffix":""},{"id":471277460,"identity":"b68207e4-6dfe-4115-b2a3-49e452fe52f3","order_by":4,"name":"Xiao Liu","email":"","orcid":"","institution":"Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences","correspondingAuthor":false,"prefix":"","firstName":"Xiao","middleName":"","lastName":"Liu","suffix":""},{"id":471277461,"identity":"ad88b92b-a591-4f82-944f-7d13e2183aec","order_by":5,"name":"Yunduan Cui","email":"","orcid":"","institution":"Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences","correspondingAuthor":false,"prefix":"","firstName":"Yunduan","middleName":"","lastName":"Cui","suffix":""},{"id":471277462,"identity":"ffb6529e-38db-4aa2-bfe4-e54045d89e7a","order_by":6,"name":"Wang Weijun","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA2klEQVRIiWNgGAWjYFCCA2BSjoGZVC3GQC2MDSTZlQhUTqQWg4MH2CR+7qhNn+/O/PwBQ40dA/9sAjoNDhxgk+w9czx342E2wwaGY8kMEncO4NdiBtRyg7ftWO7GZgagFrYDDAYSCYS13PzbdizdsJn9YwPDPyK13OZtq0mQZ+YxbGBsI0KL/YGD7b9l2w4YbmDmKZyR2JfMI3GDgBbJGYcPG75tq5OX7z++4cOHb3Zy/DMIaGGQONgAJA8Dgw5IARXzEFAPBPwgHQx1DPINhNWOglEwCkbBCAUAMYFHtbvfbIMAAAAASUVORK5CYII=","orcid":"","institution":"Guangzhou Institute of Advanced Technology, Chinese Academy of Sciences","correspondingAuthor":true,"prefix":"","firstName":"Wang","middleName":"","lastName":"Weijun","suffix":""}],"badges":[],"createdAt":"2025-05-24 04:08:10","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6736564/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6736564/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s41315-025-00507-6","type":"published","date":"2025-12-05T15:58:16+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":97725050,"identity":"076e2dd9-1415-4aeb-9d2b-ef6a47df2195","added_by":"auto","created_at":"2025-12-08 16:14:12","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2374956,"visible":true,"origin":"","legend":"","description":"","filename":"SafeRLHumanCenteredControl.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6736564/v1_covered_89a250c9-afa7-43f2-9199-997f6d490fa2.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Safe Reinforcement Learning for Vision-Based Robotic Manipulation in Human-Centered Environments","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"international-journal-of-intelligent-robotics-and-applications","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jira","sideBox":"Learn more about [International Journal of Intelligent Robotics and Applications](https://www.springer.com/journal/41315)","snPcode":"41315","submissionUrl":"https://submission.springernature.com/new-submission/41315/3","title":"International Journal of Intelligent Robotics and Applications","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Safe RL, Robotic grasping, Autonomous learning, safety-critical coordination","lastPublishedDoi":"10.21203/rs.3.rs-6736564/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6736564/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Autonomous systems performing object manipulation in human-robot collaboration scenarios face fundamental challenges in balancing adaptability with safety constraints. We present a RL framework that addresses these challenges through safety-aware policy learning. Building upon OpenAI Safety Gym, we extend its capabilities by implementing a robotic arm model for object manipulation tasks. Our approach employs end-to-end policy learning, comparing a constrained Lagrangian variant of Proximal Policy Optimization (cPPO) against standard PPO and Soft Actor-Critic (SAC) baselines. To handle high-dimensional 1 visual inputs, we develop a structured representation learning method that effectively captures multiple skills, objects, and their interactions. The framework enables goal-conditioned manipulation across object configurations, demonstrating strong compositional generalization: the agent trained on simple two-cubic object scenarios successfully generalized to tasks with three distinct objects in more cluttered settings. Due to computational constraints of high-mass objects in the simulation environment, testing was limited to scenarios with up to three objects. Experimental results show that cPPO achieves superior safety performance with an average episode cost of 15.26 compared to 18.03 for PPO and 19.48 for SAC. While cPPO’s task performance (average episode reward ∼30) is slightly lower than PPO’s (∼35), it significantly outperforms SAC (∼12). The algorithms demonstrate convergence by 200,000 environment steps, with cPPO achieving rapid safety compliance while exhibiting steady convergence in learning performance. These findings demonstrate the effectiveness of integrating safety constraints within RL for autonomous manipulation, advancing the practical deployment of collaborative robotic systems.","manuscriptTitle":"Safe Reinforcement Learning for Vision-Based Robotic Manipulation in Human-Centered Environments","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-06-16 18:13:50","doi":"10.21203/rs.3.rs-6736564/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-09-09T01:30:36+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-07-07T15:10:03+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-06-17T13:49:38+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"263342253157405531417914423200254821427","date":"2025-06-17T13:36:51+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"199828691214753858359843055313454047863","date":"2025-06-14T13:34:56+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-06-12T13:32:01+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-05-29T03:02:05+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-05-24T06:08:50+00:00","index":"","fulltext":""},{"type":"submitted","content":"International Journal of Intelligent Robotics and Applications","date":"2025-05-24T03:57:41+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"international-journal-of-intelligent-robotics-and-applications","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jira","sideBox":"Learn more about [International Journal of Intelligent Robotics and Applications](https://www.springer.com/journal/41315)","snPcode":"41315","submissionUrl":"https://submission.springernature.com/new-submission/41315/3","title":"International Journal of Intelligent Robotics and Applications","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"b6d586b9-342c-46ed-8613-1b68666be6b7","owner":[],"postedDate":"June 16th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2025-12-08T16:13:14+00:00","versionOfRecord":{"articleIdentity":"rs-6736564","link":"https://doi.org/10.1007/s41315-025-00507-6","journal":{"identity":"international-journal-of-intelligent-robotics-and-applications","isVorOnly":false,"title":"International Journal of Intelligent Robotics and Applications"},"publishedOn":"2025-12-05 15:58:16","publishedOnDateReadable":"December 5th, 2025"},"versionCreatedAt":"2025-06-16 18:13:50","video":"","vorDoi":"10.1007/s41315-025-00507-6","vorDoiUrl":"https://doi.org/10.1007/s41315-025-00507-6","workflowStages":[]},"version":"v1","identity":"rs-6736564","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6736564","identity":"rs-6736564","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}