Robust Deterministic Policy Gradient for Disturbance Attenuation and Its Application to Quadrotor Control

Robust Deterministic Policy Gradient for Disturbance Attenuation and Its Application to Quadrotor Control | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Robust Deterministic Policy Gradient for Disturbance Attenuation and Its Application to Quadrotor Control Taeho Lee, Donghwan Lee This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7300041/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Practical control systems pose significant challenges in identifying optimal control policies due to uncertainties in the system model and external disturbances. While \(H_\infty\) control techniques are commonly used to design robust controllers that mitigate the effects of disturbances, these methods often require complex and computationally intensive calculations. To address this issue, this paper proposes a reinforcement learning algorithm called Robust Deterministic Policy Gradient (RDPG), which formulates the $ H_\infty $ control problem as a two-player zero-sum dynamic game. In this formulation, one player (the user) aims to minimize the cost, while the other player (the adversary) seeks to maximize it. We then employ deterministic policy gradient (DPG) and its deep reinforcement learning counterpart to train a robust control policy with effective disturbance attenuation. In particular, for practical implementation, we introduce an algorithm called robust deep deterministic policy gradient (RDDPG), which employs a deep neural network architecture and integrates techniques from the twin-delayed deep deterministic policy gradient (TD3) to enhance stability and learning efficiency.To evaluate the proposed algorithm, we implement it on an unmanned aerial vehicle (UAV) tasked with following a predefined path in a disturbance-prone environment. The experimental results demonstrate that the proposed method outperforms other control approaches in terms of robustness against disturbances, enabling precise real-time tracking of moving targets even under severe disturbance conditions. Deep reinforcement learning Two players zero-sum game Unmanned aerial vehicles Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7300041","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":497287277,"identity":"47c53689-2cf1-482c-b2a0-06af66400fb1","order_by":0,"name":"Taeho Lee","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA30lEQVRIie3PPwrCMBTH8SeCU0rWJwV7hZSCU9GrJBTSRfeCg4VCvEJF8SyBQF0UV0eLFyg4K/7bHNKODvluL/CBXwBcrn8MAXSTxSOEXv59Ie2kV5cHGX2I7kj6kaeMKN9HJ0L9okIykOl6ZYpbA5MAyPFiJcNtJRFJPN8QoVBDEubeilkJO8/GyFDOdyDUa1ifAx3Yh03fhDOTjmhdNBqW7YThLGKaG+6jyF/DDAdP2QmepahzLcN1WSs8sH2oSGUntEy0uT/iAE/ptcmyRUCJtJPfnQAtP3G5XC5Xl551EUPYCn1lwgAAAABJRU5ErkJggg==","orcid":"","institution":"Korea Advanced Institute of Science and Technology","correspondingAuthor":true,"prefix":"","firstName":"Taeho","middleName":"","lastName":"Lee","suffix":""},{"id":497287278,"identity":"52dc3c93-0f6f-4396-9200-118cdd1990e4","order_by":1,"name":"Donghwan Lee","email":"","orcid":"","institution":"Korea Advanced Institute of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Donghwan","middleName":"","lastName":"Lee","suffix":""}],"badges":[],"createdAt":"2025-08-05 11:23:36","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7300041/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7300041/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":95527871,"identity":"3e26d971-3147-47ef-8abc-060f3fe7ba28","added_by":"auto","created_at":"2025-11-10 10:15:06","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1254098,"visible":true,"origin":"","legend":"","description":"","filename":"SpringerDynamicgame.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7300041/v1_covered_2d76d720-8a48-4bb9-8b23-0238af80315e.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Robust Deterministic Policy Gradient for Disturbance Attenuation and Its Application to Quadrotor Control","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Deep reinforcement learning, Two players zero-sum game, Unmanned aerial vehicles","lastPublishedDoi":"10.21203/rs.3.rs-7300041/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7300041/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003ePractical control systems pose significant challenges in identifying optimal control policies due to uncertainties in the system model and external disturbances. While \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(H_\\infty\\)\u003c/span\u003e\u003c/span\u003e control techniques are commonly used to design robust controllers that mitigate the effects of disturbances, these methods often require complex and computationally intensive calculations. To address this issue, this paper proposes a reinforcement learning algorithm called Robust Deterministic Policy Gradient (RDPG), which formulates the \u003cspan\u003e$\u003c/span\u003eH_\\infty\u003cspan\u003e$\u003c/span\u003e control problem as a two-player zero-sum dynamic game. In this formulation, one player (the user) aims to minimize the cost, while the other player (the adversary) seeks to maximize it. We then employ deterministic policy gradient (DPG) and its deep reinforcement learning counterpart to train a robust control policy with effective disturbance attenuation. In particular, for practical implementation, we introduce an algorithm called robust deep deterministic policy gradient (RDDPG), which employs a deep neural network architecture and integrates techniques from the twin-delayed deep deterministic policy gradient (TD3) to enhance stability and learning efficiency.To evaluate the proposed algorithm, we implement it on an unmanned aerial vehicle (UAV) tasked with following a predefined path in a disturbance-prone environment. The experimental results demonstrate that the proposed method outperforms other control approaches in terms of robustness against disturbances, enabling precise real-time tracking of moving targets even under severe disturbance conditions.\u003c/p\u003e","manuscriptTitle":"Robust Deterministic Policy Gradient for Disturbance Attenuation and Its Application to Quadrotor Control","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-08-12 04:14:06","doi":"10.21203/rs.3.rs-7300041/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"43f564bb-b454-42e8-82f8-68700985fe3e","owner":[],"postedDate":"August 12th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-11-07T22:53:20+00:00","versionOfRecord":[],"versionCreatedAt":"2025-08-12 04:14:06","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7300041","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7300041","identity":"rs-7300041","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}