UAV Navigation using Reinforcement Learning: A Systematic Approach to Progressive Reward Function Design

UAV Navigation using Reinforcement Learning: A Systematic Approach to Progressive Reward Function Design | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article UAV Navigation using Reinforcement Learning: A Systematic Approach to Progressive Reward Function Design Christos Tsourveloudis, Lefteris Doitsidis This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8060158/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 9 You are reading this latest preprint version Abstract Fixed-wing unmanned aerial vehicles (UAVs) present significant path-following control challenges due to underactuation, coupled dynamics and stall constraints. These challenges complicate traditional control design and motivate the application of reinforcement learning (RL), which can learn effective policies without explicit aerodynamic models. A key difficulty in RL is reward function design: simple reward functions based solely on position and heading errors frequently produce oscillatory policies that struggle to generalize beyond trained paths. We address these limitations through systematic reward function decomposition, evaluating four progressively complex designs: (I) goal-distance minimization, (II) sequential waypoint navigation, (III) control-smoothness penalties, and (IV) 3D altitude tracking. Each policy is trained on a kinematic fixed-wing simulator and evaluated using reward-agnostic metrics—Path Deviation (mean distance to reference trajectory) and Oscillation Index (variance of control-rate changes). Across three RL algorithms—Proximal Policy Optimization (PPO), Soft Actor-Critic (SAC), and Twin Delayed Deep Deterministic Policy Gradient (TD3)—waypoint-based navigation (Stage II) reduces path deviation by 78–88% compared to goal-based rewards (Stage I), while smoothness penalties (Stage III) decrease control oscillations by 45–82%. The resulting policies maintain 100% success under wind disturbances despite being trained in zero-wind conditions. The framework extends to 3D trajectories (Stage IV), achieving 100% success on both seen and unseen paths while handling wind disturbances. Our results demonstrate that waypoint observations and control-rate penalties are essential components for stable fixed-wing RL control, while goal-only rewards consistently produce unstable behavior regardless of the underlying algorithm. This systematic decomposition provides a principled methodology for reward function design in RL based control of underactuated aerial systems. Reinforcement Learning Fixed-wing UAV Path Following Safe Autonomy Reward Function Design Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 06 Mar, 2026 Reviewers agreed at journal 07 Jan, 2026 Reviewers agreed at journal 06 Jan, 2026 Reviews received at journal 03 Dec, 2025 Reviewers agreed at journal 13 Nov, 2025 Reviewers invited by journal 12 Nov, 2025 Editor assigned by journal 12 Nov, 2025 Submission checks completed at journal 12 Nov, 2025 First submitted to journal 07 Nov, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8060158","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":549288575,"identity":"55c996c6-e935-4817-9adc-ac88b6353dfb","order_by":0,"name":"Christos Tsourveloudis","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA9ElEQVRIiWNgGAWjYDACduYDBxL/2MgxMDA2HiBOCzNb4oGPDWnGQC0NQC0GxGjhMT44s+FwYgOQTZwWfmYGg8O8Ow6nr20/3HCAccefxP4G9msf8GmRbGZIOMx7Jj1325lEoJYzBokzDvAUz8CnxeAww4HDPGzWudsOgLS0GQBJnmS8DrM/zNgA1MKcbnb+IUTLfEJaDJiZGQ7ObHNOMLsBtWXDAfbDeLVIHGZjOPDhTJrhthtAWxLbjI03HuZhxquFv73/84eECht5s/PpDx98bJOTnXe8/TFeLaggAUQw8xATn6iA/QHJWkbBKBgFo2BYAwCSF1XK3Le7KQAAAABJRU5ErkJggg==","orcid":"","institution":"National Technical University of Athens","correspondingAuthor":true,"prefix":"","firstName":"Christos","middleName":"","lastName":"Tsourveloudis","suffix":""},{"id":549288576,"identity":"334ce142-c5bb-4e25-baac-f9b4cf27d272","order_by":1,"name":"Lefteris Doitsidis","email":"","orcid":"","institution":"Technical University of Crete","correspondingAuthor":false,"prefix":"","firstName":"Lefteris","middleName":"","lastName":"Doitsidis","suffix":""}],"badges":[],"createdAt":"2025-11-07 21:23:06","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8060158/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8060158/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":96590182,"identity":"75960d68-a7eb-409f-a17a-4e47256162d1","added_by":"auto","created_at":"2025-11-24 06:14:00","extension":"json","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":4833,"visible":true,"origin":"","legend":"","description":"","filename":"a2585b450b244bc0990965cff994ffd6.json","url":"https://assets-eu.researchsquare.com/files/rs-8060158/v1/e18788b7dce41b9b31f58147.json"},{"id":96605217,"identity":"1ba2eb30-8c67-418c-a867-3707c71c2955","added_by":"auto","created_at":"2025-11-24 09:21:32","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":14765282,"visible":true,"origin":"","legend":"","description":"","filename":"vbffwvxyvthxvgvrtwwgcbwxxrmnsdmt.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8060158/v1_covered_b6ccf578-f2a5-4f04-8765-10b0293558fc.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"UAV Navigation using Reinforcement Learning: A Systematic Approach to Progressive Reward Function Design","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"journal-of-intelligent-and-robotic-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [Journal of Intelligent \u0026 Robotic Systems](https://link.springer.com/journal/10846)","snPcode":"10846","submissionUrl":"https://submission.springernature.com/new-submission/10846/3","title":"Journal of Intelligent \u0026 Robotic Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Open","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Reinforcement Learning, Fixed-wing UAV, Path Following, Safe Autonomy, Reward Function Design","lastPublishedDoi":"10.21203/rs.3.rs-8060158/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8060158/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Fixed-wing unmanned aerial vehicles (UAVs) present significant path-following control challenges due to underactuation, coupled dynamics and stall constraints. These challenges complicate traditional control design and motivate the application of reinforcement learning (RL), which can learn effective policies without explicit aerodynamic models. A key difficulty in RL is reward function design: simple reward functions based solely on position and heading errors frequently produce oscillatory policies that struggle to generalize beyond trained paths. We address these limitations through systematic reward function decomposition, evaluating four progressively complex designs: (I) goal-distance minimization, (II) sequential waypoint navigation, (III) control-smoothness penalties, and (IV) 3D altitude tracking. Each policy is trained on a kinematic fixed-wing simulator and evaluated using reward-agnostic metrics—Path Deviation (mean distance to reference trajectory) and Oscillation Index (variance of control-rate changes). Across three RL algorithms—Proximal Policy Optimization (PPO), Soft Actor-Critic (SAC), and Twin Delayed Deep Deterministic Policy Gradient (TD3)—waypoint-based navigation (Stage II) reduces path deviation by 78–88% compared to goal-based rewards (Stage I), while smoothness penalties (Stage III) decrease control oscillations by 45–82%. The resulting policies maintain 100% success under wind disturbances despite being trained in zero-wind conditions. The framework extends to 3D trajectories (Stage IV), achieving 100% success on both seen and unseen paths while handling wind disturbances. Our results demonstrate that waypoint observations and control-rate penalties are essential components for stable fixed-wing RL control, while goal-only rewards consistently produce unstable behavior regardless of the underlying algorithm. This systematic decomposition provides a principled methodology for reward function design in RL based control of underactuated aerial systems.","manuscriptTitle":"UAV Navigation using Reinforcement Learning: A Systematic Approach to Progressive Reward Function Design","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-11-24 06:13:55","doi":"10.21203/rs.3.rs-8060158/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-03-06T23:01:46+00:00","index":"","fulltext":""},{"type":"reviewerAgreed","content":"190620565270197803503145230622490163132","date":"2026-01-07T09:17:15+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"174881694634794009760309475332546551117","date":"2026-01-06T21:46:32+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-12-03T14:00:53+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"82871867593722538995103700452400604515","date":"2025-11-14T02:40:01+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-11-13T03:24:02+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-11-12T05:41:32+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-11-12T05:39:59+00:00","index":"","fulltext":""},{"type":"submitted","content":"Journal of Intelligent \u0026 Robotic Systems","date":"2025-11-07T21:09:52+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"journal-of-intelligent-and-robotic-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [Journal of Intelligent \u0026 Robotic Systems](https://link.springer.com/journal/10846)","snPcode":"10846","submissionUrl":"https://submission.springernature.com/new-submission/10846/3","title":"Journal of Intelligent \u0026 Robotic Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Open","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"ea24bf5e-9a2c-4f6a-8411-6550c95cea4d","owner":[],"postedDate":"November 24th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-05-12T04:25:08+00:00","versionOfRecord":[],"versionCreatedAt":"2025-11-24 06:13:55","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8060158","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8060158","identity":"rs-8060158","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}