On the Design of Reward Function for Reinforcement Learning–Based Path Following Control Using the TD3 Algorithm

On the Design of Reward Function for Reinforcement Learning–Based Path Following Control Using the TD3 Algorithm | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article On the Design of Reward Function for Reinforcement Learning–Based Path Following Control Using the TD3 Algorithm Mahdi Khodaparast, Jafar Roshanian This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8335282/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Reward design is one of the most influential factors in determining the performance of reinforcement learning (RL) algorithms in continuous-control tasks. This paper presents a comparative study of three reward formulations for training a TD3 agent to perform three-dimensional path following using a particle kinematic system modeled as double-integrator dynamics. The first reward is a linear formulation that assigns proportional penalties based on normalized position and velocity tracking errors. The second reward employs an exponential decay based on the same error terms to emphasize high-precision tracking and reduce sensitivity to smaller deviations. The third reward, a Piecewise Quadratic Reward (PQR), combines quadratic penalties on position and velocity errors with a positive discrete progress bonus, forming a hybrid structure that balances guidance and corrective feedback. All three reward functions are evaluated across three representative trajectories—vertical ascent, straight-line cruise, and helical motion—and analyzed with respect to convergence behavior, tracking accuracy, and control smoothness. The results show that the exponential reward produces more stable trajectories with higher tracking accuracy leading to smoother actions in complex paths, while the PQR formulation penalize the agent heavily leading to less stable learning. These findings provide practical insights into how reward shaping influences the behavior of TD3-based path-following agents and offer guidelines for designing effective reward functions in continuous-control RL applications. Reinforcement Learning TD3 Algorithm Reward Function Design Path Following Control Trajectory Tracking Continuous Control Autonomous Aerial Robots Full Text Additional Declarations No competing interests reported. Supplementary Files mainManuscript.zip Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8335282","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":589099496,"identity":"9f5039fe-9b1a-41cf-90b2-c7c3ed7b848f","order_by":0,"name":"Mahdi Khodaparast","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA+ElEQVRIiWNgGAWjYFACHgaJhIr/ciDmgQcMCTBhCQJazjAbg7UkALXwEKWFsY05sQHERtKCG+i2nz1442EbW3q/2OGHQFvS8uwlEhg//GCwyMelxexMXrJFwjme3Jmz0wyAWnKKeSQSmCV7GCQsG3BpOZBjJpFQJpG74XYCSEtFYo9EAoM00C8GOG05/waohc0g3f52+geYFubfeLXcANnSlpBgIJ0DdhhICxt+W268MbZIOHPAcMbtnIIDCQZpiT1nHrZZ9hjgc1iO4c0fFQfk+Wenb/7woSI5sb09+fCNHxV1OLWgAbA6xgYoYxSMglEwCkYBuQAAqChWX1Q4AU8AAAAASUVORK5CYII=","orcid":"","institution":"Intelligent Control Systems Institute (ICSI), K.N. Toosi University of Technology","correspondingAuthor":true,"prefix":"","firstName":"Mahdi","middleName":"","lastName":"Khodaparast","suffix":""},{"id":589099497,"identity":"7f762829-33a5-4b0c-8764-b4c5f0604efc","order_by":1,"name":"Jafar Roshanian","email":"","orcid":"","institution":"Intelligent Control Systems Institute (ICSI), K.N. Toosi University of Technology","correspondingAuthor":false,"prefix":"","firstName":"Jafar","middleName":"","lastName":"Roshanian","suffix":""}],"badges":[],"createdAt":"2025-12-11 10:09:40","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8335282/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8335282/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":109296183,"identity":"fa9a8385-aac0-48d6-9317-cb29f0be7f33","added_by":"auto","created_at":"2026-05-15 08:45:56","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2129107,"visible":true,"origin":"","legend":"","description":"","filename":"mainManuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8335282/v1_covered_f04458f9-0c84-42e3-a82e-01a49e710356.pdf"},{"id":102599684,"identity":"ac764b6e-4730-43cb-bc8c-ef050bba8961","added_by":"auto","created_at":"2026-02-13 12:43:44","extension":"zip","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":8688362,"visible":true,"origin":"","legend":"","description":"","filename":"mainManuscript.zip","url":"https://assets-eu.researchsquare.com/files/rs-8335282/v1/a53f728ec46fb3443c45d983.zip"}],"financialInterests":"No competing interests reported.","formattedTitle":"On the Design of Reward Function for Reinforcement Learning–Based Path Following Control Using the TD3 Algorithm","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Reinforcement Learning, TD3 Algorithm, Reward Function Design, Path Following Control, Trajectory Tracking, Continuous Control, Autonomous Aerial Robots","lastPublishedDoi":"10.21203/rs.3.rs-8335282/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8335282/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Reward design is one of the most influential factors in determining the performance of reinforcement learning (RL) algorithms in continuous-control tasks. This paper presents a comparative study of three reward formulations for training a TD3 agent to perform three-dimensional path following using a particle kinematic system modeled as double-integrator dynamics. The first reward is a linear formulation that assigns proportional penalties based on normalized position and velocity tracking errors. The second reward employs an exponential decay based on the same error terms to emphasize high-precision tracking and reduce sensitivity to smaller deviations. The third reward, a Piecewise Quadratic Reward (PQR), combines quadratic penalties on position and velocity errors with a positive discrete progress bonus, forming a hybrid structure that balances guidance and corrective feedback. All three reward functions are evaluated across three representative trajectories—vertical ascent, straight-line cruise, and helical motion—and analyzed with respect to convergence behavior, tracking accuracy, and control smoothness. The results show that the exponential reward produces more stable trajectories with higher tracking accuracy leading to smoother actions in complex paths, while the PQR formulation penalize the agent heavily leading to less stable learning. These findings provide practical insights into how reward shaping influences the behavior of TD3-based path-following agents and offer guidelines for designing effective reward functions in continuous-control RL applications.","manuscriptTitle":"On the Design of Reward Function for Reinforcement Learning–Based Path Following Control Using the TD3 Algorithm","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-02-13 12:43:39","doi":"10.21203/rs.3.rs-8335282/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"cf9637a2-fdd1-479d-b875-c46104d867ec","owner":[],"postedDate":"February 13th, 2026","published":true,"recentEditorialEvents":[{"type":"decision","content":"Rejected","date":"2026-05-14T09:35:58+00:00","index":"","fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-05-14T09:43:29+00:00","versionOfRecord":[],"versionCreatedAt":"2026-02-13 12:43:39","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8335282","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8335282","identity":"rs-8335282","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}