Multi-AGV Path Planning Using Deep Reinforcement Learning with Internal Curiosity

Multi-AGV Path Planning Using Deep Reinforcement Learning with Internal Curiosity | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Multi-AGV Path Planning Using Deep Reinforcement Learning with Internal Curiosity Huilin Yin, Shengkai Su, Yinjia Lin, Karin Festl, Jun Yan, Daniel Watzenig This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4453111/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Deep Reinforcement Learning (DRL) is promising for multi-agent path planning problems in which sparse external environmental rewards may cause the agent group to make overly conservative decisions and explore the environment inefficiently. In general, the reward shaping mechanism is used to mitigate the above problems with the additional reward function setting. However, it requires specific domain knowledge, which limits the general applicability, and the added reward functions are not necessarily applicable to all environments. This paper aims to improve the path planning efficiency of single agents and groups of agents with the Internal Curiosity Module (ICM) mechanism to boost the generalization abilities of the agents in different environments. To this end, we incorporate the internal curiosity mechanism into the soft actor-critic model for enhancing exploration strategies, adapting to environmental changes, and improving learning effectiveness. Then, we propose a multi-agent path planning method in which the curiosity mechanism is integrated with the Multi-Agent POsthumous Credit Assignment (MA-POCA) algorithm. The neural networks can automatically calculate the additional intrinsic rewards based on observed information about the environment and the actions taken by the group of agents. Based on our experiments, we make qualitative and quantitative analyses of the performance of the proposed methods and the baseline DRL methods. The experimental results show that our proposed methods can decline the number of learning episodes and the training time of path planning, so the proposed algorithms can accelerate the exploration of single agents or agent groups in the sparse reward environment. Path planning Multi-agent Deep reinforcement learning Curiosity mechanism Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4453111","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":313800483,"identity":"23fa105f-5a01-4710-9647-613973cd0274","order_by":0,"name":"Huilin Yin","email":"","orcid":"","institution":"Tongji University","correspondingAuthor":false,"prefix":"","firstName":"Huilin","middleName":"","lastName":"Yin","suffix":""},{"id":313800484,"identity":"7f1e7aa6-b063-401f-a6cc-fd305ea11406","order_by":1,"name":"Shengkai Su","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA/ElEQVRIiWNgGAWjYHACNhBhwMDMfADEYAZxJYjUwpbYQKIWBh7DBpgQXi3yM9KfPfi4o9aYv53nO4jBzsfAfPA2D4NdHi4tBjdyzA1nnjluJnGYd2MjkAF0GFuyNQ9DcjFOLRI5bNK8bcdsGIBamoEMoBYeM2kehgOJDTi0gBwm/ReoRf4wz0OoFv5veLUw3Egwk2ZsqzEzOMzDCNRSA7KFDa8WgzNvzCR72w4YGx5mM5w5s+0AMxszm7HlHINk3A5rT38m8bOtznDe+cMPPnxsq0uWb29+eONNhR1uh0HAYTgjmYEZbDt+9UBQB2fYEVQ7CkbBKBgFIw4AAG3tUbUJ+Yh4AAAAAElFTkSuQmCC","orcid":"","institution":"Tongji University","correspondingAuthor":true,"prefix":"","firstName":"Shengkai","middleName":"","lastName":"Su","suffix":""},{"id":313800485,"identity":"fa1c34db-e431-4fca-bfcf-a4065a040f4c","order_by":2,"name":"Yinjia Lin","email":"","orcid":"","institution":"Tongji University","correspondingAuthor":false,"prefix":"","firstName":"Yinjia","middleName":"","lastName":"Lin","suffix":""},{"id":313800486,"identity":"edc45c7b-95ba-4716-a530-841d3e63c0a9","order_by":3,"name":"Karin Festl","email":"","orcid":"","institution":"Graz University of Technology","correspondingAuthor":false,"prefix":"","firstName":"Karin","middleName":"","lastName":"Festl","suffix":""},{"id":313800488,"identity":"39bb89c7-299f-4b83-8095-452b0dd14b10","order_by":4,"name":"Jun Yan","email":"","orcid":"","institution":"Tongji University","correspondingAuthor":false,"prefix":"","firstName":"Jun","middleName":"","lastName":"Yan","suffix":""},{"id":313800489,"identity":"5e141e12-8818-4921-8857-f6c0f24309b9","order_by":5,"name":"Daniel Watzenig","email":"","orcid":"","institution":"Graz University of Technology","correspondingAuthor":false,"prefix":"","firstName":"Daniel","middleName":"","lastName":"Watzenig","suffix":""}],"badges":[],"createdAt":"2024-05-21 08:02:22","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4453111/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4453111/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":60836328,"identity":"0152e058-a0b4-4e5d-b871-7e3a638bbc40","added_by":"auto","created_at":"2024-07-22 16:03:43","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":3280824,"visible":true,"origin":"","legend":"","description":"","filename":"AAMAS.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4453111/v1_covered_97b1d22a-1786-4d87-823e-5028a2e35b9f.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Multi-AGV Path Planning Using Deep Reinforcement Learning with Internal Curiosity","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Path planning, Multi-agent, Deep reinforcement learning, Curiosity mechanism","lastPublishedDoi":"10.21203/rs.3.rs-4453111/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4453111/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Deep Reinforcement Learning (DRL) is promising for multi-agent path planning problems in which sparse external environmental rewards may cause the agent group to make overly conservative decisions and explore the environment inefficiently. In general, the reward shaping mechanism is used to mitigate the above problems with the additional reward function setting. However, it requires specific domain knowledge, which limits the general applicability, and the added reward functions are not necessarily applicable to all environments. This paper aims to improve the path planning efficiency of single agents and groups of agents with the Internal Curiosity Module (ICM) mechanism to boost the generalization abilities of the agents in different environments. To this end, we incorporate the internal curiosity mechanism into the soft actor-critic model for enhancing exploration strategies, adapting to environmental changes, and improving learning effectiveness. Then, we propose a multi-agent path planning method in which the curiosity mechanism is integrated with the Multi-Agent POsthumous Credit Assignment (MA-POCA) algorithm. The neural networks can automatically calculate the additional intrinsic rewards based on observed information about the environment and the actions taken by the group of agents. Based on our experiments, we make qualitative and quantitative analyses of the performance of the proposed methods and the baseline DRL methods. The experimental results show that our proposed methods can decline the number of learning episodes and the training time of path planning, so the proposed algorithms can accelerate the exploration of single agents or agent groups in the sparse reward environment. ","manuscriptTitle":"Multi-AGV Path Planning Using Deep Reinforcement Learning with Internal Curiosity","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-06-17 05:28:46","doi":"10.21203/rs.3.rs-4453111/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"4327c538-5b25-4f2e-926c-7799d8fe5d75","owner":[],"postedDate":"June 17th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-07-22T15:55:32+00:00","versionOfRecord":[],"versionCreatedAt":"2024-06-17 05:28:46","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4453111","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4453111","identity":"rs-4453111","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}