Entropy Based Blending of Policies for Multi-Agent Coexistence

Entropy Based Blending of Policies for Multi-Agent Coexistence | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Entropy Based Blending of Policies for Multi-Agent Coexistence Rother David, Herbert Franziska, Kalter Fabian, Koert Dorothea, and 3 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4562541/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 16 May, 2025 Read the published version in Autonomous Agents and Multi-Agent Systems → Version 1 posted 10 You are reading this latest preprint version Abstract Research on multi-agent interaction involving humans is still in its infancy. Most approaches have focused on environments with collaborative human behavior or a small, defined set of situations. When deploying robots in human-inhabited environments in the future, the diversity of interactions surpasses the capabilities of pre-trained collaboration models. "Coexistence" environments, characterized by agents with varying or partially aligned objectives, present a unique challenge for robotic collaboration. Traditional reinforcement learning methods fall short in these settings. These approaches lack the flexibility to adapt to changing agent counts or task requirements without undergoing retraining. Moreover, existing models do not adequatelysupport scenarios where robots should exhibit helpful behavior toward others without compromising their primary goals.To tackle this issue, we introduce a novel framework that decomposes interaction and task-solving into separate learning problems and blends the resulting policies at inference time using a theory of mind model for task estimation. We create impact-aware agents and linearly scale the cost of training agents with the number of agents and available tasks. To this end, a weighting function blending action distributions for individual interactions with the original task action distribution is proposed. To support our claims we demonstrate that our framework scales in task and agent count across several environments and considers collaboration opportunities when present.The new learning paradigm opens the path to more complex multi-robot, multi-human interactions. Reinforcement Learning Multi-Agent Systems Policy Blending Maximum Entropy cooperative Intelligence Full Text Additional Declarations Competing interest reported. This work was supported by the Honda Research Institute Europe, Germany Dorothea Koert was funded by German Federal Ministry of Education and Research (project IKIDA 01IS20045) Joni Pajarinen was supported by Research Council of Finland (formerly Academy of Finland) (decision 345521) Thomas H. Weisswange is an employee of the Honda Research Institute Europe GmbH. Cite Share Download PDF Status: Published Journal Publication published 16 May, 2025 Read the published version in Autonomous Agents and Multi-Agent Systems → Version 1 posted Editorial decision: Revision requested 22 Sep, 2024 Reviews received at journal 21 Sep, 2024 Reviews received at journal 19 Sep, 2024 Reviewers agreed at journal 08 Sep, 2024 Reviewers agreed at journal 05 Sep, 2024 Reviewers agreed at journal 04 Sep, 2024 Reviewers invited by journal 17 Jun, 2024 Editor assigned by journal 17 Jun, 2024 Submission checks completed at journal 11 Jun, 2024 First submitted to journal 11 Jun, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4562541","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":317776620,"identity":"8317a2ea-d9bd-4d8f-82ca-a0b844244c43","order_by":0,"name":"Rother David","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABHklEQVRIie2PMUvDQBTHXzg4l6NzirafIXJQBat+lRcC1yXgkEVwqJChy32Agv0Q5hN4JdAula6RgAjFTh0KXdLNFzXbWR0d7scdPP7cj/87AIfjP2LoiHrw6W6/kxrvHiA4qAhSvDE2CoKnDyjQKEz8RWnNF6dmA6/d6+N0trusckqe37N9ddV5Apa9WZT2Ig6mE0ikOJlFDzHmlAx6pcBIauCJrSYwCnMBGGo/loyU4aNRvARkoQbR823Kct0oNzt2Ti2U8JcKh7VyVtmUIjJNC2NQK4XihcD8s8X2/XaxMtNJgFIUSnpaDShZ81KoudQ5T2yLtZZhut3cYvdoHK2g6l9Qomix/l1nNEqzra3maz1ryn5873A4HI5f+ABQamiYJvvivgAAAABJRU5ErkJggg==","orcid":"","institution":"TU Darmstadt","correspondingAuthor":true,"prefix":"","firstName":"Rother","middleName":"","lastName":"David","suffix":""},{"id":317776622,"identity":"b4e898cb-502d-4bae-a234-826cd3d82028","order_by":1,"name":"Herbert Franziska","email":"","orcid":"","institution":"TU Darmstadt","correspondingAuthor":false,"prefix":"","firstName":"Herbert","middleName":"","lastName":"Franziska","suffix":""},{"id":317776624,"identity":"773241b6-22b4-414d-af7a-9c9c8bac23a9","order_by":2,"name":"Kalter Fabian","email":"","orcid":"","institution":"TU Darmstadt","correspondingAuthor":false,"prefix":"","firstName":"Kalter","middleName":"","lastName":"Fabian","suffix":""},{"id":317776629,"identity":"c84c3a5a-697c-4126-86d5-c32eebe20f7a","order_by":3,"name":"Koert Dorothea","email":"","orcid":"","institution":"TU Darmstadt","correspondingAuthor":false,"prefix":"","firstName":"Koert","middleName":"","lastName":"Dorothea","suffix":""},{"id":317776631,"identity":"bb4296d1-e434-4945-8730-4f1f4de4151c","order_by":4,"name":"Pajarinen Joni","email":"","orcid":"","institution":"Aalto University","correspondingAuthor":false,"prefix":"","firstName":"Pajarinen","middleName":"","lastName":"Joni","suffix":""},{"id":317776632,"identity":"8c9593d2-72fd-46b7-b742-47508ede6e48","order_by":5,"name":"Peters Jan","email":"","orcid":"","institution":"TU Darmstadt","correspondingAuthor":false,"prefix":"","firstName":"Peters","middleName":"","lastName":"Jan","suffix":""},{"id":317776633,"identity":"fe492688-8aad-4c5c-96b1-b0d269b58cf2","order_by":6,"name":"H. Weisswange Thomas","email":"","orcid":"","institution":"Honda Research Institute Europe GmbH","correspondingAuthor":false,"prefix":"","firstName":"H.","middleName":"Weisswange","lastName":"Thomas","suffix":""}],"badges":[],"createdAt":"2024-06-11 08:37:28","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4562541/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4562541/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s10458-025-09707-7","type":"published","date":"2025-05-16T15:57:45+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":83067819,"identity":"75532cb5-2549-4454-8e7f-2dad8bf73c13","added_by":"auto","created_at":"2025-05-19 16:06:38","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1625385,"visible":true,"origin":"","legend":"","description":"","filename":"EntropyBasedBlendingofPoliciesforMultiAgentCoexistence.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4562541/v1_covered_37dced20-1236-4f4a-a449-ea993938873d.pdf"}],"financialInterests":"Competing interest reported. This work was supported by the Honda Research Institute Europe, Germany\nDorothea Koert was funded by German Federal Ministry of Education\nand Research (project IKIDA 01IS20045)\nJoni Pajarinen was supported by Research Council of Finland\n(formerly Academy of Finland) (decision 345521)\nThomas H. Weisswange is an employee of the Honda Research Institute Europe GmbH.","formattedTitle":"Entropy Based Blending of Policies for Multi-Agent Coexistence","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"autonomous-agents-and-multi-agent-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"agnt","sideBox":"Learn more about [Autonomous Agents and Multi-Agent Systems](http://link.springer.com/journal/10458)","snPcode":"10458","submissionUrl":"https://submission.nature.com/new-submission/10458/3","title":"Autonomous Agents and Multi-Agent Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Reinforcement Learning, Multi-Agent Systems, Policy Blending, Maximum Entropy, cooperative Intelligence","lastPublishedDoi":"10.21203/rs.3.rs-4562541/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4562541/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Research on multi-agent interaction involving humans is still in its infancy. Most approaches have focused on environments with collaborative human behavior or a small, defined set of situations. When deploying robots in human-inhabited environments in the future, the diversity of interactions surpasses the capabilities of pre-trained collaboration models. \"Coexistence\" environments, characterized by agents with varying or partially aligned objectives, present a unique challenge for robotic collaboration. Traditional reinforcement learning methods fall short in these settings. These approaches lack the flexibility to adapt to changing agent counts or task requirements without undergoing retraining. Moreover, existing models do not adequatelysupport scenarios where robots should exhibit helpful behavior toward others without compromising their primary goals.To tackle this issue, we introduce a novel framework that decomposes interaction and task-solving into separate learning problems and blends the resulting policies at inference time using a theory of mind model for task estimation. We create impact-aware agents and linearly scale the cost of training agents with the number of agents and available tasks. To this end, a weighting function blending action distributions for individual interactions with the original task action distribution is proposed. To support our claims we demonstrate that our framework scales in task and agent count across several environments and considers collaboration opportunities when present.The new learning paradigm opens the path to more complex multi-robot, multi-human interactions.","manuscriptTitle":"Entropy Based Blending of Policies for Multi-Agent Coexistence","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-06-27 10:12:07","doi":"10.21203/rs.3.rs-4562541/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-09-22T04:31:29+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-09-21T22:26:44+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-09-19T07:14:05+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"268791121600908033241023235847807810875","date":"2024-09-08T05:20:17+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"12360919161195965340056157421026550113","date":"2024-09-06T02:44:09+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"16641740887841474620071165212119539669","date":"2024-09-04T21:58:34+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-06-18T02:47:39+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-06-17T18:35:39+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-06-12T02:32:06+00:00","index":"","fulltext":""},{"type":"submitted","content":"Autonomous Agents and Multi-Agent Systems","date":"2024-06-11T08:36:15+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"autonomous-agents-and-multi-agent-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"agnt","sideBox":"Learn more about [Autonomous Agents and Multi-Agent Systems](http://link.springer.com/journal/10458)","snPcode":"10458","submissionUrl":"https://submission.nature.com/new-submission/10458/3","title":"Autonomous Agents and Multi-Agent Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"5af73a21-95f1-49d0-bf18-9c4d94d001ea","owner":[],"postedDate":"June 27th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2025-05-19T16:01:34+00:00","versionOfRecord":{"articleIdentity":"rs-4562541","link":"https://doi.org/10.1007/s10458-025-09707-7","journal":{"identity":"autonomous-agents-and-multi-agent-systems","isVorOnly":false,"title":"Autonomous Agents and Multi-Agent Systems"},"publishedOn":"2025-05-16 15:57:45","publishedOnDateReadable":"May 16th, 2025"},"versionCreatedAt":"2024-06-27 10:12:07","video":"","vorDoi":"10.1007/s10458-025-09707-7","vorDoiUrl":"https://doi.org/10.1007/s10458-025-09707-7","workflowStages":[]},"version":"v1","identity":"rs-4562541","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4562541","identity":"rs-4562541","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}