Model-based Individual Learning for Competitive Agents

Model-based Individual Learning for Competitive Agents | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Model-based Individual Learning for Competitive Agents Yinghui Pan, Fanke Chen, Biyang Ma, Yifeng Zeng, Prashant Doshi This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6756780/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Competitive multiagent reinforcement learning is complicated since training individual agents' policies is highly coupled with the prediction of other agents' actions in the learning process. It is rather difficult for the subject agent to reason with their actions, which however is particularly useful when the subject agent fails to execute the policy. In this article, we propose a myopic modeling-to-adaptation (MTA) framework to cope with competitive agent learning from the perspective of individual agents. A subject agent first learns its baseline policy while maintaining a set of candidate models of other agents. After that, it adapts the policy when interacting with the other agents and predicting their behaviours from the candidate models. Theoretically, an infinite number of candidate models shall be considered. We adapt a value equivalence approach to compress the model space. The difficulty lies in computing value equivalence when there is no explicit representation of agents' policy. We develop a scenario-based technique to evaluate the value equivalence of their candidate models. We demonstrate the new framework with the value equivalence based model compression approach in multiple problem domains. Competitive Multiagent Reinforcement Learning Value Equivalence Model Compression Policy Adaptation Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6756780","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":470011601,"identity":"f304eeb6-bba3-43c1-9805-ad5dd578dafb","order_by":0,"name":"Yinghui Pan","email":"","orcid":"","institution":"Shenzhen University","correspondingAuthor":false,"prefix":"","firstName":"Yinghui","middleName":"","lastName":"Pan","suffix":""},{"id":470011602,"identity":"b01d06f8-33a2-4c4d-ae52-56c192124db4","order_by":1,"name":"Fanke Chen","email":"","orcid":"","institution":"Shenzhen University","correspondingAuthor":false,"prefix":"","firstName":"Fanke","middleName":"","lastName":"Chen","suffix":""},{"id":470011603,"identity":"a3037be1-48dc-4616-8d71-065adbcb4b93","order_by":2,"name":"Biyang Ma","email":"","orcid":"","institution":"Minnan Normal University","correspondingAuthor":false,"prefix":"","firstName":"Biyang","middleName":"","lastName":"Ma","suffix":""},{"id":470011604,"identity":"d5d7fbd2-8c2b-49a6-aaa8-0f9c1ca9c08f","order_by":3,"name":"Yifeng Zeng","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABLUlEQVRIie2RMUvDQBiGv3BwXS5mvSPQ+hMuOEhBzF+JCO1gA4GCBAp6ILRLcY7gj+jUrpVAsxy4SaQgLZ2duohm8BKprXjF1eGe4ePlOx7uXg7AYPiPIGuxiV+JA+BqViuhUxD/jmWifyuwVTDdKpuLNQrPkLWK3sF33MFTL4pfro7pRWsVR6fgDKaYJRolRejo7hYQu5eX80R2aTMJJ57k50BlgNlIpzgz1x4C5nmnNbf7AeV5OGaCI4AcMFvoFFT7UArxldLdUa6hsV/BiLyp1rQ9QztKCrxUNA9jqotrC2XkHeQSGbCRfJ14gmfEk2c3TU39g8fUWpPixHeS9nJN4sDhWTheiqJXr2fpw/Pwt3I4VcPqlz9C+I8Tsu8jG9W2KEdNU9VgMBgMik9IA2FHLL4yaAAAAABJRU5ErkJggg==","orcid":"","institution":"Northumbria University","correspondingAuthor":true,"prefix":"","firstName":"Yifeng","middleName":"","lastName":"Zeng","suffix":""},{"id":470011605,"identity":"4217a017-7b9c-4b4a-bea4-45d60ebbcde5","order_by":4,"name":"Prashant Doshi","email":"","orcid":"","institution":"University of Georgia","correspondingAuthor":false,"prefix":"","firstName":"Prashant","middleName":"","lastName":"Doshi","suffix":""}],"badges":[],"createdAt":"2025-05-27 07:53:14","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6756780/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6756780/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":97672941,"identity":"3cc35023-6626-4fee-80d8-d568cca0f82e","added_by":"auto","created_at":"2025-12-08 09:39:07","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1191357,"visible":true,"origin":"","legend":"","description":"","filename":"AIreview.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6756780/v1_covered_e089c6a8-c386-4c74-be74-7d4a514205aa.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Model-based Individual Learning for Competitive Agents","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Competitive Multiagent Reinforcement Learning, Value Equivalence, Model Compression, Policy Adaptation","lastPublishedDoi":"10.21203/rs.3.rs-6756780/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6756780/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Competitive multiagent reinforcement learning is complicated since training individual agents' policies is highly coupled with the prediction of other agents' actions in the learning process. It is rather difficult for the subject agent to reason with their actions, which however is particularly useful when the subject agent fails to execute the policy. In this article, we propose a myopic modeling-to-adaptation (MTA) framework to cope with competitive agent learning from the perspective of individual agents. A subject agent first learns its baseline policy while maintaining a set of candidate models of other agents. After that, it adapts the policy when interacting with the other agents and predicting their behaviours from the candidate models. Theoretically, an infinite number of candidate models shall be considered. We adapt a value equivalence approach to compress the model space. The difficulty lies in computing value equivalence when there is no explicit representation of agents' policy. We develop a scenario-based technique to evaluate the value equivalence of their candidate models. We demonstrate the new framework with the value equivalence based model compression approach in multiple problem domains.","manuscriptTitle":"Model-based Individual Learning for Competitive Agents","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-06-13 17:56:01","doi":"10.21203/rs.3.rs-6756780/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"e92c7f76-2a03-4dff-8cd5-8e39b6f4e086","owner":[],"postedDate":"June 13th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-12-08T05:53:46+00:00","versionOfRecord":[],"versionCreatedAt":"2025-06-13 17:56:01","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6756780","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6756780","identity":"rs-6756780","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}