CTMAPPO-Clip: A CTCE-Based Approach to Mitigate Policy Overfitting in Multi-Agent Reinforcement Learning

CTMAPPO-Clip: A CTCE-Based Approach to Mitigate Policy Overfitting in Multi-Agent Reinforcement Learning | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article CTMAPPO-Clip: A CTCE-Based Approach to Mitigate Policy Overfitting in Multi-Agent Reinforcement Learning Qile Bo, Dongsheng Wang, Xinghao Han, Liang Shi, Yue Niu, Huige Li, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9241599/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Achieving efficient coordination among multiple agents has become a key research focus in reinforcement learning. However, in complex collaborative environments such as SMAC, existing approaches often suffer from limited policy generalization and a tendency toward policy overfitting during multi-agent coordination. To address these limitations, we propose CTMAPPO-Clip, a novel algorithm based on the CTCE paradigm, designed to mitigate policy overfitting in MARL. First, we model and optimize the joint policy by decomposing it into a set of conditional probability distributions. This allows each agent to independently optimize its own policy function given the observed states and actions of other agents, thereby improving policy generalization and decision-making stability. Second, we incorporate a Transformer-based architecture into the policy network, leveraging self-attention mechanisms to capture inter-agent dependencies and collaborative patterns, thereby enhancing the expressive power and coordination modeling of the joint policy. Finally, we introduce an advantage clipping mechanism, which dynamically truncates excessively high advantage values during training. This suppresses noisy gradient updates caused by outlier advantages, reduces the risk of converging to suboptimal policies, and enhances the robustness of policy learning. Experimental results on the SMAC benchmark demonstrate that CTMAPPO-Clip outperforms several state-of-the-art baselines, including QMIX and MAPPO, achieving superior performance and validating the effectiveness of CTMAPPO-Clip in mitigating policy overfitting in MARL. Reinforcement Learning Multi-Agent PPO CTCE paradigm Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9241599","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":626179955,"identity":"36bcf526-62cb-49a3-aea6-95c3cf92c91f","order_by":0,"name":"Qile Bo","email":"","orcid":"","institution":"Jiangsu University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Qile","middleName":"","lastName":"Bo","suffix":""},{"id":626179958,"identity":"c510fbf6-8185-49ad-b342-0681609803ec","order_by":1,"name":"Dongsheng Wang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA7klEQVRIie2PMYrCQBSG/2zANLPWEwb1CoaU62FeCEztDRzYwjKtASFXyBFmsdgmB4jYKIKVnU1s1IkpthtTLjhfMz/D++b9Azgc/xBPgbozUNRd6d4K0yZQD+VP5e14H8XP9EkwbEZFvj+F1wbjYU3eZW57fUVpbJS43JEUjBCHNfliZVWQHBl2SSmMYoolZU0Dn9mUTNOmVYpcy7AhLF4rirotqobkphhNXyrmL9Ea97isKP1ikkd5dfgWNiXKKsnPkKNiWSXbZjabDH/Tn4tVUYw+Pm9tNK0A/ixrEYAJAu01zxho66TD4XC8Lw/lh08lbnc3qwAAAABJRU5ErkJggg==","orcid":"","institution":"Jiangsu University of Science and Technology","correspondingAuthor":true,"prefix":"","firstName":"Dongsheng","middleName":"","lastName":"Wang","suffix":""},{"id":626179960,"identity":"51643865-6782-4081-879e-f22f2101591c","order_by":2,"name":"Xinghao Han","email":"","orcid":"","institution":"JiangSu Automation Research Institute","correspondingAuthor":false,"prefix":"","firstName":"Xinghao","middleName":"","lastName":"Han","suffix":""},{"id":626179961,"identity":"cefe0a23-022e-4e90-aa74-d443e2d5b187","order_by":3,"name":"Liang Shi","email":"","orcid":"","institution":"Jiangsu University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Liang","middleName":"","lastName":"Shi","suffix":""},{"id":626179962,"identity":"812e7df6-7885-4a10-9fcd-ddcaebbc7576","order_by":4,"name":"Yue Niu","email":"","orcid":"","institution":"Jiangsu University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Yue","middleName":"","lastName":"Niu","suffix":""},{"id":626179964,"identity":"32b68ad6-da1a-4d9e-975c-3db3d876c16f","order_by":5,"name":"Huige Li","email":"","orcid":"","institution":"Jiangsu University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Huige","middleName":"","lastName":"Li","suffix":""},{"id":626179965,"identity":"72df9515-ceb8-4613-9390-4d0558736fd7","order_by":6,"name":"Tianning Zhang","email":"","orcid":"","institution":"Jiangsu University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Tianning","middleName":"","lastName":"Zhang","suffix":""},{"id":626179966,"identity":"5d0665e1-d858-4f0c-bd18-85ac08fa6c8b","order_by":7,"name":"Yun Cui","email":"","orcid":"","institution":"Jiangsu University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Yun","middleName":"","lastName":"Cui","suffix":""}],"badges":[],"createdAt":"2026-03-27 07:38:44","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9241599/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9241599/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":108713781,"identity":"710145c3-f2a7-4b0b-a391-27841ac6618b","added_by":"auto","created_at":"2026-05-07 14:42:36","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2429093,"visible":true,"origin":"","legend":"","description":"","filename":"JAAMASv21.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9241599/v1_covered_4aadb0d9-b954-4baf-94c1-73b1983ce015.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"CTMAPPO-Clip: A CTCE-Based Approach to Mitigate Policy Overfitting in Multi-Agent Reinforcement Learning","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Reinforcement Learning, Multi-Agent, PPO, CTCE paradigm","lastPublishedDoi":"10.21203/rs.3.rs-9241599/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9241599/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eAchieving efficient coordination among multiple agents has become a key research focus in reinforcement learning. However, in complex collaborative environments such as SMAC, existing approaches often suffer from limited policy generalization and a tendency toward policy overfitting during multi-agent coordination. To address these limitations, we propose CTMAPPO-Clip, a novel algorithm based on the CTCE paradigm, designed to mitigate policy overfitting in MARL. First, we model and optimize the joint policy by decomposing it into a set of conditional probability distributions. This allows each agent to independently optimize its own policy function given the observed states and actions of other agents, thereby improving policy generalization and decision-making stability. Second, we incorporate a Transformer-based architecture into the policy network, leveraging self-attention mechanisms to capture inter-agent dependencies and collaborative patterns, thereby enhancing the expressive power and coordination modeling of the joint policy. Finally, we introduce an advantage clipping mechanism, which dynamically truncates excessively high advantage values during training. This suppresses noisy gradient updates caused by outlier advantages, reduces the risk of converging to suboptimal policies, and enhances the robustness of policy learning. Experimental results on the SMAC benchmark demonstrate that CTMAPPO-Clip outperforms several state-of-the-art baselines, including QMIX and MAPPO, achieving superior performance and validating the effectiveness of CTMAPPO-Clip in mitigating policy overfitting in MARL.\u003c/p\u003e","manuscriptTitle":"CTMAPPO-Clip: A CTCE-Based Approach to Mitigate Policy Overfitting in Multi-Agent Reinforcement Learning","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-27 02:44:50","doi":"10.21203/rs.3.rs-9241599/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"1462b594-e2d2-48ea-b7b7-febf7c1484bf","owner":[],"postedDate":"April 27th, 2026","published":true,"recentEditorialEvents":[{"type":"decision","content":"Rejected","date":"2026-05-07T14:36:43+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-07T09:03:48+00:00","index":11,"fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-30T14:53:39+00:00","index":10,"fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-05-07T14:41:52+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-27 02:44:50","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9241599","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9241599","identity":"rs-9241599","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}