TSPPO: Transformer-Based Sequential Proximal Policy Optimization for Multi-Agent Systems

TSPPO: Transformer-Based Sequential Proximal Policy Optimization for Multi-Agent Systems | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article TSPPO: Transformer-Based Sequential Proximal Policy Optimization for Multi-Agent Systems Tao YANG, Xinhao SHI, Cheng XU, Yulin YANG, Qinghan ZENG, Hongzhe LIU This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6780777/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 03 Feb, 2026 Read the published version in Multimedia Systems → Version 1 posted 12 You are reading this latest preprint version Abstract Multi-agent reinforcement learning has emerged as a transformative approach for solving complex tasks in dynamic and cooperative environments, such as resource allocation, robotics, and swarm control. However, integrating long-term strategic planning with immediate reactive decision-making remains a significant challenge due to the inherent non-stationarity, partial observabil-ity, and scalability issues in multi-agent systems. In this paper, we propose a novel framework, Transformer-Based Sequential Proximal Policy Optimiza-tion(TSPPO). Specifically, we introduce Contextual State Encoding with Transformers to capture both long-term dependencies and fine-grained temporal dynamics, enabling agents to dynamically balance strategic planning and reactive decision-making. Furthermore, we develop a Pre-order Advantage Correction mechanism to mitigate non-stationarity by correcting the advantage function during sequential policy updates, ensuring stable convergence. To enhance learning efficiency, we propose Sequential Decisions on Marginal Contributions. This approach prioritizes agents for policy updates based on their estimated contributions to team performance. Extensive experiments conducted on benchmark environments, including the Star-Craft Multi-Agent Challenge and Multi-Agent MUJOCO, demonstrate that TSPPO consistently outperforms state-of-the-art baselines in terms of convergence speed, stability, and final performance. These results validate the effectiveness of our proposed framework in handling the complex interplay of cooperation and competition in multi-agent systems, setting a new standard for scalable and robust MARL approaches. Multi-Agent Reinforcement Learning Transformers Recurrent Neural Networks Sequential Decision-Making Non-Stationarity Proximal Policy Optimization Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 03 Feb, 2026 Read the published version in Multimedia Systems → Version 1 posted Editorial decision: Revision requested 29 Aug, 2025 Reviews received at journal 05 Aug, 2025 Reviews received at journal 04 Aug, 2025 Reviewers agreed at journal 09 Jul, 2025 Reviewers agreed at journal 09 Jul, 2025 Reviewers agreed at journal 09 Jul, 2025 Reviewers agreed at journal 09 Jul, 2025 Reviewers agreed at journal 09 Jul, 2025 Reviewers invited by journal 09 Jul, 2025 Editor assigned by journal 30 Jun, 2025 Submission checks completed at journal 30 May, 2025 First submitted to journal 29 May, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6780777","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":483227193,"identity":"8d6b13e1-63ae-4915-a525-08e2c752f477","order_by":0,"name":"Tao YANG","email":"","orcid":"","institution":"Beijing Union University","correspondingAuthor":false,"prefix":"","firstName":"Tao","middleName":"","lastName":"YANG","suffix":""},{"id":483227194,"identity":"a43e1546-bf5a-4409-80fa-c9bc3a3516e4","order_by":1,"name":"Xinhao SHI","email":"","orcid":"","institution":"Beijing Union University","correspondingAuthor":false,"prefix":"","firstName":"Xinhao","middleName":"","lastName":"SHI","suffix":""},{"id":483227195,"identity":"b0598b47-e545-487d-a382-9389e3034905","order_by":2,"name":"Cheng XU","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAx0lEQVRIiWNgGAWjYHACNhAhZ4BgE6nFmHQtiRuI1mJwI/nZg487atO3S+QYMHwoO8zAP7sBvxbJGWnmhjPPHM/dOSPHgHHGucMMEncO4NfCL5HDJs3bdix3w40cA2betsMMBhIJBDwC0vK37Vi6AUjLX2K0gG1hbKtJAGthJEaLZM8zM8netgOGO3ueFRzsOZfOI3GDgBaD48nPJH621cmbsydvfPCjzFqOfwYBLVBwGEweAGIeotQDQR2xCkfBKBgFo2AkAgA7KEBdc5C6pgAAAABJRU5ErkJggg==","orcid":"","institution":"Beijing Union University","correspondingAuthor":true,"prefix":"","firstName":"Cheng","middleName":"","lastName":"XU","suffix":""},{"id":483227196,"identity":"4815cf75-98bd-4fed-b1ee-8cd34c194cf7","order_by":3,"name":"Yulin YANG","email":"","orcid":"","institution":"Unit 32178 of the PLA ,Science and Technology Innovation Research Center of ARI","correspondingAuthor":false,"prefix":"","firstName":"Yulin","middleName":"","lastName":"YANG","suffix":""},{"id":483227197,"identity":"fa8ba6b0-9829-4cdb-9665-8ab560f7a98e","order_by":4,"name":"Qinghan ZENG","email":"","orcid":"","institution":"Unit 32178 of the PLA ,Science and Technology Innovation Research Center of ARI","correspondingAuthor":false,"prefix":"","firstName":"Qinghan","middleName":"","lastName":"ZENG","suffix":""},{"id":483227198,"identity":"1ffdac82-4244-4928-be2e-3609b9f77c37","order_by":5,"name":"Hongzhe LIU","email":"","orcid":"","institution":"Beijing Union University","correspondingAuthor":false,"prefix":"","firstName":"Hongzhe","middleName":"","lastName":"LIU","suffix":""}],"badges":[],"createdAt":"2025-05-30 04:08:20","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6780777/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6780777/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s00530-025-02153-1","type":"published","date":"2026-02-03T15:59:39+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":102234304,"identity":"c7ee9e44-8f44-4253-8ade-aeb1dd1f393b","added_by":"auto","created_at":"2026-02-09 16:09:35","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":5134299,"visible":true,"origin":"","legend":"","description":"","filename":"TSPPOTransformerBasedSequentialProximalPolicyOptimizationforMultiAgentSystems.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6780777/v1_covered_b91759ca-1361-497a-b094-0cc686f989a6.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"TSPPO: Transformer-Based Sequential Proximal Policy Optimization for Multi-Agent Systems","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"multimedia-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"mmsj","sideBox":"Learn more about [Multimedia Systems](http://link.springer.com/journal/530)","snPcode":"530","submissionUrl":"https://submission.nature.com/new-submission/530/3","title":"Multimedia Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Multi-Agent Reinforcement Learning, Transformers, Recurrent Neural Networks, Sequential Decision-Making, Non-Stationarity, Proximal Policy Optimization","lastPublishedDoi":"10.21203/rs.3.rs-6780777/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6780777/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Multi-agent reinforcement learning has emerged as a transformative approach for solving complex tasks in dynamic and cooperative environments, such as resource allocation, robotics, and swarm control. However, integrating long-term strategic planning with immediate reactive decision-making remains a significant challenge due to the inherent non-stationarity, partial observabil-ity, and scalability issues in multi-agent systems. In this paper, we propose a novel framework, Transformer-Based Sequential Proximal Policy Optimiza-tion(TSPPO). Specifically, we introduce Contextual State Encoding with Transformers to capture both long-term dependencies and fine-grained temporal dynamics, enabling agents to dynamically balance strategic planning and reactive decision-making. Furthermore, we develop a Pre-order Advantage Correction mechanism to mitigate non-stationarity by correcting the advantage function during sequential policy updates, ensuring stable convergence. To enhance learning efficiency, we propose Sequential Decisions on Marginal Contributions. This approach prioritizes agents for policy updates based on their estimated contributions to team performance. Extensive experiments conducted on benchmark environments, including the Star-Craft Multi-Agent Challenge and Multi-Agent MUJOCO, demonstrate that TSPPO consistently outperforms state-of-the-art baselines in terms of convergence speed, stability, and final performance. These results validate the effectiveness of our proposed framework in handling the complex interplay of cooperation and competition in multi-agent systems, setting a new standard for scalable and robust MARL approaches.","manuscriptTitle":"TSPPO: Transformer-Based Sequential Proximal Policy Optimization for Multi-Agent Systems","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-07-10 07:22:19","doi":"10.21203/rs.3.rs-6780777/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-08-29T13:18:34+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-08-05T06:53:04+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-08-04T09:07:36+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"314053395766916564606343802034490901950","date":"2025-07-09T19:29:15+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"288632902722161245553013872402722074181","date":"2025-07-09T17:56:54+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"60608142506004214460147959658660569709","date":"2025-07-09T09:46:45+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"197235200179552422435911103586314838078","date":"2025-07-09T09:06:26+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"126372892503419884034530448729981796321","date":"2025-07-09T07:50:01+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-07-09T06:46:27+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-06-30T15:56:02+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-05-30T08:25:01+00:00","index":"","fulltext":""},{"type":"submitted","content":"Multimedia Systems","date":"2025-05-30T03:58:33+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"multimedia-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"mmsj","sideBox":"Learn more about [Multimedia Systems](http://link.springer.com/journal/530)","snPcode":"530","submissionUrl":"https://submission.nature.com/new-submission/530/3","title":"Multimedia Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"7b9b03af-4667-4016-8a15-75afab40f5cb","owner":[],"postedDate":"July 10th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2026-02-09T16:06:31+00:00","versionOfRecord":{"articleIdentity":"rs-6780777","link":"https://doi.org/10.1007/s00530-025-02153-1","journal":{"identity":"multimedia-systems","isVorOnly":false,"title":"Multimedia Systems"},"publishedOn":"2026-02-03 15:59:39","publishedOnDateReadable":"February 3rd, 2026"},"versionCreatedAt":"2025-07-10 07:22:19","video":"","vorDoi":"10.1007/s00530-025-02153-1","vorDoiUrl":"https://doi.org/10.1007/s00530-025-02153-1","workflowStages":[]},"version":"v1","identity":"rs-6780777","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6780777","identity":"rs-6780777","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}