HAMRL: A Hierarchical Attention-based Multi-agent Reinforcement Learning Approach for IoT-based Service Composition

HAMRL: A Hierarchical Attention-based Multi-agent Reinforcement Learning Approach for IoT-based Service Composition | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article HAMRL: A Hierarchical Attention-based Multi-agent Reinforcement Learning Approach for IoT-based Service Composition Rezaei Sara, Goli-Bidgoli Salman, Dehghani Fereshteh This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7312675/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 9 You are reading this latest preprint version Abstract The dynamic and heterogeneous nature of Internet of Things (IoT) environments present considerable challenges for efficient service composition, particularly with respect to scalability, adaptability, and Quality of Service (QoS). This paper proposes a novel framework—Hierarchical Attention-based Multi-agent Reinforcement Learning (HAMRL)—designed to address these challenges by integrating a centralized critic, a hierarchical attention mechanism, and decentralized actors. HAMRL enables intelligent IoT agents to collaboratively and dynamically compose services in response to diverse user requests. By fusing local and global contextual information, the framework enhances decision-making, improves scalability, and increases adaptability across distributed IoT networks. Extensive experiments demonstrate that HAMRL significantly outperforms state-of-the-art approaches—including MAAC, COMA, MAA2C, and MAPPO—in terms of QoS, learning stability, and adaptability under dynamic conditions. This study underscores the essential role of adaptive coordination mechanisms in optimizing service composition in complex IoT ecosystems, establishing HAMRL as a promising solution for next-generation scalable and resilient IoT applications. Multi-Agent Hierarchical Attention Reinforcement Learning QoS-aware Internet of Things Service Composition Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 19 Mar, 2026 Reviews received at journal 03 Nov, 2025 Reviews received at journal 31 Oct, 2025 Reviewers agreed at journal 23 Oct, 2025 Reviewers agreed at journal 22 Oct, 2025 Reviewers invited by journal 10 Sep, 2025 Editor assigned by journal 10 Sep, 2025 Submission checks completed at journal 08 Aug, 2025 First submitted to journal 06 Aug, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7312675","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":515232021,"identity":"04dbe967-c473-4c67-866c-6fd590b80a4a","order_by":0,"name":"Rezaei Sara","email":"","orcid":"","institution":"University of Kashan","correspondingAuthor":false,"prefix":"","firstName":"Rezaei","middleName":"","lastName":"Sara","suffix":""},{"id":515232022,"identity":"a5d93b5b-2690-4d08-8990-198cb07400e9","order_by":1,"name":"Goli-Bidgoli Salman","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABAklEQVRIiWNgGAWjYFAC5gYwxcfMfBhIHYAJS+DRwgjRwsbMlkyqFgYeY2QtuIF8+8HWDT931MqxsfN8Nvjx5468uUQC44cfDBb5uLQYnElsu9l75rgxGzPv5sTetmeGO2ckMEv2MEhYNuDSwpDYdoO37VhiG1DLAd6Gw4wbbiQwSAP9YoDTYf0P227+BWvheXzwz5/D9kAtzL/xaWG4kdh2m7etBqSFOZmH7XAiUAsbXlsMbjxsuy3bdgDoFzZjY9m2w8kbzjxss+wxwOew5GM337bVyfHzH34s+ebPYdsNx5MP3/hRUYfbYRBwGJkDiilCGhgY6giqGAWjYBSMghEMAH2KWX2vMtVaAAAAAElFTkSuQmCC","orcid":"","institution":"University of Kashan","correspondingAuthor":true,"prefix":"","firstName":"Goli-Bidgoli","middleName":"","lastName":"Salman","suffix":""},{"id":515232023,"identity":"2211d4d8-3a34-4aeb-9d7e-4d1a5231750b","order_by":2,"name":"Dehghani Fereshteh","email":"","orcid":"","institution":"University of Kashan","correspondingAuthor":false,"prefix":"","firstName":"Dehghani","middleName":"","lastName":"Fereshteh","suffix":""}],"badges":[],"createdAt":"2025-08-06 19:38:16","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7312675/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7312675/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":91515499,"identity":"c59e4d5b-165a-4631-9efb-41ecda7a6705","added_by":"auto","created_at":"2025-09-17 09:14:29","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1114282,"visible":true,"origin":"","legend":"","description":"","filename":"0HAMRL20250806.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7312675/v1_covered_a4a71f4a-c33f-4bd1-8de2-b9cb8b1d7aa1.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"HAMRL: A Hierarchical Attention-based Multi-agent Reinforcement Learning Approach for IoT-based Service Composition","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"cluster-computing","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [Cluster Computing](https://www.springer.com/journal/10586)","snPcode":"10586","submissionUrl":"https://submission.nature.com/new-submission/10586/3","title":"Cluster Computing","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Multi-Agent, Hierarchical Attention, Reinforcement Learning, QoS-aware, Internet of Things, Service Composition","lastPublishedDoi":"10.21203/rs.3.rs-7312675/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7312675/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe dynamic and heterogeneous nature of Internet of Things (IoT) environments present considerable challenges for efficient service composition, particularly with respect to scalability, adaptability, and Quality of Service (QoS). This paper proposes a novel framework\u0026mdash;Hierarchical Attention-based Multi-agent Reinforcement Learning (HAMRL)\u0026mdash;designed to address these challenges by integrating a centralized critic, a hierarchical attention mechanism, and decentralized actors. HAMRL enables intelligent IoT agents to collaboratively and dynamically compose services in response to diverse user requests. By fusing local and global contextual information, the framework enhances decision-making, improves scalability, and increases adaptability across distributed IoT networks. Extensive experiments demonstrate that HAMRL significantly outperforms state-of-the-art approaches\u0026mdash;including MAAC, COMA, MAA2C, and MAPPO\u0026mdash;in terms of QoS, learning stability, and adaptability under dynamic conditions. This study underscores the essential role of adaptive coordination mechanisms in optimizing service composition in complex IoT ecosystems, establishing HAMRL as a promising solution for next-generation scalable and resilient IoT applications.\u003c/p\u003e","manuscriptTitle":"HAMRL: A Hierarchical Attention-based Multi-agent Reinforcement Learning Approach for IoT-based Service Composition","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-09-17 08:42:22","doi":"10.21203/rs.3.rs-7312675/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-03-19T13:20:17+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-11-03T09:16:11+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-10-31T19:52:24+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"27343997621727236094758625744492147113","date":"2025-10-24T01:56:01+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"11767664653968920195684697431234624610","date":"2025-10-22T13:14:50+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-09-10T09:09:25+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-09-10T08:35:59+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-08-08T09:11:38+00:00","index":"","fulltext":""},{"type":"submitted","content":"Cluster Computing","date":"2025-08-06T19:30:57+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"cluster-computing","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [Cluster Computing](https://www.springer.com/journal/10586)","snPcode":"10586","submissionUrl":"https://submission.nature.com/new-submission/10586/3","title":"Cluster Computing","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"dba487bc-6580-486f-9489-587b21eed9a2","owner":[],"postedDate":"September 17th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-05-12T12:37:24+00:00","versionOfRecord":[],"versionCreatedAt":"2025-09-17 08:42:22","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7312675","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7312675","identity":"rs-7312675","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}