FedTLRec: Federated Recommendation with Transformer-based Parameter Aggregation and LoRA Compression

FedTLRec: Federated Recommendation with Transformer-based Parameter Aggregation and LoRA Compression | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article FedTLRec: Federated Recommendation with Transformer-based Parameter Aggregation and LoRA Compression WangXudong This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7994476/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Federated learning has emerged as a key paradigm in privacy-preserving computing due to its "data usable but not visible" property, enabling users to collaboratively train models without sharing raw data. Motivated by this, federated recommendation systems offer a promising architecture that balances user privacy with recommendation accuracy through distributed collaborative learning. However, existing federated recommendation systems face significant challenges in balancing model performance, communication efficiency, and user privacy. In this paper, we propose FedTLRec (Federated Recommendation with Transformer-based Parameter Aggregation and Collaborative LoRA), a novel federated recommendation framework that combines Low-Rank Adaptation (LoRA) parameter compression with Transformer-based parameter aggregation. Our approach addresses the communication bottleneck in federated learning by employing LoRA to compress client model updates, which significantly reduces data transmission overhead. Additionally, we introduce a Transformer-based aggregation model on the server side to integrate parameter updates from multiple clients effectively. This model leverages attention mechanisms to capture relationships between clients. To further enhance efficiency, we implement a K-means clustering strategy to group clients with similar characteristics before the parameter aggregation process. Extensive experiments conducted on real-world datasets demonstrate that FedTLRec achieves superior recommendation performance while substantially reducing communication costs compared to state-of-the-art federated recommendation methods. Furthermore, our method effectively manages client dropout scenarios, maintaining robust performance even when a portion of clients are offline. The code will be open sourced https://github.com/trueWangSyutung/Kmean-Lora-PFedRec Computer Architecture and Engineering federated recommendation low-rank adaptation transformer models collaborative learning parameter efficiency Full Text Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7994476","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":537825486,"identity":"610b9f75-20cf-4c91-8b2d-be50d573684f","order_by":0,"name":"WangXudong","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABFklEQVRIie2QsWrDMBCGzxjsRcWrOjTPoGJwCBryKvZSLSbQpWTooElTIavzDh0yZVYwpIscrx4DGZuCgvdSxXHIYqcdC9EH4n4d93FCABbL/8U9HtkGee78qnixGXYBgXT4HxVEWgWuK+SjyD/vBB0Mg/e6PggKY0cNtzClCfcL2amoydNoLlg4yr6WeCWY2aIeOSiWcDSJu5RIphE5iDxZVMUSViI/KY7pcGye2qWU+4gkjaJ2+qJ8X1GqNNw2W8o3wBeF9yvjah852YaFpPIirDYMIbl+zuI1CwVKO5X7WRrW6IUOSJnv9NQEP8sXWr/Sh5mvOhWDh5uCT9+DAEuAY/Z65g2ubkog23vA+2ctFovlJvkBfO5p0Q7tO0QAAAAASUVORK5CYII=","orcid":"https://orcid.org/0009-0000-7453-0395","institution":"Tianjin University of Technology","correspondingAuthor":true,"prefix":"","firstName":"","middleName":"","lastName":"WangXudong","suffix":""}],"badges":[],"createdAt":"2025-10-31 04:57:01","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":true,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":true},"doi":"10.21203/rs.3.rs-7994476/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7994476/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":95312255,"identity":"de54b152-6d95-4bae-9d68-943b0ea8555b","added_by":"auto","created_at":"2025-11-06 15:48:28","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":553235,"visible":true,"origin":"","legend":"","description":"","filename":"IEEEConferenceTemplate2.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7994476/v1_covered_899f3bc1-57d8-4b94-b7f9-9278529f3342.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003eFedTLRec: Federated Recommendation with Transformer-based Parameter Aggregation and LoRA Compression\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"Tianjin University of Technology","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"federated recommendation, low-rank adaptation, transformer models, collaborative learning, parameter efficiency","lastPublishedDoi":"10.21203/rs.3.rs-7994476/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7994476/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eFederated learning has emerged as a key paradigm in privacy-preserving computing due to its \"data usable but not visible\" property, enabling users to collaboratively train models without sharing raw data. Motivated by this, federated recommendation systems offer a promising architecture that balances user privacy with recommendation accuracy through distributed collaborative learning. However, existing federated recommendation systems face significant challenges in balancing model performance, communication efficiency, and user privacy. In this paper, we propose FedTLRec (Federated Recommendation with Transformer-based Parameter Aggregation and Collaborative LoRA), a novel federated recommendation framework that combines Low-Rank Adaptation (LoRA) parameter compression with Transformer-based parameter aggregation. Our approach addresses the communication bottleneck in federated learning by employing LoRA to compress client model updates, which significantly reduces data transmission overhead. Additionally, we introduce a Transformer-based aggregation model on the server side to integrate parameter updates from multiple clients effectively. This model leverages attention mechanisms to capture relationships between clients. To further enhance efficiency, we implement a K-means clustering strategy to group clients with similar characteristics before the parameter aggregation process. Extensive experiments conducted on real-world datasets demonstrate that FedTLRec achieves superior recommendation performance while substantially reducing communication costs compared to state-of-the-art federated recommendation methods. Furthermore, our method effectively manages client dropout scenarios, maintaining robust performance even when a portion of clients are offline. The code will be open sourced https://github.com/trueWangSyutung/Kmean-Lora-PFedRec\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e","manuscriptTitle":"FedTLRec: Federated Recommendation with Transformer-based Parameter Aggregation and LoRA Compression","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-11-05 19:02:50","doi":"10.21203/rs.3.rs-7994476/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"1468e0c3-14a8-4761-b902-ea8ec2c163c5","owner":[],"postedDate":"November 5th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":57197992,"name":"Computer Architecture and Engineering"}],"tags":[],"updatedAt":"2025-11-05T19:02:50+00:00","versionOfRecord":[],"versionCreatedAt":"2025-11-05 19:02:50","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7994476","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7994476","identity":"rs-7994476","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}