Hybrid Transformer-Based Recommender System with LLM-Assisted Semantic Modeling for Sequential and Federated Learning

Hybrid Transformer-Based Recommender System with LLM-Assisted Semantic Modeling for Sequential and Federated Learning | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Hybrid Transformer-Based Recommender System with LLM-Assisted Semantic Modeling for Sequential and Federated Learning Lakshmi Bai Maddala, Rajendra Pamula, Katteda Subbarao This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9316077/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 7 You are reading this latest preprint version Abstract Sequential recommender systems based on transformers have shown good performance to model the user interaction dynamics, although they usually depend mainly on the interaction data and fail to use rich semantic data that exists in item metadata. This is a major constraint especially in cases of sparsity and cold-start when the interaction histories are not enough to model preference accurately. This paper presents the suggestion of LLMTransRec, a hybrid Transformer-based recommendation model combining collaborative interaction cues, semantic representations generated by pretrained language models, and content-based features into a single model. In order to successfully integrate these heterogeneous modalities, we propose a context-sensitive gating system, which dynamically weighs the contributions of these modalities on the context of the sequence of interactions between a user. We test the suggested framework on a variety of benchmark datasets having different sparsity and richness of content. The experimental findings prove that the model attains stable improvements when compared to strong sequential and graph-based baselines on an integrated sampled evaluation protocol. Other studies, such as ablation analysis and cold-start analysis indicate that semantic features and adaptive fusion can help enhance the robustness of recommendations. On the whole, the findings indicate that the integration of semantic representations into sequential recommendation models is a viable way of improving the performance in the sparse-data context. Sequential Recommendation Transformer-based Models Large Language Models Multimodal Fusion Federated Learning Cold-Start Problem Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Reviews received at journal 18 May, 2026 Reviewers agreed at journal 23 Apr, 2026 Reviewers agreed at journal 20 Apr, 2026 Reviewers invited by journal 20 Apr, 2026 Editor assigned by journal 11 Apr, 2026 Submission checks completed at journal 11 Apr, 2026 First submitted to journal 03 Apr, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9316077","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":628257222,"identity":"68a4a6ff-f917-417a-872b-384b8a0b86b4","order_by":0,"name":"Lakshmi Bai Maddala","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABUUlEQVRIie2Rv2rCUBSHT7hwXa7JekPEPEHhiBAoFfsqKQGnWyg4GLBgIBAXH8DHUArOgYB2CHRNtxbByUKtSwqlNE20BLV07XC/8d7zcf78ACSS/wlyuAGgFQ/BRt7WKn74/cwoKF5RQU4oCKCyEOHVbTn6aG4XCv1DqXMblXHcUSaJwOKDnp7qbBhNkxTbQFk0W1aDiOhjsdmkLtRU0/ffEmibUKmGJcWKO93zETrZLkGvmSmqxl7uDBbngwWGAKfhEdUuK6GwOEOSdYktI+8yvJ4RJSgUIoDYQBiWlYe1pX/gACgXuaJMQrFS3j9zxd8KGBwpSVbJMMoVPV//XlCoesXFssGiY2XVvajhglE27+HuyE2DzXmmXGW74KIRHA7mTB/Xbr9uDv3Z0y7K501627o0/cVyK9y+qWlxWdnDDnPg+9R+zQcOFYlEIpH88AWIKmzVymq48AAAAABJRU5ErkJggg==","orcid":"","institution":"Indian Institute of Technology Dhanbad","correspondingAuthor":true,"prefix":"","firstName":"Lakshmi","middleName":"Bai","lastName":"Maddala","suffix":""},{"id":628257224,"identity":"6ca7e76d-4e2b-4762-ab5c-cc2201855b8f","order_by":1,"name":"Rajendra Pamula","email":"","orcid":"","institution":"Indian Institute of Technology Dhanbad","correspondingAuthor":false,"prefix":"","firstName":"Rajendra","middleName":"","lastName":"Pamula","suffix":""},{"id":628257228,"identity":"40913483-816b-4f45-9227-e3949d793cd4","order_by":2,"name":"Katteda Subbarao","email":"","orcid":"","institution":"St. Ann's College of Engineering Technology","correspondingAuthor":false,"prefix":"","firstName":"Katteda","middleName":"","lastName":"Subbarao","suffix":""}],"badges":[],"createdAt":"2026-04-03 20:38:17","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9316077/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9316077/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":108006886,"identity":"87b07054-81c7-4070-a0c3-f7100469984d","added_by":"auto","created_at":"2026-04-28 12:57:49","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1494487,"visible":true,"origin":"","legend":"","description":"","filename":"UpdatedPaper.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9316077/v1_covered_4676f103-60a4-4cf4-ab11-d1d154e13414.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Hybrid Transformer-Based Recommender System with LLM-Assisted Semantic Modeling for Sequential and Federated Learning","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"neural-processing-letters","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"nepl","sideBox":"Learn more about [Neural Processing Letters](http://link.springer.com/journal/11063)","snPcode":"11063","submissionUrl":"https://submission.nature.com/new-submission/11063/3","title":"Neural Processing Letters","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Sequential Recommendation, Transformer-based Models, Large Language Models, Multimodal Fusion, Federated Learning, Cold-Start Problem","lastPublishedDoi":"10.21203/rs.3.rs-9316077/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9316077/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eSequential recommender systems based on transformers have shown good performance to model the user interaction dynamics, although they usually depend mainly on the interaction data and fail to use rich semantic data that exists in item metadata. This is a major constraint especially in cases of sparsity and cold-start when the interaction histories are not enough to model preference accurately.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThis paper presents the suggestion of LLMTransRec, a hybrid Transformer-based recommendation model combining collaborative interaction cues, semantic representations generated by pretrained language models, and content-based features into a single model. In order to successfully integrate these heterogeneous modalities, we propose a context-sensitive gating system, which dynamically weighs the contributions of these modalities on the context of the sequence of interactions between a user.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eWe test the suggested framework on a variety of benchmark datasets having different sparsity and richness of content. The experimental findings prove that the model attains stable improvements when compared to strong sequential and graph-based baselines on an integrated sampled evaluation protocol. Other studies, such as ablation analysis and cold-start analysis indicate that semantic features and adaptive fusion can help enhance the robustness of recommendations.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eOn the whole, the findings indicate that the integration of semantic representations into sequential recommendation models is a viable way of improving the performance in the sparse-data context.\u003c/p\u003e","manuscriptTitle":"Hybrid Transformer-Based Recommender System with LLM-Assisted Semantic Modeling for Sequential and Federated Learning","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-27 11:30:34","doi":"10.21203/rs.3.rs-9316077/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"editorInvitedReview","content":"","date":"2026-05-18T13:56:59+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"24317964155321407631571747260412620232","date":"2026-04-23T04:12:39+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"107172106215714922239497326950730197692","date":"2026-04-20T19:56:32+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-04-20T11:55:47+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-04-11T13:49:37+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-04-11T13:49:16+00:00","index":"","fulltext":""},{"type":"submitted","content":"Neural Processing Letters","date":"2026-04-03T20:33:09+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"neural-processing-letters","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"nepl","sideBox":"Learn more about [Neural Processing Letters](http://link.springer.com/journal/11063)","snPcode":"11063","submissionUrl":"https://submission.nature.com/new-submission/11063/3","title":"Neural Processing Letters","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"087f17f3-2328-45e3-9e3b-5e41b7660585","owner":[],"postedDate":"April 27th, 2026","published":true,"recentEditorialEvents":[{"type":"editorInvitedReview","content":"","date":"2026-05-18T13:56:59+00:00","index":28,"fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-04-27T11:30:35+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-27 11:30:34","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9316077","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9316077","identity":"rs-9316077","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}