Relative Position Enhanced Self-Attention with Co-Aware Distillation

Relative Position Enhanced Self-Attention with Co-Aware Distillation | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Relative Position Enhanced Self-Attention with Co-Aware Distillation Qinglong Chu, Haitao Wang, Jianfeng He This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5023471/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract The self-attention mechanism has been widely adopted in sequential recommendation due to its powerful sequence modeling capabilities. However, the inherent limitations of the self-attention mechanism pose challenges in effectively capturing global collaborative information across sequences and the relative positional information between items within a sequence. Firstly, the core operation of the self-attention mechanism involves calculating the similarity between item pairs within a single user’s sequence via the dot product of Query and Key. This approach is inherently limited in its ability to capture collaborative information across multiple user sequences. Secondly, due to the permutation-invariance property of the attention mechanism, it naturally lacks the capability to capture relative positional information. To address these challenges, this paper proposes a novel sequential recommendation method called Relative Position Enhanced Self-Attention with Collaborative-Aware Distillation (RPESA-CAD). First, we introduce a Collaborative-Aware Distillation module, which captures collaborative information across sequences by constructing a global item transition graph and distills this information into item embeddings, thereby enabling the model to effectively leverage global collaborative information. Additionally, a Relative Position Awareness module is proposed to model the relative positional relationships between item pairs. Unlike previous approaches, our method unifies the modeling of both positional and semantic differences between items, addressing the issue of performance degradation caused by disparate modeling strategies. We conduct extensive experiments on four public datasets, and the results demonstrate that the proposed model outperforms several baseline models across various evaluation metrics, validating its effectiveness. sequential recommendation self-attention knowledge distillation position encoding Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5023471","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":350563871,"identity":"45d9f181-62d3-45f3-8f7e-bee19b23564a","order_by":0,"name":"Qinglong Chu","email":"","orcid":"","institution":"Kunming University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Qinglong","middleName":"","lastName":"Chu","suffix":""},{"id":350563872,"identity":"16388731-a830-486f-868f-ea4dd364e829","order_by":1,"name":"Haitao Wang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAnElEQVRIiWNgGAWjYBACCQYGNgaGCgk5edK0HDhjYWzYQJKWg20ViQwHiNUiOe2M2eOP8yQSGBuYHz66QYwWaekcc4OD2yTy2BnYjI1ziNEiJ527TQKopZixgYdNmgQtcyQSGw4Qq0UarKWBFC2Ss/O/SZw5JmFs2EysXyRup6VJVNTUycmzNz98TJQWBGAmTfkoGAWjYBSMAnwAABYiLONdTU7bAAAAAElFTkSuQmCC","orcid":"","institution":"Kunming University of Science and Technology","correspondingAuthor":true,"prefix":"","firstName":"Haitao","middleName":"","lastName":"Wang","suffix":""},{"id":350563873,"identity":"493022db-6567-4d4b-bd80-e71a0e5e9e04","order_by":2,"name":"Jianfeng He","email":"","orcid":"","institution":"Kunming University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Jianfeng","middleName":"","lastName":"He","suffix":""}],"badges":[],"createdAt":"2024-09-03 08:54:38","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5023471/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5023471/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":67124391,"identity":"b4513780-e51c-4621-b4b3-74623ab87fab","added_by":"auto","created_at":"2024-10-21 11:47:05","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":5383238,"visible":true,"origin":"","legend":"","description":"","filename":"RelativePositionEnhancedSelfAttentionwithCoAwareDistillation.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5023471/v1_covered_b2bf2e0f-7827-48c1-bb42-da6fb60f2d7a.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Relative Position Enhanced Self-Attention with Co-Aware Distillation","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"sequential recommendation, self-attention, knowledge distillation, position encoding","lastPublishedDoi":"10.21203/rs.3.rs-5023471/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5023471/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"The self-attention mechanism has been widely adopted in sequential recommendation due to its powerful sequence modeling capabilities. However, the inherent limitations of the self-attention mechanism pose challenges in effectively capturing global collaborative information across sequences and the relative positional information between items within a sequence. Firstly, the core operation of the self-attention mechanism involves calculating the similarity between item pairs within a single user’s sequence via the dot product of Query and Key. This approach is inherently limited in its ability to capture collaborative information across multiple user sequences. Secondly, due to the permutation-invariance property of the attention mechanism, it naturally lacks the capability to capture relative positional information. To address these challenges, this paper proposes a novel sequential recommendation method called Relative Position Enhanced Self-Attention with Collaborative-Aware Distillation (RPESA-CAD). First, we introduce a Collaborative-Aware Distillation module, which captures collaborative information across sequences by constructing a global item transition graph and distills this information into item embeddings, thereby enabling the model to effectively leverage global collaborative information. Additionally, a Relative Position Awareness module is proposed to model the relative positional relationships between item pairs. Unlike previous approaches, our method unifies the modeling of both positional and semantic differences between items, addressing the issue of performance degradation caused by disparate modeling strategies. We conduct extensive experiments on four public datasets, and the results demonstrate that the proposed model outperforms several baseline models across various evaluation metrics, validating its effectiveness.","manuscriptTitle":"Relative Position Enhanced Self-Attention with Co-Aware Distillation","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-10-10 04:18:09","doi":"10.21203/rs.3.rs-5023471/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"64d16a32-0b6c-4a11-a289-2471984062a3","owner":[],"postedDate":"October 10th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-10-21T11:38:53+00:00","versionOfRecord":[],"versionCreatedAt":"2024-10-10 04:18:09","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5023471","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5023471","identity":"rs-5023471","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}