Decentralized Collaborative Filtering for Distributed Hash Tables

Decentralized Collaborative Filtering for Distributed Hash Tables | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Decentralized Collaborative Filtering for Distributed Hash Tables Krishna Shukla This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4316139/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract We implement a decentralized method for collaborative filtering via matrix factorization that can rest on peer-to-peer Distributed Hash Table architectures like Pastry and Chord, for the purpose of realizing a scalable and robust decentralised search engine. We begin by introducing a centralized method for matrix factorization that involves minimizing a cost function consisting of the ratings and user and item latent factors. We introduce biases and then derive the gradients for the error update step. We then introduce gossip learning; a matrix-factorization based collaborative filtering approach for a decentralized network. Since we could not obtain a dataset of browsing activity of users where the rating would constitute the number of times a website is visited, we used a dataset of movie ratings to test our method. This preserves the user and item biases and provides user-item ratings for rating to latent factor error computation. We performed a train-test split and loaded the training and validation set onto each node separately. This method is then tested using the PeerSim peer-to-peer network simulator to obtain the best learning rate $\eta$, regularization parameter $\lambda$ and latent factors $k$ for a $10,000$ node peer-to-peer network. The optimization metric used is the Root Mean Squared Error measuring the discrepancy between the ratings in the validation set and their respective predictions by model updates on the training set at each node. We found that by the fortieth cycle of the PeerSim simulator, the training set is fully distributed across each node. The best performing parameter setting over our $63$ experiment runs has learning rate $\eta = 0.001$, regularization parameter $\lambda = 1$ and latent factors $k = 20$. These settings are then built into a basic working prototype of a Pastry algorithm based peer-to-peer distributed search engine with message passing tested for $50$ nodes. Collaborative Filtering Matrix Factorization Peer-toPeer Pastry Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4316139","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":297595889,"identity":"3e3396d0-a016-4067-bf02-800bbaca04b1","order_by":0,"name":"Krishna Shukla","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA1ElEQVRIiWNgGAWjYBACCQbGBmYQw37+4YMHEkjSYiDBlkCsFgYGqBYegwNEOUxy9uHmz4U77OzNpXs+HHjYZsPA396N3zJpvsQ26ZlnkhN3zjm74UBiWxqDxJmzG/BqkeNhbGPmbWNOYDiQu+FAwpnDQBfmEtTS/Jm3rd6e4UDOA6CW/4S1SPMwNkjzth1m3HAjh+FAQsUBwlokexjbgFqOJ87sOWYA1JLMQ9AvEmfYHwMdVm3Pz9788OEPAzs5/vZe/FowAA9pykfBKBgFo2AUYAUA3Y9IxpCR96kAAAAASUVORK5CYII=","orcid":"","institution":"Vrije Universiteit Amsterdam","correspondingAuthor":true,"prefix":"","firstName":"Krishna","middleName":"","lastName":"Shukla","suffix":""}],"badges":[],"createdAt":"2024-04-24 07:13:47","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4316139/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4316139/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":62990063,"identity":"a449f7e1-a194-48b8-8fee-334301e787b5","added_by":"auto","created_at":"2024-08-21 21:44:41","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":586654,"visible":true,"origin":"","legend":"","description":"","filename":"DecentralisedCollaborativeFilteringforDHTs.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4316139/v1_covered_ff9e3ae2-8300-4866-ba8f-27ec70d1f9cb.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Decentralized Collaborative Filtering for Distributed Hash Tables","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Collaborative Filtering, Matrix Factorization, Peer-toPeer, Pastry","lastPublishedDoi":"10.21203/rs.3.rs-4316139/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4316139/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"We implement a decentralized method for collaborative filtering via matrix factorization that can rest on peer-to-peer Distributed Hash Table architectures like Pastry and Chord, for the purpose of realizing a scalable and robust decentralised search engine. We begin by introducing a centralized method for matrix factorization that involves minimizing a cost function consisting of the ratings and user and item latent factors. We introduce biases and then derive the gradients for the error update step. We then introduce gossip learning; a matrix-factorization based collaborative filtering approach for a decentralized network. Since we could not obtain a dataset of browsing activity of users where the rating would constitute the number of times a website is visited, we used a dataset of movie ratings to test our method. This preserves the user and item biases and provides user-item ratings for rating to latent factor error computation. We performed a train-test split and loaded the training and validation set onto each node separately. This method is then tested using the PeerSim peer-to-peer network simulator to obtain the best learning rate $\\eta$, regularization parameter $\\lambda$ and latent factors $k$ for a $10,000$ node peer-to-peer network. The optimization metric used is the Root Mean Squared Error measuring the discrepancy between the ratings in the validation set and their respective predictions by model updates on the training set at each node. We found that by the fortieth cycle of the PeerSim simulator, the training set is fully distributed across each node. The best performing parameter setting over our $63$ experiment runs has learning rate $\\eta = 0.001$, regularization parameter $\\lambda = 1$ and latent factors $k = 20$. These settings are then built into a basic working prototype of a Pastry algorithm based peer-to-peer distributed search engine with message passing tested for $50$ nodes.","manuscriptTitle":"Decentralized Collaborative Filtering for Distributed Hash Tables","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-05-07 16:26:27","doi":"10.21203/rs.3.rs-4316139/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"f95564a4-f940-4da0-bd8a-7d176a9e6290","owner":[],"postedDate":"May 7th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-08-21T21:36:34+00:00","versionOfRecord":[],"versionCreatedAt":"2024-05-07 16:26:27","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4316139","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4316139","identity":"rs-4316139","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}