Large-Scale Stream K-means Based on Product-Quantized Codes

Large-Scale Stream K-means Based on Product-Quantized Codes | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Large-Scale Stream K-means Based on Product-Quantized Codes Yuqing Hang, Hongwei Yin, Wenjun Hu, Longfei Zhong This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4412715/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 24 Jan, 2025 Read the published version in International Journal of Machine Learning and Cybernetics → Version 1 posted 12 You are reading this latest preprint version Abstract In recent years, using clustering technology to process large-scale data streams is a research hotspot in the field of data mining. However, for the processing of large-scale data streams, most existing methods suffer from slow speed, insufficient memory, and lack of detection and response mechanisms for concept drift. In this paper, a Large-Scale Stream K-measn based on product-quantized codes (LS 2 K-means) is proposed. By first introducing product quantization code into the framework of incremental clustering methods, memory space consumption is reduced through the dimensionality reduction of data. Additionally, a new similarity measurement method is introduced, greatly improving the efficiency of distance calculation. A concept drift detection and response mechanism is constructed. By comparing the consistency of clustering results, concept drift can be quickly detected, and a backtracking mechanism is utilized to respond to concept drift promptly, effectively improving the algorithm’s performance. The effectiveness of the proposed method is validated through simulations on six real datasets. The method efficiently handles concept drift and outperforms DenStream and EmCStream in terms of execution efficiency. clustering data streams product quantization drift detection Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 24 Jan, 2025 Read the published version in International Journal of Machine Learning and Cybernetics → Version 1 posted Editorial decision: Revision requested 26 Aug, 2024 Reviews received at journal 20 Jun, 2024 Reviews received at journal 15 Jun, 2024 Reviews received at journal 24 May, 2024 Reviewers agreed at journal 22 May, 2024 Reviewers agreed at journal 22 May, 2024 Reviewers agreed at journal 21 May, 2024 Reviewers agreed at journal 21 May, 2024 Reviewers invited by journal 21 May, 2024 Editor assigned by journal 18 May, 2024 Submission checks completed at journal 14 May, 2024 First submitted to journal 13 May, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4412715","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":305565434,"identity":"e082d201-6a30-489b-9a41-8663ca1fab02","order_by":0,"name":"Yuqing Hang","email":"","orcid":"","institution":"Huzhou University","correspondingAuthor":false,"prefix":"","firstName":"Yuqing","middleName":"","lastName":"Hang","suffix":""},{"id":305565435,"identity":"b00c2edc-45ff-4a0a-b97e-b8b3a918a5d3","order_by":1,"name":"Hongwei Yin","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA00lEQVRIie3PvQrCMBDA8ZNAukS6SYJDXyHFQQTxWQ76BC7iIKUiZHVV/HiMzikFXYqz4KJ7hTq72ArOiZtg/ksI3I/kAFyuH4x6yyqrpkBkc2M2xGc6zHnxDREr7Om2ArAnUiNqsY+9PsfW9a4g8DvaROrGKSWDFZJwpyBcb9BAsnmiRcqIPCPt1j9EeTGRnIBub3lDvKcVEYrWJJHvV4gV8RmDnB+QyOK2ENsTN+9Cg5I8qlkcyWOUVeVkGPhdA/kUAbSS+uR2400j+1GXy+X6u156qkGQpVGArwAAAABJRU5ErkJggg==","orcid":"","institution":"Huzhou University","correspondingAuthor":true,"prefix":"","firstName":"Hongwei","middleName":"","lastName":"Yin","suffix":""},{"id":305565436,"identity":"07ec57b2-a1bc-4b22-bf96-e2c28332b7d3","order_by":2,"name":"Wenjun Hu","email":"","orcid":"","institution":"Huzhou University","correspondingAuthor":false,"prefix":"","firstName":"Wenjun","middleName":"","lastName":"Hu","suffix":""},{"id":305565437,"identity":"8e9b159e-56e0-4c45-ad03-710cf219290d","order_by":3,"name":"Longfei Zhong","email":"","orcid":"","institution":"Huzhou University","correspondingAuthor":false,"prefix":"","firstName":"Longfei","middleName":"","lastName":"Zhong","suffix":""}],"badges":[],"createdAt":"2024-05-13 10:47:27","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4412715/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4412715/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s13042-025-02531-1","type":"published","date":"2025-01-24T15:57:26+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":74858355,"identity":"08aad484-3df6-42dd-8d1e-885330f15e83","added_by":"auto","created_at":"2025-01-27 16:08:28","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":301907,"visible":true,"origin":"","legend":"","description":"","filename":"ls2ckmeans.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4412715/v1_covered_a05eae41-6903-4953-acb0-8efb0d6ddf12.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Large-Scale Stream K-means Based on Product-Quantized Codes","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"international-journal-of-machine-learning-and-cybernetics","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jmlc","sideBox":"Learn more about [International Journal of Machine Learning and Cybernetics](http://actavetscand.biomedcentral.com/)","snPcode":"13042","submissionUrl":"https://submission.nature.com/new-submission/13042/3","title":"International Journal of Machine Learning and Cybernetics","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"clustering, data streams, product quantization, drift detection","lastPublishedDoi":"10.21203/rs.3.rs-4412715/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4412715/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eIn recent years, using clustering technology to process large-scale data streams is a research hotspot in the field of data mining. However, for the processing of large-scale data streams, most existing methods suffer from slow speed, insufficient memory, and lack of detection and response mechanisms for concept drift. In this paper, a Large-Scale Stream K-measn based on product-quantized codes (LS\u003csup\u003e2\u003c/sup\u003eK-means) is proposed. By first introducing product quantization code into the framework of incremental clustering methods, memory space consumption is reduced through the dimensionality reduction of data. Additionally, a new similarity measurement method is introduced, greatly improving the efficiency of distance calculation. A concept drift detection and response mechanism is constructed. By comparing the consistency of clustering results, concept drift can be quickly detected, and a backtracking mechanism is utilized to respond to concept drift promptly, effectively improving the algorithm’s performance. The effectiveness of the proposed method is validated through simulations on six real datasets. The method efficiently handles concept drift and outperforms DenStream and EmCStream in terms of execution efficiency.\u003c/p\u003e","manuscriptTitle":"Large-Scale Stream K-means Based on Product-Quantized Codes","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-05-23 18:57:48","doi":"10.21203/rs.3.rs-4412715/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-08-27T02:10:50+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-06-20T08:47:28+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-06-15T07:57:24+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-05-25T02:57:34+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"324939896075751541648718076694740779077","date":"2024-05-22T13:57:33+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"25350233171570228317703202211433599746","date":"2024-05-22T07:46:14+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"222187593384680451662654691012211356231","date":"2024-05-22T03:35:17+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"35842778886280320989406749034979621379","date":"2024-05-22T02:52:00+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-05-22T02:50:05+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-05-18T12:29:02+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-05-14T11:49:31+00:00","index":"","fulltext":""},{"type":"submitted","content":"International Journal of Machine Learning and Cybernetics","date":"2024-05-13T10:46:10+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"international-journal-of-machine-learning-and-cybernetics","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jmlc","sideBox":"Learn more about [International Journal of Machine Learning and Cybernetics](http://actavetscand.biomedcentral.com/)","snPcode":"13042","submissionUrl":"https://submission.nature.com/new-submission/13042/3","title":"International Journal of Machine Learning and Cybernetics","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"f49ab273-433a-4b1c-8bdd-6b1f1c3daa04","owner":[],"postedDate":"May 23rd, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2025-01-27T16:01:02+00:00","versionOfRecord":{"articleIdentity":"rs-4412715","link":"https://doi.org/10.1007/s13042-025-02531-1","journal":{"identity":"international-journal-of-machine-learning-and-cybernetics","isVorOnly":false,"title":"International Journal of Machine Learning and Cybernetics"},"publishedOn":"2025-01-24 15:57:26","publishedOnDateReadable":"January 24th, 2025"},"versionCreatedAt":"2024-05-23 18:57:48","video":"","vorDoi":"10.1007/s13042-025-02531-1","vorDoiUrl":"https://doi.org/10.1007/s13042-025-02531-1","workflowStages":[]},"version":"v1","identity":"rs-4412715","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4412715","identity":"rs-4412715","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}