Distributed log anomaly detection method based on improved HDBSCAN

Distributed log anomaly detection method based on improved HDBSCAN | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Distributed log anomaly detection method based on improved HDBSCAN Xunyun Wu, Jiabin Wang, Xilong Lin, Jiexuan Zhuang, Hao Cheng This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9400892/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 9 You are reading this latest preprint version Abstract As one of the most valuable information sources in distributed systems, log data serves as the foundation for fault diagnosis and security auditing tasks. However, the explosive growth of unstructured and dynamically changing log data poses significant challenges for anomaly detection. Automated and efficient detection can not only reduce operational and maintenance costs, but also significantly improve system reliability. This paper proposes an end-to-end log anomaly detection method based on an improved HDBSCAN algorithm and distributed collaboration. To address the template inconsistency problem in distributed environments, we design a distributed Drain parsing algorithm. This algorithm incorporates dynamic parse tree expansion and global incremental synchronization mechanisms. Meanwhile, we introduce an optimized HDBSCAN algorithm that incorporates the RTree spatial index and Boruvka minimum spanning tree (MST). This optimized algorithm breaks through computational bottlenecks and enhances the ability to capture non-spherical clusters. In addition, we fuse cross-partition clustering and determine anomaly levels by combining the Gaussian kernel function and GLOSH scoring mechanism. We validate the model, which is built on the Spark framework, on three well-known datasets: HDFS, BGL, and OpenStack. Experimental results show that the proposed model outperforms most existing mainstream approaches in terms of parsing efficiency and detection performance. Log anomaly detection HDBSCAN distributed computing R-Tree spatial index Drain algorithm Spark Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 04 May, 2026 Reviews received at journal 27 Apr, 2026 Reviewers agreed at journal 27 Apr, 2026 Reviews received at journal 27 Apr, 2026 Reviewers agreed at journal 27 Apr, 2026 Reviewers invited by journal 27 Apr, 2026 Editor assigned by journal 18 Apr, 2026 Submission checks completed at journal 18 Apr, 2026 First submitted to journal 13 Apr, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9400892","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":634104808,"identity":"4375d3f5-182d-4c9e-8897-73e2608aec43","order_by":0,"name":"Xunyun Wu","email":"","orcid":"","institution":"Huaqiao University","correspondingAuthor":false,"prefix":"","firstName":"Xunyun","middleName":"","lastName":"Wu","suffix":""},{"id":634104809,"identity":"9f3d1a33-a772-4fc8-990d-7cb70f4a7657","order_by":1,"name":"Jiabin Wang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAzklEQVRIiWNgGAWjYBAC9gYILWcAptiI0MJzgBlMG5OuJXED8VrYzx+T+LmjNn07e44Bw4eywwz8sxsIaOFJZpPsPXM8d2fPGwPGGecOM0jcOYBfiz1DMtsN3rZjuRtu5Bgw87YdZjCQSCBgC/9jtpt/246lG4C0/CVKi0Qy223etpoEsBZG4rQ8Nv8t23bAcMOZZwUHe86l80jcIOiwxMeGb9vq5A2OJ2988KPMWo5/BgEtUHAYiBMYDoDMIEo9ENSBtYyCUTAKRsEowAoA4GJEAg1j+44AAAAASUVORK5CYII=","orcid":"","institution":"Huaqiao University","correspondingAuthor":true,"prefix":"","firstName":"Jiabin","middleName":"","lastName":"Wang","suffix":""},{"id":634104810,"identity":"5203901e-5823-48fe-bd93-05a4e031fa4b","order_by":2,"name":"Xilong Lin","email":"","orcid":"","institution":"Huaqiao University","correspondingAuthor":false,"prefix":"","firstName":"Xilong","middleName":"","lastName":"Lin","suffix":""},{"id":634104811,"identity":"f5765fe6-5ddd-4d13-8921-97288f46fe7b","order_by":3,"name":"Jiexuan Zhuang","email":"","orcid":"","institution":"Huaqiao University","correspondingAuthor":false,"prefix":"","firstName":"Jiexuan","middleName":"","lastName":"Zhuang","suffix":""},{"id":634104814,"identity":"2afa3f5f-1d5d-4a4d-b9df-d25374a74432","order_by":4,"name":"Hao Cheng","email":"","orcid":"","institution":"Huaqiao University","correspondingAuthor":false,"prefix":"","firstName":"Hao","middleName":"","lastName":"Cheng","suffix":""}],"badges":[],"createdAt":"2026-04-13 08:25:24","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9400892/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9400892/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":108476499,"identity":"3c482dc6-4d58-4da6-b8eb-4b8ba59eb840","added_by":"auto","created_at":"2026-05-05 06:58:56","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1628783,"visible":true,"origin":"","legend":"","description":"","filename":"ArticleTitle1.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9400892/v1_covered_efd3a704-a48b-4e09-b780-dda10bdc838d.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Distributed log anomaly detection method based on improved HDBSCAN","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"discover-computing","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [Discover Computing](https://link.springer.com/journal/10791)","snPcode":"10791","submissionUrl":"https://submission.springernature.com/new-submission/10791/3","title":"Discover Computing","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Discover Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Log anomaly detection, HDBSCAN, distributed computing, R-Tree spatial index, Drain algorithm, Spark","lastPublishedDoi":"10.21203/rs.3.rs-9400892/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9400892/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eAs one of the most valuable information sources in distributed systems, log data serves as the foundation for fault diagnosis and security auditing tasks. However, the explosive growth of unstructured and dynamically changing log data poses significant challenges for anomaly detection. Automated and efficient detection can not only reduce operational and maintenance costs, but also significantly improve system reliability. This paper proposes an end-to-end log anomaly detection method based on an improved HDBSCAN algorithm and distributed collaboration. To address the template inconsistency problem in distributed environments, we design a distributed Drain parsing algorithm. This algorithm incorporates dynamic parse tree expansion and global incremental synchronization mechanisms. Meanwhile, we introduce an optimized HDBSCAN algorithm that incorporates the RTree spatial index and Boruvka minimum spanning tree (MST). This optimized algorithm breaks through computational bottlenecks and enhances the ability to capture non-spherical clusters. In addition, we fuse cross-partition clustering and determine anomaly levels by combining the Gaussian kernel function and GLOSH scoring mechanism. We validate the model, which is built on the Spark framework, on three well-known datasets: HDFS, BGL, and OpenStack. Experimental results show that the proposed model outperforms most existing mainstream approaches in terms of parsing efficiency and detection performance.\u003c/p\u003e","manuscriptTitle":"Distributed log anomaly detection method based on improved HDBSCAN","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-05-05 06:58:38","doi":"10.21203/rs.3.rs-9400892/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-05-04T11:33:33+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-27T19:16:27+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"7376132900086992427319346054343288020","date":"2026-04-27T18:49:23+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-27T15:05:52+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"302327676793597214351486425830690373106","date":"2026-04-27T14:33:30+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-04-27T08:34:37+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-04-18T05:50:58+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-04-18T05:50:37+00:00","index":"","fulltext":""},{"type":"submitted","content":"Discover Computing","date":"2026-04-13T08:16:46+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"discover-computing","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [Discover Computing](https://link.springer.com/journal/10791)","snPcode":"10791","submissionUrl":"https://submission.springernature.com/new-submission/10791/3","title":"Discover Computing","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Discover Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"0eb19a30-cb45-4c02-8262-04a0b01de1f4","owner":[],"postedDate":"May 5th, 2026","published":true,"recentEditorialEvents":[{"type":"decision","content":"Revision requested","date":"2026-05-04T11:33:33+00:00","index":"","fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-05-14T09:03:10+00:00","versionOfRecord":[],"versionCreatedAt":"2026-05-05 06:58:38","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9400892","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9400892","identity":"rs-9400892","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}