Opt-AEDDM: Towards Optimizing Autoencoders for effective Concept Drift Detection

Opt-AEDDM: Towards Optimizing Autoencoders for effective Concept Drift Detection | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Opt-AEDDM: Towards Optimizing Autoencoders for effective Concept Drift Detection Usman Ali, Tariq Mahmood This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6578197/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 31 Jan, 2026 Read the published version in Knowledge and Information Systems → Version 1 posted 13 You are reading this latest preprint version Abstract The occurrence of concept drift is an important phenomenon in an operational scenario related to machine learning based system which affects the performance of pre-trained models. To maintain the integrity and confidence in predictions, an effective drift detection and adaptation mechanism is a vital component in these operational systems. Apart from different supervised, semi-supervised and unsupervised drift detection techniques, recently the research trend has been focused more on deep learning-based methods specifically based on autoencoders. A hallmark of autoencoders is their versatility. Different types and implementations exist, each specializing in handling specific tasks. In case of drift detection, the focus is to learn the data distribution using autoencoder and to measure the deviation of the newly arriving data in terms of reconstruction loss. While standard or vanilla autoencoders are the most used; other variations also exist and can be evaluated for better drift detection performance. Apart from the type of the autoencoder, another important consideration is the use of right set of parameters and hyperparameters for an autoencoder based drift detection mechanism. In this paper, we provide a framework to optimize the performance of autoencoder based drift detection methods. We provide a theoretical and an empirical evaluation of other applicable types of autoencoders including denoising, variational and standard, followed by a detailed mechanism for selecting the best hyperparameters using grid search and selection of drift detection method’s specific parameters using both grid search and Bayesian Optimization (BO). For experimentation, we have used AEDDM (Autoencoder-based Drift Detection Method) as the base drift detection method to produce Opt-AEDDM- the optimized version of AEDDM. Detailed experiments on 4 synthetic (Hyperplane, Gaussian, VD and RBF) and two real world datasets (NOAA and Forest Covertype) prove the applicability and effectiveness of the proposed framework in finding the best autoencoder with best set of hyperparameters and parameters with improved drift detection performance. Concept drift Machine learning Autoencoders Variational Autoencoders Hyperparameters Optimization Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 31 Jan, 2026 Read the published version in Knowledge and Information Systems → Version 1 posted Editorial decision: Revision requested 22 Aug, 2025 Reviews received at journal 09 Jul, 2025 Reviews received at journal 09 Jul, 2025 Reviews received at journal 02 Jul, 2025 Reviewers agreed at journal 11 Jun, 2025 Reviewers agreed at journal 10 Jun, 2025 Reviewers agreed at journal 10 Jun, 2025 Reviewers agreed at journal 09 Jun, 2025 Reviewers agreed at journal 08 Jun, 2025 Reviewers invited by journal 08 Jun, 2025 Editor assigned by journal 08 May, 2025 Submission checks completed at journal 03 May, 2025 First submitted to journal 02 May, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6578197","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":504350604,"identity":"ceaa03b5-10a6-4a3d-acd5-7f29d277306b","order_by":0,"name":"Usman Ali","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAu0lEQVRIiWNgGAWjYFAC5oYDHwwOgFgGQJxAjBbGhoMzSNbCzMNAihaD4wcbD9sU3JFnYG/eJsG4I40ILWcSGw7nGDwzbOA5VibBeCaHCC0HwFoOMzZI5JhJMLZVEKHl/MOGwxYGh+0b5N8Qq+UG0BYGg8OJDRI8IC1EOEzyxsOGgz0Gz5LbeNKKLRLPEOF9vvPJhz/8+HPHtp/98MYbH3ckE9YCB2wgIrGBBB0QwEi6llEwCkbBKBgBAABS0j+n9cWl/gAAAABJRU5ErkJggg==","orcid":"","institution":"Institute of Business Administration (IBA)","correspondingAuthor":true,"prefix":"","firstName":"Usman","middleName":"","lastName":"Ali","suffix":""},{"id":504350605,"identity":"d2948050-da42-4075-b205-71b32d757bf4","order_by":1,"name":"Tariq Mahmood","email":"","orcid":"","institution":"Institute of Business Administration (IBA)","correspondingAuthor":false,"prefix":"","firstName":"Tariq","middleName":"","lastName":"Mahmood","suffix":""}],"badges":[],"createdAt":"2025-05-02 11:53:26","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6578197/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6578197/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s10115-025-02664-3","type":"published","date":"2026-01-31T15:58:23+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":101690650,"identity":"f57bf9b8-336e-4993-9a9c-7daf311d42a2","added_by":"auto","created_at":"2026-02-02 16:06:45","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1336479,"visible":true,"origin":"","legend":"","description":"","filename":"OptAEDDMTowardsOptimizingAutoencodersforeffectiveConceptDriftDetection.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6578197/v1_covered_e4299f34-28a8-483c-8a77-e8b5edb287f1.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Opt-AEDDM: Towards Optimizing Autoencoders for effective Concept Drift Detection","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"knowledge-and-information-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"kais","sideBox":"Learn more about [Knowledge and Information Systems](http://link.springer.com/journal/10115)","snPcode":"10115","submissionUrl":"https://submission.nature.com/new-submission/10115/3","title":"Knowledge and Information Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Concept drift, Machine learning, Autoencoders, Variational Autoencoders, Hyperparameters, Optimization","lastPublishedDoi":"10.21203/rs.3.rs-6578197/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6578197/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe occurrence of concept drift is an important phenomenon in an operational scenario related to machine learning based system which affects the performance of pre-trained models. To maintain the integrity and confidence in predictions, an effective drift detection and adaptation mechanism is a vital component in these operational systems. Apart from different supervised, semi-supervised and unsupervised drift detection techniques, recently the research trend has been focused more on deep learning-based methods specifically based on autoencoders. A hallmark of autoencoders is their versatility. Different types and implementations exist, each specializing in handling specific tasks. In case of drift detection, the focus is to learn the data distribution using autoencoder and to measure the deviation of the newly arriving data in terms of reconstruction loss. While standard or vanilla autoencoders are the most used; other variations also exist and can be evaluated for better drift detection performance. Apart from the type of the autoencoder, another important consideration is the use of right set of parameters and hyperparameters for an autoencoder based drift detection mechanism. In this paper, we provide a framework to optimize the performance of autoencoder based drift detection methods. We provide a theoretical and an empirical evaluation of other applicable types of autoencoders including denoising, variational and standard, followed by a detailed mechanism for selecting the best hyperparameters using grid search and selection of drift detection method\u0026rsquo;s specific parameters using both grid search and Bayesian Optimization (BO). For experimentation, we have used AEDDM (Autoencoder-based Drift Detection Method) as the base drift detection method to produce Opt-AEDDM- the optimized version of AEDDM. Detailed experiments on 4 synthetic (Hyperplane, Gaussian, VD and RBF) and two real world datasets (NOAA and Forest Covertype) prove the applicability and effectiveness of the proposed framework in finding the best autoencoder with best set of hyperparameters and parameters with improved drift detection performance.\u003c/p\u003e","manuscriptTitle":"Opt-AEDDM: Towards Optimizing Autoencoders for effective Concept Drift Detection","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-09-19 06:05:39","doi":"10.21203/rs.3.rs-6578197/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-08-22T18:00:43+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-07-09T09:19:23+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-07-09T05:33:52+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-07-03T02:10:43+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"332392979776730475695263533507728960013","date":"2025-06-11T05:41:09+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"21439991465392493887355912623661267276","date":"2025-06-10T15:03:14+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"321472751799402231044714431112991395869","date":"2025-06-10T14:01:17+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"38539008420066262242283657404294876323","date":"2025-06-10T01:28:56+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"53709263627761418068697452700838891583","date":"2025-06-09T03:20:22+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-06-09T03:03:29+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-05-08T07:39:03+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-05-03T10:03:04+00:00","index":"","fulltext":""},{"type":"submitted","content":"Knowledge and Information Systems","date":"2025-05-02T11:49:52+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"knowledge-and-information-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"kais","sideBox":"Learn more about [Knowledge and Information Systems](http://link.springer.com/journal/10115)","snPcode":"10115","submissionUrl":"https://submission.nature.com/new-submission/10115/3","title":"Knowledge and Information Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"6a78583c-b87b-4f0b-bb6b-ab27a6532938","owner":[],"postedDate":"September 19th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2026-02-02T16:02:51+00:00","versionOfRecord":{"articleIdentity":"rs-6578197","link":"https://doi.org/10.1007/s10115-025-02664-3","journal":{"identity":"knowledge-and-information-systems","isVorOnly":false,"title":"Knowledge and Information Systems"},"publishedOn":"2026-01-31 15:58:23","publishedOnDateReadable":"January 31st, 2026"},"versionCreatedAt":"2025-09-19 06:05:39","video":"","vorDoi":"10.1007/s10115-025-02664-3","vorDoiUrl":"https://doi.org/10.1007/s10115-025-02664-3","workflowStages":[]},"version":"v1","identity":"rs-6578197","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6578197","identity":"rs-6578197","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}