Directional Adaptive Metric Sampling Minimal Expected Loss: A Continuous Optimisation Method

Directional Adaptive Metric Sampling Minimal Expected Loss: A Continuous Optimisation Method | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Directional Adaptive Metric Sampling Minimal Expected Loss: A Continuous Optimisation Method Rizal Purnawan, Dieky Adzkiya This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5402563/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract This paper presents an optimisation method called Directional Adaptive Metric Sampling Minimal Expected Loss (DAMSMEL) which uses a technique of minimising the expected value of the objective function across a sample of neighbourhood points in an iterative manner generated uniformly in a bidirectional configuration with respect to the vector basis with exponentially decaying distance of adjacent points as well as a reset mechanism, and this method does not rely on gradient information. DAMSMEL is intended to be a robust optimisation method which can escape local minima in non-convex landscapes, overcoming an inherent limitation associated with gradient-based methods. This paper presents the theoretical foundation of DAMSMEL within probability theory and functional analysis including a rigorous proof that DAMSMEL is convergent to a global minimum in convex optimisation landscapes. We have conducted tests on convex and non-convex optimisation problems, on a machine learning regression problem, as well as a non-linear problem, comparing DAMSMEL with GD, Stochastic Gradient Descent (SGD), Adaptive Moment Estimation (Adam) and Ordinary Least Squares (OLS). Results show that while both DAMSMEL and GD converge to the global minimum in the convex case, GD gets trapped in local minima for the non-convex problem, whereas DAMSMEL successfully converges to the global minimum, demonstrating DAMSMEL's robustness in non-convex optimisation. In the machine learning regression task, DAMSMEL, SGD, and OLS yield comparable accuracy. And in the non-linear problem, DAMSMEL outperforms GD, SGD and Adam as it provides the most accurate solution compared to GD, SGD and Adam with huge margins. However, DAMSMEL tends to require longer runtimes due to its computational complexity, making it currently more suitable for low to medium-scale optimisation problems. MSC Classification: 65K10 Computational Mathematics Applied Mathematics DAMSMEL Optimization Algorithm Machine Learning Applied Mathematics Computational Mathematics Full Text Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5402563","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":374830379,"identity":"3f5cdd0d-4b05-4391-bf72-ada6bbad4fa9","order_by":0,"name":"Rizal Purnawan","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABCklEQVRIiWNgGAWjYDACZiB+AOdV1MiBqAMPsCtGaEmA884cMwZrScCuGAHgChjbmBMbUESwAPN29ocfEncczjdnP2P4uYCNLX1+2OGHQFvs5HQbsGuROcyQLJF45rDlzp4cY+kZPDK5G2+nGQC1JBubHcCuRYKZ4YBEYtthA4MDuRukeSTYcjfOTgBpOZC4DacWxuYfYC3n327+zWPAnG44O/0DAS3MbBBbbuRuk+ZJYE6Ql84hZAsbm0XimXSglvffrHkOHDPcIJ1TcCDBAI9f+I8/vvFxhzXQYWnJt3n/1cjLz07f/OFDhZ0cLi1gwNiAxDEAqzTAoxxDi3wDDlWjYBSMglEwYgEAcRhgcp3ysHcAAAAASUVORK5CYII=","orcid":"https://orcid.org/0000-0001-8858-4036","institution":"Independent Researcher","correspondingAuthor":true,"prefix":"","firstName":"Rizal","middleName":"","lastName":"Purnawan","suffix":""},{"id":374830380,"identity":"323d85d2-653a-449f-96ab-319b94a7826b","order_by":1,"name":"Dieky Adzkiya","email":"","orcid":"https://orcid.org/0000-0002-4718-2871","institution":"Institut Teknologi Sepuluh Nopember","correspondingAuthor":false,"prefix":"","firstName":"Dieky","middleName":"","lastName":"Adzkiya","suffix":""}],"badges":[],"createdAt":"2024-11-06 12:01:26","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-5402563/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5402563/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":68411663,"identity":"189e24fa-44d0-4c90-b30c-e401ce580f88","added_by":"auto","created_at":"2024-11-07 04:08:13","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":422262,"visible":true,"origin":"","legend":"","description":"","filename":"DAMSMEL.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5402563/v1_covered_689bf520-3f26-40b7-999e-9108532136d3.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003eDirectional Adaptive Metric Sampling Minimal Expected Loss: A Continuous Optimisation Method\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"DAMSMEL, Optimization Algorithm, Machine Learning, Applied Mathematics, Computational Mathematics","lastPublishedDoi":"10.21203/rs.3.rs-5402563/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5402563/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThis paper presents an optimisation method called Directional Adaptive Metric Sampling Minimal Expected Loss (DAMSMEL) which uses a technique of minimising the expected value of the objective function across a sample of neighbourhood points in an iterative manner generated uniformly in a bidirectional configuration with respect to the vector basis with exponentially decaying distance of adjacent points as well as a reset mechanism, and this method does not rely on gradient information. DAMSMEL is intended to be a robust optimisation method which can escape local minima in non-convex landscapes, overcoming an inherent limitation associated with gradient-based methods. This paper presents the theoretical foundation of DAMSMEL within probability theory and functional analysis including a rigorous proof that DAMSMEL is convergent to a global minimum in convex optimisation landscapes. We have conducted tests on convex and non-convex optimisation problems, on a machine learning regression problem, as well as a non-linear problem, comparing DAMSMEL with GD, Stochastic Gradient Descent (SGD), Adaptive Moment Estimation (Adam) and Ordinary Least Squares (OLS). Results show that while both DAMSMEL and GD converge to the global minimum in the convex case, GD gets trapped in local minima for the non-convex problem, whereas DAMSMEL successfully converges to the global minimum, demonstrating DAMSMEL's robustness in non-convex optimisation. In the machine learning regression task, DAMSMEL, SGD, and OLS yield comparable accuracy. And in the non-linear problem, DAMSMEL outperforms GD, SGD and Adam as it provides the most accurate solution compared to GD, SGD and Adam with huge margins. However, DAMSMEL tends to require longer runtimes due to its computational complexity, making it currently more suitable for low to medium-scale optimisation problems.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMSC Classification: \u003c/strong\u003e65K10\u003c/p\u003e","manuscriptTitle":"Directional Adaptive Metric Sampling Minimal Expected Loss: A Continuous Optimisation Method","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-11-07 04:00:03","doi":"10.21203/rs.3.rs-5402563/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"fb673fbd-cf31-4924-bd1b-e11a58053287","owner":[],"postedDate":"November 7th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":39899018,"name":"Computational Mathematics"},{"id":39899019,"name":"Applied Mathematics"}],"tags":[],"updatedAt":"2024-11-07T04:00:03+00:00","versionOfRecord":[],"versionCreatedAt":"2024-11-07 04:00:03","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5402563","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5402563","identity":"rs-5402563","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}