AURA: An Adaptive Unified Regularization Approach for Gradient-Based Optimization

AURA: An Adaptive Unified Regularization Approach for Gradient-Based Optimization | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article AURA: An Adaptive Unified Regularization Approach for Gradient-Based Optimization Keshav Gupta This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7480833/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract In the engineering world, optimization plays an important role i.e. almost every industry tries to optimize their systems as much as possible in order to increase efficiency. Hence, in the machine learning universe as well, optimization is quite famous and when it comes to optimisation, one algorithm which often comes to our mind is gradient descent. Currently, there are many customized optimiser techniques which describe different kinds of techniques and processes to converge faster and obtain a low error rate. We propose AURA (Adaptive Unified Regularized Algorithm), a novel stochastic optimizer that shifts adaptation from the learning rate to the momentum parameter. Unlike conventional adaptive methods such as Adam and RMSProp, which primarily rely on per-parameter learning rate scaling, AURA maintains a fixed learning rate and instead adaptively modulates momentum ($\beta$) through three synergistic signals: (i) loss-trend awareness, which captures short-term dynamics in optimization stability, (ii) gradient-norm sensitivity, which prevents instability under varying gradient magnitudes, and (iii) cosine-similarity modulation, which aligns current updates with historical trajectories to enhance directional consistency. Empirical evaluations on classification and regression benchmarks demonstrate that AURA achieves competitive or superior convergence behavior compared to widely used optimizers. Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7480833","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":509181108,"identity":"f5a57b64-d035-4ab5-ac93-8607be14d8c2","order_by":0,"name":"Keshav Gupta","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA+klEQVRIiWNgGAWjYNCCAwwMBgwMjA/gAjz41TM2QLUwGzAkkKiFTYIoLfIRuccf/DhjY2/OfsasmveHTWJ//wHGB2/bcGsxvJGX2NhzIy1xZ0+O2W2ehDRjiRsJzIZz8WmZkWPYwPPhcILBgbQ0oJbDcgw3GNikeQloafzz4b+9wflnacU8Cf955M8fYP+NT4u8RI5hM8+NA4wbbiQfY+ZJOCBncCCBjRmfFgOed4mzZc4kJ2648fiw5Jy0ZGPDG4nNknPO4bGlPffAxzfH7IAOS2z88MbGLnHe+cMHP7wpw2PLAcwoAEYUPiDfQCCiR8EoGAWjYBQwAABkM1kv2QavtwAAAABJRU5ErkJggg==","orcid":"","institution":"The University of Texas at Austin","correspondingAuthor":true,"prefix":"","firstName":"Keshav","middleName":"","lastName":"Gupta","suffix":""}],"badges":[],"createdAt":"2025-08-28 13:38:31","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7480833/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7480833/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":90851203,"identity":"188e884e-e876-4cc9-9381-1fc8efe9a97d","added_by":"auto","created_at":"2025-09-09 03:15:56","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":479754,"visible":true,"origin":"","legend":"","description":"","filename":"AURAOptimiser.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7480833/v1_covered_b0cf676f-ba15-4662-8636-7aaa0113f420.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"AURA: An Adaptive Unified Regularization Approach for Gradient-Based Optimization","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-7480833/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7480833/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"In the engineering world, optimization plays an important role i.e. almost every industry tries to optimize their systems as much as possible in order to increase efficiency. Hence, in the machine learning universe as well, optimization is quite famous and when it comes to optimisation, one algorithm which often comes to our mind is gradient descent. Currently, there are many customized optimiser techniques which describe different kinds of techniques and processes to converge faster and obtain a low error rate. We propose AURA (Adaptive Unified Regularized Algorithm), a novel stochastic optimizer that shifts adaptation from the learning rate to the momentum parameter. Unlike conventional adaptive methods such as Adam and RMSProp, which primarily rely on per-parameter learning rate scaling, AURA maintains a fixed learning rate and instead adaptively modulates momentum ($\\beta$) through three synergistic signals: (i) loss-trend awareness, which captures short-term dynamics in optimization stability, (ii) gradient-norm sensitivity, which prevents instability under varying gradient magnitudes, and (iii) cosine-similarity modulation, which aligns current updates with historical trajectories to enhance directional consistency. Empirical evaluations on classification and regression benchmarks demonstrate that AURA achieves competitive or superior convergence behavior compared to widely used optimizers.","manuscriptTitle":"AURA: An Adaptive Unified Regularization Approach for Gradient-Based Optimization","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-09-09 02:51:51","doi":"10.21203/rs.3.rs-7480833/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"2bfe5f7b-7734-4a29-824f-809111b253dc","owner":[],"postedDate":"September 9th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-09-09T02:51:51+00:00","versionOfRecord":[],"versionCreatedAt":"2025-09-09 02:51:51","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7480833","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7480833","identity":"rs-7480833","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}