A Unified Framework for Non-Convex Optimization in Deep Learning via Adaptive Variance Reduction

A Unified Framework for Non-Convex Optimization in Deep Learning via Adaptive Variance Reduction | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A Unified Framework for Non-Convex Optimization in Deep Learning via Adaptive Variance Reduction Yunyang Zhang, Xianyu Chen, Zizheng Zhang, Shen Zhou This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7250325/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract The increasing complexity of deep learning models necessitates sophisticated optimization techniques to effectively navigate their non-convex loss landscapes. However, traditional optimization methods, such as stochastic gradient descent (SGD), often encounter challenges related to high variance in gradient estimates, leading to suboptimal convergence and performance. This paper proposes a unified framework for adaptive variance reduction in stochastic non-convex optimization, addressing these critical issues. We introduce the Adaptive Variance Reduced Gradient (AVRG) algorithm, which dynamically balances variance reduction with computational efficiency, yielding improved convergence rates and robustness. Our framework synthesizes existing adaptive variance reduction methods, providing a cohesive theoretical understanding while filling existing gaps in the literature. Comprehensive empirical evaluations across diverse deep learning benchmarks demonstrate the efficacy of AVRG compared to established methods, highlighting its faster convergence and superior performance. By equipping researchers and practitioners with enhanced optimization strategies, this work contributes significantly to the field of deep learning, paving the way for future research in more complex optimization scenarios. We explore the theoretical guarantees of our approach, aiming to foster advancements in training methodologies for deep neural networks, thereby enabling better performance across a wide array of applications. Full Text Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7250325","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":493030182,"identity":"781b8f2a-9ba8-4853-8f30-8281956c65ca","order_by":0,"name":"Yunyang Zhang","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Yunyang","middleName":"","lastName":"Zhang","suffix":""},{"id":493030183,"identity":"b1049447-9425-4186-b036-31f41445fe29","order_by":1,"name":"Xianyu Chen","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Xianyu","middleName":"","lastName":"Chen","suffix":""},{"id":493030184,"identity":"74c01944-7dae-4e8a-8b80-f81b26a7caf9","order_by":2,"name":"Zizheng Zhang","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Zizheng","middleName":"","lastName":"Zhang","suffix":""},{"id":493030185,"identity":"aafafbf0-c242-4e44-9a51-184df2f8759b","order_by":3,"name":"Shen Zhou","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABFElEQVRIiWNgGAWjYDACCSBmbEASMGAH8QwsSNHCcwBESpCiRSIBJo4dyM9uPvbw5w6bxAbpw4c/fKi5J28u+fzqhh8FEgz87d0J2LQY3DmWbiB5Ji2xgS8tTXLGsWLDnbNzym72AB0mcebsBqxaJHLMJAzbDic28PCYMfOwJSQY3M5Ju8ED1GIgkYtVi/yM/G8SiWAt/J8///kH1HLzTNrNP3i0MNzIYZM4CLGFQZqxDajlBvux2/hsMbiRZibZ2JZm3MbDZibZ25dguOFMDtttGQMJHlx+kZ+R/EzyZ5uNbD8P8+MPP74lyBscP/7s5ps/NnL87b3YHQYDbAgmjwGYxKscDbA/IEX1KBgFo2AUDH8AAA8DYpsNUdNBAAAAAElFTkSuQmCC","orcid":"","institution":"","correspondingAuthor":true,"prefix":"","firstName":"Shen","middleName":"","lastName":"Zhou","suffix":""}],"badges":[],"createdAt":"2025-07-30 08:23:15","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-7250325/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7250325/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":87996673,"identity":"34716015-0eb0-491f-96e7-c23f7e00e0ab","added_by":"auto","created_at":"2025-07-31 09:32:45","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":231900,"visible":true,"origin":"","legend":"","description":"","filename":"paper.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7250325/v1_covered_38277f61-bad7-471c-926b-ae09c3c79ceb.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003eA Unified Framework for Non-Convex Optimization in Deep Learning via Adaptive Variance Reduction\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-7250325/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7250325/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe increasing complexity of deep learning models necessitates sophisticated optimization techniques to effectively navigate their non-convex loss landscapes. However, traditional optimization methods, such as stochastic gradient descent (SGD), often encounter challenges related to high variance in gradient estimates, leading to suboptimal convergence and performance. This paper proposes a unified framework for adaptive variance reduction in stochastic non-convex optimization, addressing these critical issues. We introduce the Adaptive Variance Reduced Gradient (AVRG) algorithm, which dynamically balances variance reduction with computational efficiency, yielding improved convergence rates and robustness. Our framework synthesizes existing adaptive variance reduction methods, providing a cohesive theoretical understanding while filling existing gaps in the literature. Comprehensive empirical evaluations across diverse deep learning benchmarks demonstrate the efficacy of AVRG compared to established methods, highlighting its faster convergence and superior performance. By equipping researchers and practitioners with enhanced optimization strategies, this work contributes significantly to the field of deep learning, paving the way for future research in more complex optimization scenarios. We explore the theoretical guarantees of our approach, aiming to foster advancements in training methodologies for deep neural networks, thereby enabling better performance across a wide array of applications.\u003c/p\u003e","manuscriptTitle":"A Unified Framework for Non-Convex Optimization in Deep Learning via Adaptive Variance Reduction","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-07-31 09:24:39","doi":"10.21203/rs.3.rs-7250325/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"ed81d30f-bcb1-4766-8ad1-01209863f3ff","owner":[],"postedDate":"July 31st, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-07-31T09:24:39+00:00","versionOfRecord":[],"versionCreatedAt":"2025-07-31 09:24:39","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7250325","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7250325","identity":"rs-7250325","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}