A Mini-batch Stochastic Recursive Gradient Method with Barzilai-Borwein Step-Size for Machine Learning 

A Mini-batch Stochastic Recursive Gradient Method with Barzilai-Borwein Step-Size for Machine Learning | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A Mini-batch Stochastic Recursive Gradient Method with Barzilai-Borwein Step-Size for Machine Learning Yi-Ming Yang, Fu-Sheng Wang, Zheng Peng, Xiao-Jun Zhou This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3129748/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract As a mini-batch version of the SARAH algorithm, the MB-SARAH algorithm has received extensive attention due to its simple recursive scheme for updating stochastic gradient estimates. In this paper, we give a modification of the MB-SARAH method via cooperating with the BB step-size, shorted to MB-SARAH-BB. The MB-SARAH-BB combines some advantages of both MB-SARAH and BB methods, providing robustness in selecting initial step size during the optimization process. In the framework of MB-SARAH-BB, we propose a novel implementable method, Ada-MB-SARAH-BB, which utilizes adaptive probability for sampling in the mini-batch stochastic recursive gradient computation during the inner loop iteration. We establish the linear convergence of the MB-SARAH-BB and Ada-MB-SARAH-BB methods under some mild assumptions. Numerical experiments on standard machine learning datasets demonstrate that, the MB-SARAH-BB is effective and more competitive than the recent successful stochastic gradient methods. In addition, numerical experiments also demonstrate that the performance of Ada-MB-SARAH-BB is generally better than or comparable to MB-SARAH-BB method. MSC Classification: 90C15 · 90C25 · 90C30 Machine learning Mini batches SARAH algorithm BB step-size Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3129748","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":214871843,"identity":"34399bbe-823b-4760-b0f8-a61bbd1cb3cf","order_by":0,"name":"Yi-Ming Yang","email":"","orcid":"","institution":"Xiangtan University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Yi-Ming","middleName":"","lastName":"Yang","suffix":""},{"id":214871845,"identity":"31374c72-a9c7-4c40-a1a9-bbc0b4a424cb","order_by":1,"name":"Fu-Sheng Wang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA30lEQVRIiWNgGAWjYBAC9gYwZcPDJn/4wIEPP4jQwnMATKXJ8UuwJR6c2UO8lsPGkjN4jA9zsBGjhf+M4eeCX8yJG273fDjMwMMgzy92gIAWiRxj6Zl9bIkb7pzdcLjAgsFw5uwE/FrsJXg3SPP28CRuOJC74fAMHoYEg9sEtPDwn938m7dHAqgl58FhHjZitDDkbpPm+WEA9H4OA5FaJPK/WfM2JMjx8xwzAAayBGG/8PAfS77N8+c/Dxt78+MPH37YyPNLE9ACBoxtcKYEEcrB4A+xCkfBKBgFo2BEAgBspUfF5qBJBAAAAABJRU5ErkJggg==","orcid":"","institution":"Taiyuan Normal University","correspondingAuthor":true,"submittingAuthor":false,"prefix":"","firstName":"Fu-Sheng","middleName":"","lastName":"Wang","suffix":""},{"id":214871847,"identity":"e849834e-a3bb-4414-b29e-d74d352773a9","order_by":2,"name":"Zheng Peng","email":"","orcid":"","institution":"Xiangtan University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Zheng","middleName":"","lastName":"Peng","suffix":""},{"id":214871848,"identity":"79e0dac0-6e34-44c6-82b1-5400d9136477","order_by":3,"name":"Xiao-Jun Zhou","email":"","orcid":"","institution":"Hebei University of Technology","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Xiao-Jun","middleName":"","lastName":"Zhou","suffix":""}],"badges":[],"createdAt":"2023-07-01 01:59:12","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3129748/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3129748/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":51402448,"identity":"e985a097-b197-4de4-83b1-babbbfbb526b","added_by":"auto","created_at":"2024-02-21 00:43:23","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":528247,"visible":true,"origin":"","legend":"","description":"","filename":"AMinibatchStochasticRecursiveGradientMethodwithBarzilaiBorweinStepSizeforMachineLearning.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3129748/v1_covered_49019e32-4799-4447-adf7-ced1f6808caf.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"A Mini-batch Stochastic Recursive Gradient Method with Barzilai-Borwein Step-Size for Machine Learning ","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Machine learning, Mini batches, SARAH algorithm, BB step-size","lastPublishedDoi":"10.21203/rs.3.rs-3129748/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3129748/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eAs a mini-batch version of the SARAH algorithm, the MB-SARAH algorithm has received extensive attention due to its simple recursive scheme for updating stochastic gradient estimates. In this paper, we give a modification of the MB-SARAH method via cooperating with the BB step-size, shorted to MB-SARAH-BB. The MB-SARAH-BB combines some advantages of both MB-SARAH and BB methods, providing robustness in selecting initial step size during the optimization process. In the framework of MB-SARAH-BB, we propose a novel implementable method, Ada-MB-SARAH-BB, which utilizes adaptive probability for sampling in the mini-batch stochastic recursive gradient computation during the inner loop iteration. We establish the linear convergence of the MB-SARAH-BB and Ada-MB-SARAH-BB methods under some mild assumptions. Numerical experiments on standard machine learning datasets demonstrate that, the MB-SARAH-BB is effective and more competitive than the recent successful stochastic gradient methods. In addition, numerical experiments also demonstrate that the performance of Ada-MB-SARAH-BB is generally better than or comparable to MB-SARAH-BB method.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMSC Classification:\u003c/strong\u003e 90C15 · 90C25 · 90C30\u003c/p\u003e","manuscriptTitle":"A Mini-batch Stochastic Recursive Gradient Method with Barzilai-Borwein Step-Size for Machine Learning ","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2023-07-06 02:26:27","doi":"10.21203/rs.3.rs-3129748/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"530aaece-a603-4e01-8418-6b2f022b0cc0","owner":[],"postedDate":"July 6th, 2023","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-02-21T00:35:07+00:00","versionOfRecord":[],"versionCreatedAt":"2023-07-06 02:26:27","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-3129748","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3129748","identity":"rs-3129748","version":["v1"]},"buildId":"WrCJVZZCHTDjtuVLN7oU0","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}