When the whole is greater than the sum of its parts: Scaling black-box inference to large data settings through divide-and-conquer

When the whole is greater than the sum of its parts: Scaling black-box inference to large data settings through divide-and-conquer | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article When the whole is greater than the sum of its parts: Scaling black-box inference to large data settings through divide-and-conquer Emily C. Hector, Amanda Lenzi This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7850688/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 11 You are reading this latest preprint version Abstract Black-box methods such as deep neural networks are exceptionally fast at obtaining point estimates of model parameters due to their amortisation of the loss function computation, but are currently restricted to settings for which simulating training data is inexpensive. When simulating data is computationally expensive, both the training and uncertainty quantification, which typically relies on a parametric bootstrap, become intractable. We propose a black-box divide-and-conquer estimation and inference framework when data simulation is computationally expensive that trains a black-box estimation method on a partition of the multivariate data domain, estimates and bootstraps on the partitioned data, and combines estimates and inferences across data partitions. Through the divide step, only small training data need be simulated, substantially accelerating the training. Further, the estimation and bootstrapping can be conducted in parallel across multiple computing nodes to further speed up the procedure. Finally, the conquer step accounts for any dependence between data partitions through a statistically and computationally efficient weighted average. We illustrate the implementation of our framework in high-dimensional spatial settings with Gaussian and max-stable processes. Applications to modeling extremal temperature data from both a climate model and observations from the National Oceanic and Atmospheric Administration highlight the feasibility of estimation and inference of max-stable process parameters with tens of thousands of locations. Amortised inference Convolutional neural networks Gaussian process Generalized method of moments Max-stable process Statistical computing Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 14 Jan, 2026 Reviews received at journal 08 Jan, 2026 Reviews received at journal 04 Jan, 2026 Reviews received at journal 18 Dec, 2025 Reviewers agreed at journal 25 Nov, 2025 Reviewers agreed at journal 24 Nov, 2025 Reviewers agreed at journal 28 Oct, 2025 Reviewers invited by journal 15 Oct, 2025 Editor assigned by journal 14 Oct, 2025 Submission checks completed at journal 14 Oct, 2025 First submitted to journal 13 Oct, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7850688","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":536192926,"identity":"ab7faa23-8d48-497e-bf2c-1b0b89497d81","order_by":0,"name":"Emily C. Hector","email":"","orcid":"","institution":"University of Michigan–Ann Arbor","correspondingAuthor":false,"prefix":"","firstName":"Emily","middleName":"C.","lastName":"Hector","suffix":""},{"id":536192929,"identity":"87271404-5cff-4094-a933-788f073a38e3","order_by":1,"name":"Amanda Lenzi","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA8UlEQVRIiWNgGAWjYLACHgYGGRB94AMDM1iAsYEILTwg+uAMkrUw8xCjhX/a4WcP3tQw8PBLtz88bFNhzcDffoBNcgYeLRK308wN5xxj4JGccyDhcM6ZdAaJMwlskhvwOep2gpk0DxsDj8GNhAOHc9sOMzDcYGCTfIBHh/zt9G/SPP8YeOxvJDYctvx3mEGekBaD2zlm0rxtQFskkhkOMzYcZjAAacHnMMPbOWWSc/skeCRupDEc7DmWzmN4JrHZEp/35W6nb5N4881Gjn9G+uMPP2qs5eSOHz54swef9yFAAs7iISYiR8EoGAWjYBQQAADIs0kdmPR+HgAAAABJRU5ErkJggg==","orcid":"","institution":"University of Edinburgh","correspondingAuthor":true,"prefix":"","firstName":"Amanda","middleName":"","lastName":"Lenzi","suffix":""}],"badges":[],"createdAt":"2025-10-13 15:38:25","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7850688/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7850688/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":94636877,"identity":"36dd5b44-afe7-490d-8a13-e17170a377f1","added_by":"auto","created_at":"2025-10-29 07:18:37","extension":"json","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":4314,"visible":true,"origin":"","legend":"","description":"","filename":"1a07840f57e34a0cba9e2484cffbc8f7.json","url":"https://assets-eu.researchsquare.com/files/rs-7850688/v1/75f7219d0840452206d560cb.json"},{"id":94640532,"identity":"bc8f53d5-641d-4aa1-8c34-93345cd4199f","added_by":"auto","created_at":"2025-10-29 07:49:47","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":3850161,"visible":true,"origin":"","legend":"","description":"","filename":"snarticleHectorLenzi.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7850688/v1_covered_2b61de6b-bc0b-4fd9-8a02-99af781d0e3d.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"When the whole is greater than the sum of its parts: Scaling black-box inference to large data settings through divide-and-conquer","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"statistics-and-computing","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"stco","sideBox":"Learn more about [Statistics and Computing](http://link.springer.com/journal/11222)","snPcode":"11222","submissionUrl":"https://submission.nature.com/new-submission/11222/3","title":"Statistics and Computing","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Amortised inference, Convolutional neural networks, Gaussian process, Generalized method of moments, Max-stable process, Statistical computing","lastPublishedDoi":"10.21203/rs.3.rs-7850688/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7850688/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Black-box methods such as deep neural networks are exceptionally fast at obtaining point estimates of model parameters due to their amortisation of the loss function computation, but are currently restricted to settings for which simulating training data is inexpensive. When simulating data is computationally expensive, both the training and uncertainty quantification, which typically relies on a parametric bootstrap, become intractable. We propose a black-box divide-and-conquer estimation and inference framework when data simulation is computationally expensive that trains a black-box estimation method on a partition of the multivariate data domain, estimates and bootstraps on the partitioned data, and combines estimates and inferences across data partitions. Through the divide step, only small training data need be simulated, substantially accelerating the training. Further, the estimation and bootstrapping can be conducted in parallel across multiple computing nodes to further speed up the procedure. Finally, the conquer step accounts for any dependence between data partitions through a statistically and computationally efficient weighted average. We illustrate the implementation of our framework in high-dimensional spatial settings with Gaussian and max-stable processes. Applications to modeling extremal temperature data from both a climate model and observations from the National Oceanic and Atmospheric Administration highlight the feasibility of estimation and inference of max-stable process parameters with tens of thousands of locations.","manuscriptTitle":"When the whole is greater than the sum of its parts: Scaling black-box inference to large data settings through divide-and-conquer","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-10-29 07:18:33","doi":"10.21203/rs.3.rs-7850688/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-01-15T00:47:59+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-01-09T04:14:05+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-01-05T03:28:12+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-12-18T11:26:30+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"282761704899074018419722542595980458003","date":"2025-11-25T13:10:37+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"44990634375746024919070887421910503647","date":"2025-11-25T01:27:04+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"82589007190181609203276283814294186653","date":"2025-10-28T10:42:21+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-10-16T03:49:50+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-10-15T00:43:15+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-10-14T13:12:51+00:00","index":"","fulltext":""},{"type":"submitted","content":"Statistics and Computing","date":"2025-10-13T15:35:15+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"statistics-and-computing","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"stco","sideBox":"Learn more about [Statistics and Computing](http://link.springer.com/journal/11222)","snPcode":"11222","submissionUrl":"https://submission.nature.com/new-submission/11222/3","title":"Statistics and Computing","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"77031d71-ada4-4b9b-9a00-9a6ca5215732","owner":[],"postedDate":"October 29th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-05-19T05:24:49+00:00","versionOfRecord":[],"versionCreatedAt":"2025-10-29 07:18:33","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7850688","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7850688","identity":"rs-7850688","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}