Sequencing the Neurome: Towards Scalable Exact Parameter Reconstruction of Black-Box Neural Networks

Sequencing the Neurome: Towards Scalable Exact Parameter Reconstruction of Black-Box Neural Networks | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Sequencing the Neurome: Towards Scalable Exact Parameter Reconstruction of Black-Box Neural Networks Judah Goldfeder, Quinten Roets, Gabe Guo, John Wright, Hod Lipson This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4947064/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Inferring the exact parameters of a neural network with only query access is an NP-Hard problem, with few practical existing algorithms. Solutions would have major implications for security, verification, interpretability, and understanding biological networks. The key challenges are the massive parameter space, and complex non-linear relationships between neurons. We resolve these challenges using two insights. First, we observe that almost all networks used in practice are produced by random initialization and first order optimization, an inductive bias that drastically reduces the practical parameter space. Second, we present a novel query generation algorithm that produces maximally informative samples, letting us untangle the non-linear relationships efficiently. We demonstrate reconstruction of a hidden network containing over 1.5 million parameters, and of one 7 layers deep, the largest and deepest reconstructions to date, with max parameter difference less than 0.0001, and illustrate robustness and scalability across a variety of architectures, datasets, and training procedures. Physical sciences/Mathematics and computing/Computer science Physical sciences/Mathematics and computing/Information technology Physical sciences/Mathematics and computing/Software Biological sciences/Neuroscience/Computational neuroscience/Neural decoding Physical sciences/Mathematics and computing/Applied mathematics Deep Learning Reconstruction Full Text Additional Declarations There is NO Competing Interest. Supplementary Files ReverseengineeringnetworksNMISupplementaryInformation.pdf Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4947064","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":363322376,"identity":"2f1194c0-15b0-4d2c-8f1b-6472d6882f65","order_by":0,"name":"Judah Goldfeder","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA7klEQVRIiWNgGAWjYBACxgYGxocfKmzkGBiYgcwGCZCgASEtzMYSZ9KMIbqJ0QIEbBK8bYcSGyBaGAhrYW4/fkFCsu1Aen97Y5vUzR0W8gzszdsk8DqsJ6fAoODcndwZZw62SeeekTBs4DlWhl9LQ05CgkTZs9wNEolALW0SQO/kmOHX0v8m4QAP2+F0A6gW+wb5NwS0zEg/2MDTdjgBpiWxQYKHkJY3zMzAQDYE+qXZGuiX5DaetGILfFoM+9Of/wRGpTx/e/PB27k76mz72Q9vvIFXSwMPWiyw4VMOAsBoeEBIzSgYBaNgFIx0AAB4yU5HBzm8DgAAAABJRU5ErkJggg==","orcid":"","institution":"Columbia University","correspondingAuthor":true,"prefix":"","firstName":"Judah","middleName":"","lastName":"Goldfeder","suffix":""},{"id":363322377,"identity":"916f71a2-5a01-435b-8d27-54c2e2d0a037","order_by":1,"name":"Quinten Roets","email":"","orcid":"","institution":"Columbia University","correspondingAuthor":false,"prefix":"","firstName":"Quinten","middleName":"","lastName":"Roets","suffix":""},{"id":363322378,"identity":"95dfdd89-e2bf-43c5-ad19-2a6048093c2d","order_by":2,"name":"Gabe Guo","email":"","orcid":"","institution":"Columbia University","correspondingAuthor":false,"prefix":"","firstName":"Gabe","middleName":"","lastName":"Guo","suffix":""},{"id":363322379,"identity":"f4f66f66-c7c8-4518-83e3-36ba5af56cef","order_by":3,"name":"John Wright","email":"","orcid":"","institution":"Columbia University","correspondingAuthor":false,"prefix":"","firstName":"John","middleName":"","lastName":"Wright","suffix":""},{"id":363322380,"identity":"73da07da-b155-46cf-8a9e-c5f4dd959167","order_by":4,"name":"Hod Lipson","email":"","orcid":"https://orcid.org/0000-0003-0769-4618","institution":"Columbia University","correspondingAuthor":false,"prefix":"","firstName":"Hod","middleName":"","lastName":"Lipson","suffix":""}],"badges":[],"createdAt":"2024-08-20 19:00:08","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4947064/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4947064/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":67788162,"identity":"c2cb8996-1dfa-4cff-9aa8-ac24148a5c6a","added_by":"auto","created_at":"2024-10-29 17:29:57","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2366363,"visible":true,"origin":"","legend":"","description":"","filename":"ReverseengineeringnetworksNMI.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4947064/v1_covered_c4758f1c-52d7-4b01-aa2a-064e694b0e9e.pdf"},{"id":66297458,"identity":"b291f309-0036-4ba1-b2ef-e5e928c3d393","added_by":"auto","created_at":"2024-10-10 03:59:10","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":693256,"visible":true,"origin":"","legend":"","description":"","filename":"ReverseengineeringnetworksNMISupplementaryInformation.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4947064/v1/0d731c9ef46b895c64664a98.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"Sequencing the Neurome: Towards Scalable Exact Parameter Reconstruction of Black-Box Neural Networks","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Deep Learning, Reconstruction","lastPublishedDoi":"10.21203/rs.3.rs-4947064/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4947064/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Inferring the exact parameters of a neural network with only query access is an NP-Hard problem, with few practical existing algorithms. Solutions would have major implications for security, verification, interpretability, and understanding biological networks. The key challenges are the massive parameter space, and complex non-linear relationships between neurons.\r\nWe resolve these challenges using two insights. First, we observe that almost all networks used in practice are produced by random initialization and first order optimization, an inductive bias that drastically reduces the practical parameter space. Second, we present a novel query generation algorithm that produces maximally informative samples, letting us untangle the non-linear relationships efficiently. \r\nWe demonstrate reconstruction of a hidden network containing over 1.5 million parameters, and of one 7 layers deep, the largest and deepest reconstructions to date, with max parameter difference less than 0.0001, and illustrate robustness and scalability across a variety of architectures, datasets, and training procedures.","manuscriptTitle":"Sequencing the Neurome: Towards Scalable Exact Parameter Reconstruction of Black-Box Neural Networks","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-10-10 03:59:05","doi":"10.21203/rs.3.rs-4947064/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"21ab9767-15fa-4b7d-b07f-559c16038c1c","owner":[],"postedDate":"October 10th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":38649365,"name":"Physical sciences/Mathematics and computing/Computer science"},{"id":38649366,"name":"Physical sciences/Mathematics and computing/Information technology"},{"id":38649367,"name":"Physical sciences/Mathematics and computing/Software"},{"id":38649368,"name":"Biological sciences/Neuroscience/Computational neuroscience/Neural decoding"},{"id":38649369,"name":"Physical sciences/Mathematics and computing/Applied mathematics"}],"tags":[],"updatedAt":"2024-10-29T17:21:49+00:00","versionOfRecord":[],"versionCreatedAt":"2024-10-10 03:59:05","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4947064","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4947064","identity":"rs-4947064","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}