Discrete Weight Neural Networks: Investigating the Relationship Between Weight Precision and Generalization

Discrete Weight Neural Networks: Investigating the Relationship Between Weight Precision and Generalization | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF matters-arising Discrete Weight Neural Networks: Investigating the Relationship Between Weight Precision and Generalization Avinav Sahoo This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8499757/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Biological synapses transmit signals discretely and with noise, yet biological learners often generalize from few examples. Motivated by this contrast, we study how constraining neural-network weights to discrete grids affects fit and generalization in a controlled rule-learning task. We compare standard float32 training with binary, ternary, and small-integer weight constraints using straight-through estimators (STE), and we include a fine-grained fixed-point grid (Q16.16). We also evaluate a simple pure-integer coordinate-descent baseline to isolate optimization effects when updates are restricted to integer steps. On a 5×5 relational sum-comparison task, coarse discretization substantially reduces the train–test gap but also degrades attainable accuracy under our optimization procedures, indicating that reduced overfitting often coincides with underfitting. In contrast, Q16.16 fixed-point training preserves learnability and, in some settings, matches or exceeds our float32 baseline (e.g., 84% vs. 78% test accuracy at n = 500 in one configuration). We discuss these results in the context of prior work on quantization as regularization and on integer-only training, and we highlight optimization—rather than representational capacity—as the primary bottleneck for very low-bit weights in this setting. Biological sciences/Computational biology and bioinformatics Physical sciences/Engineering Physical sciences/Mathematics and computing Biological sciences/Neuroscience neural networks quantization generalization discrete optimization few-shot learning biological plausibility Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8499757","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"matters-arising","associatedPublications":[],"authors":[{"id":571568964,"identity":"57a1ee65-3279-4873-8573-3d7f089b6d49","order_by":0,"name":"Avinav Sahoo","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA40lEQVRIiWNgGAWjYFACxgYwZQBkPQDSPHykaGE2AGlhI9oyoHI2CRCDoBb+GcltD3621cmbszcfq/yaYyfDxsD88NENPFokbiS2G/a2HTbc2XMs7bbstmSgw9iMjXPwWXPmYJsEz5kDjBtu5JjdltzGDNTCwyaNT4s8UIvknzN19iAtxZLb6glrMTje2CbNU8GcCNLC+HHbYcJaDEFaZCoOJ284cyxZmnHbcR42ZgJ+kTvM/kzyjUGd7YbjzQc//txWbc/P3vzwMV7vIwNmHjBJrHIQYPxBiupRMApGwSgYMQAAWitH8cJHMQsAAAAASUVORK5CYII=","orcid":"","institution":"Pulseinnovas AI lab","correspondingAuthor":true,"prefix":"","firstName":"Avinav","middleName":"","lastName":"Sahoo","suffix":""}],"badges":[],"createdAt":"2026-01-02 09:53:19","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8499757/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8499757/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":100358774,"identity":"7fbcb696-99bf-4010-bae9-491da608157f","added_by":"auto","created_at":"2026-01-16 07:21:20","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":883968,"visible":true,"origin":"","legend":"","description":"","filename":"DiscreteWeightNeuralNetworksJournalUpdatedTechCheck2.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8499757/v1_covered_3ac4d2ca-0b17-4595-84ef-9dbd52d146d9.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Discrete Weight Neural Networks: Investigating the Relationship Between Weight Precision and Generalization","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"neural networks, quantization, generalization, discrete optimization, few-shot learning, biological plausibility","lastPublishedDoi":"10.21203/rs.3.rs-8499757/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8499757/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eBiological synapses transmit signals discretely and with noise, yet biological learners often generalize from few examples. Motivated by this contrast, we study how constraining neural-network weights to discrete grids affects fit and generalization in a controlled rule-learning task. We compare standard float32 training with binary, ternary, and small-integer weight constraints using straight-through estimators (STE), and we include a fine-grained fixed-point grid (Q16.16). We also evaluate a simple pure-integer coordinate-descent baseline to isolate optimization effects when updates are restricted to integer steps. On a 5\u0026times;5 relational sum-comparison task, coarse discretization substantially reduces the train\u0026ndash;test gap but also degrades attainable accuracy under our optimization procedures, indicating that reduced overfitting often coincides with underfitting. In contrast, Q16.16 fixed-point training preserves learnability and, in some settings, matches or exceeds our float32 baseline (e.g., 84% vs. 78% test accuracy at n\u0026thinsp;=\u0026thinsp;500 in one configuration). We discuss these results in the context of prior work on quantization as regularization and on integer-only training, and we highlight optimization\u0026mdash;rather than representational capacity\u0026mdash;as the primary bottleneck for very low-bit weights in this setting.\u003c/p\u003e","manuscriptTitle":"Discrete Weight Neural Networks: Investigating the Relationship Between Weight Precision and Generalization","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-01-09 15:43:16","doi":"10.21203/rs.3.rs-8499757/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"0052114b-c601-49c1-8f22-69133a3002ec","owner":[],"postedDate":"January 9th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":60824541,"name":"Biological sciences/Computational biology and bioinformatics"},{"id":60824542,"name":"Physical sciences/Engineering"},{"id":60824543,"name":"Physical sciences/Mathematics and computing"},{"id":60824544,"name":"Biological sciences/Neuroscience"}],"tags":[],"updatedAt":"2026-01-09T15:43:17+00:00","versionOfRecord":[],"versionCreatedAt":"2026-01-09 15:43:16","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8499757","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8499757","identity":"rs-8499757","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}