Spectral Compactness Ensures Robustness in Low-Precision Neural Networks

Spectral Compactness Ensures Robustness in Low-Precision Neural Networks | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Spectral Compactness Ensures Robustness in Low-Precision Neural Networks Jewon Moon This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8880704/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 4 You are reading this latest preprint version Abstract Low-precision arithmetic (FP16/INT8/INT4) is increasingly required to deploy large neural networks on resource-constrained hardware, yet practitioners frequently observe training or inference failures such as NaN divergence and unstable gradient flow after quantization. This work proposes a unifying linear-algebraic explanation: \emph{the tail class of the singular spectrum} of weight matrices governs whether finite-precision perturbations accumulate benignly or catastrophically. We formalize a \emph{spectral compactness} condition using trace-norm (nuclear-norm) mass concentration and show that exponentially decaying singular spectra induce an explicit \emph{quantization threshold} that confines dynamics to a numerically robust low-rank subspace. We then give a practical recipe---\emph{nuclear initialization} and trace-norm regularization---to enforce spectral compactness in low-precision neural networks. Synthetic experiments isolating spectral effects (diagonal spectra under float32 iteration and coarse quantization) show large gains in (i) effective rank compression (e.g., $82 \rightarrow 11$ to capture $\approx 90%$ of trace-norm mass), (ii) eigenvalue distinguishability after quantization (e.g., $36.0% \rightarrow 63.6%$), and (iii) resistance to finite-precision dissipation over long iterative depth ($t=1000$) where heavy-tailed spectra collapse toward the floating-point floor. These results suggest that spectral-tail shaping is a computational necessity for robust low-bit deployment and a principled initialization/regularization tool for quantization-aware training and low-rank adaptation. low-precision arithmetic quantization nuclear norm spectral decay effective rank low-rank adaptation numerical stability Full Text Additional Declarations No competing interests reported. Supplementary Files spectralcompactnessexample.py Cite Share Download PDF Status: Under Review Version 1 posted Reviewers invited by journal 18 May, 2026 Editor assigned by journal 20 Feb, 2026 Submission checks completed at journal 20 Feb, 2026 First submitted to journal 14 Feb, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8880704","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":594305358,"identity":"c1992074-2861-40a2-ace4-876a6365dc59","order_by":0,"name":"Jewon Moon","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABO0lEQVRIie3PPUvDQBjA8SecpEuarHcE6ycQrgRqxVi/So5CRl8QxEk6XZdC1gx+CKdODhcO2km7BlzapXOLCBEdzKXRkr7sDvdfnhy53x0HoNP9x2Q5HTB6+WgDoGKN1Ych9pIAgPQKonaiPwI7iSgJFb+Hl9es/25kj+uz6WfmX3kTyWcfHB+d9J2Re/PcvnT6dSEW0Gkc9yqEyJrXHATh6TBlfe+Q4+ajRKYbz/EtlnaQxND1WtW7qDRNFwJJW6nBXcKxEaOcWAKzJ2RRaYFgwy1S+1LEi5KCXFTJ905iIkUoME6WHLMqgW1CpInIIAwpThl34RV3c+KdxTmJ1VsGdOst9mRkLDLfp040npPszj+PnWT2di0eWBS9JNPsvtPYIJWQtZoH7mqqJd2/XWVkJX1fE51Op9PBD/iKcN3oEpQiAAAAAElFTkSuQmCC","orcid":"","institution":"Singularity principle institute","correspondingAuthor":true,"prefix":"","firstName":"Jewon","middleName":"","lastName":"Moon","suffix":""}],"badges":[],"createdAt":"2026-02-14 14:38:14","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8880704/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8880704/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":103312178,"identity":"4d842b11-8af4-4973-a3dc-fd6eec609293","added_by":"auto","created_at":"2026-02-24 10:13:41","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":375404,"visible":true,"origin":"","legend":"","description":"","filename":"nplsubmissionpackageF.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8880704/v1_covered_a97dd557-dedd-4893-8540-40f76ed07275.pdf"},{"id":103312023,"identity":"9da87583-94f8-419b-82a3-d65735bb9088","added_by":"auto","created_at":"2026-02-24 10:13:24","extension":"py","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":10692,"visible":true,"origin":"","legend":"","description":"","filename":"spectralcompactnessexample.py","url":"https://assets-eu.researchsquare.com/files/rs-8880704/v1/7f7d65f663adfafeb6c2b7a1.py"}],"financialInterests":"No competing interests reported.","formattedTitle":"Spectral Compactness Ensures Robustness in Low-Precision Neural Networks","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"neural-processing-letters","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"nepl","sideBox":"Learn more about [Neural Processing Letters](http://link.springer.com/journal/11063)","snPcode":"11063","submissionUrl":"https://submission.nature.com/new-submission/11063/3","title":"Neural Processing Letters","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"low-precision arithmetic, quantization, nuclear norm, spectral decay, effective rank, low-rank adaptation, numerical stability","lastPublishedDoi":"10.21203/rs.3.rs-8880704/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8880704/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Low-precision arithmetic (FP16/INT8/INT4) is increasingly required to deploy large neural networks on resource-constrained hardware, yet practitioners frequently observe training or inference failures such as NaN divergence and unstable gradient flow after quantization.\nThis work proposes a unifying linear-algebraic explanation: \\emph{the tail class of the singular spectrum} of weight matrices governs whether finite-precision perturbations accumulate benignly or catastrophically.\nWe formalize a \\emph{spectral compactness} condition using trace-norm (nuclear-norm) mass concentration and show that exponentially decaying singular spectra induce an explicit \\emph{quantization threshold} that confines dynamics to a numerically robust low-rank subspace.\nWe then give a practical recipe---\\emph{nuclear initialization} and trace-norm regularization---to enforce spectral compactness in low-precision neural networks.\nSynthetic experiments isolating spectral effects (diagonal spectra under float32 iteration and coarse quantization) show large gains in (i) effective rank compression (e.g., $82 \\rightarrow 11$ to capture $\\approx 90\\%$ of trace-norm mass), (ii) eigenvalue distinguishability after quantization (e.g., $36.0\\% \\rightarrow 63.6\\%$), and (iii) resistance to finite-precision dissipation over long iterative depth ($t=1000$) where heavy-tailed spectra collapse toward the floating-point floor.\nThese results suggest that spectral-tail shaping is a computational necessity for robust low-bit deployment and a principled initialization/regularization tool for quantization-aware training and low-rank adaptation.","manuscriptTitle":"Spectral Compactness Ensures Robustness in Low-Precision Neural Networks","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-02-24 10:12:03","doi":"10.21203/rs.3.rs-8880704/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"reviewersInvited","content":"","date":"2026-05-18T16:33:54+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-02-20T07:42:08+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-02-20T07:39:40+00:00","index":"","fulltext":""},{"type":"submitted","content":"Neural Processing Letters","date":"2026-02-14T14:30:22+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"neural-processing-letters","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"nepl","sideBox":"Learn more about [Neural Processing Letters](http://link.springer.com/journal/11063)","snPcode":"11063","submissionUrl":"https://submission.nature.com/new-submission/11063/3","title":"Neural Processing Letters","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"4b81ea81-5c7b-4875-a356-fd5f59fa38f4","owner":[],"postedDate":"February 24th, 2026","published":true,"recentEditorialEvents":[{"type":"reviewersInvited","content":"6","date":"2026-05-18T16:33:54+00:00","index":"","fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-05-18T16:38:26+00:00","versionOfRecord":[],"versionCreatedAt":"2026-02-24 10:12:03","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8880704","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8880704","identity":"rs-8880704","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}