Denoising Neural Models with Spectral QUEnching and Eigenvalue ZEroing (SQUEEZE)

Denoising Neural Models with Spectral QUEnching and Eigenvalue ZEroing (SQUEEZE) | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Denoising Neural Models with Spectral QUEnching and Eigenvalue ZEroing (SQUEEZE) Pierre Dantas, Waldir Junior, Lucas Cordeiro, Eulanda dos Santos This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8720929/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract \Glspl{llm} built with Transformer technology perform very well but requires substantial computational power. This paper presents \gls{squeeze}, a new framework that treats model compression as a signal-to-noise separation task. Using principles from \gls{rmt}, \gls{squeeze} identifies and preserves structural signals within weight matrices while discarding components that align with random noise. Unlike traditional methods, this approach serves as a post-training transformation that does not require retraining the model. Our analysis of fine-tuned \texttt{BERT-base} models reveals that matrix aspect ratios ($\beta$) significantly influence spectral behavior: rectangular feed-forward (\gls{ffn}) layers ($\beta =$ 0.25) adhere closely to the \gls{mp} and exhibit substantial redundancy, whereas square attention matrices ($\beta =$ 1) and highly rectangular embedding matrices ($\beta \ll$ 1) show significant departures from the null model. We implement a five-step pipeline -- standardization, \gls{svd} decomposition, baseline establishment, \gls{tw} finite-size adjustment, and rank truncation. Testing across three \gls{glue} tasks shows that \gls{squeeze} reduces model size by \SI{8.1}{\percent} with an accuracy loss of less than \SI{2.5}{\percent}. Specifically, targeting \gls{ffn} layers allows for a \SI{64}{\percent} reduction in effective rank while maintaining a cosine similarity of over 0.80. Our results show that spectral geometry is a critical factor in Transformer compressibility, positioning \gls{squeeze} as a high-fidelity alternative to aggressive methods such as quantization when model quality is the primary priority. Transformers Spectral Compression Random Matrix Theory Marchenko-Pastur Law Tracy-Widom Statistics Singular Value Shrinkage Deep Learning Interpretability Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8720929","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":591259903,"identity":"73a2b5e9-0234-4f29-9a90-24d0e5ef31e8","order_by":0,"name":"Pierre Dantas","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA+klEQVRIiWNgGAWjYDACZgjF2ACmKkAizA3EagGxzoBEGAloYUDWwtiGZCMuYM7OfOxzRQWDbP/s/oMff86rjeZvB2r5UbENpxbLZrbkmWfOMBjPuHOYWZp32/HcGYcZGxh7ztzGqcXgMI8xY2MbQ2LDjWQGacZtx3IbgFqYGdvwaeH/zNj4jyFx/o1k5p8/5xzLnU9YCw8zY2MDQ+KGG8lsErwNNbkbCGkB+sWYseGYhPHGG8lm1jzHDuRuBGo5iM8v5vyHHzM21NjIzruR+Pjmj5q63HnnDx988KMCj8MglASMfxhMHsCpHqEFDurwKR4Fo2AUjIIRCgBDwVn7swBY0AAAAABJRU5ErkJggg==","orcid":"","institution":"University of Manchester","correspondingAuthor":true,"prefix":"","firstName":"Pierre","middleName":"","lastName":"Dantas","suffix":""},{"id":591259905,"identity":"4e32f6b0-8eec-4c71-b84a-69822edccc71","order_by":1,"name":"Waldir Junior","email":"","orcid":"","institution":"Federal University of Amazonas","correspondingAuthor":false,"prefix":"","firstName":"Waldir","middleName":"","lastName":"Junior","suffix":""},{"id":591259906,"identity":"5d2bf876-73d2-4215-a2a5-d82daf59fdce","order_by":2,"name":"Lucas Cordeiro","email":"","orcid":"","institution":"University of Manchester","correspondingAuthor":false,"prefix":"","firstName":"Lucas","middleName":"","lastName":"Cordeiro","suffix":""},{"id":591259908,"identity":"a045f2e1-e11e-4807-8d0d-e6ceb6eb7b89","order_by":3,"name":"Eulanda dos Santos","email":"","orcid":"","institution":"Federal University of Amazonas","correspondingAuthor":false,"prefix":"","firstName":"Eulanda","middleName":"dos","lastName":"Santos","suffix":""}],"badges":[],"createdAt":"2026-01-28 12:53:07","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8720929/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8720929/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":103914834,"identity":"0a85e6d0-a6f9-4687-a6a9-511714dbfd2d","added_by":"auto","created_at":"2026-03-04 12:57:39","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":408312,"visible":true,"origin":"","legend":"","description":"","filename":"202601SQUEEZErev32.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8720929/v1_covered_b33d3f62-63e2-470b-8c8a-e529acbee2d9.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Denoising Neural Models with Spectral QUEnching and Eigenvalue ZEroing (SQUEEZE)","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Transformers, Spectral Compression, Random Matrix Theory, Marchenko-Pastur Law, Tracy-Widom Statistics, Singular Value Shrinkage, Deep Learning, Interpretability","lastPublishedDoi":"10.21203/rs.3.rs-8720929/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8720929/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\\Glspl{llm} built with Transformer technology perform very well but requires substantial computational power. This paper presents \\gls{squeeze}, a new framework that treats model compression as a signal-to-noise separation task. Using principles from \\gls{rmt}, \\gls{squeeze} identifies and preserves structural signals within weight matrices while discarding components that align with random noise. Unlike traditional methods, this approach serves as a post-training transformation that does not require retraining the model. Our analysis of fine-tuned \\texttt{BERT-base} models reveals that matrix aspect ratios ($\\beta$) significantly influence spectral behavior: rectangular feed-forward (\\gls{ffn}) layers ($\\beta =$ 0.25) adhere closely to the \\gls{mp} and exhibit substantial redundancy, whereas square attention matrices ($\\beta =$ 1) and highly rectangular embedding matrices ($\\beta \\ll$ 1) show significant departures from the null model. We implement a five-step pipeline -- standardization, \\gls{svd} decomposition, baseline establishment, \\gls{tw} finite-size adjustment, and rank truncation. Testing across three \\gls{glue} tasks shows that \\gls{squeeze} reduces model size by \\SI{8.1}{\\percent} with an accuracy loss of less than \\SI{2.5}{\\percent}. Specifically, targeting \\gls{ffn} layers allows for a \\SI{64}{\\percent} reduction in effective rank while maintaining a cosine similarity of over 0.80. Our results show that spectral geometry is a critical factor in Transformer compressibility, positioning \\gls{squeeze} as a high-fidelity alternative to aggressive methods such as quantization when model quality is the primary priority.","manuscriptTitle":"Denoising Neural Models with Spectral QUEnching and Eigenvalue ZEroing (SQUEEZE)","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-02-16 06:30:13","doi":"10.21203/rs.3.rs-8720929/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"3f6670cf-97bb-486f-a021-b5b89b4e5533","owner":[],"postedDate":"February 16th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-03-04T12:56:06+00:00","versionOfRecord":[],"versionCreatedAt":"2026-02-16 06:30:13","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8720929","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8720929","identity":"rs-8720929","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}