Gradient-Guided Layerwise Adaptive Noise Injection for Pre-trained Language Model Fine-tuning

Gradient-Guided Layerwise Adaptive Noise Injection for Pre-trained Language Model Fine-tuning | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Gradient-Guided Layerwise Adaptive Noise Injection for Pre-trained Language Model Fine-tuning Qinglin Jiang, Cheng Zeng, Nan Chi This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9574488/v1 This work is licensed under a CC BY 4.0 License Status: Under Revision Version 1 posted 12 You are reading this latest preprint version Abstract Fine-tuning pre-trained language models (PLMs) is crucial for achieving strong performance in downstream natural language processing tasks, but it is prone to overfitting in low-resource scenarios. Existing noise-based regularization methods typically rely on static parameter statistics or predefined heuristics to mitigate this issue. However, the noise magnitude and injection timing in these approaches depend on fixed hyperparameters or heuristic schedules, making them unable to adapt to the dynamic optimization state of the model during training.To address this limitation, this paper proposes a novel fine-tuning framework named gradient-guided noise injection (GNI). The core idea is to utilize gradient information generated during optimization to dynamically adjust the noise intensity in a layer-wise and real-time manner. Extensive experiments on multiple tasks from the GLUE benchmark demonstrate that GNI consistently improves performance across different pre-trained models and task settings, while exhibiting strong generalizability and robustness. This work provides a simple yet effective solution for optimization-state-aware dynamic regularization. Physical sciences/Engineering Physical sciences/Mathematics and computing Pre-trained Language Models Adaptive Regularization Natural Language Processing Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Revision Version 1 posted Editorial decision: Revision requested 14 May, 2026 Reviews received at journal 13 May, 2026 Reviewers agreed at journal 11 May, 2026 Reviews received at journal 10 May, 2026 Reviews received at journal 08 May, 2026 Reviewers agreed at journal 07 May, 2026 Reviewers agreed at journal 07 May, 2026 Reviewers invited by journal 07 May, 2026 Editor invited by journal 07 May, 2026 Editor assigned by journal 04 May, 2026 Submission checks completed at journal 04 May, 2026 First submitted to journal 30 Apr, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9574488","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":640039923,"identity":"77f5264f-7194-4ffe-a16c-9bff2b521a16","order_by":0,"name":"Qinglin Jiang","email":"","orcid":"","institution":"Guizhou University","correspondingAuthor":false,"prefix":"","firstName":"Qinglin","middleName":"","lastName":"Jiang","suffix":""},{"id":640039932,"identity":"abe00a2e-52ea-4edc-92e8-594343d04406","order_by":1,"name":"Cheng Zeng","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA9ElEQVRIie3RsUoDMRjA8YRAsqRmTRF9hkjglg59lYRCu4g4dhD8QuFu7Hqd+whdOl4ptEuK643nG1xnRU2dnGJGwfy3wPfjSwhCudwfTBAMyr5TIhCGcCa/k2G1gMczXLEhYAdJRPkjnFdwK1TzvSaBoNa6zWBbXOuXddWh+cgCOzVRgWu70AM/1UX7Gi7mZxb4g4kSIm2pOT1MinbnAJd7C5KrKKHSVm+cfj5v6gv5SCCc7+BuVVKihAsEEohkDlTvKZEtdrU5zHTJ7+NkvGedMvPwlctj1/dPo5sl83HyY59ByFxelzgfEk36bC6Xy/2vvgDleUwY5CQa1gAAAABJRU5ErkJggg==","orcid":"","institution":"Guizhou Institute of Technology","correspondingAuthor":true,"prefix":"","firstName":"Cheng","middleName":"","lastName":"Zeng","suffix":""},{"id":640039937,"identity":"6df199b7-b018-4ccd-8c60-c92406dc675e","order_by":2,"name":"Nan Chi","email":"","orcid":"","institution":"Guizhou Institute of Technology","correspondingAuthor":false,"prefix":"","firstName":"Nan","middleName":"","lastName":"Chi","suffix":""}],"badges":[],"createdAt":"2026-04-30 09:02:14","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9574488/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9574488/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":109472656,"identity":"359b3d0b-5524-4cd7-890a-653548420670","added_by":"auto","created_at":"2026-05-18 13:25:55","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":471953,"visible":true,"origin":"","legend":"","description":"","filename":"TemplateforsubmissionstoScientificReports.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9574488/v1_covered_6015d723-93fb-4406-aedc-0fc8e4dd5ba8.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Gradient-Guided Layerwise Adaptive Noise Injection for Pre-trained Language Model Fine-tuning","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Pre-trained Language Models, Adaptive Regularization, Natural Language Processing","lastPublishedDoi":"10.21203/rs.3.rs-9574488/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9574488/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eFine-tuning pre-trained language models (PLMs) is crucial for achieving strong performance in downstream natural language processing tasks, but it is prone to overfitting in low-resource scenarios. Existing noise-based regularization methods typically rely on static parameter statistics or predefined heuristics to mitigate this issue. However, the noise magnitude and injection timing in these approaches depend on fixed hyperparameters or heuristic schedules, making them unable to adapt to the dynamic optimization state of the model during training.To address this limitation, this paper proposes a novel fine-tuning framework named gradient-guided noise injection (GNI). The core idea is to utilize gradient information generated during optimization to dynamically adjust the noise intensity in a layer-wise and real-time manner. Extensive experiments on multiple tasks from the GLUE benchmark demonstrate that GNI consistently improves performance across different pre-trained models and task settings, while exhibiting strong generalizability and robustness. This work provides a simple yet effective solution for optimization-state-aware dynamic regularization.\u003c/p\u003e","manuscriptTitle":"Gradient-Guided Layerwise Adaptive Noise Injection for Pre-trained Language Model Fine-tuning","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-05-18 13:25:02","doi":"10.21203/rs.3.rs-9574488/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-05-14T04:08:49+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-14T01:23:57+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"114337869971572060075997848918175952733","date":"2026-05-11T14:41:28+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-10T11:42:39+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-08T04:22:40+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"75631343348269291287401028750813823175","date":"2026-05-08T01:41:55+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"86113481829393801338946476013285867496","date":"2026-05-08T00:49:29+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-05-08T00:46:53+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2026-05-07T12:39:05+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-05-04T10:54:01+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-05-04T10:53:10+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2026-04-30T08:45:32+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"aa35d59a-5556-4665-a565-7ce2adbdcdee","owner":[],"postedDate":"May 18th, 2026","published":true,"recentEditorialEvents":[{"type":"decision","content":"Revision requested","date":"2026-05-14T04:08:49+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-14T01:23:57+00:00","index":106,"fulltext":""},{"type":"reviewerAgreed","content":"114337869971572060075997848918175952733","date":"2026-05-11T14:41:28+00:00","index":98,"fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-10T11:42:39+00:00","index":97,"fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-08T04:22:40+00:00","index":90,"fulltext":""},{"type":"reviewerAgreed","content":"75631343348269291287401028750813823175","date":"2026-05-08T01:41:55+00:00","index":88,"fulltext":""},{"type":"reviewerAgreed","content":"86113481829393801338946476013285867496","date":"2026-05-08T00:49:29+00:00","index":87,"fulltext":""},{"type":"reviewersInvited","content":"73","date":"2026-05-08T00:46:53+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2026-05-07T12:39:05+00:00","index":"","fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"in-revision","subjectAreas":[{"id":68118854,"name":"Physical sciences/Engineering"},{"id":68118856,"name":"Physical sciences/Mathematics and computing"}],"tags":[],"updatedAt":"2026-05-18T13:25:02+00:00","versionOfRecord":[],"versionCreatedAt":"2026-05-18 13:25:02","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9574488","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9574488","identity":"rs-9574488","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}