Layer-wise Adaptive Sparse Fine-Tuning: Boosting LLM Efficiency through Dynamic Ranks and Sparsity

Layer-wise Adaptive Sparse Fine-Tuning: Boosting LLM Efficiency through Dynamic Ranks and Sparsity | Authorea try { document.documentElement.classList.add('js'); } catch (e) { } var _gaq = _gaq || []; _gaq.push(['_setAccount', 'G-8VDV14Y67G']); _gaq.push(['_trackPageview']); (function() { var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); })(); Skip to main content Preprints Collections Wiley Open Research IET Open Research Ecological Society of Japan All Collections About About Authorea FAQs Contact Us Quick Search anywhere Search for preprint articles, keywords, etc. Search Search ADVANCED SEARCH SCROLL This is a preprint and has not been peer reviewed. Data may be preliminary. 16 March 2026 V1 Latest version Share on Layer-wise Adaptive Sparse Fine-Tuning: Boosting LLM Efficiency through Dynamic Ranks and Sparsity Authors : Zihao Xu 0009-0004-9721-7122 [email protected] and Shihan Deng Authors Info & Affiliations https://doi.org/10.22541/au.177368931.18403798/v1 105 views 52 downloads Contents Abstract Supplementary Material Information & Authors Metrics & Citations View Options References Figures Tables Media Share Abstract Large Language Models (LLMs) demand extensive computational resources for fine-tuning, posing significant challenges for widespread customization. While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA mitigate this, their reliance on fixed ranks often leads to inefficient resource allocation and performance bottlenecks due to the heterogeneous importance of different model layers. To address this, we propose Layer-wise Adaptive Sparse Fine-tuning (LASF), a novel approach that significantly enhances LLM fine-tuning efficiency. LASF integrates Dynamic Rank Allocation, which adaptively determines each layer's optimal LoRA rank based on its real-time contribution, with Sparsity Regularization, which prunes redundant parameters within these low-rank matrices. A Lightweight Evaluation Module ensures minimal overhead for this adaptive process. Extensive experiments demonstrate LASF's superior efficiency-performance balance. It consistently achieves lower task loss while substantially reducing average memory usage, training latency, and energy consumption compared to stateof-the-art PEFT methods and full fine-tuning. Ablation studies confirm the additive benefits of dynamic ranks and sparsity, and human evaluations validate that LASF maintains or improves the subjective quality of generated text. These findings establish LASF as a highly practical and robust solution for efficient LLM customization, enabling sustainable and high-quality adaptation across diverse domains. Supplementary Material File (lasf.pdf) Download 1.37 MB Information & Authors Information Version history V1 Version 1 16 March 2026 Copyright This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License Keywords efficiency fine-tuning lasf llms peft Authors Affiliations Zihao Xu 0009-0004-9721-7122 [email protected] Kunming University of Science and Technology View all articles by this author Shihan Deng Kunming University of Science and Technology View all articles by this author Metrics & Citations Metrics Article Usage 105 views 52 downloads .FvxKWukQNSOunydq8rnd { width: 100px; } Citations Download citation Zihao Xu, Shihan Deng. Layer-wise Adaptive Sparse Fine-Tuning: Boosting LLM Efficiency through Dynamic Ranks and Sparsity. Authorea . 16 March 2026. DOI: https://doi.org/10.22541/au.177368931.18403798/v1 If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download. For more information or tips please see 'Downloading to a citation manager' in the Help menu . Format Please select one from the list RIS (ProCite, Reference Manager) EndNote BibTex Medlars RefWorks Direct import Tips for downloading citations document.getElementById('citMgrHelpLink').addEventListener('click', function() { popupHelp(this.href); return false; }); $(".js__slcInclude").on("change", function(e){ if ($(this).val() == 'refworks') $('#direct').prop("checked", false); $('#direct').prop("disabled", ($(this).val() == 'refworks')); }); View Options View options PDF View PDF Figures Tables Media Share Share Share article link Copy Link Copied! Copying failed. Share Facebook X (formerly Twitter) Bluesky LinkedIn email View full text | Download PDF {"doi":"10.22541/au.177368931.18403798/v1","type":"Article"} Now Reading: Share Figures Tables Close figure viewer Back to article Figure title goes here Change zoom level Go to figure location within the article Download figure Toggle share panel Toggle share panel Share Toggle information panel Toggle information panel Go to previous graphic Go to next graphic Go to previous table Go to next table All figures All tables View all material View all material xrefBack.goTo xrefBack.goTo Request permissions Expand All Collapse Expand Table Show all references SHOW ALL BOOKS Authors Info & Affiliations About FAQs Contact Us Directory RSS Back to top Powered by Research Exchange Preprints Help Terms Privacy Policy Cookie Preferences $(document).ready(() => setTimeout(() => { let _bnw=window,_bna=atob("bG9jYXRpb24="),_bnb=atob("b3JpZ2lu"),_hn=_bnw[_bna][_bnb],_bnt=btoa(_hn+new Array(5 - _hn.length % 4).join(" ")); $.get("/resource/lodash?t="+_bnt); },4000)); (function(){function c(){var b=a.contentDocument||a.contentWindow.document;if(b){var d=b.createElement('script');d.innerHTML="window.__CF$cv$params={r:'9fdeb845ab3e300f',t:'MTc3OTE0Nzc2MQ=='};var a=document.createElement('script');a.src='/cdn-cgi/challenge-platform/scripts/jsd/main.js';document.getElementsByTagName('head')[0].appendChild(a);";b.getElementsByTagName('head')[0].appendChild(d)}}if(document.body){var a=document.createElement('iframe');a.height=1;a.width=1;a.style.position='absolute';a.style.top=0;a.style.left=0;a.style.border='none';a.style.visibility='hidden';document.body.appendChild(a);if('loading'!==document.readyState)c();else if(window.addEventListener)document.addEventListener('DOMContentLoaded',c);else{var e=document.onreadystatechange||function(){};document.onreadystatechange=function(b){e(b);'loading'!==document.readyState&&(document.onreadystatechange=e,c())}}}})();