A Comparative Analysis of Learning-Rate Selection for Fine-Tuning Transformer Language Models

A Comparative Analysis of Learning-Rate Selection for Fine-Tuning Transformer Language Models | Authorea try { document.documentElement.classList.add('js'); } catch (e) { } var _gaq = _gaq || []; _gaq.push(['_setAccount', 'G-8VDV14Y67G']); _gaq.push(['_trackPageview']); (function() { var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); })(); Skip to main content Preprints Collections Wiley Open Research IET Open Research Ecological Society of Japan All Collections About About Authorea FAQs Contact Us Quick Search anywhere Search for preprint articles, keywords, etc. Search Search ADVANCED SEARCH SCROLL This is a preprint and has not been peer reviewed. Data may be preliminary. 3 February 2026 V1 Latest version Share on A Comparative Analysis of Learning-Rate Selection for Fine-Tuning Transformer Language Models Author : Gopakumar T- V [email protected] Authors Info & Affiliations https://doi.org/10.22541/au.177011867.70562622/v1 453 views 73 downloads Contents Abstract Information & Authors Metrics & Citations View Options References Figures Tables Media Share Abstract Learning rate is a critical hyperparameter in the fine-tuning of large language models, strongly influencing convergence, training stability, and generalization performance. However, learning-rate selection is often based on heuristics, and systematic comparisons across different model architectures remain limited. This work presents a controlled empirical study of learning-rate sensitivity in three transformer-based language models of varying scale: GPT-2, OPT-350M, and EleutherAI/Pythia-160M. All models were fine-tuned on the same public-domain dataset, Alice’s Adventures in Wonderland , under identical experimental conditions. Four learning rates (1e-05, 5e-05, 1e-04, and 5e-04) were evaluated over three training epochs while keeping all other hyperparameters fixed. Model performance was assessed using epoch-wise validation loss and final perplexity, enabling direct comparison of convergence behavior and training stability. The results indicate that moderate learning rates lead to stable convergence and improved generalization, whereas excessively large learning rates cause optimization instability, reflected by increasing validation loss and sharp rises in perplexity. OPT-350M achieves the lowest perplexity and exhibits greater robustness at lower learning rates, while GPT-2 and Pythia-160M show higher sensitivity to learning-rate selection. Overall, the study highlights the importance of careful learning-rate tuning and provides practical guidance for stable fine-tuning of transformer-based language models. 1. Introduction Transformer-based large language models (LLMs) have become central to modern natural language processing, demonstrating strong performance across a wide range of tasks such as text generation, translation, and summarization. Since the introduction of the transformer architecture by Vaswani et al. [1], subsequent work has largely focused on architectural design, scaling laws, and large-scale pretraining strategies [2–4]. While these advances have significantly improved model capacity and generalization, the fine-tuning stage remains critical for adapting pretrained models to specific datasets and task distributions [2,3,8]. During fine-tuning, optimization hyperparameters play a decisive role in determining convergence behavior and final performance. Among these, the learning rate is particularly influential, as it controls the magnitude of parameter updates during gradient-based optimization. An excessively small learning rate may lead to slow convergence, whereas an overly large learning rate can cause instability, overshooting, or divergence of the training process [7]. Validation loss is commonly used to monitor generalization during training, while perplexity serves as a standard evaluation metric in language modeling, providing an interpretable measure of predictive uncertainty [2]. Training is typically organized into epochs, allowing model behavior to be analyzed progressively as parameters are updated. Recent studies on large pretrained models such as GPT-2, OPT, and Pythia have highlighted the importance of careful optimization and learning-rate scheduling during fine-tuning [2,5,6,8]. However, many of these studies focus on single-model settings or large proprietary datasets, making it difficult to isolate the effect of learning rate across models of different sizes under controlled conditions. In this work, we present a systematic empirical analysis of learning-rate sensitivity in three transformer-based language models—GPT-2, OPT-350M, and EleutherAI/Pythia-160M. All models are fine-tuned on the same public-domain dataset, Alice’s Adventures in Wonderland by Lewis Carroll, which is freely available and has no copyright restrictions. By keeping all experimental settings fixed and varying only the learning rate, we aim to provide clear insights into convergence behavior, training stability, and perplexity trends across different model scales. 2. Methodology This study adopts a controlled experimental methodology to analyze the effect of learning-rate selection on the fine-tuning behavior of transformer-based language models. The primary objective is to isolate the impact of the learning rate on convergence behavior, training stability, and generalization performance while keeping all other factors constant. 2.1 Experimental Environment and Tools All experiments were conducted using Google Colab, leveraging its publicly available computational resources. Model implementation, training, and evaluation were carried out using Python and the Hugging Face Transformers library. This framework was selected due to its standardized and widely accepted implementations of transformer architectures, tokenizers, optimizers, and evaluation utilities, ensuring reproducibility and consistency across experiments. 2.2 Models and Dataset Three autoregressive transformer-based language models of different parameter scales were selected: GPT-2, OPT-350M, and EleutherAI/Pythia-160M. These models were chosen to represent small-, medium-, and larger-scale pretrained language models commonly used in research. All models were fine-tuned on the same textual dataset, Alice’s Adventures in Wonderland by Lewis Carroll. This dataset is in the public domain, freely available, and contains no copyright restrictions. Tokenization was performed using the Byte Pair Encoding (BPE) tokenizer associated with each respective model to ensure compatibility. 2.3 Training Configuration Each model was fine-tuned for three epochs using the AdamW optimizer. To study learning-rate sensitivity, four learning rates were evaluated: 1e-05, 5e-05, 1e-04, and 5e-04. Apart from the learning rate, all other hyperparameters—including optimizer settings, batch size, and training duration—were kept identical across experiments. This controlled setup ensures that observed performance differences are attributable solely to learning-rate variation. 2.4 Evaluation Metrics and Logging Model performance was evaluated using epoch-wise validation loss and final perplexity, which are standard metrics for assessing generalization in language modeling. Validation loss was recorded at the end of each epoch, enabling analysis of convergence behavior across training stages. Final perplexity was computed after completion of training and used for cross-model and cross-learning-rate comparison. Training stability was further assessed by examining loss monotonicity, divergence patterns, and perplexity escalation, particularly at higher learning rates. The collected metrics directly correspond to the tables and plots presented in the Results section. 3. Results and Discussion This section presents and analyzes the experimental results obtained from fine-tuning three transformer-based language models under different learning-rate settings. The focus is on understanding how learning-rate selection influences convergence behavior, generalization performance, and training stability across models of varying scale. Results are reported using epoch-wise validation loss and final perplexity, enabling both within-model and cross-model comparisons. The analysis is organized progressively. First, validation-loss trends across epochs are examined for each model to highlight learning dynamics under different learning rates. Next, training stability is assessed by identifying non-monotonic loss behavior and perplexity escalation at higher learning rates. A comparative analysis is then performed to evaluate learning-rate sensitivity across models using perplexity-based metrics. Finally, the best-performing configurations are compared to identify which model achieves superior performance when optimally tuned. Together, the results provide a coherent empirical view of how learning-rate choice affects optimization behavior and highlight important differences in robustness and stability across transformer model scales. Figure-1 illustrates the effect of different learning rates on the validation loss of GPT-2 across three training epochs. At the lowest learning rate (1e-05), validation loss decreases steadily but remains relatively high, indicating slower convergence. Moderate learning rates (5e-05 and 1e-04) lead to faster and more stable convergence, with 1e-04 achieving the lowest validation loss by the final epoch. At the highest learning rate (5e-04), validation loss decreases initially but increases at the final epoch, suggesting mild instability and overshooting. Overall, the figure shows that GPT-2 benefits from moderate learning rates, while overly small or large learning rates result in suboptimal convergence. Figure 1: Validation Loss versus Epoch for GPT-2 under Different Learning Rates From the figure-1 we can also witness that, at a learning rate of 5e-04, parameter updates are too large, causing the optimizer to overshoot the optimal region of the loss surface. This leads to unstable updates and poorer generalization, resulting in higher validation loss. In contrast, 1e-04 provides a balanced update size that allows stable convergence toward a better minimum. This figure-2 shows how OPT-350M’s validation loss changes across epochs for different learning rates (LR). The lower panel highlights stable learning rates (1e-05, 5e-05, and 1e-04), where validation loss (PPL) remains low and changes smoothly, indicating stable and effective training. Figure-2: Validation Loss versus Epoch for OPT-350M under Different Learning Rates Among these, 1e-05 and 5e-05 achieve the best performance. The upper panel shows the behaviour at a very high learning rate (5e-04), where validation loss is much larger, reflecting unstable training and poor generalization. The broken Y-axis is used to separate these two scales so that both stable and unstable behaviours are clearly visible in the same figure without compressing the smaller loss variations. This figure-3 shows how the validation loss of Pythia-160M changes across epochs for different learning rates. At the lowest learning rate (1e-05), the model shows stable training with consistently low validation loss, indicating good convergence. As the learning rate increases to 5e-05 and 1e-04, the validation loss becomes higher and less stable, suggesting reduced optimization effectiveness. At the highest learning rate (5e-04), validation loss is significantly larger across all epochs, reflecting unstable training and poor generalization. Overall, the figure demonstrates that Pythia-160M is sensitive to learning-rate selection, with stable convergence achieved only at lower learning rates, while aggressive learning rates lead to degraded performance. Figure-3: Validation Loss versus Epoch for EleutherAI/Pythia-160M under Different Learning Rates Figure-4 illustrates the learning-rate sensitivity of three transformer-based language models—GPT-2, OPT-350M, and Pythia-160M—measured using final-epoch perplexity across four learning rates on a logarithmic scale. Here, GPT-2 exhibits a relatively smooth and stable response to learning-rate variation, with perplexity gradually decreasing as the learning rate increases from 1e-05 to 1e-04, followed by a mild increase at 5e-04. This behavior indicates a moderate sensitivity to learning-rate selection and suggests that GPT-2 tolerates a wider range of learning rates without catastrophic degradation. In contrast, OPT-350M demonstrates strong performance at lower learning rates, achieving the lowest perplexity among all models at 1e-05 and 5e-05. However, its perplexity increases sharply at the highest learning rate (5e-04), indicating severe optimization instability. This abrupt divergence highlights the heightened sensitivity of larger models to aggressive learning rates. Pythia-160M shows intermediate behavior between GPT-2 and OPT-350M. While stable at lower learning rates, its perplexity increases steadily as the learning rate grows, reflecting a narrower stability margin. Overall, the figure confirms that larger models achieve superior performance at carefully chosen learning rates but are more susceptible to divergence when the learning rate is too high, underscoring the critical role of learning-rate tuning in fine-tuning transformer language models. Figure 4: Perplexity versus Learning Rate (Log Scale) for GPT-2, OPT-350M, and Pythia-160M The figure-5 compares the validation loss across training epochs for GPT-2, OPT-350M, and Pythia-160M using the best learning rate identified for each model. OPT-350M consistently achieves the lowest validation loss and shows a smooth downward trend, indicating stable and effective convergence. GPT-2 exhibits a steady reduction in validation loss but remains higher than the other models, reflecting its smaller model capacity. Pythia-160M demonstrates intermediate performance, with validation loss decreasing initially and then stabilizing. Overall, the figure highlights that when optimally tuned, larger models converge to lower validation loss and generalize better on the same dataset. Table-1 summarizes the training stability of GPT-2, OPT-350M, and Pythia-160M across different learning rates using validation loss behavior and final perplexity. GPT-2 remains stable for learning rates up to 1e-04, with consistent loss reduction and controlled perplexity, while mild instability appears at higher rates. Figure 5; Validation Loss Comparison across Models at Their Optimal Learning Rates OPT-350M achieves the best performance and stable convergence at low learning rates but exhibits strong divergence and rapid perplexity escalation at aggressive settings Pythia-160M shows stable behavior only at the lowest learning rate, with higher rates. Table 1: Training Stability Summary Across Models and Learning Rates GPT-2 1e-05 Decreasing 20.73 Stable GPT-2 5e-05 Decreasing 15.79 Stable GPT-2 1e-04 Decreasing 15.08 Stable GPT-2 5e-04 Slight increase 17.00 Mildly unstable OPT-350M 1e-05 Decreasing 11.82 Stable OPT-350M 5e-05 Decreasing 11.87 Stable OPT-350M 1e-04 Non-monotonic 13.30 Unstable OPT-350M 5e-04 Divergent 470.29 Highly unstable Pythia-160M 1e-05 Decreasing 13.82 Stable Pythia-160M 5e-05 Increasing 17.43 Unstable Pythia-160M 1e-04 Non-monotonic 21.65 Unstable Pythia-160M 5e-04 Divergent 75.34 Highly unstable leading to increasing loss and instability. Overall, the table highlights that larger models achieve superior performance but require careful learning-rate selection to maintain training stability. Figure 6: Best Achieved Perplexity for Each Model under Optimal Learning-Rate Settings Figure-5 compares the best achieved perplexity for each model under optimal learning-rate settings. For cross-model comparison, only the best stable learning-rate configuration for each model was considered; runs exhibiting divergence or severe instability were excluded from this plot. OPT-350M achieves the lowest perplexity, indicating better language modeling performance, followed by Pythia-160M and GPT-2. The results show that larger models generalize better on the dataset when properly tuned. 4. Conclusion This study presented a controlled empirical analysis of learning-rate sensitivity in transformer-based language models during fine-tuning. By evaluating GPT-2, OPT-350M, and EleutherAI/Pythia-160M under identical experimental conditions and varying only the learning rate, the work isolated the influence of this hyperparameter on convergence behavior, training stability, and generalization performance. Validation loss and perplexity were used as primary evaluation metrics, enabling consistent comparison across models and learning-rate settings. The results demonstrate that moderate learning rates lead to stable convergence and improved generalization, whereas excessively large learning rates consistently cause instability, manifested as non-monotonic validation loss and sharp increases in perplexity. Among the evaluated models, OPT-350M achieved the lowest validation loss and perplexity, indicating superior performance and greater robustness at lower learning rates. GPT-2 and Pythia-160M also benefited from careful learning-rate selection but exhibited narrower stability margins and higher sensitivity to aggressive optimization settings. Overall, the findings highlight that learning-rate selection remains a critical factor in fine-tuning transformer language models, particularly as model scale increases. By using a public-domain dataset and standardized tools, this study provides reproducible insights that can guide practical learning-rate tuning for stable and effective fine-tuning across different model sizes. References [1] Vaswani et al., Attention Is All You Need , NeurIPS, 2017. [2] Radford et al., Language Models are Unsupervised Multitask Learners , OpenAI, 2019. [3] Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers , NAACL, 2019. [4] Liu et al., RoBERTa: A Robustly Optimized BERT Pretraining Approach , arXiv, 2019. [5] Zhang et al., OPT: Open Pre-trained Transformer Language Models , Meta AI, 2022. [6] Biderman et al., Pythia: A Suite for Analyzing Large Language Models , EleutherAI, 2023. [7] Goodfellow et al., Deep Learning , MIT Press, 2016 [8] (1) Building LLMs from scratch - YouTubeTop of Form Bottom of Form Information & Authors Information Version history V1 Version 1 03 February 2026 Copyright This work is licensed under a Non Exclusive No Reuse License. Keywords empirical optimization study. large language model fine-tuning learning-rate sensitivity training stability in transformer models validation loss and perplexity analysis Authors Affiliations Gopakumar T- V [email protected] Cochin University College of Engineering Kuttanad View all articles by this author Metrics & Citations Metrics Article Usage 453 views 73 downloads .FvxKWukQNSOunydq8rnd { width: 100px; } Citations Download citation Gopakumar T- V. A Comparative Analysis of Learning-Rate Selection for Fine-Tuning Transformer Language Models. Authorea . 03 February 2026. DOI: https://doi.org/10.22541/au.177011867.70562622/v1 If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download. For more information or tips please see 'Downloading to a citation manager' in the Help menu . Format Please select one from the list RIS (ProCite, Reference Manager) EndNote BibTex Medlars RefWorks Direct import Tips for downloading citations document.getElementById('citMgrHelpLink').addEventListener('click', function() { popupHelp(this.href); return false; }); $(".js__slcInclude").on("change", function(e){ if ($(this).val() == 'refworks') $('#direct').prop("checked", false); $('#direct').prop("disabled", ($(this).val() == 'refworks')); }); View Options View options PDF View PDF Figures Tables Media Share Share Share article link Copy Link Copied! Copying failed. Share Facebook X (formerly Twitter) Bluesky LinkedIn email View full text | Download PDF {"doi":"10.22541/au.177011867.70562622/v1","type":"Article"} Now Reading: Share Figures Tables Close figure viewer Back to article Figure title goes here Change zoom level Go to figure location within the article Download figure Toggle share panel Toggle share panel Share Toggle information panel Toggle information panel Go to previous graphic Go to next graphic Go to previous table Go to next table All figures All tables View all material View all material xrefBack.goTo xrefBack.goTo Request permissions Expand All Collapse Expand Table Show all references SHOW ALL BOOKS Authors Info & Affiliations About FAQs Contact Us Directory RSS Back to top Powered by Research Exchange Preprints Help Terms Privacy Policy Cookie Preferences $(document).ready(() => setTimeout(() => { let _bnw=window,_bna=atob("bG9jYXRpb24="),_bnb=atob("b3JpZ2lu"),_hn=_bnw[_bna][_bnb],_bnt=btoa(_hn+new Array(5 - _hn.length % 4).join(" ")); $.get("/resource/lodash?t="+_bnt); },4000)); (function(){function c(){var b=a.contentDocument||a.contentWindow.document;if(b){var d=b.createElement('script');d.innerHTML="window.__CF$cv$params={r:'9fe15d4e9ca9e2c5',t:'MTc3OTE3NTQ5Mg=='};var a=document.createElement('script');a.src='/cdn-cgi/challenge-platform/scripts/jsd/main.js';document.getElementsByTagName('head')[0].appendChild(a);";b.getElementsByTagName('head')[0].appendChild(d)}}if(document.body){var a=document.createElement('iframe');a.height=1;a.width=1;a.style.position='absolute';a.style.top=0;a.style.left=0;a.style.border='none';a.style.visibility='hidden';document.body.appendChild(a);if('loading'!==document.readyState)c();else if(window.addEventListener)document.addEventListener('DOMContentLoaded',c);else{var e=document.onreadystatechange||function(){};document.onreadystatechange=function(b){e(b);'loading'!==document.readyState&&(document.onreadystatechange=e,c())}}}})();