author

The adoption of Large Language Models (LLMs) across multiple contexts has sparked interest in understanding how scaling model size might lead to behavioral changes, as LLMs can exhibit behaviors not observed in their smaller counterparts. Understanding these emergent capabilities is essential for advancing LLM development and improving their interpretability across diverse tasks. However, whether LLMs exhibit true emergence in the context of Software Engineering remains an unexplored topic, as most research has focused on NLP tasks. In this paper, we investigate the emergence of capabilities in the context of SE. We propose a model-agnostic pipeline for evaluating this phenomenon across three SE tasks: bug fixing, code translation, and commit message generation. More precisely, for each task, we present a case study instantiating our pipeline to analyze the emergence of capabilities in CodeGen1-multi across four scales, ranging from 350M to 16.1B parameters. Our findings do not provide evidence to support the idea of emergent capabilities resulting from scaling the model size in the selected set of tasks. We hope our results can pave the way to a more nuanced understanding of the emergent capabilities of LLMs within the SE domain, guiding future research to focus on task-specific evaluations and the identification of alternative factors contributing to this phenomenon. Our work underscores the importance of task diversity in examining model behaviors and highlights potential limitations in transferring prior understandings of and approaches to emergence from NLP to Software Engineering. Supplementary Material File (wtcjmtpqsnvtjmbjbbzvstcsffmczygp.pdf) - Download - 616.01 KB Information & Authors Information Version history Copyright This work is licensed under a Non Exclusive No Reuse License.

Authors Metrics & Citations Metrics Article Usage 196views 310downloads Citations Download citation Conor O’Brien, Daniel Rodriguez-Cardenas, Alejandro Velasco, et al. author. Authorea. 19 June 2025. DOI: https://doi.org/10.22541/au.175033466.63493649/v1 DOI: https://doi.org/10.22541/au.175033466.63493649/v1 If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download. For more information or tips please see 'Downloading to a citation manager' in the Help menu.