Temporal validity of software datasets for code metrics: an empirical assessment of sampling strategies

Temporal validity of software datasets for code metrics: an empirical assessment of sampling strategies | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Temporal validity of software datasets for code metrics: an empirical assessment of sampling strategies Juan Andres Carruthers, Andrea Lezcano Airaldi, Jorge Andres Diaz Pace, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7217702/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 9 You are reading this latest preprint version Abstract Context: In empirical research, drawing reliable conclusions about a target population requires working with representative samples. Representativeness refers to the degree to which a sample's properties of interest resemble those of the target population. However, a sample that was representative in the past might not be representative in the present day if the population has significantly evolved during that period. Objective: To evaluate the effectiveness of a dataset extraction tool for collecting current samples of software repositories and keeping their temporal validity over time. Method: We performed a Mining Software Repositories study utilizing three datasets: Tempero et al.’s Qualitas Corpus, a sample from Github and an updated version of the Qualitas Corpus. Based on these datasets, we generated thresholds for three source code metrics (Lines of Code, Cyclomatic Complexity and Weighted Methods per Class) and compared whether these thresholds yielded consistent results. Results: We observed significant differences in all the source code metrics under study when pairing the Qualitas Corpus and samples containing projects with recent development data, with the former registering higher thresholds. Furthermore, the thresholds obtained from the samples collected with our extraction tool recorded consistent thresholds. Conclusions: Using outdated code-based datasets in empirical studies can affect study results, therefore, it is important that researchers not only publish their datasets but also provide strategies to update those datasets over time. Additionally, we presented and validated sampling approaches implemented demonstrating their effectiveness to collect current samples. Sampling software Empirical evaluation Temporal validity Metric thresholds Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 1. Introduction The positivist paradigm is an objective approach that analyzes the world through empirical data, formal propositions, quantifiable measures of variables, hypothesis testing and drawing of inferences about a representative sample (Wohlin & Aurum, 2015 ). In this context, results are reliable if they can be repeated by another researcher and reach a similar conclusion, and findings cannot be established until the study is independently repeated. According to the ACM policy for Artifact Review and Badging (2020) replicability means that an independent group can obtain the same result using the author’s own artifacts . In that sense, the scientific community established the FAIR principles (Findability, Accessibility, Interoperability, and Reusability) (Wilkinson et al., 2016 ), which apply not only to research data but also to the algorithms, tools, and workflows used to generate the datasets. In Empirical Software Engineering, there are specific research methods rooted in the positivist paradigm (e.g., experiments, mining software repository studies, systematic literature reviews, surveys) which require a sample dataset along with precise instructions to validate the study's results. The sample dataset should aim to be representative, i.e., it must closely resemble the underlying characteristics of the target population, which enables researchers to draw generalizable conclusions and extend them to the population under study (Shull et al., 2008 ). However, replicating a study published several years ago using current data might yield inconsistent results if the population has undergone significant changes during that period, causing the original dataset to lose temporal validity. Temporal validity refers to the extent to which findings or conclusions based on data collected at one point in time can be generalized to the present day (Carruthers et al., 2024 ). Munger ( 2019 ) originally presents the concept in the domain of online social sciences and we extended to Software Engineering. In rapidly changing subjects, the time when the data is allocated can influence the results of a study. For instance, it is not the same to conduct an experiment with a dataset comprising source code developed more than 10 years ago than using a dataset with code from recently-developed projects. Several studies reported that the probability of a project remaining actively maintained for more than five years is below 50% (Ait et al., 2022 ; Carruthers et al., 2024 ; Coelho et al., 2020 ). In this sense, we argue that researchers should design and publish their datasets along with the sampling and update procedures to facilitate the assessment of these strategies and mitigate potential temporal validity issues in the future. While it is ideal to evolve datasets over time, there are circumstances in which it is not feasible. Some datasets require human intervention, making automation difficult. For example, classification and labelling tasks intended for machine learning studies cannot always be automated (Arcelli Fontana et al., 2016 ; Ohira et al., 2015 ; Schnappinger et al., 2021 ). Another reason is the limited access to the type of data needed for the study. For instance, in software estimation datasets that contain process metrics from projects (Amasaki et al., 2020 ; Jose Thiago & Oliveira, 2021 ; Rankovic et al., 2021 ), organizations must have gathered them in advance and make the data publicly accessible. In other cases the effort required to retrieve the data can pose restrictions for external investigators who might lack the necessary resources. However, there are datasets mainly containing source code, repository and code metrics, commit messages or issue reports, which can (and should) be evolved over time without the aforementioned barriers. Usually, to retrieve this kind of dataset, the designated API to make requests and fetch records is the only asset required. The main goal of this work is to evaluate the effectiveness of dataset retrieval approaches for collecting current samples from software repositories. To this end, we built a data extraction tool in the context of a Mining Software Repositories (MSR) study, aiming to assess whether the sampling techniques implemented return consistent results. The tool provides two strategies to build a dataset: sampling from scratch and dataset update, and we empirically evaluated and compared the outcomes of these strategies. The first strategy enables users to obtain a subset of the population, while the second restores a sample deemed as temporally invalid, updating the projects with recent activity and replacing the remaining ones with suitable candidates. Both strategies implement stratified random sampling based on the k-means clustering algorithm (Wu et al., 2024 ). Regarding the MSR study, we produced thresholds for three source code metrics: Lines of Code (LOC), Mccabe's Cyclomatic Complexity (McCC) and Weighted Methods per Class (WMC). Thresholds work as reference values to assess the state of the software components, indicating acceptable and abnormal ranges for specific metrics. We computed these thresholds using three datasets: the Qualitas Corpus (Tempero et al., 2010 ), a current sample from Github, and an updated version of the Qualitas Corpus, and then statistically compared them to tackle two research questions. The findings of this paper are two-fold. Firstly, like in a prior work (Carruthers et al., 2024 ), we assessed the temporal validity of an outdated dataset, but instead of repository metrics we considered code metrics. The study allowed us to reinforce what we stated previously, providing additional evidence about the importance of using updated data when conducting code-related MSR studies. Secondly, the evaluation showed that our approach is a viable option to generate current samples. We uploaded the datasets and the scripts of the investigation in a Zenodo repository following the guidelines from Gonzalez-Barahona & Robles ( 2023 ) to report research artifacts and facilitate study replication. The rest of the paper is structured as follows. Section 2 describes attempts from other authors to build similar solutions. Section 3 provides context on the MSR study in which our strategies were tested. Section 4 presents the research questions and details the methodology for our study. Section 5 reports the findings of the study. Section 6 presents the answers to the research questions. Section 7 discusses threats to the validity. Finally, Section 8 gives the conclusions and outlines future work. 2. Related works Over the years, investigators have proposed various solutions to extract development-related data available from online software forges, like Apache Software Foundation, Bitbucket, Github, or SourceForge. Linstead et al. (2009) designed Sourcerer, a platform to mine and search code snippets. The platform automatically crawls software repositories from SourceForge and Apache, parses the code and extracts certain features to improve the performance of the queries. It stores all the information into a relational database and provides access through a website. According to the foundational study, Sourcerer contains code-related data from 4632 Java projects. In 2012, Gousios & Spinellis (2012) presented GHTorrent, a service that gathers development data from Github. The authors explored and mined Github’s API to share repository data about users, organizations, teams, commits, issues, and others from all the projects inside the forge. GHTorrent distributes the data as monthly incremental MongoDB dumps. Another tool is Boa[2] (Dyer et al., 2013), a domain-specific programming language for analyzing ultra-large-scale software repositories with an API and a web-based interface to access its infrastructure. In the beginning, the service facilitated access to almost 650K SVN and CVS repositories from SourceForge, sharing their versions, metadata, commits, changelogs, development team data, and source code. Falessi et al. (2017) created STRESS, a semi-automated approach for project selection based on diversity, fit, and quality. The tool connects to Apache Software Foundation’s query service, allowing project filtering with more than 100 characteristics, including: the number of tickets, number of commits, date of the first/last commit and ticket, number of lines of code committed per programming language, and size of the team. Similarly, Dabic et al. (2021)[3] introduced Github Search, containing information from 730K GitHub public repositories across ten programming languages. The tool crawls and consumes 25 repository characteristics from GitHub’s API, and repositories’ landing page, issues page and pull requests page. Like STRESS, it provides a web application to filter the projects by the number of issues, commits, programming language, date of creation, among others. Also in 2021, Ma et al. (2021) described World of Code[4], a system that collects data from Bitbucket, Github and other forges. In contrast to the previous tools, World of Code stores the complete information provided by Git objects (e.g., all commit information, files content, file versions, etc.). Despite the usefulness of the reviewed solutions for empirical studies, most tools created before 2021 are no longer accessible. Apparently, it is difficult to allocate, maintain and scale such platforms online over an extended period of time. The required resources can be a constraint here because not all the researchers have at their disposal an infrastructure large enough to host these platforms. We think this approach is not always the best option, particularly, in cases in which most of the sampling process can be implemented as runnable scripts and delivered to the research community. It is preferable to execute these scripts and obtain fresh datasets directly from the software forge instead of relying on third-party mirrors that might have outdated records. However, this approach might still pose challenges for researchers with limited computational resources if they try to fetch data from thousands of projects. [1] https://doi.org/10.5281/zenodo.15008288 [2] https://boa.cs.iastate.edu/ [3] https://seart-ghs.si.usi.ch/ [4] https://worldofcode.org/ 3. Code metrics thresholds In this section, we provide a contextual background on the Software Engineering field in which our tool was validated. According to Fenton & Bieman (2014), Software Engineering encompasses techniques applied in the construction and support of software products, involving activities such as; managing, planning, modeling, testing, maintaining, among others. These activities adhere to an engineering approach, which means that they must be understood and controlled to reduce uncertainty while the software is specified, designed, built, and maintained. To achieve this level of control we need to monitor the status of the projects, products, processes, and resources through metrics (Lanza & Marinescu, 2006). Metrics quantify specific characteristics of the software, for example, product metrics can describe different perspectives of the source code, like its cohesion, complexity, coupling or size. When working with metrics, establishing reference points is essential to link a particular metric value to useful semantics (Lanza & Marinescu, 2006). Simply having a numeric value for a metric lacks context, and it is insufficient information to evaluate the state of the system. Therefore, we require thresholds for most of the metrics as they allow us to make informed assessments about the system being measured. Thresholds delimit which values are acceptable and abnormal in software components; providing decision-making guidance to programmers, managers and other members of the team (Lorenz & Kidd, 1994). The literature offers different strategies to set quality thresholds. For example, Mccabe (1976) defined a threshold for the Cyclomatic Complexity metric based on the experience of programmers, suggesting that a method with a value over ten should be refactored. Nejmeh (1988) took a similar approach for another complexity metric (NPATH). The author conducted empirical studies at AT&T Bell Laboratories and came up with a threshold of 200. Coleman et al. (1995) defined two levels for the Maintainability Index metric, an upper and a lower limit of 85 and 65 respectively; components scoring over 85 were considered as highly maintainable, between 85 and 65 were considered as moderate, and components below 65 were considered candidates for refactoring. Because these values rely on subjective experience, their replication and generalization can be challenging. Consequently, data-driven methods have emerged, in which threshold values depend on the data fed into the model, rather than on opinions provided by experienced users. For instance, Alves et al. (2010) proposed a method that implements the statistical properties of metrics based on data from a benchmark of systems. They generated three thresholds for five metrics: McCC, LOC, Number of Parameters, Fan-in and Number of Methods. These thresholds denoted four levels of risk for modules: low, moderate, high and very high. Also considering metrics’ distributions using a benchmark dataset, K. Ferreira et al. (2012) delimited thresholds for six object-oriented measures: Coupling Factor (Brito E Abreu & Carapuça, 1994), Number of Public Fields, Number of Public Methods, and two Chidamber & Kemerer (1994)’s metrics Lack of Cohesion in Methods and Depth of Inheritance Tree. They identified the probability distribution that best fitted each metric and calculated three range values according to value frequency: good, regular and bad. Alternatively, Herbold et al. (2011) introduced a method based on Kearns (1998)’ machine learning algorithm for thresholds calculation. They computed values of four method-level metrics (Cyclomatic Number, Nested Block Depth, Number of Function Calls and Number of Statements) and seven class-level metrics (WMC, Coupling Between Objects, Response for a Class, Number of Methods, LOC, Number of Overridden Methods and Number of Static Methods). They tested it through four case studies employing projects from different programming languages and adjusting the algorithm configuration. Nevertheless, Lavazza & Morasca (2016b) expressed concerns about assessing software quality relying solely on thresholds derived from statistical distributions of internal measures. They argued that estimating external quality must be obtained via probability models that take internal measures as independent variables (Morasca, 2009). In their article, the authors estimated fault-proneness, defining it as the probability that a module contains at least a fault, and generated thresholds for 13 metrics using different approaches. They claimed that thresholds obtained from distribution-based methods using only internal measures remain the same independently of the type of quality being targeted (e.g., reliability, maintainability, security, etc.). The authors concluded that metric thresholds should not remain constant regardless of the quality of interest, and instead they should be tailored to the objectives of practitioners in a given project. Although we think Lavazza & Morasca suggestions are reasonable, our main goal is to determine whether our tool is able to build consistent samples rather than prescribing quality thresholds for specific metrics to practitioners. In our case, the topic of metric thresholds serves as a means to validate the effectiveness of the sampling strategies designed. 4. Study design To conduct our investigation, we followed the guidelines for MSR studies proposed by Vidoni (2022). In this section, we present the research questions, provide details about the data collection process and results of the protocol execution, and describe the measures and data analysis process to answer the questions. 4.1. Planning 4.1.1. Research Questions RQ1. Is it necessary to update software samples? We compare the thresholds of three source code metrics (Lines of Code, Cyclomatic Complexity and Weighted Methods per Class) derived from the benchmark dataset Qualitas Corpus (which was assembled over a decade ago), and a current sample generated with our procedure. We aim to assess the temporal validity of a highly considered benchmark dataset, and to determine whether after some time a proper update procedure is required. RQ2. Is it possible to generate consistent samples? We determine whether samples generated with our tool, either from scratch or by updating an existing sample, exhibit consistent results. Like in RQ1, we compare the two samples using their thresholds for the same three source code metrics. 4.1.2. Data Collection We employed two sources of information: the Qualitas Corpus (Tempero et al., 2010) and Github. The former served as the outdated codebase while the latter provided recent development data. We considered Github as a source of recent development data because it allows us to query the repositories by programming language. We retrieved the projects metadata using its REST[5] and GraphQL[6] API services, and the source code from repositories’ URL. The repository data collected for our study are presented in Table 1 along with their descriptions and rationale. Table 1 Retrieved data Id. Meta-data Description Rationale NAME name Repository’s name Keyword filtering OWN owner Repository’s owner URL url Repository’s url Repository access PL primaryLanguage Programming languages Identify repository’s main programming language SIZE totalSize Code size in KB Identify repository’s java code size PRIV isPrivate Repository accessibility Determine repository’s availability MIRR isMirror Mirrored repository Determine if the repository is original CONTR contributors Number of contributors Quantify the collaboration CPR closedPullReqCount Number of closed pull-requests MPR mergedPullReqCount Number of merged pull-requests COMM commits Number of commits Quantify the history CREAT createdAt Creation date CI closedIssuesCount Number of closed issues Quantify the issues STARS stargazerCount Number of stars Quantify the popularity FORKS forkCount Number of forks COMMD dateLastCommit Last commit date Detect recent activity CPRD closedPullReqLastDate Last closed pull-request date MPRD mergedPullReqLastDate Last merged pull-request date ARCH isArchived Archived repository CODE code Project source code Calculate code metrics We filtered the repositories that might not be suitable for Software Engineering research (Munaiah et al., 2017). The filtering process consisted of discarding repositories that did not meet the thresholds in Table 2. These thresholds were based on the results of secondary studies (Carruthers et al., 2022b, 2022a), and recommendations of Munaiah et al. (2017), Kalliamvakou et al. (2014), and Lewowski & Madeyski (2020). To avoid unwanted repositories that might surpass the quality thresholds (e.g., demo, tutorials or configuration repositories), we excluded repositories including any of the keywords: benchmark, conf, demo, docs, exam, guide, sample, template, tutorial, and wiki. Table 2 Quality thresholds and criteria Id. Threshold Criterion T1 Java programming language. Java projects. T2 Public repositories. Repository meta-data available. T3 GIT repositories. Support processes tools usage. T4 50 closed issues or more T5 Repositories that are not mirrors. Original projects T6 10000 lines of code or more. Size T7 3 or more contributors Collaboration T8 50 pull-requests or more. T9 1 or more years since creation. History T10 1000 commits or more. T11 10 or more forks. Popularity T12 10 or more stars. T13 1 or more commits or merged/closed pull-request in the last month. Recent activity T14 1 or more commits per month in the last year. T15 Not archived repository 4.1.3. Generated Samples We generated three different samples to evaluate our approach: an outdated sample (Qualitas Corpus), a sample with current data (Current Sample), and an updated version of the outdated sample (Qualitas Updated). All samples were equally sized with 112 subjects each. Qualitas Corpus This sample corresponds to the last release of the Qualitas Corpus from September 2013. We acquired the source code from the r distribution package available in the official website of the dataset[7] and searched for the access links to the projects on GitHub. Current Sample Based on the dataset obtained from Github, we generated the sample using a stratified random sampling strategy. This method involves dividing the population into non-overlapping sub-populations, called strata, and randomly selecting elements from each stratum (Lohr, 2021). Stratified sampling is preferred when the sub-populations may have different mean values for the targeted variables, providing lower variance estimates for the whole population (Lohr, 2021). The number of elements selected from each stratum was proportional to the number of subjects in that group relative to the Github dataset. We implemented the selection procedure by first generating the strata (or groups) based on the repository size (SIZE) due to its correlation with the variables of interest at a project level (namely LOC, McCC and WMC metrics) reflected by the Spearman's correlation coefficients over 0.93 in Fig. 1. The groups were generated using k-means clustering algorithm (Bugayenko et al., 2023; Wu et al., 2024). K-means divides m datapoints in n dimensions into k clusters seeking to minimize the sum of squares within cluster (Hartigan & Wong, 1979). In our case, the datapoints correspond to the SIZE score of each project in the dataset retrieved from Github. Before applying the algorithm, we isolated projects with extreme SIZE values with the interquartile outlier detection rule. This rule states that any values greater than the sum of the third quartile and 1.5 times the interquartile range (third quartile – first quartile) are excluded (Vinutha et al., 2018). With the list of subjects after removing outliers, we applied k-means with five groups. We determined the optimum number of clusters using the elbow method (Tibshirani et al., 2001). Once we obtained the groups, we randomly selected the projects from each group proportionally according to the stratum size and population. For example, if the population comprised 800 subjects and the groups contained 170, 180, 220, 100, and 130 subjects, in a sample of 100 projects, we select 21, 22, 28, 13 and 16 projects from each group respectively. We then included a proportional subset of the projects that were previously separated as outliers into the resulting sample. Fig. 2 illustrates the implemented approach. Qualitas Updated The construction of the updated sample began with the selection of the outdated or original dataset, i.e., the Qualitas Corpus described earlier. We identified projects in the original sample that were also included in the Github dataset. If the elements of the original sample had recent activity, we proceeded to update them in the sample with their latest version. The remaining elements that did not have recent updates were discarded. To replace the discarded projects, we applied a replacement process similar to the stratified random sampling explained for the Current Sample. Firstly, we generated the strata from the Github dataset in the same manner as the Current Sample, separating outliers and applying k-means with five groups to create the strata. Secondly, we recorded the number of projects from each stratum existing in the updated sample (excluding the discarded repositories), and computed the difference with the proportional number of elements corresponding to the equivalent stratum in the Github dataset. Finally, we selected the recent projects according to the differences recorded by group in the previous step. The resulting sample comprised the same number of subjects as the original sample. Fig. 3 summarizes the approach implemented to update the Qualitas. 4.2. Execution For the extraction, we developed a Python script according to the specifications defined in the planning stage. An initial query on Github services resulted in 15868 Java projects, and a second filtering reduced those to 892 subjects. As already mentioned in Section 4.1.3, the samples comprised 112 projects and, in some cases (Current and Qualitas Updated datasets), they were generated from the Github dataset. The complete dataset used in the study, as well as the scripts to extract the data and generate the graphs, are hosted in Zenodo1 and Github[8] repositories. 4.2.1. Threshold derivation To obtain the thresholds for the samples, we retrieved the source code of the projects and calculated three metrics (see Table 3): two at method level, McCC and LOC, and one at class level, WMC. As reviewed in secondary studies (Carruthers et al., 2022b, 2022a), these metrics are commonly used in Software Engineering activities such as defect detection, software maintenance, effort estimation, among others. We used Sourcemeter[9] to analyze Java code, similarly to other authors on the subject (Ferenc et al., 2020; Mahdieh et al., 2020; Mehboob & Chong, 2023). Table 3 Analyzed code metrics Id Description Metric type LOC Method lines of code Size McCC Cyclomatic complexity Complexity WMC Weighted methods per class For threshold derivation of the metrics of interest, we implemented the statistical data-based approach proposed by Alves et al. (2010). The technique computes a cumulative distribution of the studied metric and establishes three reference values: lower (70%), medium (80%) and upper bound (90%). These values delimit four risk categories for classifying a component of a system (method or class): low risk ( 90%). The risk categorization refers to the degree of urgency for the code to be refactored: medium risk requires long-term attention, high risk requires medium-term attention, and very high risk requires immediate attention. For instance, using WMC thresholds from the Current Sample in Fig. 4, the values 50, 80 and 148 correspond to lower, medium and upper thresholds respectively; therefore, classes recording less than 50 are low risk, between 50 and 80 are moderate, between 80 and 148 are high risk, and higher than 148 are very high risk. The method has proven to be viable in multiple Software Engineering contexts, such as defect prediction (Boucher & Badri, 2018), code smells in test code (Spadini et al., 2020), software product lines (Vale et al., 2019) and code smell detection (F. Ferreira & Valente, 2023). 4.2.2. Outlier detection We calculated and analyzed the cumulative distributions for each system to detect possible outliers. In subfigure a) of Fig. 5, gray curves represent the cumulative distributions of each individual project, and the black curve corresponds to the distribution of the complete sample. The graph helped us identify system curves being distant from the reference, with a clear example being the red line marked in subfigure b) of Fig. 5, which corresponds to the project hapifhir/org.hl7.fhir.core [10]. Once we identified the outliers, we replaced them with similar projects. In this case, we computed the Euclidean distance (Khatibi Bardsiri et al., 2014) in SIZE between the outlier and the other projects in the Github dataset, and the one that registered the closest value to zero was chosen as the replacement. Afterwards, we computed the metrics of the new projects and recalculated the sample thresholds. 4.3. Data Analysis In order to select the appropriate statistical tests, we analyzed the probability distributions of the code metrics under study. In previous studies (Ajienka et al., 2020; Capiluppi et al., 2020; Terragni et al., 2020) authors stated these three metrics have non-normal distributions, in particular, Arar & Ayan (2016) declared they follow a power law distribution, i.e., a heavy- tailed distribution where large values are very rare and lower values are very common. The probability distributions plotted in Fig. 6 and the p-values below 0.001 from Shapiro Wilk statistic (Shapiro & Wilk, 1965) further confirmed our assumptions of non-normality, thus we decided to use non-parametric tests. We assessed the RQs conducting a hypothesis test to ensure the statistical validity of the results at a significance level of α = 0.05. We performed the Kolmogorov-Smirnov test for two independent samples (KS), a non-parametric test suitable for ordinal data which requires the construction of cumulative frequency distributions (Sheskin, 2000). The test evaluates whether two samples came from the same population, if so, their cumulative frequency distributions would be expected to be identical or reasonably close to one another. The null hypothesis of the KS test is: F 1 (X) = F 2 (X) for all values of X , where F j (X) denotes the population distribution from which the j th sample is derived. When we derived the thresholds from a sample using Alves et al. (2010) method, a cumulative distribution function of the analyzed metric is generated. We used KS test to evaluate whether two distributions from different samples in the same dimension are statistically similar. Since we are examining the consistency of the two complete samples across all dimensions, if the hypothesis test for any of the studied metrics is rejected, the null hypothesis will also be rejected, i.e., the hypothesis on all metrics must be accepted for both samples to be considered similar or consistent. Fig. 7 shows the symbolic expression of the null hypothesis applied to both RQs, where F j (X) refers to the cumulative distributions derived from the samples being compared, and the variables enclosed by parenthesis correspond to the distribution of each specific metric. We compared the samples depending on the RQ. For RQ1 we evaluated the temporal validity of an outdated dataset, therefore we paired the Qualitas Corpus and the Current Sample to identify if after ten years the thresholds produced with Qualitas can be considered as valid for the analyzed dimensions. This scenario led to the following hypothesis: · H 0 : Qualitas ( X ) = Current ( X ). The data distribution of the Qualitas Corpus and the Current Sample retrieved from Github were derived from the same population. · H 1 : Qualitas ( X ) ≠ Current ( X ). The data distribution of the Qualitas Corpus and the Current Sample retrieved from Github were not derived from the same population. The second RQ aimed at assessing the effectiveness of the tool developed to generate and update samples, hence in this case we paired the Current and Qualitas Updated samples. This scenario led to the following hypothesis: · H 0 : Qualitas' ( X ) = Current ( X ). The data distribution of the updated version of the Qualitas Corpus and the Current Sample retrieved from Github were derived from the same population. · H 1 : Qualitas' ( X ) ≠ Current ( X ). The data distribution of the updated version of the Qualitas Corpus and the Current Sample retrieved from Github were not derived from the same population. [5] https://api.github.com [6] https://api.github.com/graphql [7] http://www.qualitascorpus.com/ [8] https://github.com/juancarruthers/thresholds_experiment [9] https://sourcemeter.com/ [10] https://github.com/hapifhir/org.hl7.fhir.core 5. Results In this section we answer the research questions based on the results obtained from the data analysis. 5.1. RQ1: Is it necessary to update software samples? Initially, we calculated thresholds for LOC, McCC and WMC, for both Qualitas and Current samples and presented them in Table 4. The analysis of the metric thresholds revealed substantial differences between the two samples, observing lower values for the Current Sample. Starting with LOC, Qualitas registered 46, 71 and 132 for lower, medium and upper thresholds respectively; while the Current Sample recorded 29, 43 and 77 at the same points. These values produced ratios ranging between 0.58 and 0.63 comparing samples on the same column. We observed a similar trend on complexity metrics, recording ratios between 0.55 and 0.64 in McCC, and between 0.56 and 0.62 in WMC. Table 4 Thresholds calculated for the Qualitas Corpus and the Current Sample Sample Metric Lower Medium Upper Qualitas Corpus LOC 46 71 132 McCC 7 11 22 WMC 81 136 265 Current Sample LOC 29 43 77 McCC 4 7 12 WMC 50 80 148 We spotted interesting comparisons when contrasting the values from different columns, such as, the lower thresholds of Qualitas Corpus and the medium thresholds of the Current Sample. For instance, a method with 45 LOC would be labeled as high risk with the thresholds from the Current Sample, but as low risk with thresholds from Qualitas. This trend was noticed for the other metrics as well, a class registering 80 on WMC would be categorized as high risk with the Current Sample, and low risk in Qualitas. Moreover, we extended the analysis conducting a KS test to compare the distributions of thresholds between the Qualitas and Current samples for each metric. The results, as presented in Table 5, indicated p-values below 0.05 for all metrics, leading to the rejection of the null hypothesis. These results further confirmed our initial assumption that the thresholds generated with these two samples have significant differences. Based on these findings, we deemed the Qualitas Corpus temporally invalid for the source code metrics LOC, McCC and WMC. Table 5 Kolmogorov-Smirnov test results between the thresholds of the Qualitas Corpus and the Current Sample Metric Qualitas Corpus and Current Sample comparison KS P-value LOC 0.1296 0 McCC 0.1272 0.0084 WMC 0.1332 0 5.2. RQ2: Is it possible to generate consistent samples? Like in RQ1, we computed the thresholds of two samples: Qualitas Updated and Current Sample, but in this case both of them were created with our data extraction tool. In Table 6 we registered lower, medium, and upper thresholds of the sample for each metric. Upon comparing these thresholds across metrics, we detected marginal differences between the two. In particular, McCC thresholds were identical with lower, medium and upper values of 4, 7 and 12 respectively. Table 6 Thresholds calculated for the Qualitas Updated and the Current Sample Sample Metric Lower Medium Upper Qualitas Updated LOC 29 42 74 McCC 4 7 12 WMC 52 80 151 Current Sample LOC 29 43 77 McCC 4 7 12 WMC 50 80 148 Then again, we conducted a KS test to compare the distributions of thresholds between the Qualitas Updated and the Current Sample and summarized them in Table 7. The KS test results denoted p-values over 0.98 for all metrics, thus the null hypothesis from Fig. 7 in this case was accepted, suggesting there are not significant differences between the thresholds of the Qualitas Updated and the Current Sample. This implies that the procedure is capable of generating consistent samples that closely resemble each other in terms of the source code metrics LOC, McCC, and WMC. Table 7 Kolmogorov-Smirnov test results between the thresholds of the Qualitas Updated and the Current Sample Metric Qualitas Updated and Current Sample comparison KS P-value LOC 0.0204 0.9869 McCC 0.0118 1 WMC 0.0111 1 6. Discussion In this section we interpret the findings above and discuss them in the context of the research questions and the literature on the subject. 6.1. Metrics Thresholds We used the data-driven approach proposed by Alves et al. (2010) to derive thresholds for three source code metrics: LOC, McCC, and WMC. We applied the method to obtain a set of thresholds from two datasets, the 2013 last release of the Qualitas Corpus(Tempero et al., 2010), and a current sample from Github. The statistical analysis performed with KS tests revealed significant differences between the two datasets in the distribution of all metrics, raising concerns about Qualitas’ temporal validity and if it remains suitable for current MSR investigations that focus on these properties. Furthermore, comparing the lower, medium and higher thresholds of the datasets we observed a notable decline in these reference values to such an extent that code components categorized as low risk with Qualitas thresholds would be currently identified as high-risk, i.e., methods or classes that should be refactored in the medium term. The downward trend of the complexity and size of software components can be explained by the general consensus in the literature that development tasks get more challenging as the codebase grows in size and complexity, motivating development teams to actively monitor and reduce its complexity over time. Researchers have discovered associations between systems external quality such as, reliability, maintainability or security; and code complexity/size (Iftikhar et al., 2024). For instance, a positive correlation between software size and complexity, and error probability has been reported (Shatnawi & Li, 2008), i.e., when code complexity increases the probability of more errors in a component also increases, which turns systems less reliable (Sehgal et al., 2020). In this sense, Lavazza & Morasca conducted several reliability-related studies using these metrics to build prediction models and also metrics thresholds (Lavazza & Morasca, 2016a; Morasca & Lavazza, 2016, 2019). Regarding maintainability, the development industry has internalized that clean code is often easier to change than convoluted code. Many code smells -symptoms of poor design and implementation choices (Fowler et al., 2002)-, like Long Method , Complex Class , Spaghetti Code , Blob ,etc. (Kermansaravi et al., 2021; Palomba et al., 2018) are considered as such due to their association with the complexity or size of components. Also, code clones (another type of code smell) are worth mentioning, as they increase the complexity and scale the existing code (Mo et al., 2021). Regarding software security, experts have claimed that complexity hides bugs that may result in security vulnerabilities that attackers can take advantage of (Reis et al., 2021). Additionally, we observed that the LOC and McCC thresholds obtained from the current sample approximate the values reported by Alves et al. (2010) despite the differences in the number of subjects and their inclusion of C# projects. For LOC, they recorded lower, medium and higher thresholds of 30, 44 and 74 respectively, and for McCC 6, 8 and 14. Interestingly, even when comparing McCC to the reference value of 10 established by Mccabe (1976), the values are not that far apart. 6.2. Sampling procedure assessment In RQ2, we evaluated the effectiveness of our sampling and update strategies to generate current datasets. To that end, we constructed two datasets with our approach: one from scratch and an updated version of the Qualitas Corpus. Subsequently, we computed thresholds of three source code metrics using these datasets. Both types of sampling extracted active repositories from Github. We statistically compared these thresholds and the null hypothesis was accepted; nonetheless, these values alone do not fully convey the significance of the similarity. Fig. 8 plots the thresholds of the three samples analyzed in the study. The blue and orange curves represent the samples created with our tool showing they mostly overlap from beginning to end. In contrast, the green curve, corresponding to the thresholds of Qualitas Corpus, has a noticeable distance from the other two curves. In Section 2, we explained our position regarding other tools created for sampling and mining open-source projects, such as their low chances to remain online after some time and the resource constraints associated with deploying such large platforms. Our approach resembles that of Lewowski & Madeyski (2020), who developed and distributed a runnable script allowing the dynamic creation of the datasets. However, there are key differences in their implementation: their quality selection criteria filtered by four repository metrics (stars, forks, commits and total size), they omitted source code metrics and the implementation of an update strategy. In summary, in this study we evaluated our tool to build and update current samples. We analyzed graphically and statistically the thresholds of the samples returned from the implemented approaches for three source code metrics: LOC, McCC and WMC; demonstrating its effectiveness. Also, we highlighted its comparative advantages with other tools reported in the literature. 7. Threats to Validity In this section, we discuss the threats to validity identified for our work, according to the four types of validity suggested by Wohlin et al. (2012). 7.1. Construct Validity Construct validity is concerned about generalizing the result to the concept or theory behind the study, i.e. the relationship between theory and observation. RQ1 investigated the impact of temporal validity in software samples, and RQ2 evaluated the effectiveness of two strategies generating samples that yield similar results. To perform the comparisons, we relied on three broadly accepted measures of source code complexity and size: Lines of Code, Cyclomatic Complexity and Weighted Method per Class. We centered our analysis on the specific dimensions due to their natural tendency to evolve over time (Caneill et al., 2017; Hatton et al., 2017; Rousseau et al., 2020). While these are not the only metrics that describe software complexity and size, they provide a foundation to evaluate the implication of temporal validity and the viability of our sampling strategies. Another potential threat lies in the projects selected for analysis. On one hand, the Qualitas Corpus(Tempero et al., 2010) is a regularly cited dataset and we considered as an adequate sample to represent the population of quality software in the moment it was last updated (2013). On the other hand, to assure no toy projects were collected for the samples obtained from Github, we defined the thresholds for quality repositories in Section 4.1.2, and manually reviewed the list of 892 repositories retrieved by the data extraction tool. 7.2. Internal Validity Internal validity is threatened by external influences that we did not, or were not able to consider when trying to infer cause-effect relationships. In the context of two RQ, we compared the thresholds obtained from three datasets: Qualitas Corpus , and two samples generated with our sampling strategies. To reduce the bias probability, we took the following precautions: all the samples studied were equally sized, the subjects identified as outliers were replaced, the method applied to derive the thresholds was the same across all datasets and no additional weights were assigned on the projects (e.g., size, community, popularity, etc.) for the generated thresholds. However, RQ1 compared the thresholds obtained from two samples collected more than a decade apart, and other external factors might have influenced the reference values instead of temporal validity. These external factors could introduce changes in the dataset independently of the variables being studied. For example, changes in the software development industry, technological advancements, or shifts in user preferences. Although time itself might not be the direct cause of the observed changes, these external factors are inherently tied to the passage of time. Therefore, to some extent, time can be considered a contributing factor to these changes. Moreover, another potential threat in RQ2 stems from the approach performed to update the Qualitas Corpus. Our update strategy relied exclusively on Github as the source of current repositories, and it is possible that some projects in the Qualitas Corpus have recent versions available but in different hosting locations. 7.3. External Validity External validity is concerned with the generalizability of the conclusions of the study. We tackled this threat using a probabilistic approach called stratified random sampling strategy as recommended by Baltes & Ralph (2022) and Cosentino et al. (2017) and tested recently by other empirical studies (Gorostidi et al., 2024). Respecting the samples created with our tool, we provided evidence of them being generalizable to the target population according to the reported results in RQ2. Furthermore, our dataset only contains open-source Java projects hosted on GitHub, based on findings of our secondary studies (Carruthers et al., 2022b, 2022a). Although there are more code sharing platforms, Github established itself as one of the main sources for empirical research (Dabic et al., 2021; Munaiah et al., 2017), being the largest in terms of public open-source projects, therefore we can argue our results approximate to reality. Nevertheless, the generalization to other programming languages and proprietary projects is limited, and further studies might be needed to confirm our results. 7.4. Conclusion Validity Threats to the conclusion validity are concerned with issues that affect the ability to draw the correct conclusion about relations between the treatment and the outcome of an experiment. We addressed low statistical power using samples with 112 projects per dataset, which is enough statistical power (0.8) to detect medium to small effect sizes. This was calculated with the statistical software G*Power (Heradio et al., 2022) for goodness of fit tests. For statistical tests, we employed both graphical and quantitative approaches to determine which inferential test was suitable. Consequently, we applied two-sample KS non-parametric test to assess whether the cumulated distributions of the datasets’ thresholds came from the same population. 8. Conclusions In this article, we conducted a Mining Software Repository study to evaluate the effectiveness of a dataset retrieval tool. Employing three distinct datasets -two samples generated through our data extraction tool and the Qualitas Corpus- we derived thresholds for three source code metrics: Lines of Code, Cyclomatic Complexity and Weighted Methods per Class. The study was structured around two research questions oriented to determine the capability of our tool for generating current samples, and to assess the temporal validity of the involved datasets. To highlight the value of our data extraction tool, we paired the thresholds derived from the Qualitas Corpus, which its last update was a decade ago, with those obtained from a sample created using our tool. The results revealed significant differences between the threshold pairs reflecting a temporal validity loss for Qualitas Corpus after ten years since its last update. Notably, components that were previously categorized as low risk based on the Qualitas thresholds would be deemed as high-risk today. For the evaluation of the tool effectiveness, we compared the thresholds computed for the samples generated by the instrument. The statistical analysis of the thresholds did not show significative differences in any metric. In particular, the Cyclomatic Complexity reference values were identical for both samples. These findings were corroborated by the graphical representation of the cumulative distributions depicting overlapping curves throughout. The main contribution in this study is the introduction of a data extraction tool to construct and update samples of software projects, positioning it as a viable option to generate current samples in the context of code-related MSR studies. Additionally, we assessed the temporal validity of a benchmark dataset such as the Qualitas Corpus, highlighting how much outdated data can impact the results of a study, and the importance of implementing effective strategies to update datasets over time. As a future work, we aim to enhance our approach and validate its results through a series of replication studies in collaboration with external research groups. Declarations Author Contributions J.A.C. led the conceptualization, developed the methodology and software, curated the data, conducted validation, and wrote the original draft of the manuscript. A.L.A. was responsible for visualization and contributed to reviewing and editing the manuscript. J.A.D.P. supported data curation, supervision, and manuscript review and editing. E.I. contributed to the conceptualization and methodology, provided supervision, and participated in manuscript review and editing. All authors reviewed and approved the final version of the manuscript. Data Availability Original dataset and replication scripts for this research are publicly available https://doi.org/10.5281/zenodo.15008288. Funding This work was supported by the National Council on Scientific and Technical Research (CONICET) under a PhD Fellowship (RESOL-2021-154-APN-DIR#CONICET) and the National University of the North-East (SCyT-UNNE) under Grant 21F001. It is a part of the research conducted under the Computer Science Doctorate Program at UNNE, UNaM, and UTN. Competing interests The authors have no competing interests to declare. References Ait, A., Izquierdo, J. L. C., & Cabot, J. (2022). An empirical study on the survival rate of GitHub projects. Proceedings of the 19th International Conference on Mining Software Repositories , 365–375. https://doi.org/10.1145/3524842.3527941 Ajienka, N., Vangorp, P., & Capiluppi, A. (2020). An empirical analysis of source code metrics and smart contract resource consumption. Journal of Software: Evolution and Process . https://doi.org/10.1002/smr.2267 Alves, T. L., Ypma, C., & Visser, J. (2010). Deriving metric thresholds from benchmark data. 2010 IEEE International Conference on Software Maintenance , 1–10. https://doi.org/10.1109/ICSM.2010.5609747 Amasaki, S., Aman, H., & Yokogawa, T. (2020). An exploratory study on applicability of cross project defect prediction approaches to cross-company effort estimation. PROMISE 2020 - Proceedings of the 16th ACM International Conference on Predictive Models and Data Analytics in Software Engineering, Co-Located with ESEC/FSE 2020 , 71–80. https://doi.org/10.1145/3416508.3417118 Arar, Ö. F., & Ayan, K. (2016). Deriving thresholds of software metrics to predict faults on open source software: Replicated case studies. Expert Systems with Applications , 61 , 106–121. https://doi.org/10.1016/j.eswa.2016.05.018 Arcelli Fontana, F., Mäntylä, M. V., Zanoni, M., & Marino, A. (2016). Comparing and experimenting machine learning techniques for code smell detection. Empirical Software Engineering , 21 (3), 1143–1191. https://doi.org/10.1007/s10664-015-9378-4 Artifact Review and Badging . (2020). ACM. https://www.acm.org/publications/policies/artifact-review-and-badging-current Baltes, S., & Ralph, P. (2022). Sampling in software engineering research: a critical review and guidelines. Empirical Software Engineering , 27 (4), 94. https://doi.org/10.1007/s10664-021-10072-8 Boucher, A., & Badri, M. (2018). Software metrics thresholds calculation techniques to predict fault-proneness: An empirical comparison. Information and Software Technology , 96 , 38–67. https://doi.org/10.1016/J.INFSOF.2017.11.005 Brito, E., Abreu, F., & Carapuça, R. (1994). Object-Oriented Software Engineering: Measuring and Controlling the Development Process. 4th Int. Conf. on Software Quality , 1–8. Bugayenko, Y., Daniakin, K., Farina, M., Kholmatova, Z., Kruglov, A., Pedrycz, W., & Succi, G. (2023). Qualitative Clustering of Software Repositories Based on Software Metrics. Ieee Access : Practical Innovations, Open Solutions , 11 , 14716–14727. https://doi.org/10.1109/ACCESS.2023.3244495 Caneill, M., Germán, D. M., & Zacchiroli, S. (2017). The Debsources Dataset: two decades of free and open source software. Empirical Software Engineering , 22 (3), 1405–1437. https://doi.org/10.1007/s10664-016-9461-5 Capiluppi, A., Ajienka, N., & Counsell, S. (2020). The effect of multiple developers on structural attributes: A Study based on java software. Journal of Systems and Software , 167 , 110593. https://doi.org/10.1016/j.jss.2020.110593 Carruthers, J. A., Diaz-Pace, J. A., & Irrazábal, E. (2024). A longitudinal study on the temporal validity of software samples. Information and Software Technology , 168 , 107404. https://doi.org/10.1016/j.infsof.2024.107404 Carruthers, J. A., Diaz-Pace, J. A., & Irrazabal, E. A. (2022a). A Systematic Mapping Study of Empirical Studies Performed with Collections of Software Projects. Computación y Sistemas , 26 (4). https://doi.org/10.13053/cys-26-4-3981 Carruthers, J. A., Diaz-Pace, J. A., & Irrazabal, E. A. (2022b). How are software datasets constructed in Empirical Software Engineering studies? A systematic mapping study. 2022 48th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) , 442–450. https://doi.org/10.1109/SEAA56994.2022.00075 Chidamber, S. R., & Kemerer, C. F. (1994). A metrics suite for object oriented design. IEEE Transactions on Software Engineering , 20 (6), 476–493. https://doi.org/10.1109/32.295895 Coelho, J., Valente, M. T., Milen, L., & Silva, L. L. (2020). Is this GitHub project maintained? Measuring the level of maintenance activity of open-source projects. Information and Software Technology , 122 , 106274. https://doi.org/10.1016/j.infsof.2020.106274 Coleman, D., Lowther, B., & Oman, P. (1995). The application of software maintainability models in industrial software systems. Journal of Systems and Software , 29 (1), 3–16. https://doi.org/10.1016/0164-1212(94)00125-7 Cosentino, V., Izquierdo, C., J. L., & Cabot, J. (2017). A Systematic Mapping Study of Software Development With GitHub. Ieee Access : Practical Innovations, Open Solutions , 5 , 7173–7192. https://doi.org/10.1109/ACCESS.2017.2682323 Dabic, O., Aghajani, E., & Bavota, G. (2021). Sampling Projects in GitHub for MSR Studies. 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR) , 560–564. https://doi.org/10.1109/MSR52588.2021.00074 Dyer, R., Nguyen, H. A., Rajan, H., & Nguyen, T. N. (2013). Boa: A language and infrastructure for analyzing ultra-large-scale software repositories. 2013 35th International Conference on Software Engineering (ICSE) , 422–431. https://doi.org/10.1109/ICSE.2013.6606588 Falessi, D., Smith, W., & Serebrenik, A. (2017). STRESS: A Semi-Automated, Fully Replicable Approach for Project Selection. International Symposium on Empirical Software Engineering and Measurement , 2017-Novem , 151–156. https://doi.org/10.1109/ESEM.2017.22 Fenton, N., & Bieman, J. (2014). Software Metrics (Vol. 55). Third. https://doi.org/10.1201/b17461 Ferenc, R., Tóth, Z., Ladányi, G., Siket, I., & Gyimóthy, T. (2020). A public unified bug dataset for java and its assessment regarding metrics and bug prediction. Software Quality Journal , 28 (4), 1447–1506. https://doi.org/10.1007/s11219-020-09515-0 Ferreira, F., & Valente, M. T. (2023). Detecting code smells in React-based Web apps. Information and Software Technology , 155 , 107111. https://doi.org/10.1016/j.infsof.2022.107111 Ferreira, K., Bigonha, M. A. S., Bigonha, R. S., Mendes, L. F. O., & Almeida, H. C. (2012). Identifying thresholds for object-oriented software metrics. Journal of Systems and Software , 85 (2), 244–257. https://doi.org/10.1016/j.jss.2011.05.044 Fowler, M., Beck, K., Brant, J., Opdyke, W., & Roberts, D. (2002). Refactoring: Improving the Design of Existing Code . Gonzalez-Barahona, J. M., & Robles, G. (2023). Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositories. Information and Software Technology , 164 , 107318. https://doi.org/10.1016/j.infsof.2023.107318 Gorostidi, J., Ait, A., Cabot, J., & Canovas Izquierdo, J. L. (2024). On the Creation of Representative Samples of Software Repositories. Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement , 434–439. https://doi.org/10.1145/3674805.3690747 Gousios, G., & Spinellis, D. (2012). GHTorrent: Github’s data from a firehose. 2012 9th IEEE Working Conference on Mining Software Repositories (MSR) , 12–21. https://doi.org/10.1109/MSR.2012.6224294 Hartigan, J. A., & Wong, M. A. (1979). Algorithm AS 136: A K-Means Clustering Algorithm. Applied Statistics , 28 (1), 100. https://doi.org/10.2307/2346830 Hatton, L., Spinellis, D., & van Genuchten, M. (2017). The long-term growth rate of evolving software: Empirical results and implications. Journal of Software: Evolution and Process , 29 (5), e1847. https://doi.org/10.1002/smr.1847 Heradio, R., Fernandez-Amoros, D., Galindo, J. A., Benavides, D., & Batory, D. (2022). Uniform and scalable sampling of highly configurable systems. Empirical Software Engineering , 27 (2), 44. https://doi.org/10.1007/s10664-021-10102-5 Herbold, S., Grabowski, J., & Waack, S. (2011). Calculation and optimization of thresholds for sets of software metrics. Empirical Software Engineering , 16 (6), 812–841. https://doi.org/10.1007/s10664-011-9162-z Iftikhar, U., Ali, N., Bin, Börstler, J., & Usman, M. (2024). A tertiary study on links between source code metrics and external quality attributes. Information and Software Technology , 165 , 107348. https://doi.org/10.1016/j.infsof.2023.107348 Jose Thiago, J. T. H., & Oliveira, A. L. I. (2021). Ensemble Effort Estimation using dynamic selection. Journal of Systems and Software , 175 , 110904. https://doi.org/10.1016/J.JSS.2021.110904 Kalliamvakou, E., Gousios, G., Blincoe, K., Singer, L., German, D. M., & Damian, D. (2014). The promises and perils of mining GitHub. Proceedings of the 11th Working Conference on Mining Software Repositories , 92–101. https://doi.org/10.1145/2597073.2597074 Kearns, M. (1998). Efficient noise-tolerant learning from statistical queries. Journal of the ACM (JACM) , 45 (6), 983–1006. https://doi.org/10.1145/293347.293351 Kermansaravi, Z. A., Rahman, M. S., Khomh, F., Jaafar, F., & Guéhéneuc, Y. G. (2021). Investigating design anti-pattern and design pattern mutations and their change- and fault-proneness. Empirical Software Engineering , 26 (1), 1–47. https://doi.org/10.1007/S10664-020-09900-0 Khatibi Bardsiri, V., Jawawi, D. N. A., Hashim, S. Z. M., & Khatibi, E. (2014). A flexible method to estimate the software development effort based on the classification of projects and localization of comparisons. Empirical Software Engineering , 19 (4), 857–884. https://doi.org/10.1007/s10664-013-9241-4 Lanza, M., & Marinescu, R. (2006). Object-Oriented Metrics in Practice . Springer. https://doi.org/10.1007/3-540-39538-5 Lavazza, L., & Morasca, S. (2016a). An Empirical Evaluation of Distribution-based Thresholds for Internal Software Measures. Proceedings of the The 12th International Conference on Predictive Models and Data Analytics in Software Engineering , 1–10. https://doi.org/10.1145/2972958.2972965 Lavazza, L., & Morasca, S. (2016b). Identifying Thresholds for Software Faultiness via Optimistic and Pessimistic Estimations. International Symposium on Empirical Software Engineering and Measurement , 08-09-Sept , 1–10. https://doi.org/10.1145/2961111.2962595 Lewowski, T., & Madeyski, L. (2020). Creating Evolving Project Data Sets in Software Engineering. Studies in Computational Intelligence (Vol. 851, pp. 1–14). Springer. https://doi.org/10.1007/978-3-030-26574-8_1 Linstead, E., Bajracharya, S., Ngo, T., Rigor, P., Lopes, C., & Baldi, P. (2009). Sourcerer: Mining and searching internet-scale software repositories. Data Mining and Knowledge Discovery , 18 (2), 300–336. https://doi.org/10.1007/s10618-008-0118-x Lohr, S. L. (2021). Sampling: design and analysis . Chapman and Hall/CRC. https://doi.org/10.1201/9780429298899 Lorenz, M., & Kidd, J. (1994). Object-oriented software metrics: a practical guide . Prentice-Hall. Ma, Y., Dey, T., Bogart, C., Amreen, S., Valiev, M., Tutko, A., Kennard, D., Zaretzki, R., & Mockus, A. (2021). World of code: enabling a research workflow for mining and analyzing the universe of open source VCS data. Empirical Software Engineering , 26 (2), 22. https://doi.org/10.1007/s10664-020-09905-9 Mahdieh, M., Mirian-Hosseinabadi, S. H., Etemadi, K., Nosrati, A., & Jalali, S. (2020). Incorporating fault-proneness estimations into coverage-based test case prioritization methods. Information and Software Technology , 121 , 106269. https://doi.org/10.1016/j.infsof.2020.106269 Mccabe, T. J. (1976). A Complexity Measure. IEEE Transactions on Software Engineering , SE-2 (4), 308–320. https://doi.org/10.1109/TSE.1976.233837 Mehboob, B., & Chong, C. Y. (2023). A metadata driven process for assessing stability and reusability based on risk of change of software systems. Software: Practice and Experience , 53 (5), 1218–1248. https://doi.org/10.1002/spe.3187 Mo, R., Zhao, Y., Feng, Q., & Li, Z. (2021). The existence and co-modifications of code clones within or across microservices. International Symposium on Empirical Software Engineering and Measurement , 1–11. https://doi.org/10.1145/3475716.3475784 Morasca, S. (2009). A probability-based approach for measuring external attributes of software artifacts. 2009 3rd International Symposium on Empirical Software Engineering and Measurement , 44–55. https://doi.org/10.1109/ESEM.2009.5316048 Morasca, S., & Lavazza, L. (2016). Slope-based fault-proneness thresholds for software engineering measures. International Conference on Evaluation and Assessment in Software Engineering , 01-03-June , 1–10. https://doi.org/10.1145/2915970.2915997 Morasca, S., & Lavazza, L. (2019). Comparing the effectiveness of using design and code measures in software faultiness estimation. International Conference on Evaluation and Assessment in Software Engineering , 112–121. https://doi.org/10.1145/3319008.3319026 Munaiah, N., Kroh, S., Cabrey, C., & Nagappan, M. (2017). Curating GitHub for engineered software projects. Empirical Software Engineering , 22 (6), 3219–3253. https://doi.org/10.1007/s10664-017-9512-6 Munger, K. (2019). The Limited Value of Non-Replicable Field Experiments in Contexts With Low Temporal Validity. Social Media + Society , 5 (3), 205630511985929. https://doi.org/10.1177/2056305119859294 Nejmeh, B. A. (1988). NPATH: a measure of execution path complexity and its applications. Communications of the ACM , 31 (2), 188–200. https://doi.org/10.1145/42372.42379 Ohira, M., Kashiwa, Y., Yamatani, Y., Yoshiyuki, H., Maeda, Y., Limsettho, N., Fujino, K., Hata, H., Ihara, A., & Matsumoto, K. (2015). A dataset of high impact bugs: Manually-classified issue reports. IEEE International Working Conference on Mining Software Repositories , 2015-Augus , 518–521. https://doi.org/10.1109/MSR.2015.78 Palomba, F., Bavota, G., Di Penta, M., Fasano, F., Oliveto, R., & De Lucia, A. (2018). A large-scale empirical study on the lifecycle of code smell co-occurrences. Information and Software Technology , 99 , 1–10. https://doi.org/10.1016/j.infsof.2018.02.004 Rankovic, D., Rankovic, N., Ivanovic, M., & Lazic, L. (2021). Convergence rate of Artificial Neural Networks for estimation in software development projects. Information and Software Technology , 138 , 106627. https://doi.org/10.1016/J.INFSOF.2021.106627 Reis, S., Abreu, R., & Cruz, L. (2021). Fixing vulnerabilities potentially hinders maintainability. Empirical Software Engineering , 26 (6), 1–27. https://doi.org/10.1007/S10664-021-10019-Z Rousseau, G., Di Cosmo, R., & Zacchiroli, S. (2020). Software provenance tracking at the scale of public source code. Empirical Software Engineering , 25 (4), 2930–2959. https://doi.org/10.1007/s10664-020-09828-5 Schnappinger, M., Fietzke, A., & Pretschner, A. (2021). Human-level Ordinal Maintainability Prediction Based on Static Code Metrics; Human-level Ordinal Maintainability Prediction Based on Static Code Metrics. Evaluation and Assessment in Software Engineering , 10 , 160–169. https://doi.org/10.1145/3463274 Sehgal, R., Mehrotra, D., & Nagpal, R. (2020). New Metrics for Predicting the Reliability of Individual Component Based on Software Design Metrics (pp. 79–94). https://doi.org/10.1007/978-981-15-2455-4_4 Shapiro, S. S., & Wilk, M. B. (1965). An Analysis of Variance Test for Normality (Complete Samples). Biometrika , 52 (3/4), 591. https://doi.org/10.2307/2333709 Shatnawi, R., & Li, W. (2008). The effectiveness of software metrics in identifying error-prone classes in post-release software evolution process. Journal of Systems and Software , 81 (11), 1868–1882. https://doi.org/10.1016/j.jss.2007.12.794 Sheskin, D. J. (2000). Parametric and Nonparametric Statistical Procedures (2nd ed.). Shull, F., Singer, J., & Sjøberg, D. I. K. (Eds.). (2008). Guide to Advanced Empirical Software Engineering . Springer London. https://doi.org/10.1007/978-1-84800-044-5 Spadini, D., Schvarcbacher, M., Oprescu, A. M., Bruntink, M., & Bacchelli, A. (2020). Investigating Severity Thresholds for Test Smells. Proceedings of the 17th International Conference on Mining Software Repositories , 311–321. https://doi.org/10.1145/3379597.3387453 Tempero, E., Anslow, C., Dietrich, J., Han, T., Li, J., Lumpe, M., Melton, H., & Noble, J. (2010). The Qualitas Corpus: A curated collection of Java code for empirical studies. Proceedings - Asia-Pacific Software Engineering Conference, APSEC , 336–345. https://doi.org/10.1109/APSEC.2010.46 Terragni, V., Salza, P., & Pezzè, M. (2020). Measuring Soft-ware Testability Modulo Test Quality . 11 (20). https://doi.org/10.1145/3387904.3389273 Tibshirani, R., Walther, G., & Hastie, T. (2001). Estimating the Number of Clusters in a Data Set Via the Gap Statistic. Journal of the Royal Statistical Society Series B: Statistical Methodology , 63 (2), 411–423. https://doi.org/10.1111/1467-9868.00293 Vale, G., Fernandes, E., & Figueiredo, E. (2019). On the proposal and evaluation of a benchmark-based threshold derivation method. Software Quality Journal , 27 (1), 275–306. https://doi.org/10.1007/s11219-018-9405-y Vidoni, M. (2022). A systematic process for Mining Software Repositories: Results from a systematic literature review. Information and Software Technology , 144 , 106791. https://doi.org/10.1016/j.infsof.2021.106791 Vinutha, H. P., Poornima, B., & Sagar, B. M. (2018). Detection of outliers using interquartile range technique from intrusion dataset. Advances in Intelligent Systems and Computing , 701 , 511–518. https://doi.org/10.1007/978-981-10-7563-6_53/FIGURES/5 Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J. W., da Silva Santos, L. B., Bourne, P. E., Bouwman, J., Brookes, A. J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C. T., Finkers, R., & Mons, B. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data , 2016 3:1 (1), 1–9. https://doi.org/10.1038/sdata.2016.18 . 3 . Wohlin, C., & Aurum, A. (2015). Towards a decision-making structure for selecting a research design in empirical software engineering. Empirical Software Engineering , 20 (6), 1427–1455. https://doi.org/10.1007/s10664-014-9319-7 Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012). Experimentation in Software Engineering (Vol. 9783642290). Springer. https://doi.org/10.1007/978-3-642-29044-2 Wu, Z., Wang, Z., Chen, J., You, H., Yan, M., & Wang, L. (2024). Stratified random sampling for neural network test input selection. Information and Software Technology , 165 , 107331. https://doi.org/10.1016/j.infsof.2023.107331 Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Reviews received at journal 09 Apr, 2026 Reviewers agreed at journal 08 Feb, 2026 Reviews received at journal 22 Oct, 2025 Reviewers agreed at journal 18 Aug, 2025 Reviewers agreed at journal 18 Aug, 2025 Reviewers invited by journal 16 Aug, 2025 Editor assigned by journal 28 Jul, 2025 Submission checks completed at journal 28 Jul, 2025 First submitted to journal 25 Jul, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7217702","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":503676863,"identity":"1b081490-6df0-4712-b80b-64e5a5c49a2e","order_by":0,"name":"Juan Andres Carruthers","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABHUlEQVRIie3RsUrDQBjA8e84uCwB1wuh9BUSApFg1GcpgeuSurhUKHJTu0SzxrcIuDocHCRLHkAnlUKmDoEsLYi2PSqInNDR4f7DcXzHjzs4AJPpH4b4YePtl0538hehe4KKY8h3imD7GIIX8r2/eZK3p9bdso+ncgh0Ur11EA9KgdtOQ1DGAqdpJY2yOnTThvmcXo39AlhQChIUOsJTcLiQ1HtmBE/mMeI0DV0b5KgUEOgehvIV3ijy2uI++qSXinzA145YvZYUKTncQsBFPB4pAiB2xNbfUrRhxMXYKRtGnKxiydxehU7mJcGDtK91xM+T5QsXZydeXeFuPZPnuZWGdD29GNzXi0ct4b8nRK3qm3QAYKgfm0wmk+lHWy7MYbdGNC1kAAAAAElFTkSuQmCC","orcid":"","institution":"Software Quality Research Group – FaCENA-UNNE","correspondingAuthor":true,"prefix":"","firstName":"Juan","middleName":"Andres","lastName":"Carruthers","suffix":""},{"id":503676864,"identity":"57d72d6f-5afc-4126-af30-1ae3b369b6ab","order_by":1,"name":"Andrea Lezcano Airaldi","email":"","orcid":"","institution":"Software Quality Research Group – FaCENA-UNNE","correspondingAuthor":false,"prefix":"","firstName":"Andrea","middleName":"Lezcano","lastName":"Airaldi","suffix":""},{"id":503676865,"identity":"6c363c3c-449a-4035-845c-6e1080208e91","order_by":2,"name":"Jorge Andres Diaz Pace","email":"","orcid":"","institution":"ISISTAN, CONICET-UNICEN","correspondingAuthor":false,"prefix":"","firstName":"Jorge","middleName":"Andres Diaz","lastName":"Pace","suffix":""},{"id":503676866,"identity":"a260539c-7434-4d82-9884-8846df08b93f","order_by":3,"name":"Emanuel Irrazábal","email":"","orcid":"","institution":"Software Quality Research Group – FaCENA-UNNE","correspondingAuthor":false,"prefix":"","firstName":"Emanuel","middleName":"","lastName":"Irrazábal","suffix":""}],"badges":[],"createdAt":"2025-07-26 00:53:08","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7217702/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7217702/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":89811020,"identity":"d027e770-804c-4507-b0bf-08319e1c2c16","added_by":"auto","created_at":"2025-08-25 09:56:18","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":53303,"visible":true,"origin":"","legend":"\u003cp\u003eScatter plots and correlation analysis of the\u003cstrong\u003e \u003c/strong\u003estudied code metrics (e.g. LOC, McCC and WMC) and repository size (SIZE)\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-7217702/v1/b03ba6d5db049f37a8ffa0f4.png"},{"id":89811141,"identity":"fc4e0c7f-8f0a-454d-ba95-fd1d9f424be3","added_by":"auto","created_at":"2025-08-25 10:04:18","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":53602,"visible":true,"origin":"","legend":"\u003cp\u003eStrategy to retrieve a current sample\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-7217702/v1/466f2d27de9a4a72072debc0.png"},{"id":89811022,"identity":"9932b922-23f3-4d68-9b15-903fae55f593","added_by":"auto","created_at":"2025-08-25 09:56:18","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":40301,"visible":true,"origin":"","legend":"\u003cp\u003eStrategy to update a sample\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-7217702/v1/b8c611e6e5d72cbe4f1b9720.png"},{"id":89811142,"identity":"8e1ca06d-85ef-4dbe-b244-008ae7953a97","added_by":"auto","created_at":"2025-08-25 10:04:18","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":15313,"visible":true,"origin":"","legend":"\u003cp\u003eWMC thresholds for the Current Sample\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-7217702/v1/8a7091a32d1a083cb300d910.png"},{"id":89811025,"identity":"4f489198-70b0-41d1-bbbf-75c3a272f5fb","added_by":"auto","created_at":"2025-08-25 09:56:18","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":60573,"visible":true,"origin":"","legend":"\u003cp\u003eVisual detection of the outliers\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-7217702/v1/03db01bcd2285fff35777c43.png"},{"id":89812919,"identity":"ae80a2e1-d068-4ea4-87e7-36eb64577bf2","added_by":"auto","created_at":"2025-08-25 10:12:18","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":35909,"visible":true,"origin":"","legend":"\u003cp\u003eProbability distributions of the metrics under study in the Current Sample\u003c/p\u003e","description":"","filename":"6.png","url":"https://assets-eu.researchsquare.com/files/rs-7217702/v1/6784c1ea84277fa7312f9295.png"},{"id":89811028,"identity":"cf134e79-a361-4c33-9105-b058c548f497","added_by":"auto","created_at":"2025-08-25 09:56:18","extension":"png","order_by":7,"title":"Figure 7","display":"","copyAsset":false,"role":"figure","size":4128,"visible":true,"origin":"","legend":"\u003cp\u003eNull hypothesis for the RQs\u003c/p\u003e","description":"","filename":"7.png","url":"https://assets-eu.researchsquare.com/files/rs-7217702/v1/d8fc5df4fd223b39cd4b9d52.png"},{"id":89813418,"identity":"0c2cbaa2-069d-4591-8710-44b594494f34","added_by":"auto","created_at":"2025-08-25 10:20:19","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1362670,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7217702/v1/879d7144-2d09-4a13-bb0b-73384ec432d1.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Temporal validity of software datasets for code metrics: an empirical assessment of sampling strategies","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eThe positivist paradigm is an objective approach that analyzes the world through empirical data, formal propositions, quantifiable measures of variables, hypothesis testing and drawing of inferences about a representative sample (Wohlin \u0026amp; Aurum, \u003cspan citationid=\"CR79\" class=\"CitationRef\"\u003e2015\u003c/span\u003e). In this context, results are reliable if they can be repeated by another researcher and reach a similar conclusion, and findings cannot be established until the study is independently repeated. According to the ACM policy for Artifact Review and Badging (2020) replicability means that \u003cem\u003ean independent group can obtain the same result using the author\u0026rsquo;s own artifacts\u003c/em\u003e. In that sense, the scientific community established the FAIR principles (Findability, Accessibility, Interoperability, and Reusability) (Wilkinson et al., \u003cspan citationid=\"CR78\" class=\"CitationRef\"\u003e2016\u003c/span\u003e), which apply not only to research data but also to the algorithms, tools, and workflows used to generate the datasets.\u003c/p\u003e\u003cp\u003eIn Empirical Software Engineering, there are specific research methods rooted in the positivist paradigm (e.g., experiments, mining software repository studies, systematic literature reviews, surveys) which require a sample dataset along with precise instructions to validate the study's results. The sample dataset should aim to be representative, i.e., it must closely resemble the underlying characteristics of the target population, which enables researchers to draw generalizable conclusions and extend them to the population under study (Shull et al., \u003cspan citationid=\"CR70\" class=\"CitationRef\"\u003e2008\u003c/span\u003e). However, replicating a study published several years ago using current data might yield inconsistent results if the population has undergone significant changes during that period, causing the original dataset to lose temporal validity.\u003c/p\u003e\u003cp\u003eTemporal validity refers to the extent to which findings or conclusions based on data collected at one point in time can be generalized to the present day (Carruthers et al., \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). Munger (\u003cspan citationid=\"CR58\" class=\"CitationRef\"\u003e2019\u003c/span\u003e) originally presents the concept in the domain of online social sciences and we extended to Software Engineering. In rapidly changing subjects, the time when the data is allocated can influence the results of a study. For instance, it is not the same to conduct an experiment with a dataset comprising source code developed more than 10 years ago than using a dataset with code from recently-developed projects. Several studies reported that the probability of a project remaining actively maintained for more than five years is below 50% (Ait et al., \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2022\u003c/span\u003e; Carruthers et al., \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Coelho et al., \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2020\u003c/span\u003e). In this sense, we argue that researchers should design and publish their datasets along with the sampling and update procedures to facilitate the assessment of these strategies and mitigate potential temporal validity issues in the future.\u003c/p\u003e\u003cp\u003eWhile it is ideal to evolve datasets over time, there are circumstances in which it is not feasible. Some datasets require human intervention, making automation difficult. For example, classification and labelling tasks intended for machine learning studies cannot always be automated (Arcelli Fontana et al., \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2016\u003c/span\u003e; Ohira et al., \u003cspan citationid=\"CR60\" class=\"CitationRef\"\u003e2015\u003c/span\u003e; Schnappinger et al., \u003cspan citationid=\"CR65\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). Another reason is the limited access to the type of data needed for the study. For instance, in software estimation datasets that contain process metrics from projects (Amasaki et al., \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2020\u003c/span\u003e; Jose Thiago \u0026amp; Oliveira, \u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e2021\u003c/span\u003e; Rankovic et al., \u003cspan citationid=\"CR62\" class=\"CitationRef\"\u003e2021\u003c/span\u003e), organizations must have gathered them in advance and make the data publicly accessible. In other cases the effort required to retrieve the data can pose restrictions for external investigators who might lack the necessary resources. However, there are datasets mainly containing source code, repository and code metrics, commit messages or issue reports, which can (and should) be evolved over time without the aforementioned barriers. Usually, to retrieve this kind of dataset, the designated API to make requests and fetch records is the only asset required.\u003c/p\u003e\u003cp\u003eThe main goal of this work is to evaluate the effectiveness of dataset retrieval approaches for collecting current samples from software repositories. To this end, we built a data extraction tool in the context of a Mining Software Repositories (MSR) study, aiming to assess whether the sampling techniques implemented return consistent results. The tool provides two strategies to build a dataset: sampling from scratch and dataset update, and we empirically evaluated and compared the outcomes of these strategies. The first strategy enables users to obtain a subset of the population, while the second restores a sample deemed as temporally invalid, updating the projects with recent activity and replacing the remaining ones with suitable candidates. Both strategies implement stratified random sampling based on the k-means clustering algorithm (Wu et al., \u003cspan citationid=\"CR81\" class=\"CitationRef\"\u003e2024\u003c/span\u003e).\u003c/p\u003e\u003cp\u003eRegarding the MSR study, we produced thresholds for three source code metrics: Lines of Code (LOC), Mccabe's Cyclomatic Complexity (McCC) and Weighted Methods per Class (WMC). Thresholds work as reference values to assess the state of the software components, indicating acceptable and abnormal ranges for specific metrics. We computed these thresholds using three datasets: the Qualitas Corpus (Tempero et al., \u003cspan citationid=\"CR72\" class=\"CitationRef\"\u003e2010\u003c/span\u003e), a current sample from Github, and an updated version of the Qualitas Corpus, and then statistically compared them to tackle two research questions.\u003c/p\u003e\u003cp\u003eThe findings of this paper are two-fold. Firstly, like in a prior work (Carruthers et al., \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2024\u003c/span\u003e), we assessed the temporal validity of an outdated dataset, but instead of repository metrics we considered code metrics. The study allowed us to reinforce what we stated previously, providing additional evidence about the importance of using updated data when conducting code-related MSR studies. Secondly, the evaluation showed that our approach is a viable option to generate current samples. We uploaded the datasets and the scripts of the investigation in a Zenodo repository\u003ca class=\"FNLink\" href=\"#Fn1\" id=\"#FNLinkFn1\"\u003e\u003c/a\u003e following the guidelines from Gonzalez-Barahona \u0026amp; Robles (\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e2023\u003c/span\u003e) to report research artifacts and facilitate study replication.\u003c/p\u003e\u003cp\u003eThe rest of the paper is structured as follows. Section 2 describes attempts from other authors to build similar solutions. Section 3 provides context on the MSR study in which our strategies were tested. Section 4 presents the research questions and details the methodology for our study. Section 5 reports the findings of the study. Section 6 presents the answers to the research questions. Section 7 discusses threats to the validity. Finally, Section 8 gives the conclusions and outlines future work.\u003c/p\u003e"},{"header":"2. Related works","content":"\u003cp\u003eOver the years, investigators have proposed various solutions to extract development-related data available from online software forges, like Apache Software Foundation, Bitbucket, Github, or SourceForge. Linstead et al. (2009) designed Sourcerer, a platform to mine and search code snippets. The platform automatically crawls software repositories from SourceForge and Apache, parses the code and extracts certain features to improve the performance of the queries. It stores all the information into a relational database and provides access through a website. According to the foundational study, Sourcerer contains code-related data from 4632 Java projects.\u003c/p\u003e\n\u003cp\u003eIn 2012, Gousios \u0026amp; Spinellis (2012) presented GHTorrent, a service that gathers development data from Github. The authors explored and mined Github’s API to share repository data about users, organizations, teams, commits, issues, and others from all the projects inside the forge. GHTorrent distributes the data as monthly incremental MongoDB dumps. Another tool is Boa[2] (Dyer et al., 2013), a domain-specific programming language for analyzing ultra-large-scale software repositories with an API and a web-based interface to access its infrastructure. In the beginning, the service facilitated access to almost 650K SVN and CVS repositories from SourceForge, sharing their versions, metadata, commits, changelogs, development team data, and source code.\u003c/p\u003e\n\u003cp\u003eFalessi et al. (2017) created STRESS, a semi-automated approach for project selection based on diversity, fit, and quality. The tool connects to Apache Software Foundation’s query service, allowing project filtering with more than 100 characteristics, including: the number of tickets, number of commits, date of the first/last commit and ticket, number of lines of code committed per programming language, and size of the team. Similarly, Dabic et al. (2021)[3] introduced Github Search, containing information from 730K GitHub public repositories across ten programming languages. The tool crawls and consumes 25 repository characteristics from GitHub’s API, and repositories’ landing page, issues page and pull requests page. Like STRESS, it provides a web application to filter the projects by the number of issues, commits, programming language, date of creation, among others. Also in 2021, Ma et al. (2021) described World of Code[4], a system that collects data from Bitbucket, Github and other forges. In contrast to the previous tools, World of Code stores the complete information provided by Git objects (e.g., all commit information, files content, file versions, etc.). \u003c/p\u003e\n\u003cp\u003eDespite the usefulness of the reviewed solutions for empirical studies, most tools created before 2021 are no longer accessible. Apparently, it is difficult to allocate, maintain and scale such platforms online over an extended period of time. The required resources can be a constraint here because not all the researchers have at their disposal an infrastructure large enough to host these platforms. We think this approach is not always the best option, particularly, in cases in which most of the sampling process can be implemented as runnable scripts and delivered to the research community. It is preferable to execute these scripts and obtain fresh datasets directly from the software forge instead of relying on third-party mirrors that might have outdated records. However, this approach might still pose challenges for researchers with limited computational resources if they try to fetch data from thousands of projects.\u003c/p\u003e\n\u003cp\u003e[1] https://doi.org/10.5281/zenodo.15008288\u003c/p\u003e\n\u003cp\u003e[2] https://boa.cs.iastate.edu/\u003c/p\u003e\n\u003cp\u003e[3] https://seart-ghs.si.usi.ch/\u003c/p\u003e\n\u003cp\u003e[4] https://worldofcode.org/\u003c/p\u003e"},{"header":"3. Code metrics thresholds","content":"\u003cp\u003eIn this section, we provide a contextual background on the Software Engineering field in which our tool was validated. According to Fenton \u0026amp; Bieman (2014), Software Engineering encompasses techniques applied in the construction and support of software products, involving activities such as; managing, planning, modeling, testing, maintaining, among others. These activities adhere to an engineering approach, which means that they must be understood and controlled to reduce uncertainty while the software is specified, designed, built, and maintained. To achieve this level of control we need to monitor the status of the projects, products, processes, and resources through metrics (Lanza \u0026amp; Marinescu, 2006). Metrics quantify specific characteristics of the software, for example, product metrics can describe different perspectives of the source code, like its cohesion, complexity, coupling or size. \u003c/p\u003e\n\u003cp\u003eWhen working with metrics, establishing reference points is essential to link a particular metric value to useful semantics (Lanza \u0026amp; Marinescu, 2006). Simply having a numeric value for a metric lacks context, and it is insufficient information to evaluate the state of the system. Therefore, we require \u003cem\u003ethresholds\u003c/em\u003e for most of the metrics as they allow us to make informed assessments about the system being measured. Thresholds delimit which values are acceptable and abnormal in software components; providing decision-making guidance to programmers, managers and other members of the team (Lorenz \u0026amp; Kidd, 1994).\u003c/p\u003e\n\u003cp\u003eThe literature offers different strategies to set quality thresholds. For example, Mccabe (1976) defined a threshold for the Cyclomatic Complexity metric based on the experience of programmers, suggesting that a method with a value over ten should be refactored. Nejmeh (1988) took a similar approach for another complexity metric (NPATH). The author conducted empirical studies at AT\u0026amp;T Bell Laboratories and came up with a threshold of 200. Coleman et al. (1995) defined two levels for the Maintainability Index metric, an upper and a lower limit of 85 and 65 respectively; components scoring over 85 were considered as highly maintainable, between 85 and 65 were considered as moderate, and components below 65 were considered candidates for refactoring. Because these values rely on subjective experience, their replication and generalization can be challenging.\u003c/p\u003e\n\u003cp\u003eConsequently, data-driven methods have emerged, in which threshold values depend on the data fed into the model, rather than on opinions provided by experienced users. For instance, Alves et al. (2010) proposed a method that implements the statistical properties of metrics based on data from a benchmark of systems. They generated three thresholds for five metrics: McCC, LOC, Number of Parameters, Fan-in and Number of Methods. These thresholds denoted four levels of risk for modules: low, moderate, high and very high. Also considering metrics’ distributions using a benchmark dataset, K. Ferreira et al. (2012) delimited thresholds for six object-oriented measures: Coupling Factor (Brito E Abreu \u0026amp; Carapuça, 1994), Number of Public Fields, Number of Public Methods, and two Chidamber \u0026amp; Kemerer (1994)’s metrics Lack of Cohesion in Methods and Depth of Inheritance Tree. They identified the probability distribution that best fitted each metric and calculated three range values according to value frequency: good, regular and bad.\u003c/p\u003e\n\u003cp\u003eAlternatively, Herbold et al. (2011) introduced a method based on Kearns (1998)’ machine learning algorithm for thresholds calculation. They computed values of four method-level metrics (Cyclomatic Number, Nested Block Depth, Number of Function Calls and Number of Statements) and seven class-level metrics (WMC, Coupling Between Objects, Response for a Class, Number of Methods, LOC, Number of Overridden Methods and Number of Static Methods). They tested it through four case studies employing projects from different programming languages and adjusting the algorithm configuration.\u003c/p\u003e\n\u003cp\u003eNevertheless, Lavazza \u0026amp; Morasca (2016b) expressed concerns about assessing software quality relying solely on thresholds derived from statistical distributions of internal measures. They argued that estimating external quality must be obtained via probability models that take internal measures as independent variables (Morasca, 2009). In their article, the authors estimated fault-proneness, defining it as the probability that a module contains at least a fault, and generated thresholds for 13 metrics using different approaches. They claimed that thresholds obtained from distribution-based methods using only internal measures remain the same independently of the type of quality being targeted (e.g., reliability, maintainability, security, etc.). The authors concluded that metric thresholds should not remain constant regardless of the quality of interest, and instead they should be tailored to the objectives of practitioners in a given project. \u003c/p\u003e\n\u003cp\u003eAlthough we think Lavazza \u0026amp; Morasca suggestions are reasonable, our main goal is to determine whether our tool is able to build consistent samples rather than prescribing quality thresholds for specific metrics to practitioners. In our case, the topic of metric thresholds serves as a means to validate the effectiveness of the sampling strategies designed.\u003c/p\u003e"},{"header":"4. Study design","content":"\u003cp\u003eTo conduct our investigation, we followed the guidelines for MSR studies proposed by Vidoni (2022). In this section, we present the research questions, provide details about the data collection process and results of the protocol execution, and describe the measures and data analysis process to answer the questions.\u003c/p\u003e\n\u003ch2\u003e4.1. Planning\u003c/h2\u003e\n\u003ch3\u003e4.1.1. \u0026nbsp;Research Questions\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eRQ1. Is it necessary to update software samples?\u0026nbsp;\u003c/strong\u003eWe compare the thresholds of three source code metrics (Lines of Code, Cyclomatic Complexity and Weighted Methods per Class) derived from the benchmark dataset Qualitas Corpus (which was assembled over a decade ago), and a current sample generated with our procedure. We aim to assess the temporal validity of a highly considered benchmark dataset, and to determine whether after some time a proper update procedure is required.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eRQ2. Is it possible to generate consistent samples?\u0026nbsp;\u003c/strong\u003eWe determine whether samples generated with our tool, either from scratch or by updating an existing sample, exhibit consistent results. Like in RQ1, we compare the two samples using their thresholds for the same three source code metrics.\u003c/p\u003e\n\u003ch3\u003e4.1.2. \u0026nbsp;Data Collection\u003c/h3\u003e\n\u003cp\u003eWe employed two sources of information: the Qualitas Corpus (Tempero et al., 2010) and Github. The former served as the outdated codebase while the latter provided recent development data. We considered Github as a source of recent development data because it allows us to query the repositories by programming language. We retrieved the projects metadata using its REST[5] and GraphQL[6] API services, and the source code from repositories\u0026rsquo; URL. The repository data collected for our study are presented in Table 1 along with their descriptions and rationale.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003e1\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp; \u0026nbsp;\u003c/strong\u003eRetrieved data\u003c/p\u003e\n\u003cdiv align=\"center\"\u003e\n \u003ctable border=\"0\" cellspacing=\"0\" cellpadding=\"0\" width=\"572\"\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 66px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eId.\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 134px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eMeta-data\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 174px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eDescription\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 198px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eRationale\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 66px;\"\u003e\n \u003cp\u003eNAME\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 134px;\"\u003e\n \u003cp\u003ename\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 174px;\"\u003e\n \u003cp\u003eRepository\u0026rsquo;s name\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd rowspan=\"2\" style=\"width: 198px;\"\u003e\n \u003cp\u003eKeyword filtering\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 66px;\"\u003e\n \u003cp\u003eOWN\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 134px;\"\u003e\n \u003cp\u003eowner\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 174px;\"\u003e\n \u003cp\u003eRepository\u0026rsquo;s owner\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 66px;\"\u003e\n \u003cp\u003eURL\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 134px;\"\u003e\n \u003cp\u003eurl\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 174px;\"\u003e\n \u003cp\u003eRepository\u0026rsquo;s url\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 198px;\"\u003e\n \u003cp\u003eRepository access\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 66px;\"\u003e\n \u003cp\u003ePL\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 134px;\"\u003e\n \u003cp\u003eprimaryLanguage\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 174px;\"\u003e\n \u003cp\u003eProgramming languages\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 198px;\"\u003e\n \u003cp\u003eIdentify repository\u0026rsquo;s main programming language\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 66px;\"\u003e\n \u003cp\u003eSIZE\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 134px;\"\u003e\n \u003cp\u003etotalSize\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 174px;\"\u003e\n \u003cp\u003eCode size in KB\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 198px;\"\u003e\n \u003cp\u003eIdentify repository\u0026rsquo;s java code size\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 66px;\"\u003e\n \u003cp\u003ePRIV\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 134px;\"\u003e\n \u003cp\u003eisPrivate\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 174px;\"\u003e\n \u003cp\u003eRepository accessibility\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 198px;\"\u003e\n \u003cp\u003eDetermine repository\u0026rsquo;s availability\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 66px;\"\u003e\n \u003cp\u003eMIRR\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 134px;\"\u003e\n \u003cp\u003eisMirror\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 174px;\"\u003e\n \u003cp\u003eMirrored repository\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 198px;\"\u003e\n \u003cp\u003eDetermine if the repository is original\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 66px;\"\u003e\n \u003cp\u003eCONTR\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 134px;\"\u003e\n \u003cp\u003econtributors\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 174px;\"\u003e\n \u003cp\u003eNumber of contributors\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd rowspan=\"3\" style=\"width: 198px;\"\u003e\n \u003cp\u003eQuantify the collaboration\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 66px;\"\u003e\n \u003cp\u003eCPR\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 134px;\"\u003e\n \u003cp\u003eclosedPullReqCount\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 174px;\"\u003e\n \u003cp\u003eNumber of closed pull-requests\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 66px;\"\u003e\n \u003cp\u003eMPR\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 134px;\"\u003e\n \u003cp\u003emergedPullReqCount\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 174px;\"\u003e\n \u003cp\u003eNumber of merged pull-requests\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 66px;\"\u003e\n \u003cp\u003eCOMM\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 134px;\"\u003e\n \u003cp\u003ecommits\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 174px;\"\u003e\n \u003cp\u003eNumber of commits\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd rowspan=\"2\" style=\"width: 198px;\"\u003e\n \u003cp\u003eQuantify the history\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 66px;\"\u003e\n \u003cp\u003eCREAT\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 134px;\"\u003e\n \u003cp\u003ecreatedAt\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 174px;\"\u003e\n \u003cp\u003eCreation date\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 66px;\"\u003e\n \u003cp\u003eCI\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 134px;\"\u003e\n \u003cp\u003eclosedIssuesCount\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 174px;\"\u003e\n \u003cp\u003eNumber of closed issues\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 198px;\"\u003e\n \u003cp\u003eQuantify the issues\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 66px;\"\u003e\n \u003cp\u003eSTARS\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 134px;\"\u003e\n \u003cp\u003estargazerCount\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 174px;\"\u003e\n \u003cp\u003eNumber of stars\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd rowspan=\"2\" style=\"width: 198px;\"\u003e\n \u003cp\u003eQuantify the popularity\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 66px;\"\u003e\n \u003cp\u003eFORKS\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 134px;\"\u003e\n \u003cp\u003eforkCount\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 174px;\"\u003e\n \u003cp\u003eNumber of forks\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 66px;\"\u003e\n \u003cp\u003eCOMMD\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 134px;\"\u003e\n \u003cp\u003edateLastCommit\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 174px;\"\u003e\n \u003cp\u003eLast commit date\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd rowspan=\"4\" style=\"width: 198px;\"\u003e\n \u003cp\u003eDetect recent activity\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 66px;\"\u003e\n \u003cp\u003eCPRD\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 134px;\"\u003e\n \u003cp\u003eclosedPullReqLastDate\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 174px;\"\u003e\n \u003cp\u003eLast closed pull-request date\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 66px;\"\u003e\n \u003cp\u003eMPRD\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 134px;\"\u003e\n \u003cp\u003emergedPullReqLastDate\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 174px;\"\u003e\n \u003cp\u003eLast merged pull-request date\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 66px;\"\u003e\n \u003cp\u003eARCH\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 134px;\"\u003e\n \u003cp\u003eisArchived\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 174px;\"\u003e\n \u003cp\u003eArchived repository\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 66px;\"\u003e\n \u003cp\u003eCODE\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 134px;\"\u003e\n \u003cp\u003ecode\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 174px;\"\u003e\n \u003cp\u003eProject source code\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 198px;\"\u003e\n \u003cp\u003eCalculate code metrics\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003eWe filtered the repositories that might not be suitable for Software Engineering research (Munaiah et al., 2017). The filtering process consisted of discarding repositories that did not meet the thresholds in Table 2. These thresholds were based on the results of secondary studies (Carruthers et al., 2022b, 2022a), and recommendations of Munaiah et al. (2017), Kalliamvakou et al. (2014), and Lewowski \u0026amp; Madeyski (2020). To avoid unwanted repositories that might surpass the quality thresholds (e.g., demo, tutorials or configuration repositories), we excluded repositories including any of the keywords: benchmark, conf, demo, docs, exam, guide, sample, template, tutorial, and wiki.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003e2\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp; \u0026nbsp;\u003c/strong\u003eQuality thresholds and criteria\u003c/p\u003e\n\u003cdiv align=\"center\"\u003e\n \u003ctable border=\"0\" cellspacing=\"0\" cellpadding=\"0\" width=\"435\"\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 38px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eId.\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 230px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eThreshold\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 167px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eCriterion\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 38px;\"\u003e\n \u003cp\u003eT1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 230px;\"\u003e\n \u003cp\u003eJava programming language.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 167px;\"\u003e\n \u003cp\u003eJava projects.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 38px;\"\u003e\n \u003cp\u003eT2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 230px;\"\u003e\n \u003cp\u003ePublic repositories.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 167px;\"\u003e\n \u003cp\u003eRepository meta-data available.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 38px;\"\u003e\n \u003cp\u003eT3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 230px;\"\u003e\n \u003cp\u003eGIT repositories.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd rowspan=\"2\" style=\"width: 167px;\"\u003e\n \u003cp\u003eSupport processes tools usage.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 38px;\"\u003e\n \u003cp\u003eT4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 230px;\"\u003e\n \u003cp\u003e50 closed issues or more\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 38px;\"\u003e\n \u003cp\u003eT5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 230px;\"\u003e\n \u003cp\u003eRepositories that are not mirrors.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 167px;\"\u003e\n \u003cp\u003eOriginal projects\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 38px;\"\u003e\n \u003cp\u003eT6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 230px;\"\u003e\n \u003cp\u003e10000 lines of code or more.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 167px;\"\u003e\n \u003cp\u003eSize\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 38px;\"\u003e\n \u003cp\u003eT7\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 230px;\"\u003e\n \u003cp\u003e3 or more contributors\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd rowspan=\"2\" style=\"width: 167px;\"\u003e\n \u003cp\u003eCollaboration\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 38px;\"\u003e\n \u003cp\u003eT8\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 230px;\"\u003e\n \u003cp\u003e50 pull-requests or more.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 38px;\"\u003e\n \u003cp\u003eT9\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 230px;\"\u003e\n \u003cp\u003e1 or more years since creation.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd rowspan=\"2\" style=\"width: 167px;\"\u003e\n \u003cp\u003eHistory\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 38px;\"\u003e\n \u003cp\u003eT10\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 230px;\"\u003e\n \u003cp\u003e1000 commits or more.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 38px;\"\u003e\n \u003cp\u003eT11\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 230px;\"\u003e\n \u003cp\u003e10 or more forks.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd rowspan=\"2\" style=\"width: 167px;\"\u003e\n \u003cp\u003ePopularity\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 38px;\"\u003e\n \u003cp\u003eT12\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 230px;\"\u003e\n \u003cp\u003e10 or more stars.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 38px;\"\u003e\n \u003cp\u003eT13\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 230px;\"\u003e\n \u003cp\u003e1 or more commits or merged/closed pull-request in the last month.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd rowspan=\"3\" style=\"width: 167px;\"\u003e\n \u003cp\u003eRecent activity\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 38px;\"\u003e\n \u003cp\u003eT14\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 230px;\"\u003e\n \u003cp\u003e1 or more commits per month in the last year.\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 38px;\"\u003e\n \u003cp\u003eT15\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 230px;\"\u003e\n \u003cp\u003eNot archived repository\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003ch3\u003e4.1.3. \u0026nbsp;Generated Samples\u003c/h3\u003e\n\u003cp\u003eWe generated three different samples to evaluate our approach: an outdated sample (Qualitas Corpus), a sample with current data (Current Sample), and an updated version of the outdated sample (Qualitas Updated). All samples were equally sized with 112 subjects each.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eQualitas Corpus\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eThis sample corresponds to the last release of the Qualitas Corpus from September 2013. We acquired the source code from the \u003cem\u003er\u003c/em\u003e \u003cem\u003edistribution package\u003c/em\u003e available in the official website of the dataset[7] and searched for the access links to the projects on GitHub.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eCurrent Sample\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eBased on the dataset obtained from Github, we generated the sample using a stratified random sampling strategy. This method involves dividing the population into non-overlapping sub-populations, called strata, and randomly selecting elements from each stratum (Lohr, 2021). Stratified sampling is preferred when the sub-populations may have different mean values for the targeted variables, providing lower variance estimates for the whole population (Lohr, 2021). The number of elements selected from each stratum was proportional to the number of subjects in that group relative to the Github dataset.\u003c/p\u003e\n\u003cp\u003eWe implemented the selection procedure by first generating the strata (or groups) based on the repository size (SIZE) due to its correlation with the variables of interest at a project level (namely LOC, McCC and WMC metrics) reflected by the Spearman\u0026apos;s correlation coefficients over 0.93 in Fig. 1. The groups were generated using k-means clustering algorithm (Bugayenko et al., 2023; Wu et al., 2024). K-means divides \u003cem\u003em\u003c/em\u003e datapoints in \u003cem\u003en\u003c/em\u003e dimensions into \u003cem\u003ek\u003c/em\u003e clusters seeking to minimize the sum of squares within cluster (Hartigan \u0026amp; Wong, 1979). In our case, the datapoints correspond to the SIZE score of each project in the dataset retrieved from Github. Before applying the algorithm, we isolated projects with extreme SIZE values with the interquartile outlier detection rule. This rule states that any values greater than the sum of the third quartile and 1.5 times the interquartile range (third quartile \u0026ndash; first quartile) are excluded (Vinutha et al., 2018). With the list of subjects after removing outliers, we applied k-means with five groups. We determined the optimum number of clusters using the elbow method (Tibshirani et al., 2001).\u003c/p\u003e\n\u003cp\u003eOnce we obtained the groups, we randomly selected the projects from each group proportionally according to the stratum size and population. For example, if the population comprised 800 subjects and the groups contained 170, 180, 220, 100, and 130 subjects, in a sample of 100 projects, we select 21, 22, 28, 13 and 16 projects from each group respectively. We then included a proportional subset of the projects that were previously separated as outliers into the resulting sample. Fig. 2 illustrates the implemented approach.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eQualitas Updated\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eThe construction of the updated sample began with the selection of the outdated or original dataset, i.e., the Qualitas Corpus described earlier. We identified projects in the original sample that were also included in the Github dataset. If the elements of the original sample had recent activity, we proceeded to update them in the sample with their latest version. The remaining elements that did not have recent updates were discarded.\u003c/p\u003e\n\u003cp\u003eTo replace the discarded projects, we applied a replacement process similar to the stratified random sampling explained for the Current Sample. Firstly, we generated the strata from the Github dataset in the same manner as the Current Sample, separating outliers and applying k-means with five groups to create the strata. Secondly, we recorded the number of projects from each stratum existing in the updated sample (excluding the discarded repositories), and computed the difference with the proportional number of elements corresponding to the equivalent stratum in the Github dataset. Finally, we selected the recent projects according to the differences recorded by group in the previous step. The resulting sample comprised the same number of subjects as the original sample. Fig. 3 summarizes the approach implemented to update the Qualitas.\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003e4.2. Execution\u003c/h2\u003e\n\u003cp\u003eFor the extraction, we developed a Python script according to the specifications defined in the planning stage. An initial query on Github services resulted in 15868 Java projects, and a second filtering reduced those to 892 subjects. As already mentioned in Section 4.1.3, the samples comprised 112 projects and, in some cases (Current and Qualitas Updated datasets), they were generated from the Github dataset. The complete dataset used in the study, as well as the scripts to extract the data and generate the graphs, are hosted in Zenodo1 and Github[8] repositories.\u003c/p\u003e\n\u003ch3\u003e4.2.1. \u0026nbsp;Threshold derivation\u003c/h3\u003e\n\u003cp\u003eTo obtain the thresholds for the samples, we retrieved the source code of the projects and calculated three metrics (see Table 3): two at method level, McCC and LOC, and one at class level, WMC. As reviewed in secondary studies (Carruthers et al., 2022b, 2022a), these metrics are commonly used in Software Engineering activities such as defect detection, software maintenance, effort estimation, among others. We used Sourcemeter[9] to analyze Java code, similarly to other authors on the subject (Ferenc et al., 2020; Mahdieh et al., 2020; Mehboob \u0026amp; Chong, 2023).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003e3\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp; \u0026nbsp;\u003c/strong\u003eAnalyzed code metrics\u003c/p\u003e\n\u003cdiv align=\"center\"\u003e\n \u003ctable border=\"0\" cellspacing=\"0\" cellpadding=\"0\"\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eId\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 152px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eDescription\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eMetric type\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 46px;\"\u003e\n \u003cp\u003eLOC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 152px;\"\u003e\n \u003cp\u003eMethod lines of code\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 101px;\"\u003e\n \u003cp\u003eSize\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 46px;\"\u003e\n \u003cp\u003eMcCC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 152px;\"\u003e\n \u003cp\u003eCyclomatic complexity\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd rowspan=\"2\" style=\"width: 101px;\"\u003e\n \u003cp\u003eComplexity\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 46px;\"\u003e\n \u003cp\u003eWMC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 152px;\"\u003e\n \u003cp\u003eWeighted methods per class\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003eFor threshold derivation of the metrics of interest, we implemented the statistical data-based approach proposed by Alves et al. (2010). The technique computes a cumulative distribution of the studied metric and establishes three reference values: lower (70%), medium (80%) and upper bound (90%). These values delimit four risk categories for classifying a component of a system (method or class): low risk (\u0026lt;= 70%), moderate risk (70% - 80%), high risk (80% - 90%), and very high risk (\u0026gt; 90%). The risk categorization refers to the degree of urgency for the code to be refactored: medium risk requires long-term attention, high risk requires medium-term attention, and very high risk requires immediate attention. For instance, using WMC thresholds from the Current Sample in Fig. 4, the values 50, 80 and 148 correspond to lower, medium and upper thresholds respectively; therefore, classes recording less than 50 are low risk, between 50 and 80 are moderate, between 80 and 148 are high risk, and higher than 148 are very high risk.\u003c/p\u003e\n\u003cp\u003eThe method has proven to be viable in multiple Software Engineering contexts, such as defect prediction (Boucher \u0026amp; Badri, 2018), code smells in test code (Spadini et al., 2020), software product lines (Vale et al., 2019) and code smell detection (F. Ferreira \u0026amp; Valente, 2023).\u003c/p\u003e\n\u003ch3\u003e4.2.2. \u0026nbsp;Outlier detection\u003c/h3\u003e\n\u003cp\u003eWe calculated and analyzed the cumulative distributions for each system to detect possible outliers. In subfigure a) of Fig. 5, gray curves represent the cumulative distributions of each individual project, and the black curve corresponds to the distribution of the complete sample. The graph helped us identify system curves being distant from the reference, with a clear example being the red line marked in subfigure b) of Fig. 5, which corresponds to the project \u003cem\u003ehapifhir/org.hl7.fhir.core\u003c/em\u003e[10].\u003c/p\u003e\n\u003cp\u003eOnce we identified the outliers, we replaced them with similar projects. In this case, we computed the Euclidean distance (Khatibi Bardsiri et al., 2014) in SIZE between the outlier and the other projects in the Github dataset, and the one that registered the closest value to zero was chosen as the replacement. Afterwards, we computed the metrics of the new projects and recalculated the sample thresholds.\u003c/p\u003e\n\u003ch2\u003e4.3. Data Analysis\u003c/h2\u003e\n\u003cp\u003eIn order to select the appropriate statistical tests, we analyzed the probability distributions of the code metrics under study. In previous studies (Ajienka et al., 2020; Capiluppi et al., 2020; Terragni et al., 2020) authors stated these three metrics have non-normal distributions, in particular, Arar \u0026amp; Ayan (2016) declared they follow a power law distribution, i.e., a heavy- tailed distribution where large values are very rare and lower values are very common. The probability distributions plotted in Fig. 6 and the p-values below 0.001 from Shapiro Wilk statistic (Shapiro \u0026amp; Wilk, 1965) further confirmed our assumptions of non-normality, thus we decided to use non-parametric tests.\u003c/p\u003e\n\u003cp\u003eWe assessed the RQs conducting a hypothesis test to ensure the statistical validity of the results at a significance level of \u0026alpha; = 0.05. We performed the Kolmogorov-Smirnov test for two independent samples (KS), a non-parametric test suitable for ordinal data which requires the construction of cumulative frequency distributions (Sheskin, 2000). The test evaluates whether two samples came from the same population, if so, their cumulative frequency distributions would be expected to be identical or reasonably close to one another.\u003c/p\u003e\n\u003cp\u003eThe null hypothesis of the KS test is: \u003cem\u003eF\u003csub\u003e1\u003c/sub\u003e(X) = F\u003csub\u003e2\u003c/sub\u003e(X) for all values of X\u003c/em\u003e, where \u003cem\u003eF\u003csub\u003ej\u003c/sub\u003e(X)\u003c/em\u003e denotes the population distribution from which the \u003cem\u003ej\u003csup\u003eth\u003c/sup\u003e\u003c/em\u003e sample is derived. When we derived the thresholds from a sample using Alves et al. (2010) method, a cumulative distribution function of the analyzed metric is generated. We used KS test to evaluate whether two distributions from different samples in the same dimension are statistically similar.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eSince we are examining the consistency of the two complete samples across all dimensions, if the hypothesis test for any of the studied metrics is rejected, the null hypothesis will also be rejected, i.e., the hypothesis on all metrics must be accepted for both samples to be considered similar or consistent. Fig. 7 shows the symbolic expression of the null hypothesis applied to both RQs, where \u003cem\u003eF\u003csub\u003ej\u003c/sub\u003e(X)\u003c/em\u003e refers to the cumulative distributions derived from the samples being compared, and the variables enclosed by parenthesis correspond to the distribution of each specific metric.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eWe compared the samples depending on the RQ. For RQ1 we evaluated the temporal validity of an outdated dataset, therefore we paired the Qualitas Corpus and the Current Sample to identify if after ten years the thresholds produced with Qualitas can be considered as valid for the analyzed dimensions. This scenario led to the following hypothesis:\u003c/p\u003e\n\u003cp\u003e\u0026middot; \u003cem\u003eH\u003c/em\u003e\u003csub\u003e0\u003c/sub\u003e: \u003cem\u003eQualitas\u003c/em\u003e(\u003cem\u003eX\u003c/em\u003e) = \u003cem\u003eCurrent\u003c/em\u003e(\u003cem\u003eX\u003c/em\u003e). The data distribution of the Qualitas Corpus and the Current Sample retrieved from Github were derived from the same population.\u003c/p\u003e\n\u003cp\u003e\u0026middot; \u003cem\u003eH\u003csub\u003e1\u003c/sub\u003e\u003c/em\u003e: \u003cem\u003eQualitas\u003c/em\u003e(\u003cem\u003eX\u003c/em\u003e) \u0026ne; \u003cem\u003eCurrent\u003c/em\u003e(\u003cem\u003eX\u003c/em\u003e).\u0026nbsp;The data distribution of the Qualitas\u003cem\u003e\u0026nbsp;Corpus\u003c/em\u003e and the Current Sample retrieved from Github were not derived from the same population.\u003c/p\u003e\n\u003cp\u003eThe second RQ aimed at assessing the effectiveness of the tool developed to generate and update samples, hence in this case we paired the Current and Qualitas Updated samples. This scenario led to the following hypothesis:\u003c/p\u003e\n\u003cp\u003e\u0026middot; \u003cem\u003eH\u003c/em\u003e\u003csub\u003e0\u003c/sub\u003e: \u003cem\u003eQualitas\u0026apos;\u003c/em\u003e(\u003cem\u003eX\u003c/em\u003e) = \u003cem\u003eCurrent\u003c/em\u003e(\u003cem\u003eX\u003c/em\u003e). The data distribution of the updated version of the Qualitas Corpus and the Current Sample retrieved from Github were derived from the same population.\u003c/p\u003e\n\u003cp\u003e\u0026middot; \u003cem\u003eH\u003csub\u003e1\u003c/sub\u003e\u003c/em\u003e: \u003cem\u003eQualitas\u0026apos;\u003c/em\u003e(\u003cem\u003eX\u003c/em\u003e) \u0026ne; \u003cem\u003eCurrent\u003c/em\u003e(\u003cem\u003eX\u003c/em\u003e). The data distribution of the updated version of the Qualitas Corpus and the Current Sample retrieved from Github were not derived from the same population.\u003c/p\u003e\n\u003cp\u003e[5] https://api.github.com\u003c/p\u003e\n\u003cp\u003e[6] https://api.github.com/graphql\u003c/p\u003e\n\u003cp\u003e[7] http://www.qualitascorpus.com/\u003c/p\u003e\n\u003cp\u003e[8] https://github.com/juancarruthers/thresholds_experiment\u003c/p\u003e\n\u003cp\u003e[9] https://sourcemeter.com/\u003c/p\u003e\n\u003cp\u003e[10] https://github.com/hapifhir/org.hl7.fhir.core\u003c/p\u003e"},{"header":"5. Results","content":"\u003cp\u003eIn this section we answer the research questions based on the results obtained from the data analysis.\u003c/p\u003e\n\u003ch2\u003e5.1. RQ1: Is it necessary to update software samples?\u003c/h2\u003e\n\u003cp\u003eInitially, we calculated thresholds for LOC, McCC and WMC, for both Qualitas and Current samples and presented them in Table 4. The analysis of the metric thresholds revealed substantial differences between the two samples, observing lower values for the Current Sample. Starting with LOC, Qualitas registered 46, 71 and 132 for lower, medium and upper thresholds respectively; while the Current Sample recorded 29, 43 and 77 at the same points. These values produced ratios ranging between 0.58 and 0.63 comparing samples on the same column. We observed a similar trend on complexity metrics, recording ratios between 0.55 and 0.64 in McCC, and between 0.56 and 0.62 in WMC.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003e4\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp; \u0026nbsp;\u003c/strong\u003eThresholds calculated for the Qualitas Corpus and the Current Sample\u003c/p\u003e\n\u003cdiv align=\"center\"\u003e\n \u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"246\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 47px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eSample\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 57px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eMetric\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eLower\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eMedium\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eUpper\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"3\" style=\"width: 47px;\"\u003e\n \u003cp\u003eQualitas Corpus\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 57px;\"\u003e\n \u003cp\u003eLOC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 47px;\"\u003e\n \u003cp\u003e46\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 47px;\"\u003e\n \u003cp\u003e71\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 47px;\"\u003e\n \u003cp\u003e132\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 57px;\"\u003e\n \u003cp\u003eMcCC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 47px;\"\u003e\n \u003cp\u003e7\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 47px;\"\u003e\n \u003cp\u003e11\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 47px;\"\u003e\n \u003cp\u003e22\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 57px;\"\u003e\n \u003cp\u003eWMC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 47px;\"\u003e\n \u003cp\u003e81\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 47px;\"\u003e\n \u003cp\u003e136\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 47px;\"\u003e\n \u003cp\u003e265\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"3\" style=\"width: 47px;\"\u003e\n \u003cp\u003eCurrent Sample\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 57px;\"\u003e\n \u003cp\u003eLOC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e29\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e43\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e77\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 57px;\"\u003e\n \u003cp\u003eMcCC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e7\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e12\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 57px;\"\u003e\n \u003cp\u003eWMC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e50\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e80\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e148\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003eWe spotted interesting comparisons when contrasting the values from different columns, such as, the lower thresholds of Qualitas Corpus and the medium thresholds of the Current Sample. For instance, a method with 45 LOC would be labeled as high risk with the thresholds from the Current Sample, but as low risk with thresholds from Qualitas. This trend was noticed for the other metrics as well, a class registering 80 on WMC would be categorized as high risk with the Current Sample, and low risk in Qualitas.\u003c/p\u003e\n\u003cp\u003eMoreover, we extended the analysis conducting a KS test to compare the distributions of thresholds between the Qualitas and Current samples for each metric. The results, as presented in Table 5, indicated p-values below 0.05 for all metrics, leading to the rejection of the null hypothesis. These results further confirmed our initial assumption that the thresholds generated with these two samples have significant differences. Based on these findings, we deemed the Qualitas Corpus temporally invalid for the source code metrics LOC, McCC and WMC.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003e5\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp; \u0026nbsp;\u003c/strong\u003eKolmogorov-Smirnov test results between the thresholds of the Qualitas Corpus and the Current Sample\u0026nbsp;\u003c/p\u003e\n\u003cdiv align=\"center\"\u003e\n \u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"265\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"2\" style=\"width: 56px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eMetric\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 208px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eQualitas Corpus and Current Sample comparison\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 76px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eKS\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 132px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eP-value\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 56px;\"\u003e\n \u003cp\u003eLOC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 76px;\"\u003e\n \u003cp\u003e0.1296\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 132px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 56px;\"\u003e\n \u003cp\u003eMcCC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 76px;\"\u003e\n \u003cp\u003e0.1272\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 132px;\"\u003e\n \u003cp\u003e0.0084\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 56px;\"\u003e\n \u003cp\u003eWMC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 76px;\"\u003e\n \u003cp\u003e0.1332\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 132px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003e\u003cimg src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAApUAAABuCAYAAAB2m+hUAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAADwYSURBVHhe7Z1/bFtXdue/WXSbTWeGme6mcBybtJSRgC4KRBsPNRkZ5mYjonGypesoURaytEUsrawE9iIrCdiXzh8RIQXYbbiAxAZrI5G0lINWEhehf6Ri11YGlMala9YjjgwJKFqATCSTcuyg6c6EM53Z7BbQ/uF3bu67uu+RFPUzPh/gQeJ9990f55577nn3PPI9sLq6ugqGYRiGYRiGqYB/oiYwDMMwDMMwTLmwU8kwDMMwDMNUDDuVDMMwDMMwTMWwU8kwDMMwDMNUDDuVDMMwDMMwTMU8sBHf/v7p0hKWryWx8pOf4Jef/x1++fnn+NUXX2ADimYYhmEYhrkveOCBB/DQww/jNx55BN945Lew77vfRdVhH36zulrNuiOpyKnMfPQR0ufG8P9+8XPsf/IgHvzWN7D/oBcPffvbeOjhh9XsDMMwDMMwjAO/+uIL/OpnP8PKfBr/5+f/gNs35/Hr3/wWvnuiHbXPPqtm31Gsy6n8dH4eP37vXfzjl1/idwJHsf/Jf6VmYRiGYRiGYTaAlfmb+Jv/9ef4J7/+6/jeq6/hsYMH1Sw7grKdyr+d+jP86O0/wvc7u/Ad32H1NMMwDMMwDLMJLF27hr8cfg/P/OEP8NtHf189ve2U5VT++L13kfnoIxx69VU88p3vqKcZhmEYhmGYTeTvP/4Y14eH8Z3f/V089epr6ultpeRvf//t1J8h+8Mf4pmeHnYoGYZhGIZhtoF/8Z3v4N90d+PjH/4Qf/NnH6qnt5WSnMpP5+fxo7f/CIe6uvCtvXvV0wzDMAzDMMwW8a29e9HQ1YUfvf1H+HR+Xj29bZQU/r70aheqDh3Cdw7zM5QMwzAMwzA7gU+SSSylUnjhvWH11LZQdKcy89FH+Mcvv2SHkmEYhmEYZgfxuM+Hf/zyS2Q++kg9tS0UdSrT58bwO78XUJMZhmEYhmGYbeZ3fi+An4xF1ORtwdGp/OnS0r0fNj/4pHqKYRiGYRiG2Wb2H3wS//cffo6fLi2pp7YcR6dy+VoS+w9+V01mGIZhGIZhdgj7n/wulpN/oSZvOY5O5cpPfoIHv/kbajLDMAzDMAyzQ3jwW9/Ayk9+oiZvOY5O5S8//zvsP+hVkxmGYRiGYZgdwv6DXvzy7z9Xk7ecIk7l53jo299Wk0uiNxiEq6YWrppapNJpNLV3iM/h4REAQCqdFmmxeFwt4mvNcj4PV00tlvN59ZSWcvKWQ12jH6l0Wk12hNquIxaPo67RryZvG+HhEfQGg2pyWcTicTS1d6jJG8pGja8872juVUpTe4dlfq5HZ8pBbn8sHkd4eGRDZLNTILvHbA4bMedlNqK83mBQrHuVEh4eKWqP1PV3q4jF45b5u5zPr+m3k61zOmeHap+2it5gsGS92Mjxt+Ohb38bv/x8hzuVv/riCzz08MNqckkM9vejs60VkfAQGrxeXByLwO/zYcAw0N11EgDQ4PViOjqJSHgIzYHd/Q3zUpWLOPbKCTXJQiqdFhNls5y0pvYOLOVyanJRnnimUU0CTGezo7tHTd42YvE4+kIhNblsmgMBXNzgb9ZtxvjWNfqxlMujkM2gkM1gcXYGR1qOV2TMeoNBJJJJS9rCTAIN3nsRDLkflULOViQ8JPpwfW5uQ8ZwJ3Gk5biatKvZSB2olI2a80Sx8krt+2B/v1j3KiGVTju2B2abyQ50trXig6kpNcumEB4eQUd3DxZnZ8T8PfbKCYxFo5Z8hWwGVW43sAF2UGeftoJYPI7R8Qk12QL5BOHhkaJ5N4KHHn4Yv/zZz9TkLcfRqayUl48exfW5OfHZOH1qjYJ9MDW16x3K5Xy+bKX58P1zapKF0Jmz4v+FmYTl3EZxcSyCao9HTS7K4uyMmgQAqHK7EQkPqcnbRnMggAHDUJN3BBs9vuQ4ys4vjUdfKFT23T8x2N8Pv8+nJgvkflRK6MxZdLa1WuzBYH8/BgwDdz77zJJ3NzMdnVSTdjUbqQOVstFzvlh5W933Bq/XsT0AsPLpHVR77jltg/39GOzvV7NsOMv5PPpCIUxHJ4XDCNO2+X32v3FdqR0sZp82i+ZAAJ1trWqyIJVOI5G8BgDo7jrpmHcjeUBN2AY21als8HqFYAFg7549WMrlLAucZ99+8T/M3TN569wOOZ+8vV/X6LeEzuT0VDotzlNYzVVTawklUKiDyqdzFE6IxeNiR6Wu0Y/lfF7s3Ml12vWD0t8ZHRVpKk3tHUgkk+jo7rG07c5nn63pc8wMzVK/oDxWIO+g1jX6UdfoF32Q0cmM5KPKSAflk28iKL2pvcMxhOnU3krGTIeuT3Lacj4vwvvyeWoXnaO/9D9Babr+EOWML+XX1SVzNZXSGm9y0NILC7Y6TMj1OO1uUjt0/bArw1VED5bzeSSSSbx89Kh6Ct1dJy07o7J8CdITqp/aF5PCcdQe+VqSCZ2j/LF4XDt2hM7OUBvkNuquKbZzRG1y2egPIdejyj9l7gDJ5+xkRPVRXTp9ozmgzj+dDqjo7I5uHlLezZzzsn1TsWsTodal67tOLygvfbaTJST74pSeu70i0lV6g0H0hUIYHZ+Ay9SDXjNMK5dpp2PULlkn7PRZ5tLlK4C55qvITi31Azbyg8YOUv2qjGUbCWleO9kuGdk2qHaQ1lW1vFJkkUqncaTlOJZyuTXtpDJlmctlOs333cKmOpUA4PcdFkK9dPkK/D6fUMBUOo39j331LvHeYBBPNzSgkM0gEh6yDRHL+QYMQ9zt9AaDeDf0tgj9dXT3IGU+z7mUy+FIy3F8+P45DBiGCNMuzs4gkUxiOZ9HzAx1jI5PwDh9CoVsBolkErF43HJH1OD1ih25Krdb7NwVshk0BwK2/ZDTVWdahh4ViISHLDtPrxlviJDGjfmbSKXT6OjuQSKZxIfvn0Mhm8FyPo8jLcdF+CGRvIZYPI5YPI43e3vw4fvn1oSoKf+AYQinMJVOI3d7RZQDhxB/XaMf09FJFLIZLOW+chjCwyOYjk7COH3KNmRj196NGDMVuz51d51EJDyEzrZWVLndqHK7MWAYuDgWEeUTdANx7JUTKGQz8Pt8SC8siLIGDEOMu9/n0+4SlDq+KGNOfHLrlppkYeXTO7Y6DFM2n9y6hUI2g+no5JqIAiEbX7UfdmWUogel7kSSjEhf68wbOzLipAPvjI6KBYvaQzu28k67utMxfv4CCmbYXdZlmVg8jscPHBBjMn7+gqUNoTNnUchmUO3xCNvX1N6BN3vvtcWJZTPqUTBtmF0EZO28SSKVTluiD/Ium5OMKFQ62N+v1Td5jkGaf9DogIqd3ekLhUSYlObyZsz5WDyOq6mUkGdHd49wlGXsbAOhq0vtu04vqBySl5MsYdoXkgukCIScLm/UqNDOfmdbKwrZDG7M38To+AQSyWsoZDOinZ59+yXduYbw8AjCZpi2LxTCofp6TEcnMTo+YdFnnYwBIHd7ZU3UK6Z5vlJ+dEqVH6HaQdVW6XY0dfO8GG8NDol1C6YO0Fh1dPfAOH0KkfAQrqZSgGbO2c1Neqyv2uNBIZsRjnZfKATj9ClMRyfFGKpl0vq3m9l0p/JQfb3lzlwOgd+Yv2kJdSWS19AXCsFVU4uO7h7b5/0SyWt44fnnAHMXgxRydHxCDCA5BjfmbwpDK2/N+30+dHedtGzVkxHubGsV5ciOVqnY9WN0fEK0m/6Wgxwyz91eEZPN7/OJfpCDQxN5KZfDyqd34K2rQ0d3D94ZHbUoOpRwHC2kH0xN4VB9vUgfGujXGjNaNKm8tpdeFOeeOvgkjrQcx435myhIz9HI2LV3M8bMqU/NgQASyWtYNncq6WZHDX+RQyIbtpVP7wDmrru8i1DM0VNRxxcOuqTy+IEDapIF+eZNR4PXi4WZBMLDI47P/OkMOmFXRil6sHfPHjVpDbF43LIbKz+jRkacdICcRVqMGrxeyw2AHdUeN1w1tXi9s1PrJEF6xrY3GBROT5XbLdogX3f77l0sm7uwZOt0u7FElduNQjaDWDxu++wyzHkjh9TUOa0it0+VkTxndfomzzFZ5qUs3HZ2p5DNiB0pYjPm/PW5OSSSSbhqaoU8dTcwTrYBQEl16fQCyo2LkyzJlj7xTCNcNbVIJJPI3V4x9f4rG9/e0iKuKwaFXuVrxs9fwFPSC03eDb2Nq6mUyDtgGJZ1mfTZycZ49u1fY5uaAwFhLxdnZ1AlbcA4obODxSh3nsO0ZXv37BHrDhS7QeNNdvzS5SuWOVduSHvAMESZVJ/d+reb2XSnUl6sYQ76khkC1ykM3TnQYYfOMKBEQ7cVlNqPzcDv81nqJqNcyGZwqL4eriJhIKJS5W7wekXfXZpwBaFr72bh1Kf2lhZcunwFly5fsRjVUunuOolE8ppYkIs9N1sqpejS0w0N2jtnkrm3rk49ZWFZCkvJNxnlYFdGKXpQ5Xaj2uPRhoZj5k47pJuezWKwvx8F8wsG8q6sTMoMVx2qr7fsoGwULvMxkmIL8GbJohR9KxU7u1PX6Bc7YJvNgBk9oMPO+XayDSq6SNNG6AXtbtGhi3RsBLfv3lWTKoKc1FJDz5vNvkcfVZPW0NTeISJO6i7rVrKV699WsOlOJcw7nHdGR8XuXGdbK94ZHbXcGcIMlb9mvCE+24Vb/b7Dlgd8SZH9Pp/lWcWrqdS6dgTlO9SxaNSys0CG5154LKdto10/qj0e0T66Q7ELZ64Xb12dJRS0bP6kQ8x8Nqk5EEAkPKS905Y5VF9vCVWmFxa0d8j0nCzVJ8uF6u3uOil2jVXs2lsuTmNGFOtTd9dJ9IVC2pudUggPj4jHLwo2O3LlYqdLKmSI1OewjrQcx4BhWNqi02G6C6/EoNmVUYoewNwxGR2fWPMM0/j5C2gOBISukFO6nM/j8QMHHOVMeq7uFlIaACSSSfSFQkiZz/JB2pHV3aR+MDW1ZjfHCWof9Yv6L48VQbtSxZwJnSzkG0VyGq6mUkgkkyXPqVL1rVR0dofabLcTXCrlzHkax5jmeXI5H6HaBpm+UEi7rpSrFyq04SKPVW8wiH2PPmoZ63LHVOXphga8NfiV03tj/qZlt3o9NHi96GxrRV8oVNKGxXqgHUOSgy6ikjIfwbG7cSDIHjhFXlT2P7YXo+MTkt24htHxiYr6u1Hr345i1YEz3//eaiGbqfgYMIzVao9HfJ6OTq4CWJOvkM2s+n2+VQCrAFYHDGPNeTqqPR5tPjl9OjqpLZP+72xrFf9TewYMY01+KjsSHhLpfp/P0ic1v66MxdkZy/UAVhdnZ9b0rZDNiLZVezxr+kr/y+3vbGsV15J86aC2y21anJ1Z00a1LF2a3Ffqv51cVFmqfXRqr1Pb1LxqPSRvuV2UppaltqWzrXU1Eh4Sn9Uy6P9qj8fSlkh4yHJeTlfroHqoHLvxddIlu0MuS5ffbqxk3aQy/D6fpY/T0UlL+Wo/7MpQx0dts3zIZdD18nlZV3Rtl/P7fT5LvWQP6Jx8DcnJbk45tQHA6r879vuW8tR65WvonFouHWrZ1E/1kMdSLk/WIWqLTkbq2NL1ctsHDMN2DlCdsg7o2iiXR/ZO1iNZV3T1bOScV/VJPtQ65XPqeFK63HedXnS2tVra/4PXX7e0j/6H2S9V/0leah/s+qHKSr5OtkVyPp2c5P+rPR7b/quHqpMosnaUYwfpM7VFbodcn9oWOxss90k3j1WZy+2lNFVP6NDZQrVMkoFu/VvvceapetWN23IeWF1dXVUdTeJsw1P4gz8dV5O/1oSHR5C7vVJ0p4DZOWzkmPUGg+suJzw8ot2lU9M2m2XpgXh6lul+pKm9A20vvbjunSNm57KRc575erMdNni7+JO2Vpz6qx+ryVvKloS/GebrzrL5O20yKeXXDbYKeo5tcXZGPPRfSYiGYRhmt0HPuN4vDuVOgZ1KiZj0cw+7/rmG+4SNGDMyPq6a2nXvfFSZvzZA5bhqahE6c3Zbd8nIuSyYP3V1PyH/NIjuGTpm97IRc575+iN/QZDZOjj8zTAMwzAMs8vh8DfDMAzDMAzztYCdSoZhGIZhGKZi2KlkGIZhGIZhKoadSoZhGIZhGKZiNtWppLeq7Haa2ju29SdZ7F5tJ+OqqdW+/WO3sN0y3u3E4nHLG1rCwyPrehvKeq8rB/nNNVtBbzCofXsNER4eEd/Y34k6SL9OwGwOdY3+ssZ9PevaVsyrjWKz2lquHqv5N6JdchkbZYd47bKyaU5leHhE+y5iHbKi7MTXFF0cixT9SZZKld0Op8WQ2IiJsZ3Qz78w6yOVTqOju8eS1t11cl0/j6Ret9F6Ta+O3EoSyWtIJJPam65UOo2xaBSFbAYDhoG3Bod2nP3ZanndT/QGg1jK5dTkNdA8KGddk1Hn1U6Ffq6pFMq1DeXqsZq/FBkWaxOVUYkdUl/1WYp/cD+xaU5ld9dJdLa1qslriMXjWMp9Zezld3fvFtQ+bCSlvB+3nPeX7kQG+/vh9/nUZKZEGrxeRMJfvct3o0il05b3K28EVW43pqOTavKmEYvH8W7obVR7PLh0+Yp6Grfv3sXjBw4Aps3y+w6rWbadrZTX/cZgfz+qPR412YI8D0pd13YrzYEABgxDTV7DemxDuXpcbv7lfL5kh78SOyS/N51Zy6Y5lTrkH4ZOpdOIxePo6O5BIpmEq6YWTe0dGB2fQF8oJHbfYvG4uEbeQXDV1KI3GHQMWdH2dlN7hwgPyz90Ld/VyPXIbYS5E0j/U75YPI7eYHBNH2g3hMqQdxGb2jtEm6lu+izXJ6cXu/PSsZzPW/pBbapr9MNlylEO+S3n85Z+wWyrej2FDtR+yajlkrypH3K5uh0hyk87tNRmko3cN3kXl8olvXLCrg29ZlirHNlTOSSPYv2Xz1M7qU+yLshl0tjo5GUH9QVSaJzKoXKpHlXvwsMjSKXTONJyHEu53Jo8MlSGOr7qXHPZhL3UcXMp42o3P0rl+twcGrxetLe04GoqZTkXHh6xzF1XTe0a+2NnL+izq6YW/3ngLUsf1fx1jX5Rntwfu/JIL0j3P5iaEvmgkVkpVDr37ZDLUueSrp+kX3S+Nxi0yJhImbZbLl8+R2lyem8wKMolHbLTHypX1jUdTvOA5CT3z05fqN+UR12X7KBxo3zU7vDwiMUWEpRP7rMsa0h9tytDhc5TeXYyoTzq2mCnx3bY5ZdlKNvM3mAQy9KraV2mLpPdo/LUMghVx5zkU9fox1IuhyMtx4U86yT/gD7TNfL8pHx0Xmf/ZZ3ZrWyZUxkeHkFnW6sIM30wNYXmQACR8BD8Ph8K2QwujkXQ2daKAcPAwkwCqXQabw0OibeC9IVCYlAAwLNvv+3bQpbN7e2lXA7G6VPil/WPtBwX5SWS18TAdnT3YHF2BoVsBn6fD5HwEBq8XjS1d1jCI+PnL6CQzeD63ByWcvk1fahyu1HX6Md0dNIs67BQlkQyiaVcHoVsBoP9/UK5C9kMpqOTYjteTj9UXy/qLpVjr5wQfYyEh8RkW5hJoNrjwQvPP4furpOIhIfQ2daKKrdb9KM5EEBvMIinGxrE9cdeOYGY9BaLQjZjuztK5Q4YBqrcbjR4vZiOTmKwvx+pdBqf3Lol+jsWjaqXr9l1U+t54plGMU4wZZVKp0V7i4U0wsMjaHvpRSGfvlAIsXgcYTOs1RcK4VB9Paajk0XvxHXj7NT/WDyOq6kUCuYrFDu6eyzG8FB9vWhXZ1sr3uy9F9Km3YNSXzdGfYEUGk8kk7g+N4dCNiMWA6qHjLd8HbW72uNBIZtBg9drqYMoZDOo9nhEWGpooF+8bzx05iwi4Xvz1+/zaR2gi2MRsVOk7pLYzY9y8OzbDwB44fnn1oTAaaxo7pI8yP6QDaFzZC9U+/Pf+t4UcwmSTF7v7AQAvNnbI8qj+bM4OyNkrZbXHAigqb0Db/b2oKC8EaQcXZepZO7bIevzdHRShE1j8bjoC8ktbDqz6hwbHZ9A6MxZIbNYPG6x3TDl6ff5xIKr0ysqO5G8hoK5ltjpj9xu4/Qpx/C33TzoC4VgnD5lsRN2+iLPK926VOV2W+okZB3obGvFO6Ojlp1SdbeN9OiJZxrFNUdajltkDWW3VS1DRbdu62Sis4VQ+lAKdvllGcKMaC7OzuBQfT1GxydQ5XZjcXYGMMd736OPCrv34fvnUMhm1pQBQKtjTvKheUQ2XfUPeoNBvBt6GwXJxqfSaZHvSMtxfPj+OQwYBq7PzQGavux2tsyppGcZmto7Sn5m48b8TbEA0p3C7bt3haPxwvPPKVd8BSlDtccjDEF6YQGQ7qiWcjmsfHoHAFDt8eDOZ5+J6yldXvTu5XPDVVOL1zs7taHp5XxeKI/L3PlYyuVFiLftpRdF3qupFEbHJ+CqqRUGbzmfx1g0ipePHgXMhbYcUum0COfBvN7v84k7Kb/vsAgDrnx6RyhxLB6Ht64OMJ9B6wuF4KqpRUd3D5ZyObHglxL68dbVWXaFbt+9C5gGemEmgfDwSFkLIkF9oPdZJ5JJ5G6voMHrRV8ohKb2DhRsbjKIsWhU9BMAIuEhXJ+bE4ZkwDDE9U6Ljd04w6H/1+fmxK4YOZJ3PvtMGEO53a93doowSywed9R1FdkokpPu9/kszyORoSSnS72uHPy+w2Js0gsLYpGkZ41orMrFbn6UCt0IyfLWhcDtsLMXOvvjrauzOA6Q6iJbQq/NjMXjoj2QbpyovOV8HolkUugD2QKY41mqrqusd+7bcX1uTtizBumVeOPnL+Cpg0+KfO+G3sbVVEo7xyA94kN2S7bddCNlnD4lnDedXlHZ7S0tolw7/Rk/f8HS7mLhbx0DhiHWFZKRnb7onBR5XbJD1oHB/v6izxOSHpE9AYDOtlY0BwLYu2ePlLN0Slm37Wyhkx7rcMqv2qZD9fViDqkOKBS7R/ZILQPmuq/TsVJR/YPR8QkxrlXmq3tvzN8U+aajk6I9tF4U68tuY8ucypgZYjFOnyrpmQ2C7pLoKMeI6pB3JQrZjFCod0Nvi0kBUwF1DPb3o5DN4NgrJ9Zs88vIdeicT4LuuOmwu2sth09u3VKTBC8fPWpxeOhOf+XTO5a66a6TjnKgcpbN0BqN2bK5zY91PC9D0N0xHWRoC+aug6uEMLF881ApunG26z/MxUi+xm5hqXK78fiBA0il02vGZqfx8tGja0JVkEKPBXMXYD1UMj9WPr1juXbAMLS7407Y2QuVKrdb3LxdunwF74bexlg0iuV8Hvsf2yvyuWpqcX1uzrLwl0uhDF2X2Yy5Tw6zCt1IbQbl6FUl+rMeStWXUinnJmozKGfdlvvttOZtBM2BAApmxNBV5BGCcpA3ZNZLuW3ZrL5sF1vmVI6fv4CIGVIulacOPonR8QmxC5Iq4xkiHd66OiSSSVHGsvRNcwrBFJsQ5EjSXaGqAFVuN6o9HsuzEXbPSTzd0GD51m7YfIbj8QMHEDpzFpB250rd2WvwerGUy1nk9MmtW0LuDV4vEskkwsMjeOH559D20otiG57w+w7jNeMN8dmu/U60vfTimi9dXbp8BZ1trSUZWnKM5f5T3+SFtNd8Jqs3GESDGZZRn52T8fsOC9nC3G0pdgeto9g46/p/qL4efaGQ0JmY8i1CFeP0Kcs47FTu6dS9UJ+8Czw6PlGy80ROyNVUSuin3fwohVQ6bXHmYO4ELuVyjjKXcbIXOp5uaBDONc23S5evCJnE4vE1O8Y6yPGhum7M3wTM0GA5uq6y0XOf9JmImaHrpxsaLF9muDF/0xKhKZWlXE6M9wdTU2IXslS9stOfao9btG/Z3GWjR1EqoVx9KYbf57PYELms3O0VQNGNcimljFLWbTtb6KTHOsrJT3Z/sL8fnW2tYpe4EkJnzlr0tBT5qKhjdjWVKhpl2oy+bCurDpz5/vdWC9nMuo4Bw1gFsApgNRIeWo2Eh8Tnao9nFcDqgGGsTkcnRfp0dNJyXSGbWXNdIZsR11Me3bE4OyPy+H0+kS7XJ18vlymf8/t84vPi7MxqZ1ur+NzZ1rqmzOno5Gohm7GUMx2dtFxHeQrZjLY8+XqqX75GPVR5yH2nNPnobGu1yERXvtzvAcOwjMOAYawpU3fAHHv6LLeL2uz3+bSyoc9q/9W+Lc7OrE5HJy3tVfuiHnJeap+sd/L/pHN2h9wWtV65fDrksmkMnOqq9nhWF2dn1qTLh6x/nW2tljp+8PrrlnOyrshyHzCMNXNWlrXaD/UYMIw17ZfLp3rPvfPHIo36r8pE1k3d/KD8ahvkttA18pjI467KCaYuqfZHZy/U+UYHyYvqHDAMS18KyljTX115cr3UbkpXdZ3qle2H3bGeua+WoZYnt1OXTmU4zTG1T9PRSYts7Mqm83KarKs6/SlIY0h1OOm3zm5B04eCjb7I+UJvvin+V3VDd8hlURvtdMOubaq8nMpQ7by6/lK6zjbQZ0g6ZVeP3WGXX+5PJDwk9JjapcpLN+5qGWqarOt27ShI88Nv2irKR3ZaHgeSgzqn5LbZ9WU9x5mn6lU3bst5YHV1dVV1NImzDU/hD/50XE3+WhIeHrHsoNFuhtMdGsNsNr3BYNGdLYaBxobtZlLpNF4z3ljzRT2GYez5k7ZWnPqrH6vJW8qWhb93MuHhEbHVTdC33Bhmu0il0+v69j9z/+GqqbV8OYZhGGY7YKfS/FJOInlNfGvPVVPLizmzbYTN30kLnTlb8RfTmPuDgsOXvnYby9LP7pTyHBvDMDsHDn8zDMMwDMPscjj8zTAMwzAMw3wtYKeSYRiGYRiGqRh2KhmGYRiGYZiKYaeSYRiGYRiGqRh2KhmGYRiGYZiK2VSnMjw8Yvuar7pGf0WvXFSpa/RbXr+2bL5n2lVTW/RnKbbifZvUnkrQlUHvZqX3AG9FX5wID49Y5N0bDK55VRm1l155Se121dRuqE5sBU46vh5Ueal6bUcqnXZ8F/39jKqTOwV1rLcTV01tSXq2VTS1d1Qkm1Q6LWyKepAu7CT57yTWa0sqXdNj8fiOnKcbzXrlu1vYVKeyu+uk9m0gvcEglnI5Nbkswsp7gBdmEpbfaTv2yglMRyexODuDRDLp6GgVshnx3tHNosrtRiGbUZPLQi1jOZ9HR3cPCtkMIuEhjEWjm9aXUhynVDpteRcwAAz291ve8tHU3oFIeAiFbEa8x5vaPWAYGD9/wXL9TsdOx9dDeHgEo+MT4nNTe4fjPKF3h9Pv+jFr0enkVlFszqhzY7vY7IW8mBxUwsMjSCSTanJZ3L57V9iZSHgIfp8PhWxG2E91rjH3WK8tWe+aTut4Kp22vKe9VFLpdEWO7FazXvnuJjbVqbRjsL8f1R6PmlwWxRaKpVwOe/fsEY7YZjha282dzz4TcmwOBDbtlWapdBqJ5DU1eQ0NXi8GDENNtvDJrVvY9+ijgOnM739sLx4/cAAwHbSLYxHlivuH7q6T6GxrFZ8vjkUc58lbg0OAebMxHZ1UTzMl6uRmEIvHsZSzv5HdSWzmnFvPTmB310n4fT41uSyaAwHbFwdcHIusmWvMPdZrS9a7ptM63uD1IhK+Z8/KIXTmrJq0o1mvfHcTm+pUquEFCnPq7ozVkCjMO2jaEqfwLuUFgCeeabSkLefzlhDxE880WsIedEdT1+gXedSQMoUbKY98F9QbDFrKk6+ToXypdFq0T93ylsPWdKTSaZFPDt8QchmxeFy8dcJlykbti/wIgFy33A95F0HX91Q6baknpQmRyfWor7ukMYQ5Rku5HI60HEddox/h4RF0dPcgkUyK+iifrs3Ubjl8RXmpXaQvJF81zED5XTZyknWT9M7pbljWcaexU7Ebg1Kpa/QLWcrXy3ql5tely+j6K8vGJT1aIZdFfSE50BjI80xul117nRwQuf1y1EFOl8fISSd1eqNC40p5e4NB23FVy4vF4xa9Xs7ntforzw3Y9JHkE4vHtXpCNlWe/3L7qCxdf+W+qVB+ef40tXc4XkNtSaXTQn59oRBGxyfENdQXkqVcvm7OyvpBbVfzVArpPfVJnQcuacxIljSexaC20/jRZyjyglSvy+ERJlkHZVslp1P5lO5kk3TXyJBOLufzor1yv8td02WoTnkdh1SmnR5QelN7BxLJJDq6e7T129kuFRqXlLRe6+qjvKXYBZI79UU+pyJfX4pO7XQ2zakMK+GFWDyOq6kUCtkMjNOnLFvldY1+TEcnUchm4PcdFhOWFMY4fQqR8BCuplKAucMFAIuzM+juOmkZdDlEvDg7g0I2g8XZGVR7POLOtb2lReR54plGcW2TGW480nIcH75/DgOGgetzc4C0WyeHUBZnZ8S1BCluIZvBa8YbIk3d8u7o7hHt8/t8iISHsHfPHuHAhc6cRSGbQbXHg5QmxNkcCGA6OolqjweFbAbdXSctfYHZN6oD5hgs5/MYHZ8QcqExsut7g9drqUf3Kji5HnlHk8aQoP5MRyexMJNAd9dJS2iqORDQ6gLpEsn/4lhETP5CNoPp6CSOtBxHygyhJJJJXJ+bQ8EMsdOYWMv2iTJUOYWHR5BKp/F0QwMK2cyasSNkHafx0Y2dit0YlMPCTELIksLvS7mc6Ldcd1N7B97svfeYxIBhaA2wXX+PvXICBVPnI+EhoWOy7g/294udJXkMPnz/HArZDFLpNN4avBeKpP4umyEvSi9kM+gLhbTyktvf2daKd0ZHAVO/3g29Lcrt6O4R19vppE5vVGhc+0IhHKqvx3R0EqPjE5ZxJWdQV15zIGDR60uXr6zRX3Vu2PVx/PwFFLIZXJ+b0+58yjtu6i4I2cUnnmkU5d6Yvwko7VZfSaubg9TepVwehWxG+8gH2XfaPeruOokBw0BnW6u4hvRE3ZmSdeHD988J2ZCd8/t8wvYMGMaGRWX6QiEYp09hOjop9ESOEDQHApad7ndGR7E4O4ND9fUlzduCaVOaAwEM9vdjwDBEeS88/xymo5No8HrRGwyK+RcJD+HYKyeUkr6yMQVzviSSSaHvrxlviPQBw0Bdo7+oTbKzhzKynNXd3XLXdBWSDa3jAMS4FxTb/dbgkCgPpr5cHIuItVO32/7O6CgGDAMFU/cvXb6iZtGu17Cpr1S7IMudyvb7fFoZrB3Ta5Ybzd3IpjmVqgKOn7+AtpdeBEyDQpN2OZ8Xzoyrphaj4xNYyuWFAYqEh4Qx+eTWLVGeTDEDU+V2w+87rB0seXEkYzIdnRThcjLke/fssUwamOFnlSq3G4nkNdQ1+oXjpBp7AKj2eCzXr3x6R+Sr9ngsk+T23bvaMlTkvqTSafh9PtGPhZkEmgMB4XTH4nGLE+rUdydi8bilnvaWFnFOXkRKwU4XSJfksq+mUhg1d0DIMdi7Z49YzOVF785nnwnjQbpEITDZCXHV1CKRTCJ3ewUNXi/6QiE0tXegYDq8KroFXTd2KnZjUCnVHo+l31Q33Zy5amrRFwpp55Guv6l0WjyaAHNh9Pt8WsePIGdB1okb8zfF2FHfq9xu3Ji/iSVzB5zu5HXySiSTQv6D/f2ij6PjE2I8q9xuDBgGbszfdNRJnd6QbhA0rgOGYRl3GldZJuWU5zQ37PpY7XHDVVOL1zs7tQunE2QXZbtAu7Zj0ShePnoUMMeVsJuD1F6y4Xa4ampxcSyidTqd+GBqyqIjsmxojJ30br0MGIYoX7XvOg7V14s5Sw5HMfw+n1h7crdXMBaNAgAuXb4i6k4kr6EvFIKrphYd3T3atqQXFixrasG8yb+n74dFOjloAGxtkp09LIdy1/RS8Pt8lnbQGrkwk8DePXvgMqNdpTBoPrNc1+i3vQHQrdewqa9UuyCvBVSecfqU9hGy9MICIO3qLuVyWPn0jpptV7FpTmW5kKdeMO/kN5qXjx7F+PkLSKXTeOrgk+rpotCiRYMvGyOVhZkEPnz/HFwO29nvht4Wkw6KIdgodM4DTAW+Pjen3WndCZSqC/QgPh3kRNhhZ4yqzV1YOmhBLJh34K4iodn1sNVjQLt2hWzG9iZM1187HSoXNQRN0C4WHTrnHRpHjbBLd6JcvSnGRpWn68tgfz8K2QyOvXLCEpHZbOT+OM1BmYtjERSyGbhswqGV0PbSi/hgagqxeBwvPP+cenrLaA4EUDB3jl0OIVWZtpdexPW5OSzn82JXWOcg084YHTrsnDO7dCfs7OFGsR4dsqOpvUNETch5LQaFnj98/5zj87O69Xo99a0Xv/QlsoIZddzNbJlTWe1xiy8W0J0Mfdur2uOxOF92jlglkAP4mvGGrTPoxHI+j9ztlaIDv5zPo6m9A1VuNxZnZ7R3JzAfMN6oCaejwevFUi5n2Z0ND4+IXZxydxLs2Pfoo5YwzNVUColkcl1OWJXbXbIuPN3QYPm2YFj5NQAVKltuV3h4RMhJTu81n5PpDQbRYIb/6dGLjWCjx6AYfp8PPX3OMtX1V6dDn9y6ZZk/JPNEMmkbvn7q4JMi5A2zLrq5Gx2fENekbJ5d9ft8IhwMKXSrpl9NpfDC88856mS5elOMjSpP7Qv1kRxJuhGwK5ucdgpvF3PqHj9wQISpSU5HWo6XNQdVqK0FMzRrB+3E3Avp59AbDMKzb7/ly5eqPjUHAkgkr2H8/IV1O+3lQrvmsv6QbRjs70dnW6vYaXLCW1eH0fEJXLp8Bc2BANpbWsQ4EX7fYUv4VSdzb12dRa+X83nE4nFt+uMHDjjKyc4e2kE7honkNYyOTyAWj2/Zmr6czyORTNreDNsxFo1aIm86dOv1eutTWZLC9/JOvAyNHdm9ZfPZ1V3NqgNnvv+91UI2s65jwDBWAawCWI2Eh1YL2cxqtcezCmC12uNZrfZ4RHohmxF5AaxORydXO9taxWe5LL/PZymrs61V/A9gdXF2xlLWgGGIOiLhodXOtlZLOylftcez6vf5tHV2trWuKVdui3wszs5YyomEhyzXqu2XDzWfXM65d/7Yci4SHrJcOx2dFP9XezyrhWzGkkblq32mv6X2XR4zOtTxoT7KY6i2pbOt1XIdNO2j63S6pJbf2dZqqUOnF+oYTkcnxZjJ6YuzM6vT0UmLTCivXb9Db74p/lfHTnctnZPHQO2nXAbJRj7ovNxPu7rlPLoxtOuvKhtd/VTvgGGsGQPKK/eN9LNgzkldunrIbZDbL/dLlrOdThY0eqPWJV+rtlsnI115qhzof2q7Ojfs+qgrWz3kuqh9BUU2ap/k+ugaWX5yW6YVe6zT54KiD1SHPL6LszOWz36fzzLmOn2Syx8wjDVp1R6PrVwKivwg6a8qD/qf2mOnP51traKdal47uRTMvlE7aU6pc1onP/VQ7T6lyzpA7XJaT6ajk2vmtpom91lOk+Ut26/qImu62hf1esrbqbHdctvp3OLsjGibznao4wqN7VPLpvO6+npe7RJpatmqbKejk0ImkGSpk688dlBsbLnHmafqVTduy3lgdXV1VXU0ibMNT+EP/nRcTd61xOJx7Hv00XXvVKYXFizhufDwiO2OZTHUa+kucz1tYxiG+boTHh7BC88/57jztF3o1gfm/iWVTuM1442KdzvL5U/aWnHqr36sJm8pWxb+3gnQt5nXgxw+JOyeEytGeHhkzbUfTE2tu20MwzBfd3K3V3akQxmLx3HslRPsUDIMgPtip5K+DDNt/nzDeljO59d8U3dxdmbdRq7O/J1BIhIeYqPEMAwjIdvdSuwtw2wVss76fb5N+c6EHTthp/K+cCoZhmEYhmG+zuwEp/K+Cn8zDMMwDMMwmwM7lQzDMAzDMEzFsFPJMAzDMAzDVAw7lQzDMAzDMEzFsFPJMAzDMAzDVMyudCqb2ju0r3O7X0ml0xv6XuDeYNDyqqi6Rr/29XupdFr8XBPM39+s5HVcG0FvMGjbhnLaxzq283HSeyc9YIqjzu2NRLUvDMN8fdh1TmVvMOj4XtmNYrcsSMv5PI60HFeT1014eASj4xOWtIWZhPb3PdV6u7tOWt5nvdUyjMXja9ouU077Lo5F+HdDTWLxuPamYrtp8Hq1b6wopge7Cbv3oW826ty2w2kO2THY3295m9h6ymAYZmey65zKwf5++H0+NXlDicXjWMrdexH8TqfK7cZ0dFJNXjfdXSfR2daqJmtxqjeVTiORvKYmbyrNgUDJbd+O9u1W3hocUpN2NOXowU4ndOasmrQlOM1tYjmfr9h53022lmGY4myqUxmLx+GqqYWrptZyN1rX6BfpdBe+nM/DVVMr/tL/paArD+YdsKumFql02hJuoby60FksHkdHdw8SyaSlDbprKETa1N4BV00twsMjos9yPspD7ZHPyf1V++wy5Ub9kuWpa7uKnD+VTou6mto7ALP9dN4pHKW2i+T9wdSUJR+FtVLpNI60HMdSLifqk/untkOGwm50EFQ2yUPWJ/kaJ+zap9uFk0P+JMdYPG67q0J9o3bIZar916XL+WV9JsLDIyIftYH6rYb16Vq53JQZKpbrdEljQOWru2L05qcjLcdFHZRXvl6FZCHrIPVdHkfSY51uy/Ik/STZ0DVy/U56YDcGhJ39kee2mtel9N9OLqRL1HbV5vXahOqb2juQSCbR0d0jytPJSYXqWc7nRZuofCdbJF+rzm3VVixLbw2R+1RMzpDsZszG1jIMs3vZNKcylU7jrcEhFLIZLM7OYHR8Asv5PGLxOB4/cACFbAaR8BDGz18AAGGgjr1yAoVsBn6fD+mFBaXUtfQGg3g39Laop6O7RzhQAFDIZvCa8YbIX9fox3R00qzj8Bpj3hwIIBIegt/nQyGbQZXbrb2m1wzDd3T3wDh9CpHwEPpCIVyfm0Mhm8FSLodUOo1UOi0M56H6ehSyGcBcgCD1l+RBciBj79m3H4VsBs2BAN4aHBLtgCljJ5oDAUxHJ9HZ1ooGr1fsal4ciyCVTuOTW7dQyGYwHZ3EWDSqXg5I7SCa2jvwZm+PaAMRlsLmDV4vpqOTqPZ4UMhmcHEsgkh4CJ1trahyu1HldmPAMLSvrwqdOYtI+J7e+H0+xOJxUXZfKIRD9fWYjk6KXcZlM/xPMrTbOXFqnxrab2rvsLxCc/z8BRSyGVyfm9PuqpCMnnimEYVsBp1trbgxfxMw9fPphgYxvsdeObEmfcAwxI6ULN8BwxDOxNVUCoVsxrJzFTpzVowDySM8PILOtlZx/QdTU0JGS7kcjNOnhKz8Ph+M06cAAC88/5z2VaELMwlUezyYjk5isL8fqXQaudsrogzYhC9Jj2k+dLa1oqcviFg8jr5QCKPjEyhkM1iYSWA5nxcyvteuw2hq70CV241CNoNqj0eES2mu07winPTAbgxkdPbniWcaxdyW58cTzzRicXZG9J/mcl8oJNITyaRwSskp//D9cxgwDFyfmwMAvDM6isXZGRyqr9fq7cWxCPw+HyLhIVwci9jKSUV+JECOPBSzRXZzW2crqtxuLM7OAKaNbQ4ESpIz2U3Y2FqGYXY3m+ZU3pi/ifaWFsAM0ZLRaA4EcHEsgt5g0LIokIGSDeLKp3fE/3aMjk8Ip4CclRvzN1HldiORvIa6Rj8WZhLo7jqJ5XxeGHhXTS1Gxye0ToKM3TUUho+Eh0T91R6PeGav2uMBTAeGDCct2G/29uBqKoVUOo3HDxwQdTUHAvD7fEil00IOLzz/nDi/MJPA3j174KqptTg9TjR4vZYw7+27d0X6wkwC4eERx+en5PFYzueRSCZFP14+elScKxY2bw4EkEhew7K5U7n/sb1qFkB6ltFVUysWHyp7wDBE3dT/S5evWOq1a0Ox9slcHIuI8QOAao8brppavN7ZqXWESUakwwCQu70CmM5eXygEV00tOrp7RLsTyWtibLu7Topy6UbFVVOLvlAIn9y6Jcp01dTi4lhE6NjTDQ1w1dTiqYNPijbQc6NN7R3oC4UA6RGJao/H4kAbp08JJ/XS5StrHEodH0xN4VB9vfg8NNCvfYyAZEFtfb2zU+jOgGFYxuLS5St4uqFBfB7s77c8N/34gQPiJlHW30j4q7C8kx7YjYGMzv4MGAYavF7se/RRkUY3ck880yh0lMa6kM3gzmefwSXtlJIuTUcnhdNENudQfb1wZlVHTkcxORXDyRY5ze1SbUUpct6Kx5cYhtk+Ns2phLSwylCI6lB9vWVRqAS7sMnCTAIfvn8OLiVcSnf6BXMXrRTWc00pyE5DMZraO8ROiuz0FMPvOyx2Tb11dYAUekOJz09tBO0tLbh0+YqjA0NhuYK5W7QTGOzvRyGbwbFXTqzZuS0F2l2mg7jz2WeWfATtdhXMnTyYzkkhm4FLCq12d51Ewdy9pLGk8Khx+hQGDMNSrkqD14tPbt2ynT92lHKzVy46W0G0vfQiLl2+oiaXhd0YrAfa4aaDHOe6Rr9l97gYzYEACuYOuKvE8K+TnDaLcmzFRsqZYZjdx6Y5lU8dfFKEvGE6k6l0Gh9MTVl2nCrF7/PhndFR8flqKoUXnn9OhJ4oTJNIXkOV241qj8fiYOpCdzLruUaHvKMwfv4C2l56EQ1eL5ZyOctzbJ/curUmHAtpl1D3bddivHz0KEJnzuL23btit4R2duRvYRaDrqVwGYV4dSE4Hd1dJ9EXCjkujKPjE5Ydv2Lsf2yvRc8SyWsYHZ9Y82xgJZAjSbIvZfEn/L7DlscvSHf8vsOWUDbJ1O/zoadvra5RGwpmaBWm3JfzeREmTaXTGD9/wbJ7Xow3e3tw7JUTtjvHKofq68UOKACkFxZEREIH7eypO4kyOlsh520OBNAXCiEWj9vaDSc9sBuD9UBzVn7GsjcYFP0s54aTrhvs70dnW2vRx32KyUmFblrUOaGzRU5zu1RbsZFyZhimfFbVhG3ggdXVVdt2jD33LH7vv/xXPPTww+qpkggPj4gFqNrjwcJMAinzSxKUtpTLobOtVTxTVO3xwO87LD6rz3n1BoPi3HR0Eg1eL+rMLxPIacv5PHr6vnp+Ry5HDk9Rfhm5jXReveaDqSnRjgHDEP30+3z45NYt0Z7F2RmkFxYwfv6CaEtnW6vY3ViWHniHFAaT+0RpTeaD+5Bk978mxvFvW9sAs267Ra2u0Y/2lhaxMMj1Ull+nw9PNzSIvkTCQ3hrcMjSDlk2fp8PiWQShWzGMtaR8BC8dXWifFn2vcEgDtXX2zoH8vjq9EOWNemUfI3f50O1x2356SAouujUPihyXpydwTujo6J8eewIeazk9g0YBrq7TlrKozTddbp0apuujN5gEInkNTF2F8ci4ssPkOTX82oXht4bBmx0hHaG7aC66VpZljp5QNIv0hEaK7l9cp/ldF0be4NBLOXuOdBQ5ii1wUkPdPKTofmt2h9Z96hd6pxdnJ1BlfnsNY2bPKfkemW5wQyFy/JRoT7p5KeTE2Eni1g8bmuL7Oa2na24OBYRcium64RqvyH9fNF0dBKvGW/A7zus1SmGYez51RdfIP6DP0TH9A/VU1uKo1P5P/99K77f8R/wz6Xn/pjyIUNutwDcT/QGg7xg7DA2Y0zIEXFyVpmth20Rw3w9+d+3buFG5H/g3/3p2i/9bSWO4e9vPPJbWJl3/oYxwzC7l1g8bvlSBsMwDLP7WJlP46F/8YiavOU4OpX7Dh7El7/4BzWZKYOU9DMe9+szRinp9wM3ekeMWR+95heirs/NrXn8YyNQfxqL2X7YFjHM15f/8/NfYN+TT6rJW45j+PunS0uYev0/oin8x+ophmEYhmEYZgdwsfs/4eg7/x2/WV2tntpSHHcqf7O6Gr/+LRdWzG8CMgzDMAzDMDuHlfmb+Kff/Na2O5Qo5lQCwHdPtOOv/3zjfp6FYRiGYRiG2Rj++s/j8J5oV5O3haJOZe2zz+LXHnwQn2jemsEwDMMwDMNsDx//xV/g1x58ELVHjqintoWiTiUAfO/V13B95D18/vHH6imGYRiGYRhmi/n7jz/GX42O4Huvvqae2jZKciofO3gQz/zhD5AaHsbP72z8K9oYhmEYhmGY0vj5nTv4y/few9PGG3js4EH19LZRklMJAL999Pfxnd/9XfxoaAh/zzuWDMMwDMMwW87nH3+MHw0NoebZZ/Evj72gnt5WHH9SSMff/NmH+NHbf4Tvd3ah5l/71NMMwzAMwzDMJvBJMonUyDCefuMN/Mvf31kOJdbjVALAp/Pz+PF77+Ifv/wSv/N7Aew/uP0/uMkwDMMwDPN1ZGX+Jv76z+P4tQcfxPdefW1Hhbxl1uVUEpmPPsJPxiL4v7/4OfY9eRD/zPVN7D/oxUPf/jYeevhhNTvDMAzDMAzjwK+++AK/+tnPsDKfxpeFX2Dl5jz+6Te/BW97B2qffVbNvqOoyKkkfrq0hOVrSayk0/jl53+Hf/j8c/zqiy/UbAzDMAzDMIwDDz38ML7xyCP4jUd+C/u9XlQd9u2IHzYvhQ1xKhmGYRiGYZj7m5K//c0wDMMwDMMwdrBTyTAMwzAMw1QMO5UMwzAMwzBMxbBTyTAMwzAMw1QMO5UMwzAMwzBMxbBTyTAMwzAMw1QMO5UMwzAMwzBMxbBTyTAMwzAMw1QMO5UMwzAMwzBMxbBTyTAMwzAMw1QMO5UMwzAMwzBMxbBTyTAMwzAMw1QMO5UMwzAMwzBMxfx/K9HQ0N88d+oAAAAASUVORK5CYII=\"\u003e\u003c/p\u003e\n\u003ch2\u003e5.2. RQ2: Is it possible to generate consistent samples?\u003c/h2\u003e\n\u003cp\u003eLike in RQ1, we computed the thresholds of two samples: Qualitas Updated and Current Sample, but in this case both of them were created with our data extraction tool. In Table 6 we registered lower, medium, and upper thresholds of the sample for each metric. Upon comparing these thresholds across metrics, we detected marginal differences between the two. In particular, McCC thresholds were identical with lower, medium and upper values of 4, 7 and 12 respectively.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003e6\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp; \u0026nbsp;\u003c/strong\u003eThresholds calculated for the Qualitas Updated and the Current Sample\u003c/p\u003e\n\u003cdiv align=\"center\"\u003e\n \u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"255\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 57px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eSample\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 57px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eMetric\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eLower\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eMedium\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eUpper\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"3\" style=\"width: 57px;\"\u003e\n \u003cp\u003eQualitas Updated\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 57px;\"\u003e\n \u003cp\u003eLOC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 47px;\"\u003e\n \u003cp\u003e29\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 47px;\"\u003e\n \u003cp\u003e42\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 47px;\"\u003e\n \u003cp\u003e74\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 57px;\"\u003e\n \u003cp\u003eMcCC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 47px;\"\u003e\n \u003cp\u003e4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 47px;\"\u003e\n \u003cp\u003e7\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 47px;\"\u003e\n \u003cp\u003e12\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 57px;\"\u003e\n \u003cp\u003eWMC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 47px;\"\u003e\n \u003cp\u003e52\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 47px;\"\u003e\n \u003cp\u003e80\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 47px;\"\u003e\n \u003cp\u003e151\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"3\" style=\"width: 57px;\"\u003e\n \u003cp\u003eCurrent Sample\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 57px;\"\u003e\n \u003cp\u003eLOC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e29\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e43\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e77\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 57px;\"\u003e\n \u003cp\u003eMcCC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e7\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e12\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 57px;\"\u003e\n \u003cp\u003eWMC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e50\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e80\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e148\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003eThen again, we conducted a KS test to compare the distributions of thresholds between the Qualitas Updated and the Current Sample and summarized them in Table 7. The KS test results denoted p-values over 0.98 for all metrics, thus the null hypothesis from Fig. 7 in this case was accepted, suggesting there are not significant differences between the thresholds of the Qualitas Updated and the Current Sample. This implies that the procedure is capable of generating consistent samples that closely resemble each other in terms of the source code metrics LOC, McCC, and WMC.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003e7\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp; \u0026nbsp;\u003c/strong\u003eKolmogorov-Smirnov test results between the thresholds of the Qualitas Updated and the Current Sample\u003c/p\u003e\n\u003cdiv align=\"center\"\u003e\n \u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"293\" class=\"fr-table-selection-hover\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"2\" style=\"width: 56px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eMetric\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 237px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eQualitas Updated and Current Sample comparison\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 114px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eKS\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 123px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eP-value\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 56px;\"\u003e\n \u003cp\u003eLOC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 114px;\"\u003e\n \u003cp\u003e0.0204\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 123px;\"\u003e\n \u003cp\u003e0.9869\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 56px;\"\u003e\n \u003cp\u003eMcCC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 114px;\"\u003e\n \u003cp\u003e0.0118\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 123px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 56px;\"\u003e\n \u003cp\u003eWMC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 114px;\"\u003e\n \u003cp\u003e0.0111\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 123px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003e\u0026nbsp;\u003cimg src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAApYAAABpCAYAAACAqWPvAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAADUMSURBVHhe7Z1/cFvXdee/djKNkziI06SRHIkQ6SGnjd2IkQNWgSKsKqK7ShoqDm25I5FJbNIk45F2PSRnF55tx8SS3h9j7C6J9a40LsmAVhKKnDEl2SukNncLMhqoQly+WCZbe5MBxqQIyZKcuHFfnF+dJNg/9M7tfRf3AQ8g+EP2+cy8GeC+++7Pc889954LvJtyuVwODMMwDMMwDLNCblYDGIZhGIZhGKYc2LBkGIZhGIZhKgIblgzDMAzDMExFYMOSYRiGYRiGqQhsWDIMwzAMwzAVgQ1LhmEYhmEYpiKwYckwDMMwDMNUhJvW8n8s33j1VVw8/ze48vLL+NmPf4Sfv/kmfvX221jDIjAMwzAMw7zjuOmmm/C+W2/FBz76UXzwY7+H2z/9aWzb9Tl8/M471airypoYlq+cOoWXx78F3Hwzqu6+G7d/8k7ccttteP9HPoL33XqrGp1hGIZhGIYpkX96+2f4+U/+Ab986y1cfuUVvD7/MnK/+Q0+3fpV3HXvvWr0VWFVDcvFs2dhjAzjfR/+MO764p/i47//B2oUhmEYhmEYZpV444c/wCvf+Sv88q230ND1ddTs2aNGqSirZli+fGIcF755HJ996CFsvfsz6m2GYRiGYRhmjch+38D3vvENfOaBB1Hf0qrerhirYlgm/9t/xdW/W8Dnur6OD23erN5mGIZhGIZh1hjz6lWkhv8Smz61HYF/++/U2xWh4r8Kf/nEON545e/xr/78L9ioZBiGYRiG2SB4Nm/Gv/zzv8C1v1/Ay+PfVm9XhIoalotnz153f3d04ub3vle9zTAMwzAMw6wjN7/3vdjV+XW89M3jeO2731Vvr5iKGpbGyDB2PvQQPLxTyTAMwzAMsyH50ObN+OxDD8EYGVZvrZiKGZavnDyJ9334w6jiH+owDMMwDMNsaKo+48Mtt30Yr0xNqbdWRMUMy5dPfBt3ffFP1WCGYRiGYRhmA3LXF7+ICycqe9ayIoblG6++Ctx8M/9PJcMwDMMwzA3Cx3//D3Dze95z3Y6rEBUxLC/+zTn+r0qGYRiGYZgbjC077sbSuaQaXDYVMSyvzM/jE5/8pBrMMAzDMAzDbGA+ceeduLqwoAaXTUUMy5/9+Ee45bbb1GCGYRiGYRhmA3PLbbfhZz/+kRpcNhUxLH/+5pv4wEd+Vw12RW84DE9tHTy1dUgZBprb2sX36PAIACBlGCJsKh5Xk1h1mtva1zzf5rZ2Uf8bgejwCHrDYTV4w9MbDt9Q7eyWqXg8bxzpqG8MImUYavCKKbddPbV1WMpm1WChAypNobG9Wm2zkVntOlcifScZKYf6xqBj/8vzzlrotql4HM1t7WrwquJ2XK1H2Vab6PBIyXUqJC86bpR2e/9HPoKfv/mmGlw2FTEsf/X22/idWz+oBrtisL8fHa0tiEWH4Pf5cHoshmAggIFQCN1dnQAAv8+H6ckJxKJDONDUpCaxqvSGw0gkV3b2IGUYJQljdHhkxXmWQnR4ZEWKeioeR18kogZvSNS+GOzvF3K2ESlnQlvKZtHe3QMzk0YsOoSxyUnbPTL4mtvasbi8LD1ZGaLDIxgdP6EGF6W+MagGCfYdPKQGVYTTYzGbTqH2LqVtyumjjUgpdS6HYum7bUczk0Z1VZUaXDK94XDB8uw7eAgLszOYnpwoS55L5UBTE06PxcR3t+2xEgqNK8o/ZRho7+5Rb9/QpAyj5DmrmLyorGW7TcXjK1qwve/WW/HLn/5UDS6bihiWK+X+/ftxfm5OfA8dOWybDAHgmTNn1tyohGV4BAMBNbgkIkePqUEF6e7qXHGepVDqAFM50NSEgVBIDd6QlNoX60nKMJBInlODi3Ll2jXUeL2A1TfzMwlx78nRUfH59FhMxKsk3V2d6GhtUYOLIpdTZXpyQg2qOPIOq9u2KbePNiJu61wuhdJfymbXxHiTGezvL1geAKiuqoLf54OZSatRVpW1ag+ncSXn7/f5EIsOqVFuaPw+X8lzViF50bGW7fb44MrzuUkNWAEbwrD0+3w25Xz7pk1YXF627aJ5t2wVn2GtfslN4bTbRvdpJ0R2D6YMA0vZLDy1dWKrWueGlyG3AcWvbwyKtGBNTPS8nGYimUR7d48Ik+PJu2dUvkI7N5DqJedNZaEdKY/kvkkZBuobg6J+cnt5LDfI9r2Nos7y8QR11azLuxhyu6r104U1W+5Jek7uC52Ll/qRyk27kmraur6gvCht+q6WifpevnS70NQPTvflssptq8pEyjCw7+AhLC4vizqpyGWh+1PxuO05ue16w2GMjp9AXySSJ2NqmXUyQ2X3SPINqX+p3WXonlxXXd84Qe35zJkzeeH1lltKbRs5TzmvlGGIfqR2qbdcs9HhEfRFIhgdPwGP4hpU24ZQ++i/PPk/RVz5Oao7lZmepftqfoRcVrn9dDJG/SWnK8eVy07tpqahQ86LIFlNGUaefiB0ZSQuX71qu7eUzWL73kbAajPqU4oj500ySFCddfVUx5QaLsuwTMowbOXpDYe146FYGeXxspTNavWZDPWzrj0KIZeBxsJUAV2GAuMKVvmd8qe2k9PT6QWqK8Wjcqh117WrLj2KqwuXZVkuV7G0ly9fEnGxAnlBgTLIlCIXchyKR9Qruq++MYjF5WXsO3jIcTyuNRvCsASAYGC3GBTPPv8CgoEAnn3+BcASkK2fuF3E7Q2HscfvF66+ex54UNwj6huDmJ6cgJlJIxjYjd5wGAeamjA9OYGO1hb4fT5UV1VhenICp8diSBkGXrt4EWYmjenJibwdU2hWIOoOS18kgoXZGZiZNBLJJJayWeHaj0WHcHoshql4HGdTKZiZNBZmZ9De3YMlS0k9PjgEM5PGc8efdnSFN7e1izymJyew7+AhLGWztrLIO0ZL2ayY/EJHDue5kWglvjA7g+6uTkzF4/Bu2Qozk7bqcU4IvFPehaABSvWF1G66Puq1jh60d/cgdOQwYtEhnE2lAEsOqI3MTBp9kYhtEtjV0AAzk4bf58Pjg0MibXpW7QvKi+63d/cgkUzi/NwczEzatrh5OPSoSK+jtQUDoVDeDvpUPI47tm2Dacnl+MlTtvuwdgwXZmewq6FB7AjoZOL2TZswPTmBGq9X1EnGqS9Ixuk52c1Px04GQiGbvOw7eAhmJo2BUAjn5+YcZWb73kaRJyyFmzIMMRZVt1pfJILQkcOYnpwQC8elbBbjJ0+JPgwGdjsq7Oa2djzWe92lLzMVj+Ox3h48d/xprauJ+vn+/ftFewQDAfh9PjGGu7s60Sy5Zru7OjEQCqGjtcWWn9o2MnREh9r63z/ybxCLDomxZ2bSqPF68UhHBwDgsd4e0e4Phx4VbTAQCmkno8jRY6Is1H46GZP7i56p8Xrhqa3Dc8efxkAoJGSR6kzjKBYd0rYhlPYfCIVEP5GsOu3+94bDeCryBExJnkm3w9pdkfOurqoSusHMpHGgqQmRo8cQi16PFwwEhB6hsQ6pLvsOHhL1pD7SjamlbNYWHjpyWOva9Pt8tvI80tGhHQ9OfUhlvOeBB0X5t+9tFPpMN7dMSceJdO3hRHR4RMjsQCiEZ86cKarLnMYV4ZQ/6Uo1PZ1eOD0WQ0drCx7rvS5b5NWS9VEpeobiUnsnkkmkrM0hnT5xk7a8mbUSeYEl0+p8o1KKXFAc0xon9KxO983PJFDj9WJ6cgKD/f0ijfVkwxiWuxoabKsn2R3+4ksXbIMrkTyHvkgEnto6tHf35HX2UjYrFI6ntg6j4yewuHx9EKi7o5evXhXh8zMJRIdH8iZIt5iZNK5cuwaPww4EAJyfm0MimYSntk4Iy5Vr1/DMmTNoO3gQsAa2zhW+lM3itYsXhWHo9/nQ0doCY35ejSog47nG680zTnSMnzyFnXfvEN+fijyBs6lUWXkDwJbNm/HaxYu2sCVr5ajrIzp6ELPO3AIQz7/40gUsWrtD1MaXr14VSlCWkfmZBG7ftAme2ro8+SDkYw5kcAQDAdvgvHLtGgDgjm3bhKwAyFvtQjoj1RsOO07WuxoaRL+TEnKSCSfK7QsnpiV32OJyViszpCi3722Ep7YOiWQSy5cvwe/zoS8SQXNbO0xlEhwIhcTz1AfPPv8C9vj9Is5gf792EbWUzSKRTIr07t+/X9zz1dejvbsHT46OwtQY3QCwx+8X+uTFly7gtYsXsWQt4LZs3gwUcc0SatsUw1dfLxYMNPHSAvnS61cAa3IIBnaLZ2iypfjEHr8fnto67Lx7hzBIdTIm95d8Ri8WHUJ1VZVtUU51firyhEivxuvVToS0wPPU1qEvErGNY09tHU6PxbQT2ej4CdEn1VVVGAiF8OJLF8R9N3mfts6+kqwRNNYpDk2oNBaoj5zG1PjJU2i9717AGjfF+h8OOrRQH1IZ5cUbjQWSPRUyvEqlu6sTg/39aG5rF4ZpIV1WaFwVIxgI2AzDK9euOeoFAHiko0O4aKficXz5C58Xz8KhXZ3SM+bnbcdraNw76RNd2tf7LCBkheZbVEBe3Mw3buUiZRi4Y9s28f1AUxOCgQBShuFK920ENoxheaCpCYnkOaFc/T6fWBXpJnFaHdClQ74vK1zaHaWOgrT1jALnTopR3xi07TI4MRAK2cpWinA4CW0lkQ0omXLy9vt8CAZ2iwFLkx3h1EdO0OqcLqfVfHNbu1j1OSmDUhga6BeTbCJ5TjuhpixXza6GBsQcztYcaGqCmUnj/NwcPJKLo1SZKKcvVkqNtTNHF7WBaa3mPRpXlw7deC6F6qoqmJk0djU0wOPgKvzyFz5vW0AGA7vx7PMv4MWXLhRt25VAi8KUYeDZ51/AU5EnMDY5iaVs1mbguTFSu7s6YVo7g6Sb3MhYJaHdHTOTFhPi6bEYzEwangLuQdVILsTtmzapQei1jrWY1s5OOZQ6pkrFTR+uNlOWCzZ05HBZhmklcNIL1VVVuGPbNqQMA5dev2LT+4VwSs+pvVeqT4iVyEul5xt1M4Zwo/s2AhvGsIS1K/Tk6KhY2XS0tuDJ0VHsamiwxQsGduPh0KPiu3quoLqqCjVery1c/nz//v2IHD2Gy1evCmF/9vkX0NHa4uoXwtTptLrad/CQ+FzMONrV0IC+SEQo3inrnIR3y1bbj2gSyaRw9RJUL3nyTiTP2Ywr2ulKJM9hdPxEyYK3x++3HQR+8aULaL3vXld566C60WCl+MX6SMfOu3dgdPyEaJOU5kwfpN0u9ajCSujpC4s6OKX7zJkzGNC4yGV6rfNa5JY25ucdZcKJcvtiJdBCT86T6tIbDsNvuYXp2IIT1IdU15Rh2HYiCBqXlB/teDVb58aiwyM40NSEWHQoz0UN5fmdd+/A/fv3Fy1bpZB3S2lyevb5F8Qi1ldfL1x5sOT1jm3b8iZecumRaz9lGK5kzA3UnlQG3SQaDATQ05c/Psnla1ouSZVgIGD7kdjZVCpvtwqSW1+tN6xdT3l3slScxlSNt0rotyXLa0Juz1Jw24erzfjJUzbvTjEKjatycNILROjIYdtcXQyn9HTtPRWPu9YnsLxnchpnUykkkklEh0dWJC+Vnm+oDeS57bWLF+H3+Vzpvg1BrgIc/ewf5cxMesXXQCiUq/F6xffpyYkcgLx4ZiadCwYCOQA5ALmBUCjvvplJi/sActOTE7Z7NV6v7bmF2RkRt8brzQHIBQOBXEdrS14a9J3KQOH0nJyGmUmLNKhuA6GQLQ1dnYKBgKt6LczOiHC5rFR2uV5yXvJFZe1obclLRy2DLu9YdMgxPvWhfFE+anrTkxN5edNnKrucF7Wn+t1U2pLqtzA7Y+sLOa//3v8fxOeO1hZbXy7MztjSk8Od6qq2KV0drS0iLbm8urrKfReLDtnSkestl0VuH2jkXs2HPsvhf3bPl/LKopaH8pyenLClMz05YUtL/kz1lcsopy+3uam0J+VBz8t5qv0g11Vu4xqv19aOahqqbNFnuQ5qf+r6iMKo7QdCobyxp8qKWnbTkhUqBz2vPgdNf8n1kmWcyl7j9ebVXdceFJfCqH6q3KjlVp+TZVBuL7XecppyuSktte5qOegz1VOVdbVsNV5vnkyYGp3llI5aHrUeNYqOkT+r8qDTofJ3Kqv8jPoc1UvNU+4LGrNyOeAwx8ppFEpPpxfkdGq83rwwU5EFuT2c0lN1G8XX6ROntNW+LHRPLj8KyIvpMN88/eT/EGGqPKttquartoFcV904pTBVrkq5ju5sUE27srkpl8vlVGOzVI75d+Kr3x5XgxlGuyMSHR5xtTO8kVDLvJTNwpifX/HOEcOsNfWNQTwVecL1LhezMVB10I1CbzisPTrEbCy+1dqCw9/7WzW4LDaUK5x556Fzg1TqTMxaMWX9MlBGdm0yDMOsJuW6qteblGHkHWVj3vmwYcmsKk9FnhC//KbrRlu90q6kXAdI55UY5kZB/oueUs8VMuvH6bHYDbVbGbX++zFy9Bh7dd6FsCucYRiGYRjmXQy7whmGYRiGYZgNBxuWDMMwDMMwTEVgw5JhGIZhGIapCGxYMgzDMAzDMBVhQxiW0eERx7eu1DcGtW9XKZf6xmDBt5oUYioeL+tvH8p9rhhL0msoS6FQeZrb2m1vPSgX+lUgvXaKXtHmqa0TfU3l97h8FeB6sBJ5KQVPbV1Z+Xik10JuJAqNaTesVbsTq92OpaQvj5W1bAO3fVZKXSpFbzhcso5Yj3IWo1z9KstCyjDE249UesNhV3241lR6Hl8N6K1eK0GXBo3l+sag6zF2o7MhDMvurk7tX9D0hsMrfidydHjEplzmZxJl/TFwyjDQ3t2jBjtC+Zb6XCls39uoBhWlUHmiwyPaV7SVQ3dXJwZCIfH6OXqFYUdri+1dsguzMxgIhTbMX2mog75ceSkFJyNfR0p6jaXT5LLeTMXjtteTlsNatLuMmUmv6t9HuU1/Kh7H4nIWZiaNjtYW8WrI1UCWJRTQwzKVkDl1jOmgV+oRg/39RXWEmq7bNl8rytWvqn7w+3za1wdOxeMYHT+hBlcUdT51g9t5XJXHUlH7v1ROj8VW/NdIahrNbe2IRYdgZtJYXF7Gzrt3FB1j5aCOl/VmQxiWTgz296/4he4rneAIv8+HWPSf36FdDMq31OdKoZx36RYqT3dXJ4KBgBpcNl/+wudtfyz+SEdHnuJ79vkXtO8RXg+Wstm88q0Fxd4vLxM5ekx81k0uG4EDTU0YCIXUYMYFl16/ghrvdWNosL9/VSYhQpYlt6xU5tzu1tH7md2SMgwkkufU4A1FufrVrX440NTk+J7sSlHOfOp2Hi9HHon10t3FeO3iRWzZvBmwFjqrtVgudbysNhvCsFTdHORGVVdqULaVCdp+bm5rt7lVPZabePveRluYvOKi9CguFDeurgzF0OULKV217OT6ohWXzhWWMgxRN3VlRvfUutU3BkV4oZXgVDyeVy45nFzZpUK7BVQm+q6urORdBV3di0H9rj4jh8v9oJOXpWxW7ADL7SW3Kcmp2l/UzkvZrOhjub3kcsj9o6ZTjOa2diSSSbR399jk8sq1a0Xr72aXYUk6miA/Q3XWjUu5v3T1kGVT3mn1WGPEScbU/KluujxkKD8qI+VF7ULlTRmGkAmqN0FueHpWHjtym3o0Y0aHmr5O/mCVrS8Swej4CVuZdXnp9IauvykfdYzrZIn6GAVkoRTUfosOj9jqJ8fxKH1Gf+Iu1436QZWZlGFg38FDWFxeFvmpbS7LoSy/1GYpl7tluvKiQJ+igH4thtq/BOVDyHVzi5NcEJQetQ2FQTOvOaHTFwSlXUgeqWxqe8p9EC2gu52g9iK9RtC4J+Q28Cj63kk/yGl4auuEHFM95TEGh7nLaexRedV21Y2X9WbdDcvo8IhtpTFlvT7PzKQROnLYtoVe3xjE9OQEzEwawcBu9FrnSUggQ0cOIxYdErtkZiYNWDt73V2deYPHnl5AdHhfJIKF2RmYmTQSyWTJSlXNF4BwgdCW+FI2K8pOrq/B/n5RBjOTxvTkBPYdPARYqzlKV12Z071gIABjfh6wBPipyBMwM2kszM6gvbvHNmiIlGHg8cHrW/XPHX/a5qoZP3kKZiaN83NzWFwurQ2IPX6/KNNUPI5gICDce2q7OtW9EL3hMPb4/TAzaQyEQmLVGx0eQet998LMpGFm0uiLRMREpJMXcsvDyv9AU5NNXkhO+yIR7GpowPTkhOgHeRenu6vTtmsgly8WHcI9Dzwo0oOVl9tXnp0eiyEYCCAWHbLtYjwcehSm5Tp98aULQIF8C/Hk6CgGQiGR1rPPv4Apy61NY3RhdkbIyJK1S0Ayptsx8Pt8WJidQY3XK1xEbQcPClnWyZg6TkkfuNnRUHfk5b4heTMzadurRuUjJc3Sm2meO/40BkIhnJ+bAyz5pedJVtzs4MnpO8kfrJ2dgVAIHa0tMK3djUI6T9Ybuv5OWcdeEskkzs/NwZR0jypLqh7WyUKpqP1Gx2OofnDQtfMzCdR4vZienBB1K6SX/D4fpicnUOP1inaT23wpm8W+g4dgWrogkUwiZRhIGYZoMze6Bg7lLdSnhfRrIZz0A/UpodZNNwZVCskFrDFAdSQ9vJS9LmtQ5jUnSp3HVXmU28209HfKMv4p3enJCfRFIlrdXQjdXErjnugNh4X8x6JDCAYCGOzvL6gf1DTMTFrI8fxMIm+MOc1durFXSA+r42UjsO6GpToRj588hdb77gWsSYK20JeyWdGhnto6jI6fwOJyFoP9/UIgaZv5tYsXRXoyukmGnpFfmWVm0mIXqFIEAwHbYLxy7ZooO9UXljKmFT0pu6VsFnv8fnhq67Dz7h15k5lsZFx6/QoAYHT8hKhbdVUVBkIhYXTIPHPmDNoOHhTxZFdNjbcKnto6PNLR4dodo7Lz7h0YP3kKsMo2NNAvBocxP29zgzvVvRCJ5DmRRndXpyjn2OSk7V3esegQzs/NlS0vJKd0ZhSAq3NDieQ59EUi8NTWob27RzwzNjmJ+/fvB6RXRpbLc8efFp/pPexO+RaCzrHVNwZFH5Fbu6O1xSa/S9ksqquqYGbSmIrHC573vS5Xu7U7CToZU+Ub1ur/9FhsRYqzuqoKieQ51DcGMT+TEPWRj5ScHosJJU076WTwbtm8OU9WisknlPRLkb9iOk/WG7r+JiObJkXiyrVr4jOh6mGdLJRDsX5zo2upvoROZlTkNjfm5211My3j0+/zoS8SQXNbuyuDBA7lLdSnhfRrIZz0g7pwevb5F2x1c+MKLyQXS9ksXrt4Uci+3+dDR2uL2BxwS6nzuMqLL13AorUDTW19+epVnJ+bs6VLBmIp6OZSGveEd8tWoUsh9Wch/aCmoaKOMae5Szf2Cunhjci6G5alQisYM5N2VCpucZps6xuDtlXNWhOzDvvSVV1Vhe6uTpjW6r+QEpZZqdAN9vfDzKRxzwMP5u0iucXv84lBuXz5EqqrqlDj9SJlGDg/N2dzg8Oh7sXQTZQoEL7W0OqcrrWi1HzJxfLc8addTVCwDIfzc3O2iVzH/fv3Y/zkKaQMAzvv3iHC3cjY6bEYzEwaHge3WinMzyTw3PGn4dG4GIvh9/kQDOyGp7YO2/c2IhYdciWfK0Xuw0I6r9T+LkQ5sqDipt/K0bVuZEZFZ7zAatvQkcPwuPxninLKe6PhNC9WEjcyTTvbdJGBTZsn5eJmLu3u6kQieQ4ea6EmL94riW6OqsTYW282nGFZ460SB1FpdUNb/zVer20yKHVikCEDR1Ym0eER4S52EvbVZo/fb3N1RK0zJM1t7ViSXFg6t7ZMMBDAk6Oj4vvZVEr7Ixnvlq22A9mJZFK4HUhp06quXEOV3B20Am87eBDPnDkD75attnhOdS9EMLDb5iKl/lTDz8/NifxXC1ISieQ5jI6fsFz/u21uV5LZO7ZtE+WjvnTrjnODU76FGJuctK3Ei0FHG5x2o2RoJ+fh0KPiMyS3dyEZozim5X50Ay1m5LalcVRtuc7UIyXFSBkGvFu25k10qwXpKDc6r5z+LkSpsqCjWL+Vq2vdyIyMr75euL9hxZ+yfkXbGw7Db7nS5R8a6iinvIX0ayHc6oetn7gdo+MnRBvIuqccdPNiInmuZFlf6Ty+8+4dGB0/IepObvBdDQ229pyKx4v2v4qbuTQ6PCKOkpkuNzhKRZ2jqM0rMfbWnVwFOPrZP8qZmXRZ10AolAOQA5CLRYdyZiadq/F6cwByNV5vrsbrFeFmJi3iAshNT07kOlpbxHc5rWAgYEuro7VFfAaQMzPp3MLsTF568jNUBrpHYR2tLXn1UC+5DvJz8vc/u+dLeXmbmbStTpSX/CzVTS6j/IzajnL6unoEAwERFgwEcgOhkGM5YtGhHIDcwuxMXp2dLnqGvlO769LQ5VnskutJZTeVelGbFJMX+Z6aru5zjdebl24wELCVXS6HXD45PhQZcLooHxobujJRHk75Ol1qvQDkvv61r2nzgNWfavwaa7zq8o1Fh/L6VNffcr3KqYdcLrltF2ZntDIh10HNSy6bPHbkcDVv9ZLTLyR/8j1I40UOU3WeLDNq2dWxLrfrwuyMTZbkssSiQ1pZiEWH8vqm0KWWx5R0AZVBTo8+y8+q7aLWn9pf1uVyHjQ+5TDKY3pywlZGN+NPV95Cfaq2Q1DSr8Uu+Rkqn9qnZhHdo7vUNFS5kPOWw0xlPlXTVS9ZJ9QUmcdNRbeZSp9RmBwPUhvLaRZrX7nOuj5amJ3J03Ww5EqVafrc0dqSl4b8bEdrS94YMx3mLt3YK6aH5fGi1tftdXRng2ralc1NuVwupxqbpXLMvxNf/fa4GswwDCOYisexZfNm247ljQTtbMjljw6P2M48MQxz46Mb17qwdxLfam3B4e/9rRpcFhvOFc4wzDuT83NzN6xRCcuNryIf8GcY5sZnKZu1udthLSq3fuJ2WxjjDBuWDMOsKvTLztU+47raPBV5QvyalS4350sZhrlxqLb+RUUe55Gjx0o+Z/puhl3hDMMwDMMw72LYFc4wDMMwDMNsONiwZBiGYRiGYSoCG5YMwzAMwzBMRWDDkmEYhmEYhqkIbFgyDMMwDMMwFWHDGZa94bDtFU/1jUHbK5eWslnxFwD0/tnmtnYRVurrndab5rb2kl6/JdffU+T9tlPxuC2u7tVZUNqvNxwWr88qBL3P1FNbp30lFqR03bzPV60XvW5NJmUYtjhL2WxenEqhyiEht2l0eMTWDvL7kOX21N2Xkeslt7ucNrWhrkxrhToWVwPq19VCbtNiMs7ks5r9MxWPO46RGwUay+80SD87sZpywdx4bCjDcioex+j4CVvY/EzC9qfK9zzwIKYnJ6x3/SbRGw6jxlsFM5NGMBDAs8+/YHt+o3N6LFbS/2NVV12va43Xi+nJCcc3AfSGw3h8cEi86/T6+3rP5SluT22daD8zk8auhgbtO2lVurs6MT05AQB45swZ9TZShoFEMomO1hbxTl8npuJxbN/biOnJCVGOyNFjNlmIDo9g38FDWJidEXHueeDBkt/37AadHBIHmpoQiw6ho7UF3V2d6O7qxEAohGAgYHuHsJlJo6O1BYP9/UI2E8mkVvnS+2IXZmeELDS3tWNsclLU9bnjTxdU7JWgmNGqjsVKsJTNisWRmwXISkgZhmjTgVAI4ydPqVEYhUr0TyG5onspwxDvkr6RcaM7b0S2721UgwSrIRfrwUYrTylMxeOrvugvhQ1lWB5oakJHa4sabGNxeRm3b9okDCwA8G7ZClhGmpOh9W4iZRgYHT+B544/bQufn0kgkUyKnZro8AhqvF7bnzwfaGrC9OQELr1+RXrSmY7WFoyOn8gzmJ45c6ZoXxKPDw5hIBSyGS2nx2LoaG0R6fZFIohFh1BdVSXizM8kcMe2beJ7pSgmhweammwGbXdXZ57ROBWP2/4QvMZbhWAggCdHR0UYrIm7xvvPdYJklMv9R/K+uJxvmFaCQjvfq4ncHsUWICvl8tWrQl66uzptCwFGz0r7Zyoed5TZpWxWLOD8Ph9i0SE1yg0HLbbfaSzMzqhBgnLk4rqOq/ymQLnIsngj8vjgxho7G8KwlF2BKrTFLm/Fb9/biOa2djS3tWN0/AT6IpG8eB7F9VjfGBRuSZpEdS706PAIesNh4TKT05DTlldpcvnlVU99YxD1jcGCqwnZvVgo71J48aULqPF6bUYYEQwEcH5uDgAwNjmJtoMH1Sjw+3yuDfRdDQ1ag0mH3E5Ut5RhYHF5GV/+wufV6Bjs70d1VZUwhHU7u07GgZyXR5Iral+SBbm/CsmhSjCw29anNV4vjPl58f3S61fydvdCRw7nGeHPPv9C3htpnjlzBsFAQNt/TvWlIxUk09HhEeG2V3cUVLmPDo+gLxLB6PgJUffmtva8NqL4hFP7eqyjDMWM1d5wWIxfuYxXrl0T6cptTGnL4VTn+sagKIduPEaHR9De3YNEMgmP5Aan5+QwClf1hRxXB92PKkcklrJZ0ReUh9oHOuT6qnpFLRvFk+NTHNI/HkUW5DLKdSdK7R85/5R1nEZuc7meS9ms2AVT86dyyXk66XWZlGGgvjFoqxchj3lKV46npin3j1xHJxmgcNl7Q2lQW8j5UFkpzlI2q5VbGfn+UjYr8qQ0KS0UqJub8SnXXY1D9aAy6CjWVynDwL6Dh7C4vCzKggJjUUUeFxTPrR6Apn5OsuhmjMp5UF2pDKRb5fx1/a4bz7q8SYYpTTm/xeVl7Dt4SCs368G6G5ZL2Sz2HTwkXH7yqkFWLPIO5cLsDE6PxcSu1kAoBDOTRnVVFbbvbRTuUlid0dzWjsXlZcByUXZ3daI3HMYevx9mJo1YdAj3PPAgpuJxMcFSPolkUpRBTXsqHs8rfyJ5DlPxOKbicTzW24Pnjj/t6OKRy1Us71Io9v5i2kGgvFcKGUxEdHgEj3R02OLkt1MSKcPA5atXbfF0uN09lYkcPYZY9PpRgGAggKl4HNHhETFR7mpowPTkhFg1q+Urtnrd1dCAF1+6AFjK4o5t22yuVV0f+H2+vOMay5cv4fZNm2zxnHZ4nOgNh5FIJtHe3YPQkcOIRYfQF4ng/Nyctcu5LJS3Tu7Jnd/R2gIzkxbpLS5nYWbSGOzvt41FWGOTji4EAwGhoGGNMd17tVUG+/vF+JV3PR4OPQrTOkpAbUwK18ykMT05gX0HDyFluU9pd9fMpPP6kcZjd1cnYtEhBAMBmJk0DjQ1oTccxlORJ2Bm0liYnUF7dw9ShqHVF81t7Xistwem5UbXTZjzMwnUeL348hc+L/LraG1BdVWVOEJB+ap9oLJk7aBQ2UgedWXrDYeFDqQ6Dvb3Y7C/H8FAANDsCE7F4zibStnqrk6epfYPyc9AKIRnzpwRdaY2lxdK1VVVYheM+gOA0HmmJbdUJlX3qgYP9bvcNsFAAL3hsE23mpk05mcSSBkGli9fgmnJCSRXqNw/A6GQOKriJANyuMzpsRhqvF7AWhQPhEKAUtbQkcPiOZ3cyvh9PkxPTohF5/xMAsFAAKEjhwEArffdK9Lqi0REe5E3xc34TBkGXrt4EaY1zsYmJ233Sad0tLagp09vxBTrK6pHjdcLM5OG3+dzHIsqsh6IRYfw+OBQSXpAVz+dLLodo7o5TR4r3V2dwvul6/eevrB2PKt5F7IPSO9MT05smFfMrrth+ezzL9jcjvLnUrfYSRC3722Ep7YOiWQSy5cviQEu7wwlkufETmd7dw8Wl5fF4O+wzs8RS9ZqUt5Fmp9J4EBTk9ilotXF4vIyLr1+Bb76erR39+DJ0VExeFR0ikeXd6nQ0QAnyPVKea8Uv8+HGq9XDPrly5fydtuM+Xlb31KbbNm82RZPx9ZP3K4GFYXOrpIcQBrkA6GQmMhoUBeSQx2++nqhdF986QJOj8WEAk8ZRt4uJNF6373oi0QAjbucUF3jxSADIhYdEnImH3GQ+1kn9yqUXut994oweSySTFJedASluqoKieQ51DcGMT+TcL3rrSIfASAD/WwqJXZU6Rzb7Zs2CcOF5M1pPOoYHT8h6lBtvR+Y+jJfX1w33D21deiLRPDaxYtSSv9MMLBbLBwuvX5FTART8Th89fWAyz6ghfSUdf6Y0JXNu2WrbSHjVDaZ83NzYieR0r9y7ZoaTYuuf7q7OjHY34/mtnYh3+UQDARscnPl2jVHvS5TXVUljBV6PnTkMBLJczbdSjxz5gx2NTSI70MD/WKRmUieEx6UbunIhE4GlrJZJJJJoU9041lFLivJn1u59ft8Qs/A6mvaJZXjm5m02Fkm3IxPv8+H+ZkEotaZdhXSKY90dAi9KuOmr3Q4jUWVs6mUaOMDTU2Yt859u9UDxepHuBmjTnOaE7p+141nXd6VtA/WgnU3LCsNrYLoKmTByz8WMZXVpg4nhU0rcrpokjWtH8N4imztV5qdd+/AorRLJZNIJoVCDQZ2561IYQlrqVvqj/X2oC8SKWhU6XbiaIDpfnQVHR5ByjDEhKxrw95wWDu4ei3XgmntXFSa6qoq3LFtG5ayWaE4g4EAjPl5PHPmjKOCOdDUJIxwnbsc1m6oPHnI6HbKSqVUudehU7SwDFD6oVGpMlQM2oGmS128ELrx6ISujZ2gXRjT2vXScf/+/TibSonvQWu3/NLrV2zlddMHnto6nJ+bE7spTnR3dSKRPCcmIvVstRO0y0mXThbdMmW5SENHDouduUpSil53i85wI5yMbDcyUC5u5bajtQXG/Dym4nE8FXlC612pbwwicvRYnmwVG59L0pGzYudFnTYmyu0rt2PRjZcLDu1ZSv3cjFHdnFYJ3OS9kVl3w3LrJ27HqHTuLJE8h9HxE1ojohh+nw+Ly8u2rXfd4IFlVMnuAKd4BKUtlys6PAJffT0S0g9ilqxzFVOW65XcQXSucTUho8Pv86GjtSVvRVbfGEQwEBCr68H+fiwuL+cZK9v3NrpWBgQZTA+HHtVOUNROZOwuWefOYBkMfZGIrW2n4nEsX74Ev88nVrCqeyQ6PALvlq1aA2N0/ETRCVmmHDnc4/fjydFRYai33ncvxk+eKrpjTEa4007sgaYmBAOBvF9iNre1C7dXuZQq9zqqq6psO9Sw+mIpm0VzWzuqLdcS7QBVgj1+P9qlIyWUn4rTeNQRVM4Gn02ltGd9YcWVXX9O7UY7StHhEXz5C59H63335o19N30wFY8jaLm0ixEdHhFuRFNjcJMBdX5uDovLy+gNh7GroQF9kYhow6kC58DdMH7ylG3HvJKUotcXJff5M2fOaM+Qw1q8yTurxvy8iBsM7Bbub0juV50MUFtTHNplk3UqGUJnUykhGyqlyO2uhgaMnzwlFqZB6yjKzrt3ANKuoXoe2834JM+Nk1FLPDk6qm3bUvpKxu1Y3OP3236o4tRGTu3ptn5uxmihOQ3S4sTNXCLjJu8NT64CHP3sH+XMTLrsq6O1JQcgByAXDARyHa0tOTOTztV4vSJ8YXZGfAaQGwiFcsFAQHyv8XpzZiadF29hdsYWb2F2RuQrhw+EQrlYdMj2XU7HzKRz05MTeWnrws1MOheLDjnmq8s/8thjBfOmS62ffFG70aWmMxAK5ZXBVNpZzQ9Wn6jPyGlTvgOhkMhDTpP6Rm5fNR+1DdW66J53qo+pyBSVRQ6Ty0/lc5JDp4vKLPctgNz05IQtnlxmigtJTuT7seiQeE4ujy5dXTy5XsFAIG8MmRq5V8vxZ/d8KS9POR1TI4fTkxN5Y43qUuP1ijZWL7Uf5HLJn9V6drS22GRG7itVltR8ILWFnCfV1WncynHlflKvjtYW25iR06ZL1wfqRfcp3xqvV1s2tW5y+eR+JXmg9FVZUfNX4xTqHzkfijcQCtn6Qm0DuY5q36pyq8qbTp9OT04IWYNUJ1Wv6+qmjnW1rrpwamO5jtQ/ujyCgUAuGAjY6iK3u05unS45/1h0KG98yeWkz07jU77kshVqR7m95LzUNCjfQvnIeoLCdLJCl1yHUvVAofpROPW3mzGq6m8K180lun7XjWc1XB1f6niX4zuNYzfX0Z0NqmlXNjflcrmcamyWyjH/Tnz12+NqMPMOITo8UnSFxzCFYBlaPXRtqwt7p5MyDDwcerTiLmqGeTfwrdYWHP7e36rBZbHurnBmY1PfGHR02TKMG9SjFkzlWMpm834skzIMHrMMw6wbvGPJMAxzAxO1/oeUCCpvgXo3sCT9F+G7sf4Ms1IquWPJhiXDMAzDMMy7mEoaluwKZxiGYRiGYSoCG5YMwzAMwzBMRWDDkmEYhmEYhqkIbFgyDMMwDMMwFYENS4ZhGIZhGKYivCsNy+jwyI35miSGYRiGYZgNzLvOsJyKx/P+UJhhGIZhGIZZOe86w/JAUxMGQiE1mGEYhmEYhlkhG8KwrG8Mor4xiKl4HCnDACx3tae2Dp7aOtsr4eobg0gZBuobg/DU1mEqHhdx5Xjk7m5ua8+7pyLnNRWPA9abHDy1degNh9ltzjAMwzAM44J1Nyyn4nE81tuD544/jfbuHhHeF4lgYXYGZiaNRDKJpWwWzW3tWFxexr6Dh/Dc8acxEAqJZxZmZ0Q8cnePjp9A6MhhkQYZjTJT8TjOplIwM2kszM6gvbsHS9ksnhwdxcLsDHY1NGB0/IT6GMMwDMMwDKOw7oalr74e7d09eHJ0FGYmDb/PBwAwM2lcuXYNnto6Eff0WAw1Xi+mJydQXVUFWO+F7e7qFN8hubs7WltEegOhEM7PzYk4xPm5OSSSSXhq68S7Zq9cu4ZdDQ3iu5lJK08xDMMwDMMwKutuWFZXVcHMpLGrocHmiq5vDCJy9NiaGHUDoRDMTFpcfp8PB5qaYGbSOD83B09tHZayWfUxhmEYhmGYG56cGrACKmJY3nLrrfjV22+rwa6gM5IHmpoQiw7h/NycOGd5eiymRi+JRPKc+Dw2OYn79++33QeAXQ0N6ItEhOFI5zx7w2GkDAOD/f3oaG2BMT+vPsowDMMwDHND809v/wy3fOhDanDZVMSw/MDHPoZf/OQnarBrzqZS8NTWob27B490dAj3Nf2gpsbrxfa9jbYzltHhEfRFIkgkk+gNh4XLnNzXAHDHtm0ijbaDB+H3+WznL8mgHQiFsH1vIzy1dRg/eUrkHzl6DJ7aOiSS53CgqUn8oEd3VpNhGIZhGOZG4+c/+Qd84KMfVYPL5qZcLrfiHdD//a+P4JN/8ifY/KlPqbfWjejwCJYvX8Jgf796i2EYhmEYhgFw9e/+Dv/vr/8aX/pfR9VbZVGRHcvN9fW4+oMfqMEMwzAMwzDMBubyK69g8/bKbQxWxLCs/txuZF/6vhq8bqjuboZhGIZhGCafyy9fQPXuf6EGl01FDMuP33knfvvrX+ONH26MXUv6RbeZSaO7q1O9zTAMwzAM867njR/+ALnf/AYfv/NO9VbZVMSwBIAdrV/FK9/5KzWYYRiGYRiG2YC8Ev8OdrR8RQ1eERUzLO+67z788q23kP3+9b8KYhiGYRiGYTYm2e8b+OU//iPuOnBAvbUiKmZYAkBD19fxvW98A+bVq+othmEYhmEYZgNgXr2KF2PfgK+zS721YipqWNbs2YPPPPAgUiPD+O2vf63eZhiGYRiGYdaR3/7610iN/CV2fOVruOOP/1i9vWIqalgCQH1LKzb94afwf/7zf+KdS4ZhGIZhmA2CefUq/u9//o/Y9Ifb8emvfFW9XREq8gfpOl4e/zZe+uZxfPahh1D1metvsmEYhmEYhmHWnuz3DbwY+wZ2fOVrq2ZUYjUNSwBY/O53MTcyjFs+/GHc1fRFfPz3/0CNwjAMwzAMw6wSb/zwB3j1O3+FX7z1FnydXavi/pZZVcOSeOXUKVz49jdx03vegy2f/jS23PWHuOW22/CBj/wufufWD6rRGYZhGIZhmBL51dtv4xc/+Ql++dZbeP3VV3D5wgX89je/wY6vfA133XuvGn1VWBPDknjj1Vex9DfncHV+Hj/78Y/wizffxC9++lM1GsMwDMMwDFMi7//Qh/D+j34UH/zY72FzfT2qP7e7on9+7oY1NSwZhmEYhmGYdy4V/1U4wzAMwzAM8+6EDUuGYRiGYRimIrBhyTAMwzAMw1QENiwZhmEYhmGYisCGJcMwDMMwDFMR2LBkGIZhGIZhKgIblgzDMAzDMExF+P/iCWBlpkoPvwAAAABJRU5ErkJggg==\"\u003e\u003c/p\u003e"},{"header":"6. Discussion","content":"\u003cp\u003eIn this section we interpret the findings above and discuss them in the context of the research questions and the literature on the subject.\u003c/p\u003e\n\u003ch2\u003e6.1.\u0026nbsp;Metrics Thresholds\u003c/h2\u003e\n\u003cp\u003eWe used the data-driven approach proposed by Alves et al. (2010)\u0026nbsp;to derive thresholds for three source code metrics: LOC, McCC, and WMC. We applied the method to obtain a set of thresholds from two datasets, the 2013 last release of the Qualitas Corpus(Tempero et al., 2010), and a current sample from Github. The statistical analysis performed with KS tests revealed significant differences between the two datasets in the distribution of all metrics, raising concerns about Qualitas\u0026rsquo; temporal validity and if it remains suitable for current MSR investigations that focus on these properties.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eFurthermore, comparing the lower, medium and higher thresholds of the datasets we observed a notable decline in these reference values to such an extent that code components categorized as low risk with Qualitas thresholds would be currently identified as high-risk, i.e., methods or classes that should be refactored in the medium term. The downward trend of the complexity and size of software components can be explained by the general consensus in the literature that development tasks get more challenging as the codebase grows in size and complexity, motivating development teams to actively monitor and reduce its complexity over time. Researchers have discovered associations between systems external quality such as, reliability, maintainability or security; and code complexity/size (Iftikhar et al., 2024). For instance, a positive correlation between software size and complexity, and error probability has been reported (Shatnawi \u0026amp; Li, 2008), i.e., when code complexity increases the probability of more errors in a component also increases, which turns systems less reliable (Sehgal et al., 2020). In this sense, Lavazza \u0026amp; Morasca conducted several reliability-related studies using these metrics to build prediction models and also metrics thresholds (Lavazza \u0026amp; Morasca, 2016a; Morasca \u0026amp; Lavazza, 2016, 2019).\u003c/p\u003e\n\u003cp\u003eRegarding maintainability, the development industry has internalized that clean code is often easier to change than convoluted code. Many code smells -symptoms of poor design and implementation choices (Fowler et al., 2002)-, like \u003cem\u003eLong Method\u003c/em\u003e,\u003cem\u003e\u0026nbsp;Complex Class\u003c/em\u003e,\u003cem\u003e\u0026nbsp;Spaghetti Code\u003c/em\u003e,\u003cem\u003e\u0026nbsp;Blob\u003c/em\u003e,etc. (Kermansaravi et al., 2021; Palomba et al., 2018) are considered as such due to their association with the complexity or size of components. Also, code clones (another type of code smell) are\u0026nbsp;worth mentioning, as they increase the complexity and scale the existing code (Mo et al., 2021). Regarding software security, experts have claimed that complexity hides bugs that may result in security vulnerabilities that attackers can take advantage of (Reis et al., 2021).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAdditionally, we observed that the LOC and McCC thresholds obtained from the current sample approximate the values reported by Alves et al. (2010) despite the differences in the number of subjects and their inclusion of\u0026nbsp;C# projects. For LOC, they recorded lower, medium and higher thresholds of 30, 44 and 74 respectively, and for McCC 6, 8 and 14. Interestingly, even when comparing McCC to the reference value of 10 established by Mccabe (1976), the values are not that far apart.\u003c/p\u003e\n\u003ch2\u003e6.2.\u0026nbsp;Sampling procedure assessment\u003c/h2\u003e\n\u003cp\u003eIn RQ2, we evaluated the effectiveness of our sampling and update strategies to generate current datasets. To that end, we constructed two datasets with our approach: one from scratch and an updated version of the Qualitas Corpus. Subsequently, we computed thresholds of three source code metrics using these datasets. Both types of sampling extracted active repositories from Github. We statistically compared these thresholds and the null hypothesis was accepted; nonetheless, these values alone do not fully convey the significance of the similarity.\u0026nbsp;Fig. 8 plots the thresholds of the three samples analyzed in the study. The blue and orange curves represent the samples created with our tool showing they mostly overlap from beginning to end. In contrast, the green curve, corresponding to the thresholds of Qualitas Corpus, has a noticeable distance from the other two curves.\u003c/p\u003e\n\u003cp\u003eIn Section 2, we explained our position regarding other tools created for sampling and mining open-source projects, such as their low chances to remain online after some time and the resource constraints associated with deploying such large platforms. Our approach resembles that of Lewowski \u0026amp; Madeyski (2020), who developed and distributed a runnable script allowing the dynamic creation of the datasets. However, there are key differences in their implementation: their quality selection criteria filtered by four repository metrics (stars, forks, commits and total size), they omitted source code metrics and the implementation of an update strategy.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eIn summary, in this study we evaluated our tool to build and update current samples. We analyzed graphically and statistically the thresholds of the samples returned from the implemented approaches for three source code metrics: LOC, McCC and WMC; demonstrating its effectiveness. Also, we highlighted its comparative advantages with other tools reported in the literature.\u003c/p\u003e"},{"header":"7. Threats to Validity","content":"\u003cp\u003eIn this section, we discuss the threats to validity identified for our work, according to the four types of validity suggested by Wohlin et al. (2012).\u003c/p\u003e\n\u003ch2\u003e7.1.\u0026nbsp;Construct Validity\u003c/h2\u003e\n\u003cp\u003eConstruct validity is concerned about generalizing the result to the concept or theory behind the study, i.e. the relationship between theory and observation. RQ1 investigated the impact of temporal validity in software samples, and RQ2 evaluated the effectiveness of two strategies generating samples that yield similar results. To perform the comparisons, we relied on three broadly accepted measures of source code complexity and size: Lines of Code, Cyclomatic Complexity and Weighted Method per Class. We centered our analysis on the specific dimensions due to their natural tendency to evolve over time (Caneill et al., 2017; Hatton et al., 2017; Rousseau et al., 2020). While these are not the only metrics that describe software complexity and size, they provide a foundation to evaluate the implication of temporal validity and the viability of our sampling strategies.\u003c/p\u003e\n\u003cp\u003eAnother potential threat lies in the projects selected for analysis. On one hand, the Qualitas Corpus(Tempero et al., 2010) is a regularly cited dataset and we considered as an adequate sample to represent the population of quality software in the moment it was last updated (2013). On the other hand, to assure no toy projects were collected for the samples obtained from Github, we defined the thresholds for quality repositories in Section 4.1.2, and manually reviewed the list of 892 repositories retrieved by the data extraction tool.\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003e7.2.\u0026nbsp;Internal Validity\u003c/h2\u003e\n\u003cp\u003eInternal validity is threatened by external influences that we did not, or were not able to consider when trying to infer cause-effect relationships. In the context of two RQ, we compared the thresholds obtained from three datasets: Qualitas Corpus\u003cem\u003e,\u003c/em\u003e and two samples generated with our sampling strategies. To reduce the bias probability, we took the following precautions: all the samples studied were equally sized, the subjects identified as outliers were replaced, the method applied to derive the thresholds was the same across all datasets and no additional weights were assigned on the projects (e.g., size, community, popularity, etc.) for the generated thresholds.\u003c/p\u003e\n\u003cp\u003eHowever, RQ1 compared the thresholds obtained from two samples collected more than a decade apart, and other external factors might have influenced the reference values instead of temporal validity. These external factors could introduce changes in the dataset independently of the variables being studied. For example, changes in the software development industry, technological advancements, or shifts in user preferences. Although time itself might not be the direct cause of the observed changes, these external factors are inherently tied to the passage of time. Therefore, to some extent, time can be considered a contributing factor to these changes.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eMoreover, another potential threat in RQ2 stems from the approach performed to update the Qualitas Corpus. Our update strategy relied exclusively on Github as the source of current repositories, and it is possible that some projects in the Qualitas Corpus have recent versions available but in different hosting locations.\u003c/p\u003e\n\u003ch2\u003e7.3.\u0026nbsp;External Validity\u003c/h2\u003e\n\u003cp\u003eExternal validity is concerned with the generalizability of the conclusions of the study. We tackled this threat using a probabilistic approach called stratified random sampling strategy as recommended by Baltes \u0026amp; Ralph (2022) and Cosentino et al. (2017) and tested recently by other empirical studies (Gorostidi et al., 2024). Respecting the samples created with our tool, we provided evidence of them being generalizable to the target population according to the reported results in RQ2.\u003c/p\u003e\n\u003cp\u003eFurthermore, our dataset only contains open-source Java projects hosted on GitHub, based on findings of our secondary studies (Carruthers et al., 2022b, 2022a). Although there are more code sharing platforms, Github established itself as one of the main sources for empirical research (Dabic et al., 2021; Munaiah et al., 2017), being the largest in terms of public open-source projects, therefore we can argue our results approximate to reality. Nevertheless, the generalization to other programming languages and proprietary projects is limited, and further studies might be needed to confirm our results.\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003e7.4.\u0026nbsp;Conclusion Validity\u003c/h2\u003e\n\u003cp\u003eThreats to the conclusion validity are concerned with issues that affect the ability to draw the correct conclusion about relations between the treatment and the outcome of an experiment. We addressed low statistical power using samples with 112 projects per dataset, which is enough statistical power (0.8) to detect medium to small effect sizes. This was calculated with the statistical software G*Power (Heradio et al., 2022)\u0026nbsp;for goodness of fit tests.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eFor statistical tests, we employed both graphical and quantitative approaches to determine which inferential test was suitable. Consequently, we applied two-sample KS non-parametric test to assess whether the cumulated distributions of the datasets’ thresholds came from the same population.\u0026nbsp;\u003c/p\u003e"},{"header":"8. Conclusions","content":"\u003cp\u003eIn this article, we conducted a Mining Software Repository study to evaluate the effectiveness of a dataset retrieval tool. Employing three distinct datasets -two samples generated through our data extraction tool and the Qualitas Corpus- we derived thresholds for three source code metrics: Lines of Code, Cyclomatic Complexity and Weighted Methods per Class. The study was structured around two research questions oriented to determine the capability of our tool for generating current samples, and to assess the temporal validity of the involved datasets.\u003c/p\u003e\u003cp\u003eTo highlight the value of our data extraction tool, we paired the thresholds derived from the Qualitas Corpus, which its last update was a decade ago, with those obtained from a sample created using our tool. The results revealed significant differences between the threshold pairs reflecting a temporal validity loss for Qualitas Corpus after ten years since its last update. Notably, components that were previously categorized as low risk based on the Qualitas thresholds would be deemed as high-risk today.\u003c/p\u003e\u003cp\u003eFor the evaluation of the tool effectiveness, we compared the thresholds computed for the samples generated by the instrument. The statistical analysis of the thresholds did not show significative differences in any metric. In particular, the Cyclomatic Complexity reference values were identical for both samples. These findings were corroborated by the graphical representation of the cumulative distributions depicting overlapping curves throughout.\u003c/p\u003e\u003cp\u003eThe main contribution in this study is the introduction of a data extraction tool to construct and update samples of software projects, positioning it as a viable option to generate current samples in the context of code-related MSR studies. Additionally, we assessed the temporal validity of a benchmark dataset such as the Qualitas Corpus, highlighting how much outdated data can impact the results of a study, and the importance of implementing effective strategies to update datasets over time. As a future work, we aim to enhance our approach and validate its results through a series of replication studies in collaboration with external research groups.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eAuthor Contributions\u0026nbsp;\u003c/strong\u003eJ.A.C. led the conceptualization, developed the methodology and software, curated the data, conducted validation, and wrote the original draft of the manuscript. A.L.A. was responsible for visualization and contributed to reviewing and editing the manuscript. J.A.D.P. supported data curation, supervision, and manuscript review and editing. E.I. contributed to the conceptualization and methodology, provided supervision, and participated in manuscript review and editing. All authors reviewed and approved the final version of the manuscript.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData Availability\u003c/strong\u003e Original dataset and replication scripts for this research are publicly available\u0026nbsp;https://doi.org/10.5281/zenodo.15008288.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding\u003c/strong\u003e This work was supported by the National Council on Scientific and Technical Research (CONICET) under a PhD Fellowship (RESOL-2021-154-APN-DIR#CONICET) and the National University of the North-East (SCyT-UNNE) under Grant 21F001. It is a part of the research conducted under the Computer Science Doctorate Program at UNNE, UNaM, and UTN.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests\u003c/strong\u003e The authors have no competing interests to declare.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eAit, A., Izquierdo, J. L. C., \u0026amp; Cabot, J. (2022). An empirical study on the survival rate of GitHub projects. \u003cem\u003eProceedings of the 19th International Conference on Mining Software Repositories\u003c/em\u003e, 365\u0026ndash;375. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/3524842.3527941\u003c/span\u003e\u003cspan address=\"10.1145/3524842.3527941\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eAjienka, N., Vangorp, P., \u0026amp; Capiluppi, A. (2020). An empirical analysis of source code metrics and smart contract resource consumption. \u003cem\u003eJournal of Software: Evolution and Process\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/smr.2267\u003c/span\u003e\u003cspan address=\"10.1002/smr.2267\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eAlves, T. L., Ypma, C., \u0026amp; Visser, J. (2010). Deriving metric thresholds from benchmark data. \u003cem\u003e2010 IEEE International Conference on Software Maintenance\u003c/em\u003e, 1\u0026ndash;10. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/ICSM.2010.5609747\u003c/span\u003e\u003cspan address=\"10.1109/ICSM.2010.5609747\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eAmasaki, S., Aman, H., \u0026amp; Yokogawa, T. (2020). An exploratory study on applicability of cross project defect prediction approaches to cross-company effort estimation. \u003cem\u003ePROMISE 2020 - Proceedings of the 16th ACM International Conference on Predictive Models and Data Analytics in Software Engineering, Co-Located with ESEC/FSE 2020\u003c/em\u003e, 71\u0026ndash;80. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/3416508.3417118\u003c/span\u003e\u003cspan address=\"10.1145/3416508.3417118\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eArar, \u0026Ouml;. F., \u0026amp; Ayan, K. (2016). Deriving thresholds of software metrics to predict faults on open source software: Replicated case studies. \u003cem\u003eExpert Systems with Applications\u003c/em\u003e, \u003cem\u003e61\u003c/em\u003e, 106\u0026ndash;121. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.eswa.2016.05.018\u003c/span\u003e\u003cspan address=\"10.1016/j.eswa.2016.05.018\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eArcelli Fontana, F., M\u0026auml;ntyl\u0026auml;, M. V., Zanoni, M., \u0026amp; Marino, A. (2016). Comparing and experimenting machine learning techniques for code smell detection. \u003cem\u003eEmpirical Software Engineering\u003c/em\u003e, \u003cem\u003e21\u003c/em\u003e(3), 1143\u0026ndash;1191. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10664-015-9378-4\u003c/span\u003e\u003cspan address=\"10.1007/s10664-015-9378-4\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e\u003cem\u003eArtifact Review and Badging\u003c/em\u003e. (2020). ACM. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.acm.org/publications/policies/artifact-review-and-badging-current\u003c/span\u003e\u003cspan address=\"https://www.acm.org/publications/policies/artifact-review-and-badging-current\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBaltes, S., \u0026amp; Ralph, P. (2022). Sampling in software engineering research: a critical review and guidelines. \u003cem\u003eEmpirical Software Engineering\u003c/em\u003e, \u003cem\u003e27\u003c/em\u003e(4), 94. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10664-021-10072-8\u003c/span\u003e\u003cspan address=\"10.1007/s10664-021-10072-8\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBoucher, A., \u0026amp; Badri, M. (2018). Software metrics thresholds calculation techniques to predict fault-proneness: An empirical comparison. \u003cem\u003eInformation and Software Technology\u003c/em\u003e, \u003cem\u003e96\u003c/em\u003e, 38\u0026ndash;67. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/J.INFSOF.2017.11.005\u003c/span\u003e\u003cspan address=\"10.1016/J.INFSOF.2017.11.005\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBrito, E., Abreu, F., \u0026amp; Carapu\u0026ccedil;a, R. (1994). Object-Oriented Software Engineering: Measuring and Controlling the Development Process. \u003cem\u003e4th Int. Conf. on Software Quality\u003c/em\u003e, 1\u0026ndash;8.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBugayenko, Y., Daniakin, K., Farina, M., Kholmatova, Z., Kruglov, A., Pedrycz, W., \u0026amp; Succi, G. (2023). Qualitative Clustering of Software Repositories Based on Software Metrics. \u003cem\u003eIeee Access : Practical Innovations, Open Solutions\u003c/em\u003e, \u003cem\u003e11\u003c/em\u003e, 14716\u0026ndash;14727. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/ACCESS.2023.3244495\u003c/span\u003e\u003cspan address=\"10.1109/ACCESS.2023.3244495\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eCaneill, M., Germ\u0026aacute;n, D. M., \u0026amp; Zacchiroli, S. (2017). The Debsources Dataset: two decades of free and open source software. \u003cem\u003eEmpirical Software Engineering\u003c/em\u003e, \u003cem\u003e22\u003c/em\u003e(3), 1405\u0026ndash;1437. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10664-016-9461-5\u003c/span\u003e\u003cspan address=\"10.1007/s10664-016-9461-5\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eCapiluppi, A., Ajienka, N., \u0026amp; Counsell, S. (2020). The effect of multiple developers on structural attributes: A Study based on java software. \u003cem\u003eJournal of Systems and Software\u003c/em\u003e, \u003cem\u003e167\u003c/em\u003e, 110593. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.jss.2020.110593\u003c/span\u003e\u003cspan address=\"10.1016/j.jss.2020.110593\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eCarruthers, J. A., Diaz-Pace, J. A., \u0026amp; Irraz\u0026aacute;bal, E. (2024). A longitudinal study on the temporal validity of software samples. \u003cem\u003eInformation and Software Technology\u003c/em\u003e, \u003cem\u003e168\u003c/em\u003e, 107404. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.infsof.2024.107404\u003c/span\u003e\u003cspan address=\"10.1016/j.infsof.2024.107404\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eCarruthers, J. A., Diaz-Pace, J. A., \u0026amp; Irrazabal, E. A. (2022a). A Systematic Mapping Study of Empirical Studies Performed with Collections of Software Projects. \u003cem\u003eComputaci\u0026oacute;n y Sistemas\u003c/em\u003e, \u003cem\u003e26\u003c/em\u003e(4). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.13053/cys-26-4-3981\u003c/span\u003e\u003cspan address=\"10.13053/cys-26-4-3981\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eCarruthers, J. A., Diaz-Pace, J. A., \u0026amp; Irrazabal, E. A. (2022b). How are software datasets constructed in Empirical Software Engineering studies? A systematic mapping study. \u003cem\u003e2022 48th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)\u003c/em\u003e, 442\u0026ndash;450. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/SEAA56994.2022.00075\u003c/span\u003e\u003cspan address=\"10.1109/SEAA56994.2022.00075\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eChidamber, S. R., \u0026amp; Kemerer, C. F. (1994). A metrics suite for object oriented design. \u003cem\u003eIEEE Transactions on Software Engineering\u003c/em\u003e, \u003cem\u003e20\u003c/em\u003e(6), 476\u0026ndash;493. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/32.295895\u003c/span\u003e\u003cspan address=\"10.1109/32.295895\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eCoelho, J., Valente, M. T., Milen, L., \u0026amp; Silva, L. L. (2020). Is this GitHub project maintained? Measuring the level of maintenance activity of open-source projects. \u003cem\u003eInformation and Software Technology\u003c/em\u003e, \u003cem\u003e122\u003c/em\u003e, 106274. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.infsof.2020.106274\u003c/span\u003e\u003cspan address=\"10.1016/j.infsof.2020.106274\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eColeman, D., Lowther, B., \u0026amp; Oman, P. (1995). The application of software maintainability models in industrial software systems. \u003cem\u003eJournal of Systems and Software\u003c/em\u003e, \u003cem\u003e29\u003c/em\u003e(1), 3\u0026ndash;16. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/0164-1212(94)00125-7\u003c/span\u003e\u003cspan address=\"10.1016/0164-1212(94)00125-7\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eCosentino, V., Izquierdo, C., J. L., \u0026amp; Cabot, J. (2017). A Systematic Mapping Study of Software Development With GitHub. \u003cem\u003eIeee Access : Practical Innovations, Open Solutions\u003c/em\u003e, \u003cem\u003e5\u003c/em\u003e, 7173\u0026ndash;7192. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/ACCESS.2017.2682323\u003c/span\u003e\u003cspan address=\"10.1109/ACCESS.2017.2682323\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eDabic, O., Aghajani, E., \u0026amp; Bavota, G. (2021). Sampling Projects in GitHub for MSR Studies. \u003cem\u003e2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)\u003c/em\u003e, 560\u0026ndash;564. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/MSR52588.2021.00074\u003c/span\u003e\u003cspan address=\"10.1109/MSR52588.2021.00074\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eDyer, R., Nguyen, H. A., Rajan, H., \u0026amp; Nguyen, T. N. (2013). Boa: A language and infrastructure for analyzing ultra-large-scale software repositories. \u003cem\u003e2013 35th International Conference on Software Engineering (ICSE)\u003c/em\u003e, 422\u0026ndash;431. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/ICSE.2013.6606588\u003c/span\u003e\u003cspan address=\"10.1109/ICSE.2013.6606588\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eFalessi, D., Smith, W., \u0026amp; Serebrenik, A. (2017). STRESS: A Semi-Automated, Fully Replicable Approach for Project Selection. \u003cem\u003eInternational Symposium on Empirical Software Engineering and Measurement\u003c/em\u003e, \u003cem\u003e2017-Novem\u003c/em\u003e, 151\u0026ndash;156. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/ESEM.2017.22\u003c/span\u003e\u003cspan address=\"10.1109/ESEM.2017.22\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eFenton, N., \u0026amp; Bieman, J. (2014). \u003cem\u003eSoftware Metrics\u003c/em\u003e (Vol. 55). Third. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1201/b17461\u003c/span\u003e\u003cspan address=\"10.1201/b17461\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eFerenc, R., T\u0026oacute;th, Z., Lad\u0026aacute;nyi, G., Siket, I., \u0026amp; Gyim\u0026oacute;thy, T. (2020). A public unified bug dataset for java and its assessment regarding metrics and bug prediction. \u003cem\u003eSoftware Quality Journal\u003c/em\u003e, \u003cem\u003e28\u003c/em\u003e(4), 1447\u0026ndash;1506. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s11219-020-09515-0\u003c/span\u003e\u003cspan address=\"10.1007/s11219-020-09515-0\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eFerreira, F., \u0026amp; Valente, M. T. (2023). Detecting code smells in React-based Web apps. \u003cem\u003eInformation and Software Technology\u003c/em\u003e, \u003cem\u003e155\u003c/em\u003e, 107111. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.infsof.2022.107111\u003c/span\u003e\u003cspan address=\"10.1016/j.infsof.2022.107111\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eFerreira, K., Bigonha, M. A. S., Bigonha, R. S., Mendes, L. F. O., \u0026amp; Almeida, H. C. (2012). Identifying thresholds for object-oriented software metrics. \u003cem\u003eJournal of Systems and Software\u003c/em\u003e, \u003cem\u003e85\u003c/em\u003e(2), 244\u0026ndash;257. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.jss.2011.05.044\u003c/span\u003e\u003cspan address=\"10.1016/j.jss.2011.05.044\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eFowler, M., Beck, K., Brant, J., Opdyke, W., \u0026amp; Roberts, D. (2002). \u003cem\u003eRefactoring: Improving the Design of Existing Code\u003c/em\u003e.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eGonzalez-Barahona, J. M., \u0026amp; Robles, G. (2023). Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositories. \u003cem\u003eInformation and Software Technology\u003c/em\u003e, \u003cem\u003e164\u003c/em\u003e, 107318. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.infsof.2023.107318\u003c/span\u003e\u003cspan address=\"10.1016/j.infsof.2023.107318\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eGorostidi, J., Ait, A., Cabot, J., \u0026amp; Canovas Izquierdo, J. L. (2024). On the Creation of Representative Samples of Software Repositories. \u003cem\u003eProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement\u003c/em\u003e, 434\u0026ndash;439. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/3674805.3690747\u003c/span\u003e\u003cspan address=\"10.1145/3674805.3690747\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eGousios, G., \u0026amp; Spinellis, D. (2012). GHTorrent: Github\u0026rsquo;s data from a firehose. \u003cem\u003e2012 9th IEEE Working Conference on Mining Software Repositories (MSR)\u003c/em\u003e, 12\u0026ndash;21. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/MSR.2012.6224294\u003c/span\u003e\u003cspan address=\"10.1109/MSR.2012.6224294\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eHartigan, J. A., \u0026amp; Wong, M. A. (1979). Algorithm AS 136: A K-Means Clustering Algorithm. \u003cem\u003eApplied Statistics\u003c/em\u003e, \u003cem\u003e28\u003c/em\u003e(1), 100. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.2307/2346830\u003c/span\u003e\u003cspan address=\"10.2307/2346830\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eHatton, L., Spinellis, D., \u0026amp; van Genuchten, M. (2017). The long-term growth rate of evolving software: Empirical results and implications. \u003cem\u003eJournal of Software: Evolution and Process\u003c/em\u003e, \u003cem\u003e29\u003c/em\u003e(5), e1847. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/smr.1847\u003c/span\u003e\u003cspan address=\"10.1002/smr.1847\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eHeradio, R., Fernandez-Amoros, D., Galindo, J. A., Benavides, D., \u0026amp; Batory, D. (2022). Uniform and scalable sampling of highly configurable systems. \u003cem\u003eEmpirical Software Engineering\u003c/em\u003e, \u003cem\u003e27\u003c/em\u003e(2), 44. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10664-021-10102-5\u003c/span\u003e\u003cspan address=\"10.1007/s10664-021-10102-5\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eHerbold, S., Grabowski, J., \u0026amp; Waack, S. (2011). Calculation and optimization of thresholds for sets of software metrics. \u003cem\u003eEmpirical Software Engineering\u003c/em\u003e, \u003cem\u003e16\u003c/em\u003e(6), 812\u0026ndash;841. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10664-011-9162-z\u003c/span\u003e\u003cspan address=\"10.1007/s10664-011-9162-z\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eIftikhar, U., Ali, N., Bin, B\u0026ouml;rstler, J., \u0026amp; Usman, M. (2024). A tertiary study on links between source code metrics and external quality attributes. \u003cem\u003eInformation and Software Technology\u003c/em\u003e, \u003cem\u003e165\u003c/em\u003e, 107348. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.infsof.2023.107348\u003c/span\u003e\u003cspan address=\"10.1016/j.infsof.2023.107348\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eJose Thiago, J. T. H., \u0026amp; Oliveira, A. L. I. (2021). Ensemble Effort Estimation using dynamic selection. \u003cem\u003eJournal of Systems and Software\u003c/em\u003e, \u003cem\u003e175\u003c/em\u003e, 110904. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/J.JSS.2021.110904\u003c/span\u003e\u003cspan address=\"10.1016/J.JSS.2021.110904\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eKalliamvakou, E., Gousios, G., Blincoe, K., Singer, L., German, D. M., \u0026amp; Damian, D. (2014). The promises and perils of mining GitHub. \u003cem\u003eProceedings of the 11th Working Conference on Mining Software Repositories\u003c/em\u003e, 92\u0026ndash;101. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/2597073.2597074\u003c/span\u003e\u003cspan address=\"10.1145/2597073.2597074\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eKearns, M. (1998). Efficient noise-tolerant learning from statistical queries. \u003cem\u003eJournal of the ACM (JACM)\u003c/em\u003e, \u003cem\u003e45\u003c/em\u003e(6), 983\u0026ndash;1006. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/293347.293351\u003c/span\u003e\u003cspan address=\"10.1145/293347.293351\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eKermansaravi, Z. A., Rahman, M. S., Khomh, F., Jaafar, F., \u0026amp; Gu\u0026eacute;h\u0026eacute;neuc, Y. G. (2021). Investigating design anti-pattern and design pattern mutations and their change- and fault-proneness. \u003cem\u003eEmpirical Software Engineering\u003c/em\u003e, \u003cem\u003e26\u003c/em\u003e(1), 1\u0026ndash;47. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/S10664-020-09900-0\u003c/span\u003e\u003cspan address=\"10.1007/S10664-020-09900-0\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eKhatibi Bardsiri, V., Jawawi, D. N. A., Hashim, S. Z. M., \u0026amp; Khatibi, E. (2014). A flexible method to estimate the software development effort based on the classification of projects and localization of comparisons. \u003cem\u003eEmpirical Software Engineering\u003c/em\u003e, \u003cem\u003e19\u003c/em\u003e(4), 857\u0026ndash;884. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10664-013-9241-4\u003c/span\u003e\u003cspan address=\"10.1007/s10664-013-9241-4\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eLanza, M., \u0026amp; Marinescu, R. (2006). \u003cem\u003eObject-Oriented Metrics in Practice\u003c/em\u003e. Springer. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/3-540-39538-5\u003c/span\u003e\u003cspan address=\"10.1007/3-540-39538-5\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eLavazza, L., \u0026amp; Morasca, S. (2016a). An Empirical Evaluation of Distribution-based Thresholds for Internal Software Measures. \u003cem\u003eProceedings of the The 12th International Conference on Predictive Models and Data Analytics in Software Engineering\u003c/em\u003e, 1\u0026ndash;10. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/2972958.2972965\u003c/span\u003e\u003cspan address=\"10.1145/2972958.2972965\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eLavazza, L., \u0026amp; Morasca, S. (2016b). Identifying Thresholds for Software Faultiness via Optimistic and Pessimistic Estimations. \u003cem\u003eInternational Symposium on Empirical Software Engineering and Measurement\u003c/em\u003e, \u003cem\u003e08-09-Sept\u003c/em\u003e, 1\u0026ndash;10. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/2961111.2962595\u003c/span\u003e\u003cspan address=\"10.1145/2961111.2962595\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eLewowski, T., \u0026amp; Madeyski, L. (2020). Creating Evolving Project Data Sets in Software Engineering. \u003cem\u003eStudies in Computational Intelligence\u003c/em\u003e (Vol. 851, pp. 1\u0026ndash;14). Springer. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/978-3-030-26574-8_1\u003c/span\u003e\u003cspan address=\"10.1007/978-3-030-26574-8_1\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eLinstead, E., Bajracharya, S., Ngo, T., Rigor, P., Lopes, C., \u0026amp; Baldi, P. (2009). Sourcerer: Mining and searching internet-scale software repositories. \u003cem\u003eData Mining and Knowledge Discovery\u003c/em\u003e, \u003cem\u003e18\u003c/em\u003e(2), 300\u0026ndash;336. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10618-008-0118-x\u003c/span\u003e\u003cspan address=\"10.1007/s10618-008-0118-x\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eLohr, S. L. (2021). \u003cem\u003eSampling: design and analysis\u003c/em\u003e. Chapman and Hall/CRC. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1201/9780429298899\u003c/span\u003e\u003cspan address=\"10.1201/9780429298899\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eLorenz, M., \u0026amp; Kidd, J. (1994). \u003cem\u003eObject-oriented software metrics: a practical guide\u003c/em\u003e. Prentice-Hall.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMa, Y., Dey, T., Bogart, C., Amreen, S., Valiev, M., Tutko, A., Kennard, D., Zaretzki, R., \u0026amp; Mockus, A. (2021). World of code: enabling a research workflow for mining and analyzing the universe of open source VCS data. \u003cem\u003eEmpirical Software Engineering\u003c/em\u003e, \u003cem\u003e26\u003c/em\u003e(2), 22. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10664-020-09905-9\u003c/span\u003e\u003cspan address=\"10.1007/s10664-020-09905-9\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMahdieh, M., Mirian-Hosseinabadi, S. H., Etemadi, K., Nosrati, A., \u0026amp; Jalali, S. (2020). Incorporating fault-proneness estimations into coverage-based test case prioritization methods. \u003cem\u003eInformation and Software Technology\u003c/em\u003e, \u003cem\u003e121\u003c/em\u003e, 106269. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.infsof.2020.106269\u003c/span\u003e\u003cspan address=\"10.1016/j.infsof.2020.106269\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMccabe, T. J. (1976). A Complexity Measure. \u003cem\u003eIEEE Transactions on Software Engineering\u003c/em\u003e, \u003cem\u003eSE-2\u003c/em\u003e(4), 308\u0026ndash;320. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/TSE.1976.233837\u003c/span\u003e\u003cspan address=\"10.1109/TSE.1976.233837\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMehboob, B., \u0026amp; Chong, C. Y. (2023). A metadata driven process for assessing stability and reusability based on risk of change of software systems. \u003cem\u003eSoftware: Practice and Experience\u003c/em\u003e, \u003cem\u003e53\u003c/em\u003e(5), 1218\u0026ndash;1248. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/spe.3187\u003c/span\u003e\u003cspan address=\"10.1002/spe.3187\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMo, R., Zhao, Y., Feng, Q., \u0026amp; Li, Z. (2021). The existence and co-modifications of code clones within or across microservices. \u003cem\u003eInternational Symposium on Empirical Software Engineering and Measurement\u003c/em\u003e, 1\u0026ndash;11. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/3475716.3475784\u003c/span\u003e\u003cspan address=\"10.1145/3475716.3475784\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMorasca, S. (2009). A probability-based approach for measuring external attributes of software artifacts. \u003cem\u003e2009 3rd International Symposium on Empirical Software Engineering and Measurement\u003c/em\u003e, 44\u0026ndash;55. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/ESEM.2009.5316048\u003c/span\u003e\u003cspan address=\"10.1109/ESEM.2009.5316048\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMorasca, S., \u0026amp; Lavazza, L. (2016). Slope-based fault-proneness thresholds for software engineering measures. \u003cem\u003eInternational Conference on Evaluation and Assessment in Software Engineering\u003c/em\u003e, \u003cem\u003e01-03-June\u003c/em\u003e, 1\u0026ndash;10. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/2915970.2915997\u003c/span\u003e\u003cspan address=\"10.1145/2915970.2915997\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMorasca, S., \u0026amp; Lavazza, L. (2019). Comparing the effectiveness of using design and code measures in software faultiness estimation. \u003cem\u003eInternational Conference on Evaluation and Assessment in Software Engineering\u003c/em\u003e, 112\u0026ndash;121. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/3319008.3319026\u003c/span\u003e\u003cspan address=\"10.1145/3319008.3319026\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMunaiah, N., Kroh, S., Cabrey, C., \u0026amp; Nagappan, M. (2017). Curating GitHub for engineered software projects. \u003cem\u003eEmpirical Software Engineering\u003c/em\u003e, \u003cem\u003e22\u003c/em\u003e(6), 3219\u0026ndash;3253. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10664-017-9512-6\u003c/span\u003e\u003cspan address=\"10.1007/s10664-017-9512-6\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMunger, K. (2019). The Limited Value of Non-Replicable Field Experiments in Contexts With Low Temporal Validity. \u003cem\u003eSocial Media\u0026thinsp;+\u0026thinsp;Society\u003c/em\u003e, \u003cem\u003e5\u003c/em\u003e(3), 205630511985929. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1177/2056305119859294\u003c/span\u003e\u003cspan address=\"10.1177/2056305119859294\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eNejmeh, B. A. (1988). NPATH: a measure of execution path complexity and its applications. \u003cem\u003eCommunications of the ACM\u003c/em\u003e, \u003cem\u003e31\u003c/em\u003e(2), 188\u0026ndash;200. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/42372.42379\u003c/span\u003e\u003cspan address=\"10.1145/42372.42379\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eOhira, M., Kashiwa, Y., Yamatani, Y., Yoshiyuki, H., Maeda, Y., Limsettho, N., Fujino, K., Hata, H., Ihara, A., \u0026amp; Matsumoto, K. (2015). A dataset of high impact bugs: Manually-classified issue reports. \u003cem\u003eIEEE International Working Conference on Mining Software Repositories\u003c/em\u003e, \u003cem\u003e2015-Augus\u003c/em\u003e, 518\u0026ndash;521. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/MSR.2015.78\u003c/span\u003e\u003cspan address=\"10.1109/MSR.2015.78\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003ePalomba, F., Bavota, G., Di Penta, M., Fasano, F., Oliveto, R., \u0026amp; De Lucia, A. (2018). A large-scale empirical study on the lifecycle of code smell co-occurrences. \u003cem\u003eInformation and Software Technology\u003c/em\u003e, \u003cem\u003e99\u003c/em\u003e, 1\u0026ndash;10. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.infsof.2018.02.004\u003c/span\u003e\u003cspan address=\"10.1016/j.infsof.2018.02.004\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eRankovic, D., Rankovic, N., Ivanovic, M., \u0026amp; Lazic, L. (2021). Convergence rate of Artificial Neural Networks for estimation in software development projects. \u003cem\u003eInformation and Software Technology\u003c/em\u003e, \u003cem\u003e138\u003c/em\u003e, 106627. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/J.INFSOF.2021.106627\u003c/span\u003e\u003cspan address=\"10.1016/J.INFSOF.2021.106627\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eReis, S., Abreu, R., \u0026amp; Cruz, L. (2021). Fixing vulnerabilities potentially hinders maintainability. \u003cem\u003eEmpirical Software Engineering\u003c/em\u003e, \u003cem\u003e26\u003c/em\u003e(6), 1\u0026ndash;27. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/S10664-021-10019-Z\u003c/span\u003e\u003cspan address=\"10.1007/S10664-021-10019-Z\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eRousseau, G., Di Cosmo, R., \u0026amp; Zacchiroli, S. (2020). Software provenance tracking at the scale of public source code. \u003cem\u003eEmpirical Software Engineering\u003c/em\u003e, \u003cem\u003e25\u003c/em\u003e(4), 2930\u0026ndash;2959. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10664-020-09828-5\u003c/span\u003e\u003cspan address=\"10.1007/s10664-020-09828-5\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSchnappinger, M., Fietzke, A., \u0026amp; Pretschner, A. (2021). Human-level Ordinal Maintainability Prediction Based on Static Code Metrics; Human-level Ordinal Maintainability Prediction Based on Static Code Metrics. \u003cem\u003eEvaluation and Assessment in Software Engineering\u003c/em\u003e, \u003cem\u003e10\u003c/em\u003e, 160\u0026ndash;169. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/3463274\u003c/span\u003e\u003cspan address=\"10.1145/3463274\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSehgal, R., Mehrotra, D., \u0026amp; Nagpal, R. (2020). \u003cem\u003eNew Metrics for Predicting the Reliability of Individual Component Based on Software Design Metrics\u003c/em\u003e (pp. 79\u0026ndash;94). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/978-981-15-2455-4_4\u003c/span\u003e\u003cspan address=\"10.1007/978-981-15-2455-4_4\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eShapiro, S. S., \u0026amp; Wilk, M. B. (1965). An Analysis of Variance Test for Normality (Complete Samples). \u003cem\u003eBiometrika\u003c/em\u003e, \u003cem\u003e52\u003c/em\u003e(3/4), 591. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.2307/2333709\u003c/span\u003e\u003cspan address=\"10.2307/2333709\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eShatnawi, R., \u0026amp; Li, W. (2008). The effectiveness of software metrics in identifying error-prone classes in post-release software evolution process. \u003cem\u003eJournal of Systems and Software\u003c/em\u003e, \u003cem\u003e81\u003c/em\u003e(11), 1868\u0026ndash;1882. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.jss.2007.12.794\u003c/span\u003e\u003cspan address=\"10.1016/j.jss.2007.12.794\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSheskin, D. J. (2000). \u003cem\u003eParametric and Nonparametric Statistical Procedures\u003c/em\u003e (2nd ed.). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e\u003c/span\u003e\u003cspan address=\"http://www.crcpress.com\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eShull, F., Singer, J., \u0026amp; Sj\u0026oslash;berg, D. I. K. (Eds.). (2008). \u003cem\u003eGuide to Advanced Empirical Software Engineering\u003c/em\u003e. Springer London. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/978-1-84800-044-5\u003c/span\u003e\u003cspan address=\"10.1007/978-1-84800-044-5\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSpadini, D., Schvarcbacher, M., Oprescu, A. M., Bruntink, M., \u0026amp; Bacchelli, A. (2020). Investigating Severity Thresholds for Test Smells. \u003cem\u003eProceedings of the 17th International Conference on Mining Software Repositories\u003c/em\u003e, 311\u0026ndash;321. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/3379597.3387453\u003c/span\u003e\u003cspan address=\"10.1145/3379597.3387453\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eTempero, E., Anslow, C., Dietrich, J., Han, T., Li, J., Lumpe, M., Melton, H., \u0026amp; Noble, J. (2010). The Qualitas Corpus: A curated collection of Java code for empirical studies. \u003cem\u003eProceedings - Asia-Pacific Software Engineering Conference, APSEC\u003c/em\u003e, 336\u0026ndash;345. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/APSEC.2010.46\u003c/span\u003e\u003cspan address=\"10.1109/APSEC.2010.46\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eTerragni, V., Salza, P., \u0026amp; Pezz\u0026egrave;, M. (2020). \u003cem\u003eMeasuring Soft-ware Testability Modulo Test Quality\u003c/em\u003e. \u003cem\u003e11\u003c/em\u003e(20). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/3387904.3389273\u003c/span\u003e\u003cspan address=\"10.1145/3387904.3389273\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eTibshirani, R., Walther, G., \u0026amp; Hastie, T. (2001). Estimating the Number of Clusters in a Data Set Via the Gap Statistic. \u003cem\u003eJournal of the Royal Statistical Society Series B: Statistical Methodology\u003c/em\u003e, \u003cem\u003e63\u003c/em\u003e(2), 411\u0026ndash;423. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1111/1467-9868.00293\u003c/span\u003e\u003cspan address=\"10.1111/1467-9868.00293\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eVale, G., Fernandes, E., \u0026amp; Figueiredo, E. (2019). On the proposal and evaluation of a benchmark-based threshold derivation method. \u003cem\u003eSoftware Quality Journal\u003c/em\u003e, \u003cem\u003e27\u003c/em\u003e(1), 275\u0026ndash;306. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s11219-018-9405-y\u003c/span\u003e\u003cspan address=\"10.1007/s11219-018-9405-y\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eVidoni, M. (2022). A systematic process for Mining Software Repositories: Results from a systematic literature review. \u003cem\u003eInformation and Software Technology\u003c/em\u003e, \u003cem\u003e144\u003c/em\u003e, 106791. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.infsof.2021.106791\u003c/span\u003e\u003cspan address=\"10.1016/j.infsof.2021.106791\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eVinutha, H. P., Poornima, B., \u0026amp; Sagar, B. M. (2018). Detection of outliers using interquartile range technique from intrusion dataset. \u003cem\u003eAdvances in Intelligent Systems and Computing\u003c/em\u003e, \u003cem\u003e701\u003c/em\u003e, 511\u0026ndash;518. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/978-981-10-7563-6_53/FIGURES/5\u003c/span\u003e\u003cspan address=\"10.1007/978-981-10-7563-6_53/FIGURES/5\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eWilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J. W., da Silva Santos, L. B., Bourne, P. E., Bouwman, J., Brookes, A. J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C. T., Finkers, R., \u0026amp; Mons, B. (2016). The FAIR Guiding Principles for scientific data management and stewardship. \u003cem\u003eScientific Data\u003c/em\u003e, \u003cem\u003e2016 3:1\u003c/em\u003e(1), 1\u0026ndash;9. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/sdata.2016.18\u003c/span\u003e\u003cspan address=\"10.1038/sdata.2016.18\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. \u003cem\u003e3\u003c/em\u003e.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eWohlin, C., \u0026amp; Aurum, A. (2015). Towards a decision-making structure for selecting a research design in empirical software engineering. \u003cem\u003eEmpirical Software Engineering\u003c/em\u003e, \u003cem\u003e20\u003c/em\u003e(6), 1427\u0026ndash;1455. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10664-014-9319-7\u003c/span\u003e\u003cspan address=\"10.1007/s10664-014-9319-7\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eWohlin, C., Runeson, P., H\u0026ouml;st, M., Ohlsson, M. C., Regnell, B., \u0026amp; Wessl\u0026eacute;n, A. (2012). \u003cem\u003eExperimentation in Software Engineering\u003c/em\u003e (Vol. 9783642290). Springer. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/978-3-642-29044-2\u003c/span\u003e\u003cspan address=\"10.1007/978-3-642-29044-2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eWu, Z., Wang, Z., Chen, J., You, H., Yan, M., \u0026amp; Wang, L. (2024). Stratified random sampling for neural network test input selection. \u003cem\u003eInformation and Software Technology\u003c/em\u003e, \u003cem\u003e165\u003c/em\u003e, 107331. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.infsof.2023.107331\u003c/span\u003e\u003cspan address=\"10.1016/j.infsof.2023.107331\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"software-quality-journal","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"sqjo","sideBox":"Learn more about [Software Quality Journal](http://link.springer.com/journal/11219)","snPcode":"11219","submissionUrl":"https://submission.nature.com/new-submission/11219/3","title":"Software Quality Journal","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Sampling software, Empirical evaluation, Temporal validity, Metric thresholds","lastPublishedDoi":"10.21203/rs.3.rs-7217702/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7217702/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cem\u003e\u003cstrong\u003eContext: \u003c/strong\u003e\u003c/em\u003eIn empirical research, drawing reliable conclusions about a target population requires working with representative samples. Representativeness refers to the degree to which a sample's properties of interest resemble those of the target population. However, a sample that was representative in the past might not be representative in the present day if the population has significantly evolved during that period.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e\u003cstrong\u003eObjective:\u003c/strong\u003e\u003c/em\u003eTo evaluate the effectiveness of a dataset extraction tool for collecting current samples of software repositories and keeping their temporal validity over time.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e\u003cstrong\u003eMethod: \u003c/strong\u003e\u003c/em\u003eWe performed a Mining Software Repositories study utilizing three datasets: Tempero et al.’s Qualitas Corpus, a sample from Github and an updated version of the Qualitas Corpus. Based on these datasets, we generated thresholds for three source code metrics (Lines of Code, Cyclomatic Complexity and Weighted Methods per Class) and compared whether these thresholds yielded consistent results.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e\u003cstrong\u003eResults:\u003c/strong\u003e\u003c/em\u003e We observed significant differences in all the source code metrics under study when pairing the Qualitas Corpus and samples containing projects with recent development data, with the former registering higher thresholds. Furthermore, the thresholds obtained from the samples collected with our extraction tool recorded consistent thresholds. \u003cem\u003e\u003cstrong\u003eConclusions: \u003c/strong\u003e\u003c/em\u003eUsing outdated code-based datasets in empirical studies can affect study results, therefore, it is important that researchers not only publish their datasets but also provide strategies to update those datasets over time. Additionally, we presented and validated sampling approaches implemented demonstrating their effectiveness to collect current samples.\u003c/p\u003e","manuscriptTitle":"Temporal validity of software datasets for code metrics: an empirical assessment of sampling strategies","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-08-25 09:56:13","doi":"10.21203/rs.3.rs-7217702/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"editorInvitedReview","content":"","date":"2026-04-09T05:17:53+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"276565700786096504045715328084153171873","date":"2026-02-08T17:08:44+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-10-23T02:43:13+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"129554504752835845213786314773602605286","date":"2025-08-19T00:04:30+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"182436661513681961506781616179901158521","date":"2025-08-18T23:32:00+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-08-16T12:30:46+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-07-29T01:01:45+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-07-29T01:01:26+00:00","index":"","fulltext":""},{"type":"submitted","content":"Software Quality Journal","date":"2025-07-26T00:42:09+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"software-quality-journal","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"sqjo","sideBox":"Learn more about [Software Quality Journal](http://link.springer.com/journal/11219)","snPcode":"11219","submissionUrl":"https://submission.nature.com/new-submission/11219/3","title":"Software Quality Journal","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"c139946c-4eb3-400e-a755-ee5dd766ba95","owner":[],"postedDate":"August 25th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2025-08-25T09:56:13+00:00","versionOfRecord":[],"versionCreatedAt":"2025-08-25 09:56:13","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7217702","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7217702","identity":"rs-7217702","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}