Improve Classification of Security Bug Reports using fasttext. A Machine Learning Based Approach

Improve Classification of Security Bug Reports using fasttext. A Machine Learning Based Approach | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Improve Classification of Security Bug Reports using fasttext. A Machine Learning Based Approach Sultan S. Alqahtani This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-2263306/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Software developers must handle security bug reports (SBRs) before they are widely disclosed, and the system becomes vulnerable to hackers. Bug tracking systems may contain many securities-related reports which are unlabelled as SBRs. Therefore, finding unlabelled SBRs is a challenge to help security engineers identify these security issues fast and accurately. Although many methods have been proposed for classifying SBRs, challenging issues remain due to selecting an accurate and high-performance classification algorithm. This motivates us to tackle the challenges faced by the state-of-the-art SBRs classification methods by selecting a high-performance machine learning algorithm. Therefore, the main goal of this paper is to automate the process of determining which bug report can be labeled as SBR through the use of machine learning techniques. We first extracted 45,940 bug reports from publicly available datasets of five software repositories (e.g., the work of Peters et al. and Shu et al.). Second, we conducted a study on the classification of SBRs using machine learning, where we built a fasttext classifier. We then examined the accuracy of using fasttext in detecting SBRs. Our results show that fasttext can identify SBRs with an average F1 score of 0.81. Furthermore, we investigated the generalizability of identifying SBRs by applying cross-project validation, and our results show that the fasttext classifier achieves an average F1 value of 0.65. Data and results are available at https://github.com/isultane/fasttext_classifications . maintenance bug reports machine learning security software vulnerabilities Figures Figure 1 Figure 2 Figure 3 Figure 4 1. Introduction Maintenance is an essential task in the life cycle of software engineering projects, which implies several activities. For example, any potential flaws in the source code should be removed to keep it up-to-date and preserve its performance and correctness. At the same time, software engineers (i.e., maintainers) should invest some effort to maintain the mentioned tasks and help to keep the software maintenance cost low [1]. To achieve that, software maintainers rely on tools that help them keep track of and monitor software issues. For instance, issue tracking systems play an important role which helps software maintainers maintain software products by enabling rigorous, practical software evolution tasks. One example of issue tracking systems is Bug Tracking Systems (BTRs). BTRs help the developers to report and describe bugs encountered while using these products of software projects. However, some bug reports can implicitly describe security vulnerabilities that software attackers could exploit if they are exposed before being fixed [2]. A security vulnerability is a security bug issue that allows users of the product to have unauthorized access to the systems’ capabilities and thus cause harm or damage to the software [3]. Usually, project managers request that issue reporters not disclose any suspected security issues in public BTRs. Instead, the security problems should be reported directly to the software security team to provide patches (when necessary) to the product users before the security problem is discovered and exploited by attackers. However, once the patch has been published, security issues are often disclosed via public BTRs [4]. Peters et al. [5] noted that due to the lack of security expertise knowledge, bug reports are sometimes mislabeled as non-security bug reports (NSBRs) and are often publicly disclosed before they are assessed and fixed [6]. There were real examples when developers declared the SBR as an NSBR. As is shown in Figure 1, a clear SBR from the Apache Ambari project (version.4.1) shows it is mislabelled as an NSBR. At the same time, it is mentioned in the text description (highlighted in red) that it is a security issue. Previous studies presented several approaches to machine learning-based SBR classification [2], [5], [7]–[10]. For example, Peters et al. [5] proposed the FARSEC framework, a SBR classification method. The framework [5] combines filtering and ranking methods to reduce the mislabelling of SBRs by text-based classification models. Shu et al.[2] replicated and improved the FARSEC approach by applying hyper-parameter optimization that has been used before in software engineering (e.g., for software defect classification [11], [12] or effort estimation [13]). Wu et al. [7] explored the reasons that led to the poor performance of Peters et al. [5] and Shu et al.[2] and found one main reason: the quality of labels assigned to the bug reports in the datasets. In their finding, they evaluated the impacts of label correctness of two types of datasets: noise before they correct mislabeled bug reports) and clean (i.e., after they manually fix mislabeled bug reports). Their results showed that the clean datasets improved the performance of the classification models. The approaches above applied by these three studies ([5], [2], and [7]) used traditional machine learning algorithms (e.g., Random Forest, Naïve Bayes, Logistical Regression, etc.), and additionally, they are complex and time-consuming. For example, Shue et al. [2] applied hyper-parameter tuning for the learner, costing five hours to optimize the Chromium dataset. While optimizing the learner on the datasets is essential, it is good practice to explore a simple, practical, more accurate, and efficient approach[14], [15]. Therefore, we investigate the effectiveness of simple text classification for SBRs using fasttext (machine learning-based approach) to address the problem of how to distinguish (i.e., classify) SBRs more efficiently and accurately . Our study aims to answer the following research questions: RQ1: Can we accurately detect SBR using a fasttext machine learning algorithm? We built a fasttext classifier using publicly available bug report datasets ([5], [7], [16]) and compared its performance to the baseline, which is the ratio of SBRs in the studied projects. The results show that a fasttext classifier achieves a higher F1 score of 0.81, on average. This improvement equates to an average gain of +0.15 by the fasttext classifier compared to our baseline. RQ2: How effective is the fasttext classifier when applied on cross-projects? To examine the generalizability of our proposed technique, we determined the effectiveness of our machine learning technique in classifying/predicting SBRs in cross-projects. We asked RQ2: How effective is the fasttext classifier when applied on cross-projects? We built a general classifier and evaluated its performance using cross-project validation. Our results show that our classifier achieves an average F1 score of 0.65, which is lower than the within-project classifiers. However, it is still a good classifier that can be practically used. The results also show that cross-project classifier performance corresponds to an average F1-score improvement of +0.19 over our baseline. This paper makes the following contributions: To the best of our knowledge, we are the first to experimentally evaluate the impact of the fasttext model (machine learning approach) for SBR classification. We find that (1), with clean datasets, fasttext simple text classification outperforms the five baseline approaches, and (2) fasttext performs better in terms of running time and a machine’s energy consumption compared to state-of-the-art methods. These findings provide research clues and guidance for researchers and practitioners of SBR classification and prediction. The remainder of this paper is organized as follows. Section 2 provides the background. Section 3 describes our experimental case study. Section 4 details our case study results for each research question. We discuss our findings in Section 5. The threats to validity are discussed in Section 6. Section 7 concludes our paper. 2. Background 2.1 Security Bug Report Classification In this sub-section, we recall the four recent SBR classification works that inspired our study. Peters et al. [ 5 ] pointed out the mislabel problem of the five publicly available SBR prediction datasets: Chromium, Ambari, Camel, Derby, and Wicket. They designed a framework named FARSEC to improve SBR prediction by filtering out noisy data from NSBRs and extending the text mining approach with the security keywords matrix. On the other hand, Shu et al. [ 2 ] improved the performance of SBR prediction and classification by applying a hyper-parameter optimization approach by conducting their study based on Peters et al.’s work. They optimized the control parameters of the classification algorithm by using a differential evolution algorithm and the data-preprocessing approach SMOTE [ 17 ]. However, labeling a large dataset to build a classification model is needed in the research of mining software repositories. Therefore, the labels' correctness will significantly affect the model's performance [ 7 ]. Wu et al. [ 7 ] performed a case study on SBR classification. They found that the dataset used by [ 2 ], [ 5 ] for SBR classification contains many mislabeled instances, which leads to reduced performance of SBR classification models. They first improved the label correctness of the five datasets (dataset originally published by Ohire et al. [ 16 ]). They manually analyzed each bug report to find mislabeled bug reports (749 SBRs labeled NSBRs). In this finding, they evaluated the impacts of label correctness for two types of datasets: noisy (i.e., before they corrected mislabeled bug reports) and clean (i.e., after they manually corrected mislabeled bug reports). Their results showed that the clean datasets improved the performance of the classification models. In contrast to Wu et al. [ 7 ], we relied on their clean dataset to test our approach. We are different in using a new classification model based on a machine learning algorithm, which results in more enhanced performance. 2.2 Text Classification Using fasttext classifier Deep learning techniques have achieved great success recently in software engineering and security domains by automatically extracting context-sensitive features from raw software text artifacts. This area includes various tasks in malicious software detection [ 18 ], deep learning models for IoT cybersecurity [ 19 ], applying transformers to cybersecurity [ 20 ], and more. The common issue with these deep learning models is low efficiency and hard for portability. fasttext is a machine learning method for text categorization and multi-class text classification. It is often on par with deep learning classifiers in terms of accuracy, and many orders of magnitude faster for training and evaluation [ 21 ]. The sentence/document vector in fasttext is represented and constructed by averaging the embeddings of the words in the document. And for the text classification task, multinomial logistic regression is used, where the sentence/document vector corresponds to the features. When applying fasttext on problem with large number of classes, the hierarchical softmax layer is applied to speed up the computation. The simplicity of fasttext makes it very efficient compared to the state-of-the-art performances and faster than other computing methods [ 22 ]. Model training - hierarchical classifier. fasttext uses a hierarchal classifier that represents the labels in a binary tree (as depicted in Fig. 2 ). Every node in the binary tree represents a probability, so a label is represented by the probability along the path to that given label. The reason for a binary tree is that it speeds up the search time and eliminates having to go through all the different elements being searched for. For example, Fig. 2 shows a piece of text that shows the championship decision, and we have this binary tree representation of all the labels instead of computing a score for every single possible label; what we have to calculate is just the probability of each node on the path to that one correct label. This vastly decreases the computations we have to do for each piece of text. So when we have many labels, this increases the speed of the model’s training and reduces the time complexity. And not only does this increase the speed compared to linear classifiers, but it’s also faster than some deep learning techniques, like neural networks [ 23 ]. In section 3.2, we explained in more detail how to use this machine learning algorithm in real-world examples (e.g., classifying bug reports). 3. Case Study Design The main goal of our study is to determine whether a bug report can be classified as a security bug report (SBR) or a non-security bug report (NSBR). To achieve this goal, we propose using machine learning techniques—based on a machine learning algorithm. Figure 3 depicts the whole picture of our approach. We began by downloading the dataset used by [16], which was manually annotated by [5], and reviewed and refined by [7]. The authors of the updated dataset [7] not only arranged for experienced software security experts to review SBRs but also developed a specific manual review process. They used software vulnerability types defined by CWE [1] as a basis and generated a codebook as guidelines for judging whether a bug is an SBR [7]. Then, we mined the selected software repositories and annotated bug reports to prepare SBR and NSBR features (in fasttext format) and used them as dependent variables in our machine learning classifier. In the following subsections, we detail our data extraction and preprocessing steps. 3.1 Dataset In this subsection, we introduce the dataset used in our study. To determine how practical our approach is in classifying bug reports (as SBR or NSBR), we need to have a labeled test dataset on which we can apply our system. Our first criterion for building a testing dataset is to have projects with a sufficient number of bug reports that discuss security concerns. As mentioned earlier, we resort to using the dataset [7]. We use a total of five projects datasets, which are the bug reports initially labeled as SBRs or NSBRs, and published by Ohira et al. [16]. The five projects (see Table 1) are Chromium, Amabri, Camel, Derby, and Wicket. The dataset was originally publicly published by Ohira et al. [16] and supplied as comma-separated value (CSV) files. Each row represents a bug report, and the columns are features of the reports such as bug-id, title, description, data , and time a report was submitted and fixed. Table 1: Projects in the Testing Datasets [5] Project Period # of Bug reports SBR % of SBR Chromium Aug-2008 to Jun-2010 41,940 192 0.5 Ambari Sep 2011 to Aug 2014 1,000 29 3.0 Camel Jul 2007 to Sep 2013 1,000 32 3.0 Derby Sep 2004 to Sep 2014 1,000 88 9.0 Wicket Oct 2006 to Nov 2014 1,000 10 1.0 Average 9,188 70 3.0 Median 1,000 32 3.0 To further prepare the data for our experiments, we scrubbed the files by selecting only the necessary columns from the data of Wu et al. [7], we selected description, summary, and security . The security column is a binary value of 0 or 1, 0 represents non-security, and 1 illustrates a security bug report. We then combined the description and summary columns. At this point, the data of each project is partitioned into two parts. The first part is used for training, while the second is used for testing. Preprocessing step. As shown in Figure 3 , in step 2, we preprocess the text of the bug report description with the basic text preprocess approach parse_sentence and initialize_words imported from the utility [2] library package for text tokenization and word list initialization for each bug report. This step converts the text of the bug report description (i.e., we extract the related information title and body of the bug report and concatenate them into a single textual paragraph) to a preprocessed line of words (see Figure 4 ). Stop words like a, the, and, presumed to be uninformative in representing the content of a text, are removed to avoid being constructed as a signal for prediction. We then tokenize the resulting text and derive the bag of words representation from the tokenized text. As depicted in Figure 4 , each text file line contains a list of labels, followed by the corresponding document. All the labels start with the __label__ prefix, which is how fasttext recognizes what a label is and what is a word. As shown in Figure 4 , __ label__ nonsec-report label represents the annotated bug reports as NSBR, and __ label__ sec-report model represents the manually annotated bug reports as SBR. The model is then trained to predict the labels given the word in the document. 3.2 Classification Models Different methods can be used to investigate and analyze bug report information to predict the labels assigned. Regarding time and memory consumption, most machine learning models require much time for training and use a lot of memory [24]. We looked for a model that, ideally, achieved good results while not being memory and storage intensive, as we desired to deploy the model on low-end server hardware [3] . We decided to use fasttext, an open source tool by Meta (former Facebook AI research in 2016), to perform this task. fasttext uses linear models with a rank constraint and fast loss approximation [21]. As Joulin et al. [24] discussed, fasttext’s accuracy is competitive against several deep learning-based models, it is faster for training and evaluation. However, to perform our classification, we leveraged a fasttext classifier (Figure 3 – step 3) to classify whether a bug report is a security or not since it provides an intuitive and easy-to-explain classification model. The bug report pre-processing step, as we mentioned in Section 3.1, strictly depends on the method selected to perform the automated classification of bug reports and label assignments. The fasttext linear classifier is typically trained with sentences represented as a bag of words and n-grams [24] to embed information on word order partially. This is because local word order information tends to improve the text classification performance of the linear model. Thus, each bug report submitted to our system is passed to a fasttext-based classifier (as shown in Figure 3 – step 3) to perform classification and label assignment. Hierarchical representation of bug reports . In each bug report (i.e., an extracted bag of words representation), each word is represented by a vector of character n-gram. This process step represents the desired input of fasttext . To classify the bug report through fasttext , the main objective is to minimize the following objective function over possible labels: Where is a bag of features of the bug reports, the label, represents the weight dictionary of the average text embeddings, is the weight dictionary that converts the embedding to pre-softmax values for each class, and is the hierarchical softmax function used to minimize computational complexity. For the setting of fasttext , we used the tuning range for all default parameters. Table 2 shows the learner and pre-processer options we explored in this study for fasttext . Table 2: Setting up parameters for the fasttext model Target Parameters of Target Description Parameters Default Tunning Range fasttext lr epoch dim loos wordNgrams 0.1 5 100 softmax 1 [0.1, 0.3, 0.7, .9] [5, 10] [50, 100] [“ns”,”hs”,”softmax”,”ova”] [1, 2, 3] Learning rate The number of times each training example is seen in the algorithm Size of word vectors Loss function Max length of word ngram Dealing with imbalanced data. As shown in Table 1, our testing dataset has, on average, only 3% (median = 3%) labeled as SBR, meaning most of the bug reports are NSBR. This common issue in software engineering data is called data imbalance [25]. Imbalance in the datasets occurs when there is one class higher in appearance than the other in the dataset. In this case, the fasttext trains to learn from the features affecting the majority cases (i.e., NSBR) than the minority cases (i.e., SBR). To mitigate the impact of this issue, we relied on the re-sampling process to perform the under-sampling on the training dataset only. We ensured the class distribution was uniform in the output data, and the final sample size was the same as the original data. We must note that we applied this process (re-sampling step) to the training dataset only because we want to evaluate our trained model (classifier) on real-world examples where the bug report data is imbalanced. 3.3 Performance Evaluation To make the evaluation results objective, we use our study's performance metrics applied by [5], [7]. These metrics applied by the work of Peters et al. and Shu et al. include Recall ( R ), probability of false alarm ( pf ), Precision ( P ), F1-score ( F1 ), and G-measure ( G ). The first four measures are commonly used by many studies in the mining software engineering area [26]–[28] and empirical software engineering area [15], [29]. G-measure was first introduced by Peters et al. [5], while both F1-score and G-measure are harmonic means, and the G score considers the Recalls of both the majority and the minority classes [5]. However, in our study, there are four possible outcomes of the classification results: Is SBR Is NSBR Classified as SBR True Positive (TP) False Positive (FP) Classified as NSBR False Negative (FN) True Negative (TN) Based on these results, the performance metrics R, pf, P, F1, and G can be calculated as follows: However, among these five performance metrics, the higher R, P, F1, and G are, the better, while the lower pf is, the better. The final results (higher SBR label) will be ranked top to the security engineer as depicted in Figure 3 – steps 4, 5, and 6, to be verified as the correct label to assign to the bug report. [1] https://cwe.mitre.org/ [2] https://docs.python.org/3/c-api/utilities.html [3] 16 GB RAM, 20 GB SSD, Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz 1.99 GHz. 4. Case Study Results In this section, we present the results of our case study concerning the two research questions (RQ1 and RQ2). We offer the motivation, approach to answering the question, and results for each research question. 4.1 RQ1: Can we accurately detect SBR using a fasttext machine learning algorithm? Motivation : Correctly identifying a security bug report is essential to accelerate bug triaging and fixing [10] and mitigate the security impact. Even though prior work proposed classification models and techniques to automatically classify and predict SBRs from NSBRs [2], [5], [7], the accuracy of these models shows moderate performance. Thus, the main goal of this research question is to investigate the use of fasttext models to assist developers in automatically identifying SBRs from NSBRs. Approach : For each project in the testing dataset (Section 3.1), we used the manually annotated bug reports shown in Table 1 to train the fasttext classifier to predict whether a bug report is a security bug report or not. However, we used 10-fold cross-validation for each project [30]. First, we divided the dataset for each project into ten folds. We used eight folds (i.e., 80% of the data) to train the fasttext and the remaining two folds to evaluate the classifier’s performance. We ran this process ten times for each fold (i.e., 1x10-folds). Finally, to assess the performance of the fasttext classifier in detecting SBRs, we computed the well-known evaluation metrics, precision and recall, and their combination F1-score, as explained in Section 3.3, ten times for each fold. Then, to come up with one value for the ten runs, we computed the average of the evaluation metrics for 10-folds ten times (i.e., 1x10-fold) for every project in our testing dataset. Finally, we compared fasttext results with the KNN machine learning (ML) algorithm [31] that was successfully used in the assessment of what classifier provides the best accuracy for identifying security bug reports [32]. Results : Table 3 presents the precision, recall, f1-measure, pf, and g-measure values of the fasttext classifier for the five studied projects in our test dataset. First, the precision values obtained by the fasttext classifier range between 0.73 and 0.94 with an average of 0.83 (median = 0.83), while the recall values range between 0.71 and 0.94 with an average value of 0.81 (median = 0.79). Also, the f1-score achieves values between 0.71 and 0.94, with an average of 0.81 and a median of 0.80. The values of pf and G-measure indicate that the fasttext effectively detects SBRs from NSBRs, achieving averages of 0.20 (median = 0.19) and 0.80 (median=0.80), respectively. Table 3: Performance of the fasttext Technique. Project Precision Recall F-measure pf G-measure Ambari 0.83 0.79 0.79 (+0.22) 0.15 0.81 (+0.04) Camel 0.73 0.71 0.71 (+0.07) 0.34 0.67 (-0.10) Chromium 0.94 0.94 0.94 (+0.16) 0.08 0.93 (+0.05) Derby 0.82 0.79 0.80 (+0.15) 0.19 0.80 (+0.14) Wicket 0.85 0.81 0.81 (+0.13) 0.25 0.77 (- 0.07) Average 0.83 0.81 0.81 (+0.15) 0.20 0.80 (+0.01) Median 0.83 0.79 0.80 (+0.15) 0.19 0.80 (+0.02) Also, Table 3 shows the F1-score when comparing the performance of fasttext to the baseline. The compared F1-score shows an improvement of +0.15 on average over the baseline. In particular, for all five projects, the fasttext outperforms the baseline with relative F1-score values ranging between +0.07 and +0.22. Our fasttext classifier achieves an average F1-score of 0.81 (G-Measure of 0.80). Additionally, our results show that the fasttext classifier can effectively improve the detection of SBRs with an average F-score of +0.15 compared to the baseline. 4.2 RQ2: How effective is the fasttext classifier when applied on cross-projects? Motivation : Building a machine learning (ML) classifier to identify SBR requires having labeled data to train on. However, many open-source projects do not have sufficient historically labeled SBR data to build a classifier (e.g., unlabeled, minor, or new project). Thus, we cannot train a ML classifier to detect SBRs on data from these projects. For this research question, we investigated to what extent and with what accuracy a bug report can be automatically classified as a SBR using cross-project fasttext machine learning classification. Approach : We conduct a cross-project validation to understand better the generalizability of the results achieved by the fasttext classifier on bug report data from one project and apply it to another project. In particular, we experimented with five-fold cross-project validation. We conducted an experiment that trained a fasttext classifier on bug report data from four projects and used the resulting classifier to determine whether a bug report was SBR or NSBR in the remaining project, similar to related work [8] and [33]. We repeated this process five times for each project in the testing dataset. To evaluate the performance, we employed the well-known evaluation metrics where we computed the precision, recall, and F1-score, pf, and G-measure to measure the performance of the fasttext classifier. Finally, to examine the performance of the cross-project classifier concerning the random baseline, we computed the relative F1 score and compared the results with the baseline, as shown in RQ1. Results : Table 4 presents the results of our experiment. It presents each project's precision, recall, f-score, pf, and G-measure values. Table 4 shows that the fasttext classifier achieved performance of pf and G-score values of 0.29 and 0.27, and 0.67 and 0.69 for the average and median, respectively. Four projects out of five show good performance results. The projects Ambari, Camel, Chromium, and Wicket achieved high pf and G-score. Other projects show a moderate performance, including Derby, with values of 0.53 and 0.52 for the pf and G-score, respectively. Our results show that our classifier achieves an average F1 score of 0.65, which is lower than what we expected, however, it is still a good classifier that can be practically used. Finally, when we compare the performance of the cross-projects classifier to the baseline, our results show that the cross-project classifier shows an improvement of +0.19 on average over the baseline. The results show that cross-projects machine learning classifier can provide comparable performance to within-project classifier on SBR classification. For four out of the five studied projects, cross-projects classifier achieved G-measure values between 0.52-0.74 with an overall average equal to 0.67. Table 4: Performance of Cross-projects classification. Project Precision Recall F-measure pf G-measure Ambari 0.69 0.66 0.64 (+0.16) 0.16 0.74 (+0.04) Camel 0.67 0.67 0.66 (+0.26) 0.27 0.69 (+0.08) Chromium 0.68 0.66 0.65 (+0.19) 0.27 0.69 (+0.07) Derby 0.61 0.60 0.59 (+0.00) 0.53 0.52 (-0.23) Wicket 0.72 0.70 0.70 (+0.33) 0.23 0.73 (+0.17) Average 0.67 0.66 0.65 (+0.19) 0.29 0.67 (+0.03) Median 0.68 0.66 0.65 (+0.19) 0.27 0.69 (+0.07) 5. Discussion In this section, we examine the use of other ML classifiers to classify bug reports into SBR or NSBR by answering the following question: What classifier provides the best accuracy for identifying the security bug reports? So far, we used fasttext to determine a SBR, which showed a practical improvement over the baseline. However, fasttext is not the only text classification classifier. Thus, in this subsection, we investigate the use of other ML classifiers and compare their performance in identifying SBRs. We use our dataset prepared in Section 3.1 and the same approach described in Section 3 to train five other ML classifiers, namely, Random Forest (RF), Naive Bayes (NB), KNearest Neighbor (KNN), Multilayer Perceptron (MLP), and Logistical Regression (LR). We chose to examine these ML classifiers since they have different assumptions about the analyzed data, as well as having other characteristics in terms of execution speed and dealing with overfitting [7], [34], [35]. Also, they have been commonly used in the past in other software engineering and security studies (e.g., [2], [27], [33]). Table 5: Performance of using different classifiers to detect SBRs. Project fasttext NB KNN MLP LR RF F1-score G-measure F1-score G-measure F1-score G-measure F1-score G-measure F1-score G-measure F1-score G-measure Ambari 0.79 0.81 0.75 0.67 0.57 0.77 0.72 0.73 0.83 0.87 0.77 0.86 Camel 0.71 0.67 0.67 0.61 0.63 0.77 0.69 0.75 0.70 0.77 0.81 0.86 Chromium 0.94 0.93 0.89 0.86 0.78 0.88 0.94 0.96 0.95 0.96 0.95 0.97 Derby 0.80 0.80 0.76 0.73 0.65 0.66 0.81 0.84 0.76 0.79 0.77 0.83 Wicket 0.81 0.77 0.68 0.59 0.68 0.84 0.78 0.80 0.82 0.86 0.82 0.89 Average 0.81 0.80 0.75 0.69 0.66 0.78 0.79 0.82 0.81 0.85 0.82 0.88 Median 0.80 0.80 0.75 0.67 0.65 0.77 0.78 0.80 0.82 0.86 0.81 0.86 To comprehensively compare the different classifiers, we compare them in the following scenario. We perform a within-project evaluation, where the classifiers are trained and tested using non-overlapping data from the same project. And to examine the performance of within-project classification of the different ML classifiers concerning the random baseline, we compute the F1-score and G-measure. Table 5 shows the F1-score and G-measure for the examined five classifiers and the fasttext . As Table 5 shows, on average, RF, LR, and fasttext produce the highest F1-score values with an average of 0.82, 0.81, 0.81 (median = 0.81, 0.82, 0.80), respectively. At the same time, the MLP, NB, and KNN classifiers achieve the lowest performance with average F1 scores of 0.79, 0.75, and 066, Median = 0.78, 0.75, and 0.65, respectively, across all the studied projects. The RF classifiers perform better than the fasttext with a G-measure of 0.88 (median = 88). This corresponds to a significant improvement between 0.83 – 0.97, on average, in G-measure over the baseline. With an average G-score of 0.69 (median = 0.67), one ML classifier (i.e, NB) performs significantly low. The highest values achieved are produced again by the fasttext classifier, along with RF and LR . For example, three projects in the testing dataset have G-score values greater than 0.80. The results suggest that the fasttext classifier is one of the best ML classifiers to detect SBR issues. Cross-project Classification. Table 6 shows the results of the F1-score and G-measure that are achieved by the five classifiers for the cross-project validations. In this experiment, the RF classifier achieves the best results, followed by LR, and fasttext , with 0.65 average F1-score values and 0.78, 0.74, and 0.67 G-measure values, respectively. The other classifiers show low performance by achieving average F1-score values of 0.46 – 0.62. The cases of Wicket, Camel, Chromium, Derby, and Ambari projects are interesting where the RF classifier can achieve a high-performance value range between 0.72 – 0.86 for G-score values. In contrast, fasttext shows moderate performance by completing average G-scores of 0.74 and 0.73 for projects Ambari and Wicket, respectively. Also, a project produced poor results: Derby (average G-score=0.52). Table 6: Performance of using different classifiers to detect SBRs cross-project validation Project fasttext NB KNN MLP LR RF F1-score G-measure F1-score G-measure F1-score G-measure F1-score G-measure F1-score G-measure F1-score G-measure Ambari 0.64 0.74 0.60 0.67 0.49 0.70 0.66 0.71 0.66 0.79 0.46 0.72 Camel 0.66 0.69 0.53 0.44 0.40 0.61 0.62 0.64 0.67 0.80 0.70 0.81 Chromium 0.65 0.69 0.67 0.54 0.46 0.62 0.58 0.55 0.65 0.73 0.63 0.78 Derby 0.59 0.52 0.52 0.41 0.59 0.75 0.61 0.62 0.61 0.64 0.66 0.73 Wicket 0.70 0.73 0.65 0.58 0.37 0.56 0.63 0.66 0.66 0.73 0.79 0.86 Average 0.65 0.67 0.59 0.53 0.46 0.65 0.62 0.64 0.65 0.74 0.65 0.78 Median 0.65 0.69 0.60 0.54 0.46 0.62 0.62 0.64 0.66 0.73 0.66 0.78 6. Threats To Validity This section describes the threats to the validity of our study. 6.1 Internal Validity A threat to the internal validity of our study is the potential of mislabels in the datasets [7]. We were testing the publicly available dataset [2], [5], [7], in which the authors took several measures to reduce these threats. However, it is still challenging to guarantee whether a bug report is a SBR or NSBR when there is no clear definition for SBR, as noted by Ohira et al. [16]. To mitigate this threat, the SBR is defined based on the definitions of Common Weakness Enumeration (CWE [4] ), which is the most authoritative organization of vulnerability management, as a basis to judge whether a bug report is a SBR or not [7]. In addition, we used a data set that is annotated by six annotators who conducted manual reviews and have the rich practical experience and a deep understating of different vulnerability types and characteristics [7]. Another threat to the internal validity is the impact of tuning for fasttext hyper-parameters. To mitigate this threat, a set of tuning approaches have been considered in our experiments, including selecting the optimal values for more epochs(the number of times each training example is seen in the algorithm) and more significant learning rate – lr (Good values of the learning rate are in the range 0.1 - 1.0). In addition to that, we improved the model’s performance by using word bigrams (using option -wordNgrams) instead of just unigrams and ranging the values within a standard range [1 – 5] [5] . The experiment results show that this tuning approach increased the impact of the findings. 6.2 External Validity Threats to external validity are concerned with the generality of our findings. Our study is based on specific projects (four Apache projects and the Chromium project), hence our results may not hold for project writing in other environments. Generalizing our experiment’s results requires multiple labeled bug reports from different domain projects. And to improve the outcomes and refute the false positives, replication experiments are needed. However, the ML technique can be easily generalized to other projects by analyzing the unstructured English text (e.g., bug reports’ descriptions and titles) in different project environments. In this study, we evaluated the impact of fasttext by comparing the model’s performance when applied to cross-projects validation. Three baseline methods extracted from two recent SBR prediction works proposed by [5], [7] and simple classification models were applied. Moreover, a set of performance indicators, including Precision, Recall, F1-score, and G-measure, were used for the evaluation. Another external threat is that the dataset used in our study presents only open-source projects that do not reflect proprietary projects. Furthermore, we examined projects that published bug reports for open discussions on issue tracker systems, and different issue tracker systems could have more advanced features for classifying SBRs. That said, the GitHub issues tracker is one of the most popular issues tracker systems in software development environments, with basic features for publishing bug reports. [4] https://cwe.mitre.org/ [5] https://fasttext.cc/docs/en/options.html 7. Conclusion In this paper, we studied bug reports that developers tend to produce during maintaining software projects and posting on issue tracker systems. Specifically, our goal was to use machine learning techniques to classify bug reports that can be security bug reports (SBRs) or not security bug reports (NSBRs). To do so, we proposed using publicly available bug datasets extracted from five Java projects. Then, we built a fasttext classifier. We found that the fasttext classifier can effectively improve the classification of SBRs with an average F1-score of 0.81 (median = 0.80). It also achieves an average G-measure of 0.80 (median = 0.80), representing an improvement of + 0.19 on average over the state-of-the-art approach. Additionally, we investigated the generalizability of labeling bug reports (with security or not security labels) when we used cross-projects validation. Our results show that the cross-project validation achieves, on average, 0.65 and 0.67 for F1-score and G-measure, respectively. Declarations The author Sultan S. Alqahtani, declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. No funding was received to assist with the preparation of this manuscript. The author declare he has no financial interests The data that support the findings of this study are available from the corresponding author upon reasonable request. Sultan S. Alqahtani contributed to the study conception and design, including the material preparation, data collection and analysis. Also, Dr. Sultan prepared the first draft of the manuscript and proof-reader commented on previous versions of the manuscript. The author read and approved the final manuscript. References P. Floris and H. Vogt Harald, “How to save on software maintenance costs, omnext white pape,” vol. SOURCE 2 V, 2010. S. Rui, X. Tianpei, W. Laurie, and M. Tim, “Better Security Bug Report Classification via Hyperparameter Optimization,” https://arxiv.org/pdf/1905.06872.pdf , 2019. I. Chawla and S. K. Singh, “Automatic bug labeling using semantic information from LSI,” in 2014 Seventh International Conference on Contemporary Computing (IC3) , Aug. 2014, pp. 376–381, doi: 10.1109/IC3.2014.6897203. M. Bozorgi, L. K. Saul, S. Savage, and G. M. Voelker, “Beyond heuristics: learning to classify vulnerabilities and predict exploits,” in Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD ’10 , 2010, p. 105, doi: 10.1145/1835804.1835821. F. Peters, T. T. Tun, Y. Yu, and B. Nuseibeh, “Text Filtering and Ranking for Security Bug Report Prediction,” IEEE Trans. Softw. Eng. , vol. 45, no. 6, pp. 615–631, Jun. 2019, doi: 10.1109/TSE.2017.2787653. D. Wijayasekara, M. Manic, J. L. Wright, and M. McQueen, “Mining Bug Databases for Unidentified Software Vulnerabilities,” in 2012 5th International Conference on Human System Interactions , Jun. 2012, pp. 89–96, doi: 10.1109/HSI.2012.22. X. Wu, W. Zheng, X. Xia, and D. Lo, “Data Quality Matters: A Case Study on Data Label Correctness for Security Bug Report Prediction,” IEEE Trans. Softw. Eng. , vol. 48, no. 7, pp. 2541–2556, Jul. 2022, doi: 10.1109/TSE.2021.3063727. M. Sharma, P. Bedi, K. K. Chaturvedi, and V. B. Singh, “Predicting the priority of a reported bug using machine learning techniques and cross project validation,” in 2012 12th International Conference on Intelligent Systems Design and Applications (ISDA) , Nov. 2012, pp. 539–545, doi: 10.1109/ISDA.2012.6416595. N. Limsettho, H. Hata, A. Monden, and K. Matsumoto, “Automatic Unsupervised Bug Report Categorization,” in 2014 6th International Workshop on Empirical Software Engineering in Practice , Nov. 2014, pp. 7–12, doi: 10.1109/IWESEP.2014.8. A. D. Sawadogo, T. F. Guimard, Quentin Bissyandé, J. Kader Kaboré, Abdoul Klein, and N. Moha, “Early Detection of Security-Relevant Bug Reports using Machine Learning: How Far Are We?,” eprint arXiv:2112.10123 , 2021, [Online]. Available: https://ui.adsabs.harvard.edu/abs/2021arXiv211210123S/abstract. A. Agrawal and T. Menzies, “Is ‘better data’ better than ‘better data miners’?,” in Proceedings of the 40th International Conference on Software Engineering , May 2018, pp. 1050–1061, doi: 10.1145/3180155.3180197. W. Fu, T. Menzies, and X. Shen, “Tuning for software analytics: Is it really necessary?,” Inf. Softw. Technol. , vol. 76, pp. 135–146, Aug. 2016, doi: 10.1016/j.infsof.2016.04.017. M. Tianpei, Xia Rahul, Krishna Jianfeng, Chen George, Mathew Xipeng, Shen Tim, “Hyperparameter Optimization for Effort Estimation,” arxiv , 2018, doi: https://doi.org/10.48550/arXiv.1805.00336. W. Fu and T. Menzies, “Easy over hard: a case study on deep learning,” in Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering , Aug. 2017, pp. 49–60, doi: 10.1145/3106237.3106256. Z. Liu, X. Xia, A. E. Hassan, D. Lo, Z. Xing, and X. Wang, “Neural-machine-translation-based commit message generation: how far are we?,” in Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering , Sep. 2018, pp. 373–384, doi: 10.1145/3238147.3238190. M. Ohira et al. , “A Dataset of High Impact Bugs: Manually-Classified Issue Reports,” in 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories , May 2015, pp. 518–521, doi: 10.1109/MSR.2015.78. N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: Synthetic Minority Over-sampling Technique,” J. Artif. Intell. Res. , vol. 16, pp. 321–357, Jun. 2002, doi: 10.1613/jair.953. R. Abir and A. A. Moulay, “MALBERT: USING TRANSFORMERS FOR CYBERSECURITY AND MALICIOUS SOFTWARE DETECTION,” https://arxiv.org/pdf/2103.03806.pdf , 2021. M. Roopak, G. Yun Tian, and J. Chambers, “Deep Learning Models for Cyber Security in IoT Networks,” in 2019 IEEE 9th Annual Computing and Communication Workshop and Conference (CCWC) , Jan. 2019, pp. 0452–0457, doi: 10.1109/CCWC.2019.8666588. J. Yin, M. Tang, J. Cao, and H. Wang, “Apply transfer learning to cybersecurity: Predicting exploitability of vulnerabilities by description,” Knowledge-Based Syst. , vol. 210, p. 106529, Dec. 2020, doi: 10.1016/j.knosys.2020.106529. A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov, “Bag of Tricks for Efficient Text Classification,” in Proceedings of the 15th Conference of the {E}uropean Chapter of the Association for Computational Linguistics: Volume 2, Short Papers , Apr. 2017, pp. 427–431, [Online]. Available: https://aclanthology.org/E17-2068. R. Johnson and T. Zhang, “Semi-supervised Convolutional Neural Networks for Text Categorization via Region Embedding,” in Advances in Neural Information Processing Systems , 2015, vol. 28, [Online]. Available: https://proceedings.neurips.cc/paper/2015/file/acc3e0404646c57502b480dc052c4fe1-Paper.pdf. J. Liu, W.-C. Chang, Y. Wu, and Y. Yang, “Deep Learning for Extreme Multi-label Text Classification,” in Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval , Aug. 2017, pp. 115–124, doi: 10.1145/3077136.3080834. R. Kallis, A. Di Sorbo, G. Canfora, and S. Panichella, “Ticket Tagger: Machine Learning Driven Issue Classification,” in 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME) , Sep. 2019, pp. 406–409, doi: 10.1109/ICSME.2019.00070. Q. Song, Y. Guo, and M. Shepperd, “A Comprehensive Investigation of the Role of Imbalanced Learning for Software Defect Prediction,” IEEE Trans. Softw. Eng. , vol. 45, no. 12, pp. 1253–1269, Dec. 2019, doi: 10.1109/TSE.2018.2836442. Y. M. Mileva, V. Dallmeier, M. Burger, and A. Zeller, “Mining trends of library usage,” in Proceedings of the joint international and annual ERCIM workshops on Principles of software evolution (IWPSE) and software evolution (Evol) workshops , 2009, pp. 57--62, doi: 10.1145/1595808.1595821. M. Gegick, P. Rotella, and T. Xie, “Identifying security bug reports via text mining: An industrial case study,” in 7th IEEE Working Conference on Mining Software Repositories , May 2010, pp. 11–20, doi: 10.1109/MSR.2010.5463340. R. Scandariato, J. Walden, A. Hovsepyan, and W. Joosen, “Predicting Vulnerable Software Components via Text Mining,” IEEE Trans. Softw. Eng. , vol. 40, no. 10, pp. 993–1006, Oct. 2014, doi: 10.1109/TSE.2014.2340398. Y. Yang, X. Xia, D. Lo, T. Bi, J. Grundy, and X. Yang, “Predictive Models in Software Engineering: Challenges and Opportunities,” ACM Trans. Softw. Eng. Methodol. , vol. 31, no. 3, pp. 1–72, Jul. 2022, doi: 10.1145/3503509. D. Berrar, “Cross-Validation,” in Encyclopedia of Bioinformatics and Computational Biology , Elsevier, 2019, pp. 542–545. Z. Zhang, “Introduction to machine learning: k-nearest neighbors,” Ann. Transl. Med. , vol. 4, no. 11, pp. 218–218, Jun. 2016, doi: 10.21037/atm.2016.03.37. A. Alipour, A. Hindle, and E. Stroulia, “A contextual approach towards more accurate duplicate bug report detection,” in 2013 10th Working Conference on Mining Software Repositories (MSR) , May 2013, pp. 183–192, doi: 10.1109/MSR.2013.6624026. H. Peng, L. Bing, and M. Yutao, “Towards Cross-Project Defect Prediction with Imbalanced Feature Sets,” arXiv , p. 10, 2014, doi: https://doi.org/10.48550/arXiv.1411.4228. T. J. Ostrand, E. J. Weyuker, and R. M. Bell, “Predicting the location and number of faults in large software systems,” in IEEE Transactions on Software Engineering , Apr. 2005, vol. 31, no. 4, pp. 340–355, doi: 10.1109/TSE.2005.49. T. Ye, L. Zhang, L. Wang, and X. Li, “An Empirical Study on Detecting and Fixing Buffer Overflow Bugs,” in 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST) , Apr. 2016, pp. 91–101, doi: 10.1109/ICST.2016.21. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-2263306","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":151748179,"identity":"82a2a730-0565-432a-8ad5-b8a765e5bfa7","order_by":0,"name":"Sultan S. Alqahtani","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABB0lEQVRIie3OMUvEMBTA8VcKmSJdI4L9ChFBOMTzg7gYCrq0cOOBUk8KHe/mw/M7eEvmlAf2KwS8oV2cOuhy4CImIoJKKrfdkP8SEt6PPACfbzsLFICy521z/vXE/jOWmKGg4JsSIN+TvSSeYaO661Ue3RXluE2HZ1yF1ROFPHYRri94tXh8ZmxVlVrIJHtQJDmmgAcTF2HAkRJkoIUloSH0aI+CClwkntUvSN+RxYaMhLwxJFobkp+6CKiU406JjBsCQqL9hRgSCudiOh1V91PcXWpRMCHrbI7kcLDgmPQstmy6NUb7+rJ9fZNX2bQuWt2N8xPnYjb64xZ+/t43/4f4fD6f71cfoehi9V7UFFoAAAAASUVORK5CYII=","orcid":"","institution":"Al-Imam Mohammad Ibn Saud Islamic University","correspondingAuthor":true,"prefix":"","firstName":"Sultan","middleName":"S.","lastName":"Alqahtani","suffix":""}],"badges":[],"createdAt":"2022-11-11 13:29:15","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-2263306/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-2263306/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":29101074,"identity":"45f2171c-7b42-458d-ae5a-027d5bce9224","added_by":"auto","created_at":"2022-11-15 18:59:49","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":28449,"visible":true,"origin":"","legend":"\u003cp\u003eAn example of a security bug report from the Apache Ambari project mislabeled as a non-security bug report [7].\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-2263306/v1/234a866ce959a7074cb08bdb.png"},{"id":29100669,"identity":"d9fbb98b-9cfd-453e-8259-6127b13be21b","added_by":"auto","created_at":"2022-11-15 18:51:49","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":45239,"visible":true,"origin":"","legend":"\u003cp\u003eThe model architecture of fasttext for a sentence of text\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-2263306/v1/4ab78e42aa33808a851b426c.png"},{"id":29100670,"identity":"893b4d8c-f1c4-42a3-81f3-d6a50549c83d","added_by":"auto","created_at":"2022-11-15 18:51:49","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":141185,"visible":true,"origin":"","legend":"\u003cp\u003eData setting and essential process of experiments\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-2263306/v1/fe80399c4292c92314b4ff6f.png"},{"id":29100672,"identity":"f50b0bd2-d729-434f-a01e-406738496bdd","added_by":"auto","created_at":"2022-11-15 18:51:49","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":193687,"visible":true,"origin":"","legend":"\u003cp\u003efasttext input format for one example of Ambari’s bug report after preprocessing\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-2263306/v1/ddf7744be6d597d78aff62de.png"},{"id":30872226,"identity":"c572d69f-e5a4-4698-acc4-e1522746732d","added_by":"auto","created_at":"2022-12-29 09:29:43","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1024128,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-2263306/v1/7679c9bc-685b-4819-964c-998b9a123247.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Improve Classification of Security Bug Reports using fasttext. A Machine Learning Based Approach","fulltext":[{"header":"1.\tIntroduction","content":"\u003cp\u003eMaintenance is an essential task in the life cycle of software engineering projects, which implies several activities. For example, any potential flaws in the source code should be removed to keep it up-to-date and preserve its performance and correctness. At the same time, software engineers (i.e., maintainers) should invest some effort to maintain the mentioned tasks and help to keep the software maintenance cost low [1]. To achieve that, software maintainers rely on tools that help them keep track of and monitor software issues. For instance, issue tracking systems play an important role which helps software maintainers maintain software products by enabling rigorous, practical software evolution tasks. One example of issue tracking systems is Bug Tracking Systems (BTRs). BTRs help the developers to report and describe bugs encountered while using these products of software projects. However, some bug reports can implicitly describe security vulnerabilities that software attackers could exploit if they are exposed before being fixed [2]. A security vulnerability is a security bug issue that allows users of the product to have unauthorized access to the systems\u0026rsquo; capabilities and thus cause harm or damage to the software [3]. Usually, project managers request that issue reporters not disclose any suspected security issues in public BTRs. Instead, the security problems should be reported directly to the software security team to provide patches (when necessary) to the product users before the security problem is discovered and exploited by attackers. However, once the patch has been published, security issues are often disclosed via public BTRs [4]. Peters et al. [5] noted that due to the lack of security expertise knowledge, bug reports are sometimes mislabeled as non-security bug reports (NSBRs) and are often publicly disclosed before they are assessed and fixed [6]. There were real examples when developers declared the SBR as an NSBR. As is shown in Figure 1, a clear SBR from the Apache Ambari project (version.4.1) shows it is mislabelled as an NSBR. At the same time, it is mentioned in the text description (highlighted in red) that it is a security issue. \u0026nbsp;\u003c/p\u003e\n\u003cp\u003ePrevious studies presented several approaches \u003cspan dir=\"RTL\"\u003eto\u003c/span\u003e machine learning-based SBR classification [2], [5], [7]\u0026ndash;[10]. For example, Peters et al. [5] proposed the FARSEC framework, a SBR classification method. The framework [5] combines filtering and ranking methods to reduce the mislabelling of SBRs by text-based classification models. Shu et al.[2] replicated and improved the FARSEC approach by applying hyper-parameter optimization that has been used before in software engineering (e.g., for software defect classification [11], [12] or effort estimation [13]). Wu et al. [7] explored the reasons that led to the poor performance of Peters et al. [5] and Shu et al.[2] and found one main reason: the quality of labels assigned to the bug reports in the datasets. In their finding, they evaluated the impacts of label correctness of two types of datasets: noise before they correct mislabeled bug reports) and clean (i.e., after they manually fix mislabeled bug reports). Their results showed that the clean datasets improved the performance of the classification models.\u003c/p\u003e\n\u003cp\u003eThe approaches above applied by these three studies ([5], [2], and [7]) used traditional machine learning algorithms (e.g., Random Forest, Na\u0026iuml;ve Bayes, Logistical Regression, etc.), and additionally, they are complex and time-consuming. For example, Shue et al. [2] applied hyper-parameter tuning for the learner, costing five hours to optimize the Chromium dataset. While optimizing the learner on the datasets is essential, it is good practice to explore a simple, practical, more accurate, and efficient approach[14], [15]. Therefore, we investigate the effectiveness of simple text classification for SBRs using \u003cem\u003efasttext\u003c/em\u003e (machine learning-based approach) to address the problem of how to \u003cem\u003edistinguish (i.e., classify) SBRs more efficiently and accurately\u003c/em\u003e.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eOur study aims to answer the following research questions:\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eRQ1: Can we accurately detect SBR using a \u003cem\u003efasttext\u003c/em\u003e machine learning algorithm?\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe built a \u003cem\u003efasttext\u003c/em\u003e classifier using publicly available bug report datasets ([5], [7], [16]) and compared its performance to the baseline, which is the ratio of SBRs in the studied projects. The results show that a \u003cem\u003efasttext\u003c/em\u003e classifier achieves a higher F1 score of 0.81, on average. This improvement equates to an average gain of +0.15 by the \u003cem\u003efasttext\u003c/em\u003e classifier compared to our baseline.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eRQ2: How effective is the \u003cem\u003efasttext\u003c/em\u003e classifier when applied on cross-projects?\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTo examine the generalizability of our proposed technique, we determined the effectiveness of our machine learning technique in classifying/predicting SBRs in cross-projects. We asked RQ2: \u003cem\u003eHow effective is the fasttext classifier when applied on cross-projects?\u003c/em\u003e We built a general classifier and evaluated its performance using cross-project validation. Our results show that our classifier achieves an average F1 score of 0.65, which is lower than the within-project classifiers. However, it is still a good classifier that can be practically used. The results also show that cross-project classifier performance corresponds to an average F1-score improvement of +0.19 over our baseline.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThis paper makes the following contributions:\u0026nbsp;\u003c/p\u003e\n\u003cul\u003e\n \u003cli\u003eTo the best of our knowledge, we are the first to experimentally evaluate the impact of the \u003cem\u003efasttext\u003c/em\u003e model (machine learning approach) for SBR classification. \u0026nbsp;\u003c/li\u003e\n \u003cli\u003eWe find that (1), with clean datasets, \u003cem\u003efasttext\u003c/em\u003e simple text classification outperforms the five baseline approaches, and (2) \u003cem\u003efasttext\u0026nbsp;\u003c/em\u003eperforms better in terms of running time and a machine\u0026rsquo;s energy consumption compared to state-of-the-art methods.\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eThese findings provide research clues and guidance for researchers and practitioners of SBR classification and prediction.\u0026nbsp;\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe remainder of this paper is organized as follows. Section 2 provides the background. Section 3 describes our experimental case study. Section 4 details our case study results for each research question. We discuss our findings in Section 5. The threats to validity are discussed in Section 6. Section 7 concludes our paper.\u003c/p\u003e"},{"header":"2. Background","content":"\u003cdiv id=\"Sec2\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Security Bug Report Classification\u003c/h2\u003e \u003cp\u003eIn this sub-section, we recall the four recent SBR classification works that inspired our study.\u003c/p\u003e \u003cp\u003ePeters et al. [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e] pointed out the mislabel problem of the five publicly available SBR prediction datasets: Chromium, Ambari, Camel, Derby, and Wicket. They designed a framework named FARSEC to improve SBR prediction by filtering out noisy data from NSBRs and extending the text mining approach with the security keywords matrix. On the other hand, Shu et al. [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e] improved the performance of SBR prediction and classification by applying a hyper-parameter optimization approach by conducting their study based on Peters et al.\u0026rsquo;s work. They optimized the control parameters of the classification algorithm by using a differential evolution algorithm and the data-preprocessing approach SMOTE [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eHowever, labeling a large dataset to build a classification model is needed in the research of mining software repositories. Therefore, the labels' correctness will significantly affect the model's performance [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. Wu et al. [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e] performed a case study on SBR classification. They found that the dataset used by [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e], [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e] for SBR classification contains many mislabeled instances, which leads to reduced performance of SBR classification models. They first improved the label correctness of the five datasets (dataset originally published by Ohire et al. [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]). They manually analyzed each bug report to find mislabeled bug reports (749 SBRs labeled NSBRs). In this finding, they evaluated the impacts of label correctness for two types of datasets: noisy (i.e., before they corrected mislabeled bug reports) and clean (i.e., after they manually corrected mislabeled bug reports). Their results showed that the clean datasets improved the performance of the classification models.\u003c/p\u003e \u003cp\u003eIn contrast to Wu et al. [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e], we relied on their clean dataset to test our approach. We are different in using a new classification model based on a machine learning algorithm, which results in more enhanced performance.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Text Classification Using \u003cem\u003efasttext\u003c/em\u003e classifier\u003c/h2\u003e \u003cp\u003eDeep learning techniques have achieved great success recently in software engineering and security domains by automatically extracting context-sensitive features from raw software text artifacts. This area includes various tasks in malicious software detection [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e], deep learning models for IoT cybersecurity [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e], applying transformers to cybersecurity [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e], and more. The common issue with these deep learning models is low efficiency and hard for portability.\u003c/p\u003e \u003cp\u003e \u003cem\u003efasttext\u003c/em\u003e is a machine learning method for text categorization and multi-class text classification. It is often on par with deep learning classifiers in terms of accuracy, and many orders of magnitude faster for training and evaluation [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]. The sentence/document vector in \u003cem\u003efasttext\u003c/em\u003e is represented and constructed by averaging the embeddings of the words in the document. And for the text classification task, multinomial logistic regression is used, where the sentence/document vector corresponds to the features. When applying fasttext on problem with large number of classes, the hierarchical softmax layer is applied to speed up the computation. The simplicity of \u003cem\u003efasttext\u003c/em\u003e makes it very efficient compared to the state-of-the-art performances and faster than other computing methods [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e].\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003eModel training - hierarchical classifier.\u003c/b\u003e \u003cem\u003efasttext\u003c/em\u003e uses a hierarchal classifier that represents the labels in a binary tree (as depicted in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e2\u003c/span\u003e). Every node in the binary tree represents a probability, so a label is represented by the probability along the path to that given label. The reason for a binary tree is that it speeds up the search time and eliminates having to go through all the different elements being searched for. For example, Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e2\u003c/span\u003e shows a piece of text that shows the championship decision, and we have this binary tree representation of all the labels instead of computing a score for every single possible label; what we have to calculate is just the probability of each node on the path to that one correct label. This vastly decreases the computations we have to do for each piece of text. So when we have many labels, this increases the speed of the model\u0026rsquo;s training and reduces the time complexity. And not only does this increase the speed compared to linear classifiers, but it\u0026rsquo;s also faster than some deep learning techniques, like neural networks [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eIn section 3.2, we explained in more detail how to use this machine learning algorithm in real-world examples (e.g., classifying bug reports).\u003c/p\u003e \u003c/div\u003e"},{"header":"3. Case Study Design","content":"\u003cp\u003eThe main goal of our study is to determine whether a bug report can be classified as a security bug report (SBR) or a non-security bug report (NSBR). To achieve this goal, we propose using machine learning techniques\u0026mdash;based on a machine learning algorithm. Figure 3 depicts the whole picture of our approach. We began by downloading the dataset used by\u0026nbsp;[16], which\u0026nbsp;was\u0026nbsp;manually annotated by\u0026nbsp;[5], and reviewed and refined by\u0026nbsp;[7]. The authors of the updated dataset\u0026nbsp;[7]\u0026nbsp;not only arranged for experienced software security experts to review SBRs but also developed a specific manual review process. They used software vulnerability types defined by CWE\u003csup\u003e[1]\u003c/sup\u003e as a basis and generated a codebook as guidelines for judging whether a bug is an SBR [7]. Then, we mined the selected software repositories and annotated bug reports to prepare SBR and NSBR features (in \u003cem\u003efasttext\u003c/em\u003e format) and used them as dependent variables in our machine learning classifier. In the following subsections, we detail our data extraction and preprocessing steps. \u0026nbsp;\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003e3.1 Dataset\u003c/h2\u003e\n\u003cp\u003eIn this subsection, we introduce the dataset used in our study. To determine how practical our approach is in classifying bug reports (as SBR or NSBR), we need to have a labeled test dataset on which we can apply our system. Our first criterion for building a testing dataset is to have projects with a sufficient number of bug reports that discuss security concerns. As mentioned earlier, we resort to using the dataset [7]. We use a total of five projects datasets, which are the bug reports initially \u003cspan dir=\"RTL\"\u003elabeled\u003c/span\u003e as SBRs or NSBRs, and published by Ohira et al. [16]. The five projects (see Table 1) are Chromium, Amabri, Camel, Derby, and Wicket. The dataset was originally publicly published by Ohira et al. [16] and supplied as comma-separated value (CSV) files. Each row represents a bug report, and the columns are features of the reports such as \u003cem\u003ebug-id, title, description, data\u003c/em\u003e\u003cem\u003e\u003cspan dir=\"RTL\"\u003e,\u003c/span\u003e\u003c/em\u003e\u003cem\u003e\u0026nbsp;and time\u003c/em\u003e a report was submitted and fixed.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eTable\u0026nbsp;1: Projects in the Testing Datasets\u0026nbsp;[5]\u003c/p\u003e\n\u003ctable border=\"1\" cellpadding=\"0\" cellspacing=\"0\" width=\"100%\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.346938775510203%\"\u003e\n \u003cp\u003e\u003cstrong\u003eProject\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"27.551020408163264%\"\u003e\n \u003cp\u003e\u003cstrong\u003ePeriod\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"24.489795918367346%\"\u003e\n \u003cp\u003e\u003cstrong\u003e# of Bug reports\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"15.306122448979592%\"\u003e\n \u003cp\u003e\u003cstrong\u003eSBR\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"15.306122448979592%\"\u003e\n \u003cp\u003e\u003cstrong\u003e% of SBR\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.346938775510203%\"\u003e\n \u003cp\u003eChromium\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"27.551020408163264%\"\u003e\n \u003cp\u003eAug-2008 to Jun-2010\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"24.489795918367346%\"\u003e\n \u003cp\u003e41,940\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"15.306122448979592%\"\u003e\n \u003cp\u003e192\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"15.306122448979592%\"\u003e\n \u003cp\u003e0.5\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.346938775510203%\"\u003e\n \u003cp\u003eAmbari\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"27.551020408163264%\"\u003e\n \u003cp\u003eSep 2011 to Aug 2014\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"24.489795918367346%\"\u003e\n \u003cp\u003e1,000\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"15.306122448979592%\"\u003e\n \u003cp\u003e29\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"15.306122448979592%\"\u003e\n \u003cp\u003e3.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.346938775510203%\"\u003e\n \u003cp\u003eCamel\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"27.551020408163264%\"\u003e\n \u003cp\u003eJul 2007 to Sep 2013\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"24.489795918367346%\"\u003e\n \u003cp\u003e1,000\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"15.306122448979592%\"\u003e\n \u003cp\u003e32\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"15.306122448979592%\"\u003e\n \u003cp\u003e3.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.346938775510203%\"\u003e\n \u003cp\u003eDerby\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"27.551020408163264%\"\u003e\n \u003cp\u003eSep 2004 to Sep 2014\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"24.489795918367346%\"\u003e\n \u003cp\u003e1,000\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"15.306122448979592%\"\u003e\n \u003cp\u003e88\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"15.306122448979592%\"\u003e\n \u003cp\u003e9.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.346938775510203%\"\u003e\n \u003cp\u003eWicket\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"27.551020408163264%\"\u003e\n \u003cp\u003eOct 2006 to Nov 2014\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"24.489795918367346%\"\u003e\n \u003cp\u003e1,000\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"15.306122448979592%\"\u003e\n \u003cp\u003e10\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"15.306122448979592%\"\u003e\n \u003cp\u003e1.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"2\" valign=\"top\" width=\"45.45454545454545%\"\u003e\n \u003cp\u003eAverage\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"24.242424242424242%\"\u003e\n \u003cp\u003e9,188\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.151515151515152%\"\u003e\n \u003cp\u003e70\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.151515151515152%\"\u003e\n \u003cp\u003e3.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"2\" valign=\"top\" width=\"45.45454545454545%\"\u003e\n \u003cp\u003eMedian\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"24.242424242424242%\"\u003e\n \u003cp\u003e1,000\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.151515151515152%\"\u003e\n \u003cp\u003e32\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.151515151515152%\"\u003e\n \u003cp\u003e3.0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eTo further prepare the data for our experiments, we scrubbed the files by selecting only the necessary columns from the data of Wu et al. [7], we selected \u003cem\u003edescription, summary, and security\u003c/em\u003e. The security column is a binary value of 0 or 1, 0 represents non-security, and 1 illustrates a\u003cspan dir=\"RTL\"\u003e\u0026nbsp;\u003c/span\u003esecurity bug report. We then combined the description and summary columns. At this point, the data of each project is partitioned into two parts. The first part is used for training, while the second is used for testing.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePreprocessing step.\u003c/strong\u003e As shown in Figure \u003cspan dir=\"RTL\"\u003e3\u003c/span\u003e, in step 2, we preprocess the text of the bug report description with the basic text preprocess approach parse_sentence and initialize_words imported from the utility\u003csup\u003e[2]\u003c/sup\u003e library package for text tokenization and word list initialization for each bug report. This step converts the text of the bug report \u003cem\u003edescription\u003c/em\u003e (i.e., we extract the related information title and body of the bug report and concatenate them into a single textual paragraph) to a preprocessed line of words (see Figure \u003cspan dir=\"RTL\"\u003e4\u003c/span\u003e). Stop words like a, the, and, presumed to be uninformative in representing the content of a text, are removed to avoid being constructed as a signal for prediction. We then tokenize the resulting text and derive the bag of words representation from the tokenized text. As depicted in Figure \u003cspan dir=\"RTL\"\u003e4\u003c/span\u003e, each text file line contains a list of labels, followed by the corresponding document. All the labels start with the \u003ccode\u003e__label__\u003c/code\u003e prefix, which is how \u003cem\u003efasttext\u003c/em\u003e recognizes what a label is and what is a word.\u0026nbsp;\u003c/p\u003e\n\u003cdiv id=\"ftn1\"\u003e\n \u003cp\u003eAs shown in Figure \u003cspan dir=\"RTL\"\u003e4\u003c/span\u003e, __\u003ccode\u003elabel__\u003c/code\u003e nonsec-report label represents the annotated bug reports as NSBR, and __\u003ccode\u003elabel__\u003c/code\u003e sec-report model represents the manually annotated bug reports as SBR. The model is then trained to predict the labels given the word in the document.\u003c/p\u003e\n \u003ch2\u003e3.2 Classification Models\u003c/h2\u003e\n \u003cp\u003eDifferent methods can be used to investigate and analyze bug report information to predict the labels assigned. Regarding time and memory consumption, most machine learning models require much time for training and use a lot of memory [24]. We looked for a model that, ideally, achieved good results while not being memory and storage intensive, as we desired to deploy the model on low-end server hardware\u003csup\u003e[3]\u003c/sup\u003e. We decided to use fasttext, an open source tool by Meta (former Facebook AI research in 2016), to perform this task. \u003cem\u003efasttext\u003c/em\u003e uses linear models with a rank constraint and fast loss approximation [21]. As Joulin et al. [24] discussed, fasttext\u0026rsquo;s accuracy is competitive against several deep learning-based models, it is faster for training and evaluation. However, to perform our classification, we leveraged a \u003cem\u003efasttext\u003c/em\u003e classifier (Figure 3 \u0026ndash; step 3) to classify whether a bug report is a security or not since it provides an intuitive and easy-to-explain classification model.\u0026nbsp;\u003c/p\u003e\n \u003cp\u003eThe bug report pre-processing step, as we mentioned in Section 3.1, strictly depends on the method selected to perform the automated classification of bug reports and label assignments. The \u003cem\u003efasttext\u003c/em\u003e linear classifier is typically trained with sentences represented as a bag of words and n-grams [24] to embed information on word order partially. This is because local word order information tends to improve the text classification performance of the linear model. Thus, each bug report submitted to our system is passed to a \u003cem\u003efasttext-based\u003c/em\u003e classifier (as shown in Figure \u003cspan dir=\"RTL\"\u003e3\u003c/span\u003e \u0026ndash; step 3) to perform classification and label assignment.\u0026nbsp;\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003eHierarchical representation of bug reports\u003c/strong\u003e. In each bug report (i.e., an extracted bag of words representation), each word is represented by a vector of character n-gram. This process step represents the desired input of \u003cem\u003efasttext\u003c/em\u003e. To classify the bug report through \u003cem\u003efasttext\u003c/em\u003e, the main objective is to minimize the following objective function over \u0026nbsp; possible labels:\u0026nbsp;\u003c/p\u003e\n \u003cp\u003e\u003cimg src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAALwAAABCCAYAAADt0bzCAAAJ6ElEQVR4nO2dz08TQRvHv33z3um2XE1MtxdiCEYWSKR6IIEtPw4eMNuo8WKiKVwNJIsexbSJHIw/yo0L7NZoYghFWhNM2GqgVrONMRxkG2M4ttYff8C8B95dW7oLbfmx0J1P0kN3d2af3X73mZlnnp26CCEEFIpD+I/dBlAoxwkVPMVRUMFTHAUVPMVRNJ3gx8bG4HK5MDs7CwBIp9NwuVwIBoM2W0Y5CbiaLUqTz+chiiKy2Sy2trYAAMFgEPPz8/B6vTZbR7GbpvPwmUwGN2/eBLDj3QGgWCxSsVMANKHg19bW0NPTg9u3b2NhYQHpdBoDAwN2m0U5IfzXbgMOm3w+D6/Xi9HRUXAch7Nnz6K3t9dusygnhKYSfDqdhs/nAwD4fD50d3djcnIShULBZssoJ4WmGrT6/X5omgZN0+Dz+SDLMh49eoSPHz/abRrlhNBUgqdQ9sP2QWsul0NXV5fdZlAcgq2Cn52dxfnz55HNZu00g+IgbBN8NBrFmTNnoCiKXSZQHIhtgp+YmMDQ0FBDZaPRKFwu14E/+sQUxTnY3odvhFu3boFlWQAAx3EghNT8KRQKEATB5iug2MWpFLzX68WrV68AANlsFtFotK6y09PTR2Ua5YRzKgUPAB0dHYjFYgCAycnJuronPp/PaCEozsL2OHw6ncalS5fQqBnBYBDJZBIMw+Dbt280SYyyJ7Z7+D9//gDYyYFphPn5eTAMg1KphPHx8cM0jdKE2Cb4XC6HYDCI4eFhADuDz2AwiFwuV1c9Xq8Xi4uLAIB4PG68+EGhmEKahEgkQgAQAERVVbvNORY0TSPhcJhIklRXuUKhQCKRCBFF8YgsawxJkkg4HCaFQqGm7Y3QNIInhBCe5wkAwnHcodycg8CyLAFAFEU5kvpVVSUsy1o+3KqqEo7jDCegUygUCMdxVQ+JoijGseWf/YTGsiyJRCKHc1Hk33VpmlbT9nppKsEXCgXCMIzxQ9lty1EKnuM4y7o1TSMMwxBJkox7oiOKoqVARVEkHMcZ3xVFIQzDkFgsZnp8JBIhLMse+r2WJInwPF/z9npoKsETUumpEomErbYcleATiQRhWdZyvyiKpvv1h9DKS4bD4apuDsuypoIvFAqE53kSiUQOLEIzWJY1vXdW22vF9ijNYRMIBCCKIgDgxo0bdUd/isUiurq64PF4jLL6ygcHzeqUZRl+vx8ulwt+v79qwmxqagoej6ci/cFstYWlpSXL2WK/34/p6WlomgaXy1Vxjrdv34LjOOMlmd2kUikMDg4C2ImajY2NgWEYjI6OVh17//593Lt3z7QefeWI5eVl47pcLlddv8XAwADevHlT8/aaOchTeJLR+6/lTXQtRCIRUigUqvqmPM9bNu1WoMzD6y2P3upIkkQAGH3pWCxm9FF172nVVeA4bk9bzProhOx4fkEQTMtomlbVfxcEwbT/riiKYdtuD59IJIiiKMa5JEmqe1BtVu9+22ulLg+ve7q9Pnut/1JPYtdBicfjYBgG2Wy2rlnYiYkJeL1eCIKA1dXVin1mnq5WFhYWwPO8kTAXCoXA8zxev34NAPj9+zf8fj98Ph+8Xi/6+vosPWI2m8W5c+dM9xWLRWSzWbS1tVXt+/TpEy5cuGBaLpPJVOQlKYqCeDyOzc3NqmOfPHmCu3fvAgBaWlqQyWSMfUNDQwgEAujt7UUqlcLa2hpCodAed+Z4qeud1kAg0PCMKIADla0X/Z1Wt9uNQCBQd/n29nbE43EAOw96X1/fgWZxrcT769cvAP+Ek8/n0dLSgtXVVbjd7rrPs7GxAYZh0NHRUVe5L1++VHTZAoEAeJ7Hhw8fKu6fLMuIx+PGvbGip6cHpVIJd+7cqe8Cjpim68PrRKNRZDIZPH36tKHybW1t0DQNwI53npiYOJA9brcbxWKxarvenx4dHQXLsmBZFq2trXC73Za2cxyHr1+/mu57//49uru7Tfd1dnbi8+fPpvtSqRQuX75sfC8Wi0gmk2hpaanYNjY2Bk3TKloCM2ZmZsCyrGkLYSdNKfh0Oo3JyUksLi427JV1DxkKhXDt2jUAO97N4/FgdnbWGIhNTU2Zlt8t7itXriCbzUKWZcPGZDJp1P3y5UtcvXrVEJIsy5a2d3V14cePH6b7UqkU+vr6TPe1t7cbD/FuW8u7Qfl8HtevXwfDMOjv7zeOm5mZQSgUqhj0bm9vV11vNBpFe3s7BEHA3NwccrkcZFlGsVhEMBiEx+NBLpeD3++vCA6U8/37d3R2dta8vWYa7v2fUPQB52HMIvI8X1GPJElG7FlRFJJIJCwHlvrEU3kMPBKJGPMELMtWDOYSiUTVoNFqosUqLLlf7N8qLBkOhyvOyzAMEQSh4rhYLEYAEJ7nje3lIWBRFI2Br35dehxfH2QqimIcI4qiMQlmdo1HFZa0RfBWsfLy9IBGEQSh7siMGZqmmUYD9MgDITtRj0YiEGaU16vDcZzlJBHHcVXzDPpDs9fMqCiKtqYU6DOmhPxzTrtpyoknURRNQ28syzY8YSRJEmEY5sC5NKqqWobkGIYxPBLHcURV1UMRPYAKcevCsKrbLLVgr1CmjlVqwXERi8UMG/UWUp8RJqSJUwsEQSCqqlY0+YSQhr27XtdBfki9yS9vtnefo7z14Dju0PJ2JEmqyH2pJUelPHlMt7sWW+xMHhMEwXBoeouk/2ZNnTymN03lHl1V1YabLI7jLCdVKBQdW6I0uVzOGOkLgoClpSUAwPr6umWEYS+mpqZQKpUaDkFSnIMtgl9fXzdivoODg0ao7t27d7h48WJddaXTaUxPT2Nubo6+3kfZHzuaFb3/rsMwDEkkElX9+f3QU18byccOh8OHmsdNOR3Y4uFTqVTF1HcoFMLjx4/rXklgfHwcLMs2NAuaSqXqLkM5/Ry74JeXl6uEPTIygmQyWdc/dczOziKVSu2b02GGLMvQNK1i2pziEI67ScH/w267uxN6t6YWVFU1fR2t3s9RvY1EObkc+z+AEIuMyZ8/f9Zcx8OHDw/LHIrDsH0hJgrlOGnKbMn9SKfTCAaDdPVgB9J0gg8Gg3sujS3LMra3t40/LaY4i1Mn+P1yqldWVkyXydbf2gmFQgiFQvD7/XZeBsUmTp3gNzc38ezZM5RKJbx48QIbGxt0JWBKzZw6wQcCAfz9+xcsy+LBgwcAgFKpZOTm7NeloTibU/nHxOvr68Yk1cbGBgYGBiDLMvr7+7GysrJv+WKxiK2tLWPlYopzOHUeHthJMhsZGTG+P3/+HABqSh6LRqNobW2FpmkYHh7ec1kRSvNB4/AUR3EqPTyF0ihU8BRHQQVPcRRU8BRHQQVPcRRU8BRHQQVPcRRU8BRHQQVPcRRU8BRHQQVPcRRU8BRH8T8r7FVVsTxWNAAAAABJRU5ErkJggg==\" width=\"188\" height=\"66\"\u003e\u003c/p\u003e\n \u003cp\u003eWhere \u0026nbsp; is a bag of features of the \u0026nbsp; bug reports,\u003cem\u003e\u0026nbsp;\u003c/em\u003e \u003cem\u003e\u0026nbsp;\u003c/em\u003ethe label, \u0026nbsp; \u0026nbsp;represents the weight dictionary of the average text embeddings, \u0026nbsp; is the weight dictionary that converts the embedding to pre-softmax values for each class, and \u0026nbsp; is the hierarchical \u003cem\u003esoftmax function\u003c/em\u003e used to minimize computational complexity. For the setting of \u003cem\u003efasttext\u003c/em\u003e, we used the tuning range for all default parameters. Table 2 shows the learner and pre-processer options we explored in this study for \u003cem\u003efasttext\u003c/em\u003e.\u0026nbsp;\u003c/p\u003e\n \u003cp\u003eTable 2: Setting up parameters for the fasttext model\u003c/p\u003e\n \u003ctable border=\"1\" cellpadding=\"0\" cellspacing=\"0\" width=\"100%\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"2\" width=\"7.142857142857143%\"\u003e\n \u003cp\u003e\u003cstrong\u003eTarget\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"3\" style=\"width: 66.9844%;\" width=\"42.857142857142854%\"\u003e\n \u003cp\u003e\u003cstrong\u003eParameters of Target\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd rowspan=\"2\" style=\"width: 25.8906%;\" width=\"50%\"\u003e\n \u003cp\u003e\u003cstrong\u003eDescription\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"26.829268292682926%\"\u003e\n \u003cp\u003e\u003cstrong\u003eParameters\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"19.51219512195122%\"\u003e\n \u003cp\u003e\u003cstrong\u003eDefault\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 26.1094%;\" width=\"53.65853658536585%\"\u003e\n \u003cp\u003e\u003cstrong\u003eTunning Range\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e\u003cem\u003efasttext\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"11.34020618556701%\"\u003e\n \u003cp\u003elr\u003c/p\u003e\n \u003cp\u003eepoch\u003c/p\u003e\n \u003cp\u003edim\u003c/p\u003e\n \u003cp\u003eloos\u003c/p\u003e\n \u003cp\u003ewordNgrams\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.1\u003c/p\u003e\n \u003cp\u003e5\u003c/p\u003e\n \u003cp\u003e100\u003c/p\u003e\n \u003cp\u003esoftmax\u003c/p\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 26.1094%;\" width=\"22.68041237113402%\"\u003e\n \u003cp\u003e[0.1, 0.3, 0.7, .9]\u003c/p\u003e\n \u003cp\u003e[5, 10]\u003c/p\u003e\n \u003cp\u003e[50, 100]\u003c/p\u003e\n \u003cp\u003e[\u0026ldquo;ns\u0026rdquo;,\u0026rdquo;hs\u0026rdquo;,\u0026rdquo;softmax\u0026rdquo;,\u0026rdquo;ova\u0026rdquo;]\u003c/p\u003e\n \u003cp\u003e[1, 2, 3]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 25.8906%;\" width=\"50.51546391752577%\"\u003e\n \u003cp\u003eLearning rate\u003c/p\u003e\n \u003cp\u003eThe number of times each training example is seen in the algorithm\u0026nbsp;\u003c/p\u003e\n \u003cp\u003eSize of word vectors\u003c/p\u003e\n \u003cp\u003eLoss function\u003c/p\u003e\n \u003cp\u003eMax length of word ngram\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003cp\u003e\u003cstrong\u003eDealing with imbalanced data.\u003c/strong\u003e As shown in Table 1, our testing dataset has, on average, only 3% (median = 3%) labeled as SBR, meaning most of the bug reports are NSBR. This common issue in software engineering data is called data imbalance [25]. Imbalance in the datasets occurs when there is one class higher in appearance than the other in the dataset. In this case, the \u003cem\u003efasttext\u003c/em\u003e trains to learn from the features affecting the majority cases (i.e., NSBR) than the minority cases (i.e., SBR). To mitigate the impact of this issue, we relied on the re-sampling process to perform the under-sampling on the training dataset only. We ensured the class distribution was uniform in the output data, and the final sample size was the same as the original data. We must note that we applied this process (re-sampling step) to the training dataset only because we want to evaluate our trained model (classifier) on real-world examples where the bug report data is imbalanced.\u0026nbsp;\u003c/p\u003e\n \u003ch2\u003e3.3 Performance Evaluation\u003c/h2\u003e\n \u003cp\u003eTo make the evaluation results objective, we use our study\u0026apos;s performance metrics applied by [5], [7]. These metrics applied by the work of Peters et al. and Shu et al. include Recall (\u003cstrong\u003eR\u003c/strong\u003e), probability of false alarm (\u003cstrong\u003epf\u003c/strong\u003e), Precision (\u003cstrong\u003eP\u003c/strong\u003e), F1-score (\u003cstrong\u003eF1\u003c/strong\u003e), and G-measure (\u003cstrong\u003eG\u003c/strong\u003e). The first four measures are commonly used by many studies in the mining software engineering area [26]\u0026ndash;[28] and empirical software engineering area [15], [29]. G-measure was first introduced by Peters et al. [5], while both F1-score \u0026nbsp;and G-measure are harmonic means, and the G score considers the Recalls of both the majority and the minority classes [5]. However, in our study, there are four possible outcomes of the classification results:\u0026nbsp;\u003c/p\u003e\n \u003ctable border=\"1\" cellpadding=\"0\" cellspacing=\"0\" width=\"87%\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd width=\"26.262626262626263%\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"33.333333333333336%\"\u003e\n \u003cp\u003e\u003cstrong\u003eIs SBR\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"40.4040404040404%\"\u003e\n \u003cp\u003e\u003cstrong\u003eIs NSBR\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"26.262626262626263%\"\u003e\n \u003cp\u003e\u003cstrong\u003eClassified as SBR\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"33.333333333333336%\"\u003e\n \u003cp\u003eTrue Positive (TP)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"40.4040404040404%\"\u003e\n \u003cp\u003eFalse Positive (FP)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"26.262626262626263%\"\u003e\n \u003cp\u003e\u003cstrong\u003eClassified as NSBR\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"33.333333333333336%\"\u003e\n \u003cp\u003eFalse Negative (FN)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"40.4040404040404%\"\u003e\n \u003cp\u003eTrue Negative (TN)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003eBased on these results, the performance metrics R, pf, P, F1, and G can be calculated as follows:\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAcIAAACoCAYAAABzAO7NAAAgAElEQVR4nO3d34sb190/8Pd86W1sj5SrBErQqJDgBAVn1hvSTWAN9uhZevGYTSq5gbJgk7VEKLRJ161sU4odt9IT56KkXqk0EEyNtLXNY0y01mphFyTFJOu1kcDBF9UMxjV7NbObjf+A873Qc05nRiPtT+2v+bxAsNKMRme0OvOZOT8+IzHGGAghhBCf+n/bXQBCCCFkO1EgJIQQ4msUCAkhhPgaBUJCCCG+RoGQEEKIr1EgJIQQ4msUCAkhhPgaBUJCCCG+RoGQEEKIr1EgJIQQ4msUCAkhhPgaBUJCCCG+RoGQEEKIr1EgJIQQ4msUCAkhhPgaBUJCCCG+RoGQEEKIr1EgJIQQ4msUCAkhhPgaBUJCCCG+RoGQEEKIr1EgJIQQ4msUCAkhhPgaBUJCCCG+RoGQEEKIr1EgJIQQ4msUCHuor68PANBoNBCNRsXruVwOkiR5Pmq1GgAgk8k4Xg+HwygUCtuyH4TsNLVaDWfPngXQqk+ZTEYsCwQCnnXLXgej0ahjWV9fHxqNxpbvB9kZKBD2iGEYCAaDAIBnz57hyJEjYtns7Cyq1SoYY1BVFcViEYwxyLKMgYEBAMDY2BgAoF6vgzGG3/72t0gmk1u/I4TsQE+fPsX+/fsBAMvLy3jrrbcAtE46jx07BtM0USwWoaoqGGPIZrOOOnju3DnIsgzGGBhjUBQFf/rTn7ZlX8j2+9F2F2AvikajmJqaAgBIkiRen5mZQalUEld2lmVhfn4e/f39AIB4PC7WNQwDsiwjEokAAA4ePIilpaWt2gVCdix7nTpz5oz4O51OY2xsTNSvr7/+GseOHQPQqj9Pnz4V6z59+lQsA4BDhw5hZmam10UnOxUjPZHP51k6nWaMMZZOp1m1Wm1bp1gsMlVVO74/kUgwxhjTdZ2pqiqeE+J3iURC1ClN0zzXUVWVFYvFju/P5/OMMcaq1SqTZVk8J/5DTaM98uTJE9FcMzMzgxdeeKFtHfsZq1ulUsH4+DgkSYKqqujr68OFCxd6WmZCdgvDMEQ3QrPZbFvubm1xK5fLOHHiBCRJwsjICH73u985WmSIv1DTaA/Ym244RVFQrVZF5QValfGPf/yj5zbK5XLb+oT4Xa1Ww9tvvw3AWc8kSQJjTDz/9ttvoaqq6Ke3MwwDuq471if+RleEPcAYg6ZpoiOed9jbg1q3M1ZeUSkIEuI0MDDgqF/VahWpVKotqHVrbZmbm4OmaVtRXLJLUCDsEd5cYx89anfjxg0A8Fw2PT0NRVF6W0BCdqlGo4EDBw4AcI4e5SzLwsTERNvr3K1btxAKhXpeTrJ7UCDskXA4DABYWFjAG2+84VhmWRZOnz4NAI75T/Zluq63LSOEtKYjHTp0CECrL/7VV191LP/ss8+g67pjRCk3OTmJiYkJjI+Pizm7hEiMGsoJIYT42I6+InRnV6EMELtHrVZDNBr1vKo9e/YsJElCIBBoy5bTaDQQDodFJhDLssSyyclJBAIBhMPhtt9AX1+fY13SnTuziiRJiMfj9B3uQpZlIZPJIBAItC3rVp+AznXRMAyEw2HPOppMJvfe1fQ2TdtYNQBMlmXGWGu+D4CO84bIzpBOp1k+n2eqqoq5lFw+n2eKojDTNFmxWGQAmK7rYrmiKCybzTLTNJmmaY65k/x92WzW8Xo6nW77HNIdr0uxWIwxxlg2m2UA6HvcZer1OkulUuL/59atPnWri4lEghWLRVav15miKOI91Wp1Tx5/d3Qg1HXdUVn58734j9hJeKUxTdPxej6fX9OkY03T2g6ssVjM8ZqqqiybzTLGWpXaXpmLxaI4CeLbs5eP/90pKQHpLJ/PMwDiu+cHQgqEW4+fONqZpimOe6vBT2zsVqpP3eqi/Rhrr1+apjlOXPeKHd00Ojc3BwAYHByEZVn49NNPAQDHjx/3XN+rucf+IKsTDAYxMjLiaEopFAqoVCobnnRcLpdFogH+WcvLywCAb775xjGsfd++fZ5p5ewjbc+fP4+PPvpoQ2Xyo0qlAgB48803YRgG/vCHP0CWZbz77rue61Pd6p2TJ0/i8uXLjtSL77//Pn7/+99vaLsr1aduddGO17dCoYBQKLQnR9zu6EDIK+vp06fx/PPP4969e8hmsxgdHfVcv1Qqibl7Xg+yevF4HB999BGi0ShyuRwuX768KZltuuVL9aqEds1mE4ZhiAppGAYMw8Dhw4c79mcQb+VyGQDw+uuvQ1EUKIqC+fn5jgc5qlu9EwwGUSqVcPnyZeRyOUSjUYyMjIg8w+u1Un3qVhdDoRAKhQIMwxBTwfgxgJ8U8bt/7AU7OrMMr6y9rGh0Ntvi9R3H43FUKhUxncNrzuNaybKMH374wXPZ/v37uw7WuHjxojhol8tlJJNJnDt3Dp9++in+8pe/4MUXX8Tw8DClyloBT9igqipKpRL6+/tRLpfx17/+dVM/h+rWf6x0DAsGg5iYmICiKEgkEpvyG16pPnWri6OjoxgcHAQA/OMf/0Amk8F7772H6elpvPHGG7h27Rr6+/tx8uTJPXGFuGOvCHllXUsGiPU033Q7y/XTwws/I8xms4jFYpsyovDw4cN4+PCheG5ZFn784x8DaN0hYH5+Xiz74YcfHIkF4vE4GGNoNptYWFhAKBTCwMAADMPA0NAQIpEIZFnecBn3Ot7lcOzYMQSDQaiqiqWlJTx69Kjje6hubX79srMsC8lkEtlsFvfu3duUlo2V6lO3uhiJRLC4uIjFxUX09/fj+vXrGBsbw5MnT/Bf//VfCAaDiMViWFhY2HA5d4IdGwinp6cBYE1nG9R8s3l4n2CpVMLo6KhoJl1LMPRa9/jx4/jb3/4Gy7LQaDSg6zqOHj0KoJU+S5Zl5HI5AMDVq1fxwQcfeG57ZGQEH3/8cdvrm3HVutfdunULAPDaa68BgJic/vnnn3d8D9Wt3uF9gul0GqOjo6KZdC3B0OvKbqX61K0u2nXqh++UuWdXYjuQaZoMgHh43cKI9I5pmiyVSrW9br+1VDd86gT//2maxur1ulieSCQYAKYoSttouWq1yhRFEaOF3SNXvcrBb6mj67pjqDdpx0eH8gdj/xlxSHVte2SzWUf9YKzzyG0vmqYxWZZFnbLX3ZXqU7e6yFhrpL59BCm/PRwfrb1XRpDuyEC4HarVqvgBZbNZx4GW/8jcD/sPxL2s00F8I+wVwz4Mmpffq4z2H/dWlLHXTNMUc5+4er3OZFlmsix3vP8c2T5+qlv2gMZf283BQtM0x8kRD4DYY1NtdmzT6FazJ+9dXl4Ww4objQaOHTsG0zRRLBbFnSSy2SyOHDki3p9OpxGLxcAYg2mamJ+fF4m1u4lGo6suY7PZFE1/hw8fdiwbGBiApmlIp9OiuSqRSOCVV17ZcBl3kmAw6PgeAGd/xtDQ0DaWjnjxS93SNA3//Oc/xWupVArpdHpXDyYplUqOu+AEg0Hcu3cPjDGMjY1tY8k2144eNbpV7J399kS96XQaY2Njoq3efmuXgwcP4unTp2Ldx48fi1FWwWAQ4XB4xeHLq5XJZES5ut2Drdls4ty5c+L5lStXHNvpZRkJ8eKnupVOpzE8PIxPPvkEAKDr+obnApItsj0XojtPIpEQTQCdMteoqtqx6U1RFNEEwtMdraZJZLVZcuxt9dVqta1ZgmfdYazVVOiVlWK9ZSRkI/xQt/hrsiyLslHGo92Drgj/j2EYogmATyC1W82NdPnQZFVVUSwWN7VJZGFhQTQX3b17Vwxz5viQeH5Wm06nt7yMhHjxQ93i5YvH45iensbRo0fpnqK7yXZH4u3WqSPc/dUUi8WOZ3j5fH7VZ5/dPg8dOqA1TVtx3UQiIRLq5vP5trPrtZSRkM3gp7rFB87wfSkWi3tqMMle5/tAyNmbRrymDqRSKc/XGWtVlPX+6FdbydPptKN5yT1qTlGUriMmN1JGQjbCD3XL3lQry7KjOZjsfDRqFK3RawcOHADgHOHGWZaFiYkJzwmklmWhXC63NadstgcPHuC5554DgLZRk3wy7Msvv+z53q0qIyFufqlb9vLH43GMj487RluSnY0CIYBnz56J7BpPnjzBq6++6lj+2WefQdd1x6g397ITJ070tIzff/+9SMLrTiN26tQpAK0bZnrZqjIS4rbX69b58+eh6zo+++wz8do777yzptSQZPtJjFF+JEIIIf5FV4SEEEJ8jQIhIYQQX6NASAghxNcoEBJCCPE1CoSEEEJ8jQIhIYQQX6NASAghxNcoEBJCCPE1CoSEEEJ8jQIhIYQQX6NASAghxNcoEBJCCPE1CoSEEEJ8jQIhIYQQX6NASAghxNcoEBJCCPE1CoSEEEJ8jQIhIYQQX6NASAghxNcoEBJCCPE1CoSEEEJ8jQIhIYQQX6NASAghHUSjUViWBQAIBAKOZbVaDZIktT0KhYLn8kAggLNnz275PpCVUSDskUwm46gE4XBYVBDS4j6A2A86vZBMJiFJEmq1mudzQtyazSaCwSAA4PDhw45lAwMD0DQN6XQajDEwxpBIJPDKK684lmezWTDGMDs7i0uXLqHRaGz5fpAVMNIzAFi9XmeMMZbNZpksyz37rFgsxrLZ7IrrZbNZFovFelaOtUin00zTNMYYY6ZpMk3TWDqd7ulnAmCmaXZ8Tghjrd8mAM+HnaIorFqtdtyOoijiGMBY6/fWbX2yPeiKsEcMw4Asy4hEIgCAgwcPYmlpqWefVygUMDo6uuJ6o6Ojm35lalmW59VcoVDo+lmPHz/G8ePHAQDBYBChUGhTy+VmGAZUVRVn+O7nhHBjY2PQdR2apoExhmq1Kq78OMMwoOs6BgYG0Gg0EI/HHdswDAMAEIlEYFkWkskkFEXBwMDAlu4LWRkFwh6Zm5sTFcMwDPz6179GIpEAADQaDQQCAeRyORQKBQQCASSTSfHeyclJ9PX1ieZCjlcmd18Eb3q1rxeNRsV6vF+Cv8YrKC8L/6xwOCyabbzK2NfX57mvwWAQIyMjjmBYKBRQqVTaDg525XIZb775pli/UCjg3XffdZQtHo+LstnLnclkEAgEIEmS47vjZQ0EAshkMshkMo7/ybFjxzo+J8RuYWEBR44cAQDcvXsXP/7xjx3L5+bmALTq3+uvv45Dhw61Ldd1HZIk4fnnn8fi4iJu3ry5NYUna7Pdl6R7VSKREE0psiyzRCIhmuDS6bRoFszn86xYLIomwWKxyGRZZtVqlZmm6WiKUVVVbCeVSokmlmKxyBKJhFjP3vxZr9fFMtM0maIoYj1d15ksyyyfzzPGGEulUuJ9vIy8ydVdFi/5fJ6pqsqy2SxTVbVrk6Ou647mJk3THE1IvGy8uVdVVbG/qVSKqarKdF3v+N3x9xSLRcf/pNtzQjhN0zybRe1N94lEQtQtXo/tEolEz5v6yeagQNgjK/UdpFIpR/DiVFVl+XyemabJstmsCFzFYtERxOzS6bQIZoy1goiiKI7XGGOsWq06PtNdBnufHV/OA6Ou60xV1W67zBj7zwmArutd1+NBk7FW4LZ/rrtsPMDpui4Cstf2+XfHKYriCMaKojje535OiF06nRZ1WNO0thM7RVG6nkitdAwgOwc1jfaAve+gk3K5jF/84hdtr8/Pz+PEiRP4yU9+gkajgXK5DAB4+PAhPvjgA89tXb9+3TGiLRQK4ebNm7h165aj2fDOnTt45513xPP79+87ns/MzIg+O17GDz/8EMDqmhELhQIMw0A2m0UsFus6ArRSqYjtHT16FFNTU47l9+/fx/j4OCRJwtWrV3H79m2EQiE8evQImqZ59ifOz8/j6NGjAFpD12VZdvQH8u/G6zkhbg8ePMBzzz0HwDl6FGg12+u6jpdfftnzvXz5Cy+8sCVlJRtDgbAHpqenoShKx+WWZWF+fr5joDRNE4uLi4hEIqIfAmhVTKB1kLf3fc3Pz+PZs2fI5XKo1WpIJpOIRCK4dOkSxsfHxXr379/Hvn37HHOZlpeXAQC5XA5zc3MikLjLeOvWLbz22msd50HxPsFSqYTR0VF89NFHHadDWJaFcrmM1157DUArGMmy3DaNoVgsgjGGDz/8EHfu3BGvN5tNWJYFwzAcgZ7vT61Ww507dxAMBkVwXlhYEFNYarVa23NC3L7//nsx2E2WZceyU6dOAUDb72+1y8kOs92XpHsNb7qDqz/BrlgstjUFcnzYtizLLJVKieYY3twJwNHfyBhjsiyLpptqtSrWs/exMdaaYmEfzl2v18W6qqo6+ujcZUylUkyWZcc69n1OpVJtr+fzec/vIJVKtQ1Fj8VijuYn3hwKgMViMUcTJu+/cfcr8mbZVCrF6vU6AyCaSvlzXk73c0KIf1EgJGQXMk2TxWIxcUKx2X2dnQaLcF7z7NwnU9VqtW0dr75rsjvk8/mO/Z785NY++M7+Pn5S6z7x5L8jdx+srusdLxZ6oeeBcKUKxdh/dnordpz/M70qLiG7Ba9XqxnNux78AMVbFLxaMfiBj7H/BD33Oryc9XpdjFqmhqjdh4+whkdCAH5MNU1TrMNPzPjo8Hq93jZK3T4ALxaLOQKopmlbOtCo532EfB4OTzNULBahaZpYnsvloKpq22CJXqjVarh8+TLK5TJM00QwGMSf/vSnnn8uIZup0WhgamoKmqYhGAw6JnlvlpmZGQAQ8zyHhoZQKpXEcj7YiA946jQopNlsisQSwWCwra+NbI31Jr3grly5gqGhIc9lt27dwgcffIBgMIihoSGoqorp6WkArfESmqYhEokgFAohkUjg1q1bAFrzNN977z0AwC9/+Us8efIEQOs4HQqFtjTxQM8DYbcKNTk5ieXlZfzjH//odTEAtHL/3bt3D6FQCMFgEEeOHMH333/vuS4fdOKWTCYdE7sJ2SyNRgPhcBjhcBiWZeHs2bOQJMmRlKBWq+H1118HAExNTbUlgrazJ1XwenTDB2lFIhFkMpm2gyVfPjg4CMuy8OmnnwKAY9QxHz3Ng2Uul8P8/DxSqZTnZ3ZKYs0f9gFiZG3Wm/RiNcrlMt566y3HZ/FBeLOzs+JiCAD279/veczdt2+f+HtkZAQff/zxhsq0Zr2+5ORtw4y1z3fjOjWrdIMOeQDXsp2VJrzaJ8zy59S/QXollUqJJkl74gS4+gCz2SwD0LNkAHwgkf3h7kKwJ4zA/3UzuHPd5vN5xzpe/Udka60l6YUXeDSNul+z5wx25w+2z1XmTaOmaYpjKx9gxwfLuXO19sqPehlkG42GyK/Jz0Dr9fqmbJttsDmoVqvh3r17uHDhQsd1rly5gmQy6bgy3OjZEyGdfPLJJ2J6yscff4xgMAhd19vWm52dBYCOc9g26ptvvgEApFIpfPLJJ+jr6xPTCDg+v7VbPaxUKgBa02D+/e9/4/Tp06L5a7OsdGXrJ6s5JsbjcVQqFZw+fRq6rm9Knl1ZlvHDDz94Ljtw4IC4OnQLhULo6+vD888/D03TcOHCBfT39+Pbb79Ff38//vWvf+HGjRvI5XK4cuXKhsvZVS+jLD9z5SOFOmUmWc8V4UbU63XPTBGdKIrSMauLF3S5WqXH7nhsF1VVRT0xTVOcFdt5veal00C1lfaRX+11uuLkAyBWqq98YIxpmuIqs9t7vEaZ2h+Urmzj8vk80zRtU68I3Vd99gxPXtmqvDJq8XX5+9x3pem1ntb4lSoUt5VNo/bL8dXuA//ndfoHErJZAIhA6FV/eEDp1a207CM7O03J4Ce43eoDL6c9YPP6SSO1t0c+n3f8z3gz6WqPhXyEsjsQ8lSQ/IRHlmWxTf47sI8a9hoNyudJc/Zj+K4OhKupUBzvF1nPGcpauSdhd8uf6e4TpGBIesl+QsiDiLtPjQeh1dx7cj3syQ68DkD2hBFeB0VOVdW2Kzl7IgSytdaa9MItlUo5EnW4/4f8pK3Tb1aWZZEkxEsikXD8lngeYHfw7pWeBcKVKpQowBY2gbg77+1n327VatWzY9/9DyNks/ATwm5XTHwSPV1VteN1mXd92HW60S4/SXc3y3Y7aPeijPbkCJ2asPk+2FsD7IOrdqtqtdr2/+LH6q1KjE8zW3toqyrXbubVL9TL78irf2unTPLmV0yd8D5DuqJqZ89EUq1WPU+m3Vew7pNg3nfGmLNJr5tOn7XWMvLbivErbt7M2ClBAW8547+JXrek7XXbX/v3uPVUrvWy34ewG96O30trOVjbO9t59oleXnWn02lHsK1Wq6u6xVSv8ROBTlMMeACng55Tp0FB9t8gPwHqxj1Uv1vTL7faQLiaMjLWyuBj/y3ar/R0XRf3CLXn0O1Vf7Gf0N0neqzZbIpkAnwI+rNnz3ryWaOjo6vKEhGJRLC4uNiTMqxHs9kU2YZCoVDXSeKb4fHjx/jpT38qnj99+nRH3KmetU5MO07RYYyhVCptypD3vaRUKiGfzyOdToMxhnQ6jWq16siEMzc3J35jhUKhbXI+T5IRiURgWRaSySQURdm07CarKSMAfP31147fon3awNzcHF599VUMDg6KqSmPHj3C4ODgppTRzygQ9lC3ymVZFvr6+hCPxx0ZRbhGo4F4PA5JkhAOhx3ZbDKZDAKBACRJEnMceRYR+3o8M4kkSejr6xPvlSTJETAtyxKfFQgExDKvMgYCgU3NrGMYhrjlFM+mIsuy4wA0OTmJvr4+SJKEaDTqKHcymRT7aN8nvu/hcBiZTAaTk5NiWblcRn9/v3heqVQcgZHsPk+ePBHZTWZmZtpSvlUqFUxNTUGSJJw4ccKRCQVoBRld1yFJEp5//nksLi7i5s2bW1pGoPXb7PRbrFQq6O/vx9GjR8VvvVKpiBNtsgHbej26x7kH58RiMdH0ks1mRTMIv92SPeOC/RZKqqqKJppUKiX6E4rFomiW4aN0OX6LJdM02+bi8Pfbn7vvBu8uYzqdZqZpOsrSzWqbRt3fkfsWU7w81Wq1LcE0LzcfEef1Hdn7XPh3624Gdd/Jnuwu8GhyhKtZ0z6a0f0b46+ttq9vPfMzV1NG92/V63M5VVXbmlHJ+lEg7KGVKlexWPScGG2fdMoDgf2g7jWKqlqtto0cswcwO/tnusvAB6+4l/NtdAoa650MzedpMtYaOedej0/ONU1TzFfyKjfn/o7cgc89HNs9f4nsTu4gYcf7B7ud7HSa37aStQyW6VZGxtr7B+10XXf8brPZLEskEjRwapNQ02gPuZPRun399df44IMP2l6/f/8+xsfHIUkSrl69itu3byMUCuHRo0fQNA2hUKjtPXfu3ME777zjeK1UKmF5edmRaHdyctLRB/Hw4UPH87t37yIWi7WVMRgMwjAMyLLs2Uc1MDAg+rgYY9A0zfF8bGys43fEm4L++7//WyRp5+bn53HixAn85Cc/QaPREKm9Hj586PndPXr0CKqqiu/oxo0bjv2rVCqO72lubm5H9A+SjWk2mwBaTe3u3+f09DQURenYt9poNKDresc7aGxFGQHg6tWrHcs4PT3tuHPH0aNHMT4+7khoTdaPAmGPrKZydQuUxWIRjDF8+OGHuHPnjni92WzCsiwYhuHIgXr//n3s27dP5KrkfYK/+c1vALQCBNAKIC+99BLOnj0rgiMfOFOr1fDnP/8Zv/zlLz3LOD09jb6+PuRyuU3pJ+TfEc+Z+corr3jejss0TdGPyO96AAAPHjwQ5bYPfuD5bQuFAh48eID9+/eL5YZhOL6nJ0+etH0fZPfh/esLCwt44403xOuWZYm8mp3uXnHq1CkA8LzbzFaUEWidoE5MTGBqagq1Ws2xzDAMnD59GpcuXRK/0VAoBEVRup5okzXYxqvRPY1n1ujUdNHthqq8ORT/169obwq1Z+ewD/WOxWJi+Dfvy4Ot343jWR54fwnvP4RHVgh3GflNN1dzB4HVNNl4fUfuuZZ8wjB/3d7Xx+f/2ft8+L7zfeT7y78r+/fk9X0QQvxHYqwHd/UkhBBCdglqGiWEEOJrFAgJIYT4GgVCQgghvkaBkBBCiK9RICSEEOJrFAgJIYT4GgVCQgghvkaBkBBCiK9RICSEEOJrFAgJIYT4GgVCQgghvkaBkBBCiK9RICSEEOJrFAgJIYT4GgVCQgghvkaBkBBCiK9RICQ9VSgUEA6HIUkS+vr60Gg0NrS9TCYDSZLEIxAIoFAobFJp2xUKBUiShEwmAwAwDAOSJCEajW54241GA8lk0vGd8O+rVqttePvrMTk5Kf5fmUzGs4yE7Dm9ufE9IYxVq1WmqirTdZ2Zpsk0TWOxWGzD2wXAqtUqY4yxYrHIev0z1jSNFYtF8TydTrNUKrWhbdbrdaZpGjNNU7yWSCTE/vD920q6rjMArF6vs3w+z/L5PGOMif9dvV7f8jIRshV+tL1hmOxlAwMDuHfvnnh+5MgRzMzMbGibhmFAlmUMDAwAAF5++eUNbW81ms0m+vv7xfPHjx/jZz/72Ya2OTw8jHK5jGAwKF67cuXKhra5UTdu3EAsFkMkEkEkEhGvB4NBnDt3DmfOnEGpVNrGEhLSG9Q0SrbM48ePceTIkY7La7WaaILsZG5uDseOHQPQCorJZBKpVMqxTiaTcTTvcY1GA9FoVDSp8uY+y7LE69FoFMlkEpZlic+QZdkRsMrlsiMwrnU/arUaZFlGKBTquq+bxbIs9PX1IR6Po9FoIBwOO/YfAOLxOM6cOYOJiQlIkgTDMBzbGBgYQLPZpCZSsidRICRbolar4d69ezh58qTn8ng8ju+++w7Ly8uIx+NtB2KuUqmIg7Wqqjhy5Ag++eQTsfzs2bO4fv06yuUyisWiuAI1DAODg4M4fvw4GGNQFEVc9USjUYRCIZimiV/96le4d++eCHz2wMu34w6Ma92Pu3fvOrbZazdu3MAf//hH6LqOqakpNJtNxONxTE1NiXUKhQI0TUOxWARjzDNIq6qKb775ZsvKTchWoaZR0nONRgMXL15EqVTqGED++te/IhqNQrbMJZkAAAsmSURBVNd13L59u+PVEg9w/f396O/vx1tvvSWWWZaFS5cuQdd1AMBXX30ltvPFF18gHo9jdHQUAEST7eTkJJaWlkSz5A8//OAIUpVKBe+88454Pjc3h76+vo77utr92L9/f8dtrCQajTqCmBtjzPF8dHQUtVoNwWAQY2NjAABZltveNzU1hWvXrnXc7qFDh7C8vLzOUhOyc9EVIekpwzBw6tQpXLt2rWMQBID3338fp06dQiKRwOeff+7ZBGcYBnRdR39/P4LBII4dO4a7d++K5Y8ePQIAKIqCY8eO4aWXXsKFCxcAAPfv38cvfvGLtm0+fPgQsVhMPP/yyy/x05/+VDwvl8s4fPiweO4OjOvZj40qlUpgjHV8eLl79y5GRkbE84mJCbz66qviea1Wg6IoXf9HhOxVFAhJTyWTSfz9738XB9hOV1OlUgkHDx7E/v37USgUHIM1uOnpacfBOhKJtA2+UVUVjDF8++23ePz4seMK5rvvvgMA5HI5TE5OiteXlpZgWRZyuZzoG8zlcgAAXdfx7Nkz8dwwDOzbtw9nz55d934A6HhlxT9/s83MzGDfvn0AWv8TWZYxNDQklq+mufbBgwcbupIlZMfaptGqxAfy+TwD4HioqrqubZmmKbaRTqcZY60pCAAcUxs0TWMAmKIoLJvNiteLxSKTZZnJsizez7chyzJTFIXV63UWi8WYqqpiWoMsy45pDrFYTKy7XnxaiVsqlWKKojAA4nM3i6qq4rtxT9tgrPW92b8vLxvdb0J2KomxDm0phJCeCYfDuHnzZscrxs1Uq9VEH20nkiShXq93LM9qtkHIbkVNoz5Wq9UQj8dFlpZwOOxoMiS9c/PmTZw5c6ZnTaF23333XddpK7lczjGK1s2yLFy8eBHpdLpXRXSo1WqIRqOeU1AKhQICgQAkSWprnrZPgwmHw47+2cnJSQQCgbbXgVZz/Vb8H8jORYHQpyYnJ/H2229D13Xoug7TNAEAL7744jaXzB8ikQjS6TTOnz/fcarIZpmdncXMzIznwT6TyeB//ud/cPPmTc/3NhoNnD9/Hul0ekuuXjOZDJ4+fepZVsMwcOLECczOzkLXdYyPjzvS650/fx4HDhwAYwyxWAzDw8Ni2a9+9Sv861//wm9/+1vR38s/77333qNBQn63zU2zZBuYpin6y9x9RYRsVCwWa+tLrNfra0pLp2maoy+XMcay2ayj3zSVSjlS9smyLFLT8T5lXg7+Pp4ujv+93j5rsrfQFaEP3bhxA0tLS4jH46s+E7YnunY/NiMBNdk7fv/732N4eFg0QTYaDZw5cwa/+c1vNrTd2dlZRxPv/v378f3334vPWFpaEqn3+O/62bNnjm3Yf+/nz5/HRx99tKEykb2BAqEPzc7OAsCa8mWyLvPWaAAFsYtEIrh58yaGh4dRKBQwPDyMdDq94eZHHvS8uAOeW7PZhGEYKBQKCIVCMAwDhmHg8OHDIuVcL+9iQnY2yizjQ/Pz8wAg8mVKkiSWdRs5uFb27ZK9ha0w2DwSieDixYs4ceIE8vn8pvymDhw40HH+5XPPPQegNWDGK+BevHgRiqJAURSUy2Ukk0mcO3cOn376Kf7yl7/gxRdfxPDwMOLx+IbLSXYfuiL0MX7A4KMBuw2IWE/TaLerSHrs7sdKGo0GLl++jHw+j3Pnzm1Khp1Dhw7h/v374vny8rJIYcd/tzy7EB9s88ILLwBo5YBljKHZbGJhYQGhUAgDAwMwDANDQ0OIRCKeaeeIP1Ag9CGeQaTRaMCyLPztb38DAGia1vE91DRKVov3CZZKJcTjcdFMupZg6DVqVNM0TE1Nid/txMSEI21eLBbD559/DqDVD65pmmeu15GREXz88cdtr9PIUR9jxHdM02SxWExkYEmlUjR6jmyaVCrVNhqZZ+1ZST6fZ6qqiixC7hsCZ7NZMeLZPQpV13WRPUdVVc8sOPl83jEaNZFIsHw+z3RdZ4qirHVXyR5BgZBsC7hSr3ml/dpMiUTCced3nv7NPUR/o0zTZKlUStzdnbFWSjWv6QBbpV6vt6VXSyQSjtR0jLXS0CUSiT07pcY0TaYoimP/eIo9WZbbvg/iH9Q0SrZFOp2GpmlgjInJ/F988UXPPo/fZumVV14B0Ooz0jTNcQeGTlY7PcSyLLz//vv4+c9/LgZddJsgvlWGh4dx/Phx6LqOUCiEYDCIK1eu4KuvvnKMlBwaGsLo6Cjef//9PZlpJRgMotlsOppAI5EIFhcXsbi46EhCTvyFAiHZFo8fP8bx48cBtA5Qvb5bu2EYUFXVcRBsNptd7zS/VufPn8fIyIhjwNHY2Nia5mtutlqthsXFRYyOjiIUCokTAgC4cOECzp075wh6kUgER44c6elJCSE7DQVCsi3K5TLefPNNAK38kYVCAe+++65Y3mg0RB7UcDjsSEOWyWREvslkMile53koA4EAMpmMI1flWu80vx7j4+M4evTopm1vJclkEoFAwJFj036Fl8vl8Pbbb2NpaaltGQBxT8cbN244Xn/33Xfx5z//eUv2gZCdgAIh2XL8Bruvv/46JEnCl19+idnZWXFVaBgGBgcHMTg4CMYYZFnGwsICAODs2bO4fv065ufnUSwW8dJLLwFo5U5NJpO4ffs2FhcXcf36dUezZ6VScdxw1x0YN6pWq7VdcfbS5OQkRkdHsbS0hC+++ALXrl1DPp/Hl19+KdYZHR1FOp1GKpUCY8xzjlwkEhEJFjj+f+jFTYUJ2YkoEJItNzc3J26gm81mAcDRnPjFF18gHo9jdHQUk5OT0HUdL7zwAizLwqVLlzAxMYFQKIShoSGMjY0BAP7whz/gypUrIsXW0tKSo9mzXC7j5ZdfFs8rlQpee+01z/JlMhnHPMmpqSnH81qt5vm+jQRBfkXX6eFmn/t28uRJBINBceNdu5mZGccJgNvBgwc9M7YcPnx4xWwthOwVFAjJlqtUKuJq7OjRo5iamnIsv3//PsbHxyFJEq5evYrbt28jFArh0aNHHeeGzc/Pi2bJWq3maPbkzar295XLZRw+fNizfGNjY455knxQD3/wYLuZSqXSmiew89to8f386quv8MYbbzjWmZqacpwAEELaUSAkW8qyLJTLZXE1FgqFIMty21VWsVgEYwwffvgh7ty5I15vNpuwLAuGYTj6B4FWppFarYY7d+4gGAyiUCjAMAwsLCwgHA6jUCiIz9F1Hc+ePXPckmcz9m09y9br7t27IhsK72c9efKkWF6r1aAoSteBSN999x0OHDjQ9vrc3JxIW0bInrfV8zWIv6VSKTF3kIvFYo55hMVikcmyzACwWCzGdF0X69rnw9knTPN5gqlUitXrdQZAzOXjz+0TsGVZXvXcRfutf7oB0La9lSaIb4SmaeL7VBSlbbvpdHrFSeyJRIJls1nHa7quM1mWN6WMhOwGEmOrSBxIiI9Fo9FVpZFLJpOIRCIYHR3dglK18r92q77RaBTHjx/vWB7LstDf349vv/3W0b/JR9vy/ldC9jpqGiVkBavNpXrhwgX87//+75aMtmw0Gl1zwzYaDUxNTXWdznH+/HlcvHjREQQbjQZmZmYcTayE7HUUCAnZJMFgENeuXcM///lPTE5O9vSzvvnmGzSbTc+gW6vVMDg4iHw+79k/aFkWkskkfvaznzmmVExOTiKXy+HatWuUgJr4CjWNEkII8TW6IiSEEOJrFAgJIYT4GgVCQgghvkaBkBBCiK9RICSEEOJrFAgJIYT4GgVCQgghvkaBkBBCiK9RICSEEOJrFAgJIYT4GgVCQgghvkaBkBBCiK/9f6+zKDmkjLPaAAAAAElFTkSuQmCC\" width=\"450\" height=\"168\"\u003e\u003c/p\u003e\n\u003cp\u003eHowever, among these five performance metrics, the higher R, P, F1, and G are, the better, while the lower pf is, the better. The final results (higher SBR label) will be ranked top to the security engineer as depicted in Figure \u003cspan dir=\"RTL\"\u003e3\u003c/span\u003e \u0026ndash; steps 4, 5, and 6, to be verified as the correct label to assign to the bug report.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e\n\u003cdiv id=\"ftn1\"\u003e\n \u003cp\u003e[1] https://cwe.mitre.org/\u003c/p\u003e\n \u003cp\u003e[2] https://docs.python.org/3/c-api/utilities.html\u003c/p\u003e\n \u003cp\u003e[3] 16 GB RAM, 20 GB SSD, Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz \u0026nbsp; 1.99 GHz.\u003c/p\u003e\n\u003c/div\u003e"},{"header":"4. Case Study Results","content":"\u003cp\u003eIn this section, we present the results of our case study concerning the two research questions (RQ1 and RQ2). We offer the motivation, approach to answering the question, and results for each research question.\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003e4.1 RQ1: Can we accurately detect SBR using a \u003cem\u003efasttext\u003c/em\u003e machine learning algorithm?\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eMotivation\u003c/strong\u003e: Correctly identifying a security bug report is essential to accelerate bug triaging and fixing [10] and mitigate the security impact. Even though prior work proposed classification models and techniques to automatically classify and predict SBRs from NSBRs [2], [5], [7], the accuracy of these models shows moderate performance. Thus, the main goal of this research question is to investigate the use of \u003cem\u003efasttext\u003c/em\u003e models to assist developers in automatically identifying SBRs from NSBRs.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eApproach\u003c/strong\u003e: For each project in the testing dataset (Section 3.1), we used the manually annotated bug reports shown in Table 1 to train the \u003cem\u003efasttext\u003c/em\u003e classifier to predict whether a bug report is a security bug report or not. However, we used 10-fold cross-validation for each project\u0026nbsp;[30]. First, we divided the dataset for each project into ten folds. We used eight folds (i.e., 80% of the data) to train the \u003cem\u003efasttext\u003c/em\u003e and the remaining two folds to evaluate the classifier\u0026rsquo;s performance. We ran this process ten times for each fold (i.e., 1x10-folds). Finally, to assess the performance of the \u003cem\u003efasttext\u003c/em\u003e classifier in detecting SBRs, we computed the well-known evaluation metrics, precision and recall, and their combination F1-score, as explained in Section 3.3, ten times for each fold. Then, to come up with one value for the ten runs, we computed the average of the evaluation metrics for 10-folds ten times (i.e., 1x10-fold) for every project in our testing dataset. Finally, we compared \u003cem\u003efasttext\u003c/em\u003e results with the KNN machine learning (ML) algorithm\u0026nbsp;[31]\u0026nbsp;that was successfully used in the assessment of what classifier provides the best accuracy for identifying security bug reports\u0026nbsp;[32].\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults\u003c/strong\u003e: Table 3 presents the precision, recall, f1-measure, pf, and g-measure values of the \u003cem\u003efasttext\u003c/em\u003e classifier for the five studied projects in our test dataset. First, the precision values obtained by the \u003cem\u003efasttext\u003c/em\u003e classifier range between 0.73 and 0.94 with an average of 0.83 (median = 0.83), while the recall values range between 0.71 and 0.94 with an average value of 0.81 (median = 0.79). Also, the f1-score achieves values between 0.71 and 0.94, with an average of 0.81 and a\u003cspan dir=\"RTL\"\u003e\u0026nbsp;\u003c/span\u003emedian of 0.80. The values of pf and G-measure indicate that the \u003cem\u003efasttext\u003c/em\u003e effectively detects SBRs from NSBRs, achieving averages of 0.20 (median = 0.19) and 0.80 (median=0.80), respectively.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eTable\u0026nbsp;3: Performance of the\u0026nbsp;fasttext\u0026nbsp;Technique.\u003c/p\u003e\n\u003ctable border=\"1\" cellpadding=\"0\" cellspacing=\"0\" width=\"100%\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.52577319587629%\"\u003e\n \u003cp\u003eProject\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003ePrecision\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003eRecall\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"20.61855670103093%\"\u003e\n \u003cp\u003eF-measure\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003epf\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003eG-measure\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.52577319587629%\"\u003e\n \u003cp\u003eAmbari\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.83\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.79\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"20.61855670103093%\"\u003e\n \u003cp\u003e0.79 (+0.22)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.15\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.81 (+0.04)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.52577319587629%\"\u003e\n \u003cp\u003eCamel\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.73\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.71\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"20.61855670103093%\"\u003e\n \u003cp\u003e0.71 (+0.07)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.34\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.67 (-0.10)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.52577319587629%\"\u003e\n \u003cp\u003eChromium\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.94\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.94\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"20.61855670103093%\"\u003e\n \u003cp\u003e0.94 (+0.16)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.08\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.93 (+0.05)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.52577319587629%\"\u003e\n \u003cp\u003eDerby\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.82\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.79\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"20.61855670103093%\"\u003e\n \u003cp\u003e0.80 (+0.15)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.19\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.80 (+0.14)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.52577319587629%\"\u003e\n \u003cp\u003eWicket\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.85\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.81\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"20.61855670103093%\"\u003e\n \u003cp\u003e0.81 (+0.13)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.25\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.77 (- 0.07)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.52577319587629%\"\u003e\n \u003cp\u003e\u003cstrong\u003eAverage \u0026nbsp; \u0026nbsp;\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.83\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.81\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"20.61855670103093%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.81 (+0.15)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.20\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.80 (+0.01)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.52577319587629%\"\u003e\n \u003cp\u003e\u003cstrong\u003eMedian\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.83\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.79\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"20.61855670103093%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.80 (+0.15)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.19\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.80 (+0.02)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eAlso, Table 3 shows the F1-score when comparing the performance of \u003cem\u003efasttext\u003c/em\u003e to the baseline. The compared F1-score shows an improvement of +0.15 on average over the baseline. In particular, for all five projects, the \u003cem\u003efasttext\u003c/em\u003e outperforms the baseline with relative F1-score values ranging between +0.07 and +0.22.\u0026nbsp;\u003c/p\u003e\n\u003ctable cellpadding=\"0\" cellspacing=\"0\" width=\"100%\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eOur fasttext classifier achieves an average F1-score of 0.81 (G-Measure of 0.80). Additionally, our results show that the fasttext classifier can effectively improve the detection of SBRs with an average F-score of +0.15 compared to the baseline.\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e\n\u003ch2\u003e4.2 RQ2: How effective is the \u003cem\u003efasttext\u003c/em\u003e classifier when applied on cross-projects?\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eMotivation\u003c/strong\u003e: Building a machine learning (ML) classifier to identify SBR requires having labeled data to train on. However, many open-source projects do not have sufficient historically labeled SBR data to build a classifier (e.g., unlabeled, minor, or new project). Thus, we cannot train a ML classifier to detect SBRs on data from these projects. For this research question, we investigated to what extent and with what accuracy a bug report can be automatically classified as a SBR using cross-project \u003cem\u003efasttext\u003c/em\u003e machine learning classification. \u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eApproach\u003c/strong\u003e: We conduct a cross-project validation to understand better the generalizability of the results achieved by the fasttext classifier on bug report data from one project and apply it to another project. In particular, we experimented with five-fold cross-project validation. We conducted an experiment that trained a \u003cem\u003efasttext\u003c/em\u003e classifier on bug report data from four projects and used the resulting classifier to determine whether a bug report was SBR or NSBR in the remaining project, similar to related work\u0026nbsp;[8]\u0026nbsp;and\u0026nbsp;[33]. We repeated this process five times for each project in the testing dataset. To evaluate the performance, we employed the well-known evaluation metrics where we computed the precision, recall, and F1-score, pf, and G-measure to measure the performance of the \u003cem\u003efasttext\u003c/em\u003e classifier. Finally, to examine the performance of the cross-project classifier concerning the random baseline, we computed the relative F1 score and compared the results with the baseline, as shown in RQ1.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults\u003c/strong\u003e: Table 4 presents the results of our experiment. It presents each project\u0026apos;s precision, recall, f-score, pf, and G-measure values. Table 4 shows that the \u003cem\u003efasttext\u003c/em\u003e classifier achieved performance of pf and G-score values of 0.29 and 0.27, and 0.67 and 0.69 for the average and median, respectively. Four projects out of five show good performance results. The projects Ambari, Camel, Chromium, and Wicket achieved high pf and G-score. Other projects show a moderate performance, including Derby, with values of 0.53 and 0.52 for the pf and G-score, respectively. Our results show that our classifier achieves an average F1 score of 0.65, which is lower than what we expected, however, it is still a good classifier that can be practically used.\u003c/p\u003e\n\u003cp\u003eFinally, when we compare the performance of the cross-projects classifier to the baseline, our results show that the cross-project classifier shows an improvement of +0.19 on average over the baseline.\u0026nbsp;\u003c/p\u003e\n\u003ctable cellpadding=\"0\" cellspacing=\"0\" width=\"100%\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eThe results show that cross-projects machine learning classifier can provide comparable performance to within-project classifier on SBR classification. For four out of the five studied projects, cross-projects classifier achieved G-measure values between 0.52-0.74 with an overall average equal to 0.67.\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e\n\u003cp\u003eTable\u0026nbsp;4: Performance of Cross-projects classification.\u003c/p\u003e\n\u003ctable border=\"1\" cellpadding=\"0\" cellspacing=\"0\" width=\"100%\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.52577319587629%\"\u003e\n \u003cp\u003eProject\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003ePrecision\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003eRecall\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"20.61855670103093%\"\u003e\n \u003cp\u003eF-measure\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003epf\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003eG-measure\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.52577319587629%\"\u003e\n \u003cp\u003eAmbari\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.69\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.66\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"20.61855670103093%\"\u003e\n \u003cp\u003e0.64 (+0.16)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.16\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.74 (+0.04)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.52577319587629%\"\u003e\n \u003cp\u003eCamel\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.67\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.67\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"20.61855670103093%\"\u003e\n \u003cp\u003e0.66 (+0.26)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.27\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.69 (+0.08)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.52577319587629%\"\u003e\n \u003cp\u003eChromium\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.68\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.66\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"20.61855670103093%\"\u003e\n \u003cp\u003e0.65 (+0.19)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.27\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.69 (+0.07)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.52577319587629%\"\u003e\n \u003cp\u003eDerby\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.61\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.60\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"20.61855670103093%\"\u003e\n \u003cp\u003e0.59 (+0.00)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.53\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.52 (-0.23)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.52577319587629%\"\u003e\n \u003cp\u003eWicket\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.72\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.70\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"20.61855670103093%\"\u003e\n \u003cp\u003e0.70 (+0.33)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.23\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e0.73 (+0.17)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.52577319587629%\"\u003e\n \u003cp\u003e\u003cstrong\u003eAverage\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.67\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.66\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"20.61855670103093%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.65 (+0.19)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.29\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.67 (+0.03)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" width=\"17.52577319587629%\"\u003e\n \u003cp\u003e\u003cstrong\u003eMedian\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.68\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.66\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"20.61855670103093%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.65 (+0.19)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.27\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.69 (+0.07)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e"},{"header":"5. Discussion","content":"\u003cp\u003eIn this section, we examine the use of other ML classifiers to classify bug reports into SBR or NSBR by answering the following question: \u003cem\u003eWhat classifier provides the best accuracy for identifying the security bug reports?\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eSo far, we used \u003cem\u003efasttext\u003c/em\u003e to determine a SBR, which showed a practical improvement over the baseline. However, \u003cem\u003efasttext\u003c/em\u003e is not the only text classification classifier. Thus, in this subsection, we investigate the use of other ML classifiers and compare their performance in identifying SBRs. We use our dataset prepared in Section 3.1 and the same approach described in Section 3 to train five other ML classifiers, namely, Random Forest (RF), Naive Bayes (NB), KNearest Neighbor (KNN), Multilayer Perceptron (MLP), and Logistical Regression (LR). We chose to examine these ML classifiers since they have different assumptions about the analyzed data, as well as having other characteristics in terms of execution speed and dealing with overfitting [7], [34], [35]. Also, they have been commonly used in the past in other software engineering and security studies (e.g., [2], [27], [33]).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eTable\u0026nbsp;5: Performance of using different classifiers to detect SBRs.\u003c/p\u003e\n\u003ctable border=\"1\" cellpadding=\"0\" cellspacing=\"0\" width=\"100%\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"2\" width=\"12.5%\"\u003e\n \u003cp\u003eProject\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" width=\"14.583333333333334%\"\u003e\n \u003cp\u003e\u003cem\u003efasttext\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" width=\"14.583333333333334%\"\u003e\n \u003cp\u003eNB\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" width=\"14.583333333333334%\"\u003e\n \u003cp\u003eKNN\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" width=\"14.583333333333334%\"\u003e\n \u003cp\u003eMLP\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" width=\"14.583333333333334%\"\u003e\n \u003cp\u003eLR\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" width=\"14.583333333333334%\"\u003e\n \u003cp\u003eRF\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"8.333333333333334%\"\u003e\n \u003cp\u003eF1-score\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.333333333333334%\"\u003e\n \u003cp\u003eG-measure\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.333333333333334%\"\u003e\n \u003cp\u003eF1-score\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.333333333333334%\"\u003e\n \u003cp\u003eG-measure\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.333333333333334%\"\u003e\n \u003cp\u003eF1-score\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.333333333333334%\"\u003e\n \u003cp\u003eG-measure\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.333333333333334%\"\u003e\n \u003cp\u003eF1-score\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.333333333333334%\"\u003e\n \u003cp\u003eG-measure\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.333333333333334%\"\u003e\n \u003cp\u003eF1-score\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.333333333333334%\"\u003e\n \u003cp\u003eG-measure\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.333333333333334%\"\u003e\n \u003cp\u003eF1-score\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.333333333333334%\"\u003e\n \u003cp\u003eG-measure\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12.5%\"\u003e\n \u003cp\u003eAmbari\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.79\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.81\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.75\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.67\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.57\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.77\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.72\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.73\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.83\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.87\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.77\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.86\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12.5%\"\u003e\n \u003cp\u003eCamel\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.71\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.67\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.67\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.61\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.63\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.77\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.69\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.75\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.70\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.77\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.81\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.86\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12.5%\"\u003e\n \u003cp\u003eChromium\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.94\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.93\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.89\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.86\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.78\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.88\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.94\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.96\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.95\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.96\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.95\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.97\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12.5%\"\u003e\n \u003cp\u003eDerby\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.80\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.80\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.76\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.73\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.65\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.66\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.81\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.84\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.76\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.79\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.77\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.83\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12.5%\"\u003e\n \u003cp\u003eWicket\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.81\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.77\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.68\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.59\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.68\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.84\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.78\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.80\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.82\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.86\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.82\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e0.89\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12.5%\"\u003e\n \u003cp\u003e\u003cstrong\u003eAverage\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.81\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.80\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.75\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.69\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.66\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.78\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.79\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.82\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.81\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.85\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.82\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.88\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12.5%\"\u003e\n \u003cp\u003e\u003cstrong\u003eMedian\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.80\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.80\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.75\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.67\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.65\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.77\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.78\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.80\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.82\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.86\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.81\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" width=\"7.291666666666667%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.86\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eTo comprehensively compare the different classifiers, we compare them in the following scenario. We perform a within-project evaluation, where the classifiers are trained and tested using non-overlapping data from the same project. And to examine the performance of within-project classification of the different ML classifiers concerning the random baseline, we compute the F1-score and G-measure.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eTable 5 shows the F1-score and G-measure for the examined five classifiers and the \u003cem\u003efasttext\u003c/em\u003e. As Table 5 shows, on average, \u003cem\u003eRF, LR,\u0026nbsp;\u003c/em\u003eand\u003cem\u003e\u0026nbsp;fasttext\u003c/em\u003e produce the highest F1-score values with an average of 0.82, 0.81, 0.81 (median = 0.81, 0.82, 0.80), respectively. At the same time, the MLP, NB, and KNN classifiers achieve the lowest performance with average F1 scores of 0.79, 0.75, and 066, Median = 0.78, 0.75, and 0.65, respectively, across all the studied projects. The RF classifiers perform better than the \u003cem\u003efasttext\u003c/em\u003e with a G-measure of 0.88 (median = 88). This corresponds to a significant improvement between 0.83 \u0026ndash; 0.97, on average, in G-measure over the baseline. \u0026nbsp;\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eWith an average G-score of 0.69 (median = 0.67), one ML classifier (i.e, NB) performs significantly low. \u003cstrong\u003eThe highest values achieved are produced again by the \u003cem\u003efasttext\u003c/em\u003e classifier, along with RF and LR\u003c/strong\u003e. For example, three projects in the testing dataset have G-score values greater than 0.80. The results suggest that the \u003cem\u003efasttext\u003c/em\u003e classifier is one of the best ML classifiers to detect SBR issues.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCross-project Classification.\u003c/strong\u003e Table 6 shows the results of the F1-score and G-measure that are achieved by the five classifiers for the cross-project validations. In this experiment, the RF classifier achieves the best results, followed by LR, and \u003cem\u003efasttext\u003c/em\u003e, with 0.65 average F1-score values and 0.78, 0.74, and 0.67 G-measure values, respectively. The other classifiers show low performance by achieving average F1-score values of 0.46 \u0026ndash; 0.62.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe cases of Wicket, Camel, Chromium, Derby, and Ambari projects are interesting where the RF classifier can achieve a high-performance value range between 0.72 \u0026ndash; 0.86 for G-score values. In contrast, \u003cem\u003efasttext\u003c/em\u003e shows moderate performance by completing average G-scores of 0.74 and 0.73 for projects Ambari and Wicket, respectively. Also, a project produced poor results: Derby (average G-score=0.52).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eTable\u0026nbsp;6: Performance of using different classifiers to detect SBRs cross-project validation\u003c/p\u003e\n\u003ctable border=\"0\" cellpadding=\"0\" cellspacing=\"0\" width=\"100%\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"2\" width=\"9.278350515463918%\"\u003e\n \u003cp\u003eProject\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003e\u003cem\u003efasttext\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003eNB\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003eKNN\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003eMLP\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"bottom\" width=\"15.463917525773196%\"\u003e\n \u003cp\u003eLR\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" valign=\"bottom\" width=\"13.402061855670103%\"\u003e\n \u003cp\u003eRF\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"7.954545454545454%\"\u003e\n \u003cp\u003eF1-score\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9.090909090909092%\"\u003e\n \u003cp\u003eG-measure\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.954545454545454%\"\u003e\n \u003cp\u003eF1-score\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9.090909090909092%\"\u003e\n \u003cp\u003eG-measure\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.954545454545454%\"\u003e\n \u003cp\u003eF1-score\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9.090909090909092%\"\u003e\n \u003cp\u003eG-measure\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.954545454545454%\"\u003e\n \u003cp\u003eF1-score\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9.090909090909092%\"\u003e\n \u003cp\u003eG-measure\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.954545454545454%\"\u003e\n \u003cp\u003eF1-score\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"9.090909090909092%\"\u003e\n \u003cp\u003eG-measure\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.954545454545454%\"\u003e\n \u003cp\u003eF1-score\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.818181818181818%\"\u003e\n \u003cp\u003eG-measure\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" width=\"9.278350515463918%\"\u003e\n \u003cp\u003eAmbari\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.64\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.74\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.60\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.67\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.49\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.70\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.66\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.71\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.66\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.79\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.46\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.185567010309279%\"\u003e\n \u003cp\u003e0.72\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" width=\"9.278350515463918%\"\u003e\n \u003cp\u003eCamel\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.66\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.69\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.53\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.44\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.40\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.61\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.62\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.64\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.67\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.80\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.70\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.185567010309279%\"\u003e\n \u003cp\u003e0.81\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" width=\"9.278350515463918%\"\u003e\n \u003cp\u003eChromium\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.65\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.69\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.67\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.54\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.46\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.62\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.58\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.55\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.65\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.73\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.63\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.185567010309279%\"\u003e\n \u003cp\u003e0.78\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" width=\"9.278350515463918%\"\u003e\n \u003cp\u003eDerby\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.59\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.52\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.52\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.41\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.59\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.75\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.61\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.62\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.61\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.64\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.66\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.185567010309279%\"\u003e\n \u003cp\u003e0.73\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" width=\"9.278350515463918%\"\u003e\n \u003cp\u003eWicket\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.70\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.73\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.65\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.58\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.37\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.56\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.63\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.66\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.66\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e0.73\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e0.79\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.185567010309279%\"\u003e\n \u003cp\u003e0.86\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" width=\"9.278350515463918%\"\u003e\n \u003cp\u003e\u003cstrong\u003eAverage\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.65\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.67\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.59\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.53\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.46\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.65\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.62\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.64\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.65\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.74\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.65\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.185567010309279%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.78\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" width=\"9.278350515463918%\"\u003e\n \u003cp\u003e\u003cstrong\u003eMedian\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.65\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.69\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.60\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.54\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.46\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.62\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.62\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.64\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.66\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"8.24742268041237%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.73\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.216494845360825%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.66\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.185567010309279%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.78\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e"},{"header":"6. Threats To Validity","content":"\u003cp\u003eThis section describes the threats to the validity of our study.\u003c/p\u003e\n\u003ch2\u003e6.1 Internal Validity\u003c/h2\u003e\n\u003cp\u003eA threat to the internal validity of our study is the potential of mislabels in the datasets [7]. We were testing the publicly available dataset [2], [5], [7], in which the authors took several measures to reduce these threats. However, it is still challenging to guarantee whether a bug report is a SBR or NSBR when there is no clear definition for SBR, as noted by Ohira et al. [16]. To mitigate this threat, the SBR is defined based on the definitions of Common Weakness Enumeration (CWE\u003csup\u003e[4]\u003c/sup\u003e), which is the most authoritative organization of vulnerability management, as a basis to judge whether a bug report is a SBR or not [7]. In addition, we used a data set that is annotated by six annotators who conducted manual reviews and have the rich practical experience and a deep understating of different vulnerability types and characteristics [7].\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAnother threat to the internal validity is the impact of tuning for \u003cem\u003efasttext\u003c/em\u003e hyper-parameters. To mitigate this threat, a set of tuning approaches have been considered in our experiments, including selecting the optimal values for more epochs(the number of times each training example is seen in the algorithm) and more significant learning rate \u0026ndash; lr (Good values of the learning rate are in the range 0.1 - 1.0). In addition to that, we improved the model\u0026rsquo;s performance by using word bigrams (using option -wordNgrams) instead of just unigrams and ranging the values within a standard range [1 \u0026ndash; 5]\u003csup\u003e[5]\u003c/sup\u003e. The experiment results show that this tuning approach increased the impact of the findings.\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003e6.2 External Validity\u0026nbsp;\u003c/h2\u003e\n\u003cp\u003eThreats to external validity are concerned with the generality of our findings. Our study is based on specific projects (four Apache projects and the Chromium project), hence our results may not hold for project writing in other environments. Generalizing our experiment\u0026rsquo;s results requires multiple labeled bug reports from different domain projects. And to improve the outcomes and refute the false positives, replication experiments are needed. However, the ML technique can be easily generalized to other projects by analyzing the unstructured English text (e.g., bug reports\u0026rsquo; descriptions and titles) in different project environments. In this study, we evaluated the impact of \u003cem\u003efasttext\u003c/em\u003e by comparing the model\u0026rsquo;s performance when applied to cross-projects validation. Three baseline methods extracted from two recent SBR prediction works proposed by [5], [7] and simple classification models were applied. Moreover, a set of performance indicators, including Precision, Recall, F1-score, and G-measure, were used for the evaluation.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAnother external threat is that the dataset used in our study presents only open-source projects that do not reflect proprietary projects. Furthermore, we examined projects that published bug reports for open discussions on issue tracker systems, and different issue tracker systems could have more advanced features for classifying SBRs. That said, the GitHub issues tracker is one of the most popular issues tracker systems in software development environments, with basic features for publishing bug reports.\u0026nbsp;\u003c/p\u003e\n\u003cdiv id=\"ftn1\"\u003e\n \u003cp\u003e[4] https://cwe.mitre.org/\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"ftn2\"\u003e\n \u003cp\u003e[5] https://fasttext.cc/docs/en/options.html\u003c/p\u003e\n\u003c/div\u003e"},{"header":"7. Conclusion","content":"\u003cp\u003eIn this paper, we studied bug reports that developers tend to produce during maintaining software projects and posting on issue tracker systems. Specifically, our goal was to use machine learning techniques to classify bug reports that can be security bug reports (SBRs) or not security bug reports (NSBRs). To do so, we proposed using publicly available bug datasets extracted from five Java projects. Then, we built a \u003cem\u003efasttext\u003c/em\u003e classifier. We found that the \u003cem\u003efasttext\u003c/em\u003e classifier can effectively improve the classification of SBRs with an average F1-score of 0.81 (median\u0026thinsp;=\u0026thinsp;0.80). It also achieves an average G-measure of 0.80 (median\u0026thinsp;=\u0026thinsp;0.80), representing an improvement of +\u0026thinsp;0.19 on average over the state-of-the-art approach. Additionally, we investigated the generalizability of labeling bug reports (with security or not security labels) when we used cross-projects validation. Our results show that the cross-project validation achieves, on average, 0.65 and 0.67 for F1-score and G-measure, respectively.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cul\u003e\n \u003cli\u003eThe author Sultan S. Alqahtani, declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.\u003c/li\u003e\n \u003cli\u003eNo funding was received to assist with the preparation of this manuscript.\u003c/li\u003e\n \u003cli\u003eThe author declare he has no financial interests\u003c/li\u003e\n \u003cli\u003eThe data that support the findings of this study are available from the corresponding author upon reasonable request.\u003c/li\u003e\n \u003cli\u003eSultan S. Alqahtani contributed to the study conception and design, including the material preparation, data collection and analysis. Also, Dr. Sultan prepared the first draft of the manuscript and proof-reader commented on previous versions of the manuscript. The author read and approved the final manuscript.\u003c/li\u003e\n\u003c/ul\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eP. Floris and H. Vogt Harald, \u0026ldquo;How to save on software maintenance costs, omnext white pape,\u0026rdquo; vol. SOURCE 2 V, 2010.\u003c/li\u003e\n\u003cli\u003eS. Rui, X. Tianpei, W. Laurie, and M. Tim, \u0026ldquo;Better Security Bug Report Classification via Hyperparameter Optimization,\u0026rdquo; \u003cem\u003ehttps://arxiv.org/pdf/1905.06872.pdf\u003c/em\u003e, 2019.\u003c/li\u003e\n\u003cli\u003eI. Chawla and S. K. Singh, \u0026ldquo;Automatic bug labeling using semantic information from LSI,\u0026rdquo; in \u003cem\u003e2014 Seventh International Conference on Contemporary Computing (IC3)\u003c/em\u003e, Aug. 2014, pp. 376\u0026ndash;381, doi: 10.1109/IC3.2014.6897203.\u003c/li\u003e\n\u003cli\u003eM. Bozorgi, L. K. Saul, S. Savage, and G. M. Voelker, \u0026ldquo;Beyond heuristics: learning to classify vulnerabilities and predict exploits,\u0026rdquo; in \u003cem\u003eProceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD \u0026rsquo;10\u003c/em\u003e, 2010, p. 105, doi: 10.1145/1835804.1835821.\u003c/li\u003e\n\u003cli\u003eF. Peters, T. T. Tun, Y. Yu, and B. Nuseibeh, \u0026ldquo;Text Filtering and Ranking for Security Bug Report Prediction,\u0026rdquo; \u003cem\u003eIEEE Trans. Softw. Eng.\u003c/em\u003e, vol. 45, no. 6, pp. 615\u0026ndash;631, Jun. 2019, doi: 10.1109/TSE.2017.2787653.\u003c/li\u003e\n\u003cli\u003eD. Wijayasekara, M. Manic, J. L. Wright, and M. McQueen, \u0026ldquo;Mining Bug Databases for Unidentified Software Vulnerabilities,\u0026rdquo; in \u003cem\u003e2012 5th International Conference on Human System Interactions\u003c/em\u003e, Jun. 2012, pp. 89\u0026ndash;96, doi: 10.1109/HSI.2012.22.\u003c/li\u003e\n\u003cli\u003eX. Wu, W. Zheng, X. Xia, and D. Lo, \u0026ldquo;Data Quality Matters: A Case Study on Data Label Correctness for Security Bug Report Prediction,\u0026rdquo; \u003cem\u003eIEEE Trans. Softw. Eng.\u003c/em\u003e, vol. 48, no. 7, pp. 2541\u0026ndash;2556, Jul. 2022, doi: 10.1109/TSE.2021.3063727.\u003c/li\u003e\n\u003cli\u003eM. Sharma, P. Bedi, K. K. Chaturvedi, and V. B. Singh, \u0026ldquo;Predicting the priority of a reported bug using machine learning techniques and cross project validation,\u0026rdquo; in \u003cem\u003e2012 12th International Conference on Intelligent Systems Design and Applications (ISDA)\u003c/em\u003e, Nov. 2012, pp. 539\u0026ndash;545, doi: 10.1109/ISDA.2012.6416595.\u003c/li\u003e\n\u003cli\u003eN. Limsettho, H. Hata, A. Monden, and K. Matsumoto, \u0026ldquo;Automatic Unsupervised Bug Report Categorization,\u0026rdquo; in \u003cem\u003e2014 6th International Workshop on Empirical Software Engineering in Practice\u003c/em\u003e, Nov. 2014, pp. 7\u0026ndash;12, doi: 10.1109/IWESEP.2014.8.\u003c/li\u003e\n\u003cli\u003eA. D. Sawadogo, T. F. Guimard, Quentin Bissyand\u0026eacute;, J. Kader Kabor\u0026eacute;, Abdoul Klein, and N. Moha, \u0026ldquo;Early Detection of Security-Relevant Bug Reports using Machine Learning: How Far Are We?,\u0026rdquo; \u003cem\u003eeprint arXiv:2112.10123\u003c/em\u003e, 2021, [Online]. Available: https://ui.adsabs.harvard.edu/abs/2021arXiv211210123S/abstract.\u003c/li\u003e\n\u003cli\u003eA. Agrawal and T. Menzies, \u0026ldquo;Is \u0026lsquo;better data\u0026rsquo; better than \u0026lsquo;better data miners\u0026rsquo;?,\u0026rdquo; in \u003cem\u003eProceedings of the 40th International Conference on Software Engineering\u003c/em\u003e, May 2018, pp. 1050\u0026ndash;1061, doi: 10.1145/3180155.3180197.\u003c/li\u003e\n\u003cli\u003eW. Fu, T. Menzies, and X. Shen, \u0026ldquo;Tuning for software analytics: Is it really necessary?,\u0026rdquo; \u003cem\u003eInf. Softw. Technol.\u003c/em\u003e, vol. 76, pp. 135\u0026ndash;146, Aug. 2016, doi: 10.1016/j.infsof.2016.04.017.\u003c/li\u003e\n\u003cli\u003eM. Tianpei, Xia Rahul, Krishna Jianfeng, Chen George, Mathew Xipeng, Shen Tim, \u0026ldquo;Hyperparameter Optimization for Effort Estimation,\u0026rdquo; \u003cem\u003earxiv\u003c/em\u003e, 2018, doi: https://doi.org/10.48550/arXiv.1805.00336.\u003c/li\u003e\n\u003cli\u003eW. Fu and T. Menzies, \u0026ldquo;Easy over hard: a case study on deep learning,\u0026rdquo; in \u003cem\u003eProceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering\u003c/em\u003e, Aug. 2017, pp. 49\u0026ndash;60, doi: 10.1145/3106237.3106256.\u003c/li\u003e\n\u003cli\u003eZ. Liu, X. Xia, A. E. Hassan, D. Lo, Z. Xing, and X. Wang, \u0026ldquo;Neural-machine-translation-based commit message generation: how far are we?,\u0026rdquo; in \u003cem\u003eProceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering\u003c/em\u003e, Sep. 2018, pp. 373\u0026ndash;384, doi: 10.1145/3238147.3238190.\u003c/li\u003e\n\u003cli\u003eM. Ohira \u003cem\u003eet al.\u003c/em\u003e, \u0026ldquo;A Dataset of High Impact Bugs: Manually-Classified Issue Reports,\u0026rdquo; in \u003cem\u003e2015 IEEE/ACM 12th Working Conference on Mining Software Repositories\u003c/em\u003e, May 2015, pp. 518\u0026ndash;521, doi: 10.1109/MSR.2015.78.\u003c/li\u003e\n\u003cli\u003eN. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, \u0026ldquo;SMOTE: Synthetic Minority Over-sampling Technique,\u0026rdquo; \u003cem\u003eJ. Artif. Intell. Res.\u003c/em\u003e, vol. 16, pp. 321\u0026ndash;357, Jun. 2002, doi: 10.1613/jair.953.\u003c/li\u003e\n\u003cli\u003eR. Abir and A. A. Moulay, \u0026ldquo;MALBERT: USING TRANSFORMERS FOR CYBERSECURITY AND MALICIOUS SOFTWARE DETECTION,\u0026rdquo; \u003cem\u003ehttps://arxiv.org/pdf/2103.03806.pdf\u003c/em\u003e, 2021.\u003c/li\u003e\n\u003cli\u003eM. Roopak, G. Yun Tian, and J. Chambers, \u0026ldquo;Deep Learning Models for Cyber Security in IoT Networks,\u0026rdquo; in \u003cem\u003e2019 IEEE 9th Annual Computing and Communication Workshop and Conference (CCWC)\u003c/em\u003e, Jan. 2019, pp. 0452\u0026ndash;0457, doi: 10.1109/CCWC.2019.8666588.\u003c/li\u003e\n\u003cli\u003eJ. Yin, M. Tang, J. Cao, and H. Wang, \u0026ldquo;Apply transfer learning to cybersecurity: Predicting exploitability of vulnerabilities by description,\u0026rdquo; \u003cem\u003eKnowledge-Based Syst.\u003c/em\u003e, vol. 210, p. 106529, Dec. 2020, doi: 10.1016/j.knosys.2020.106529.\u003c/li\u003e\n\u003cli\u003eA. Joulin, E. Grave, P. Bojanowski, and T. Mikolov, \u0026ldquo;Bag of Tricks for Efficient Text Classification,\u0026rdquo; in \u003cem\u003eProceedings of the 15th Conference of the {E}uropean Chapter of the Association for Computational Linguistics: Volume 2, Short Papers\u003c/em\u003e, Apr. 2017, pp. 427\u0026ndash;431, [Online]. Available: https://aclanthology.org/E17-2068.\u003c/li\u003e\n\u003cli\u003eR. Johnson and T. Zhang, \u0026ldquo;Semi-supervised Convolutional Neural Networks for Text Categorization via Region Embedding,\u0026rdquo; in \u003cem\u003eAdvances in Neural Information Processing Systems\u003c/em\u003e, 2015, vol. 28, [Online]. Available: https://proceedings.neurips.cc/paper/2015/file/acc3e0404646c57502b480dc052c4fe1-Paper.pdf.\u003c/li\u003e\n\u003cli\u003eJ. Liu, W.-C. Chang, Y. Wu, and Y. Yang, \u0026ldquo;Deep Learning for Extreme Multi-label Text Classification,\u0026rdquo; in \u003cem\u003eProceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval\u003c/em\u003e, Aug. 2017, pp. 115\u0026ndash;124, doi: 10.1145/3077136.3080834.\u003c/li\u003e\n\u003cli\u003eR. Kallis, A. Di Sorbo, G. Canfora, and S. Panichella, \u0026ldquo;Ticket Tagger: Machine Learning Driven Issue Classification,\u0026rdquo; in \u003cem\u003e2019 IEEE International Conference on Software Maintenance and Evolution (ICSME)\u003c/em\u003e, Sep. 2019, pp. 406\u0026ndash;409, doi: 10.1109/ICSME.2019.00070.\u003c/li\u003e\n\u003cli\u003eQ. Song, Y. Guo, and M. Shepperd, \u0026ldquo;A Comprehensive Investigation of the Role of Imbalanced Learning for Software Defect Prediction,\u0026rdquo; \u003cem\u003eIEEE Trans. Softw. Eng.\u003c/em\u003e, vol. 45, no. 12, pp. 1253\u0026ndash;1269, Dec. 2019, doi: 10.1109/TSE.2018.2836442.\u003c/li\u003e\n\u003cli\u003eY. M. Mileva, V. Dallmeier, M. Burger, and A. Zeller, \u0026ldquo;Mining trends of library usage,\u0026rdquo; in \u003cem\u003eProceedings of the joint international and annual ERCIM workshops on Principles of software evolution (IWPSE) and software evolution (Evol) workshops\u003c/em\u003e, 2009, pp. 57--62, doi: 10.1145/1595808.1595821.\u003c/li\u003e\n\u003cli\u003eM. Gegick, P. Rotella, and T. Xie, \u0026ldquo;Identifying security bug reports via text mining: An industrial case study,\u0026rdquo; in \u003cem\u003e7th IEEE Working Conference on Mining Software Repositories\u003c/em\u003e, May 2010, pp. 11\u0026ndash;20, doi: 10.1109/MSR.2010.5463340.\u003c/li\u003e\n\u003cli\u003eR. Scandariato, J. Walden, A. Hovsepyan, and W. Joosen, \u0026ldquo;Predicting Vulnerable Software Components via Text Mining,\u0026rdquo; \u003cem\u003eIEEE Trans. Softw. Eng.\u003c/em\u003e, vol. 40, no. 10, pp. 993\u0026ndash;1006, Oct. 2014, doi: 10.1109/TSE.2014.2340398.\u003c/li\u003e\n\u003cli\u003eY. Yang, X. Xia, D. Lo, T. Bi, J. Grundy, and X. Yang, \u0026ldquo;Predictive Models in Software Engineering: Challenges and Opportunities,\u0026rdquo; \u003cem\u003eACM Trans. Softw. Eng. Methodol.\u003c/em\u003e, vol. 31, no. 3, pp. 1\u0026ndash;72, Jul. 2022, doi: 10.1145/3503509.\u003c/li\u003e\n\u003cli\u003eD. Berrar, \u0026ldquo;Cross-Validation,\u0026rdquo; in \u003cem\u003eEncyclopedia of Bioinformatics and Computational Biology\u003c/em\u003e, Elsevier, 2019, pp. 542\u0026ndash;545.\u003c/li\u003e\n\u003cli\u003eZ. Zhang, \u0026ldquo;Introduction to machine learning: k-nearest neighbors,\u0026rdquo; \u003cem\u003eAnn. Transl. Med.\u003c/em\u003e, vol. 4, no. 11, pp. 218\u0026ndash;218, Jun. 2016, doi: 10.21037/atm.2016.03.37.\u003c/li\u003e\n\u003cli\u003eA. Alipour, A. Hindle, and E. Stroulia, \u0026ldquo;A contextual approach towards more accurate duplicate bug report detection,\u0026rdquo; in \u003cem\u003e2013 10th Working Conference on Mining Software Repositories (MSR)\u003c/em\u003e, May 2013, pp. 183\u0026ndash;192, doi: 10.1109/MSR.2013.6624026.\u003c/li\u003e\n\u003cli\u003eH. Peng, L. Bing, and M. Yutao, \u0026ldquo;Towards Cross-Project Defect Prediction with Imbalanced Feature Sets,\u0026rdquo; \u003cem\u003earXiv\u003c/em\u003e, p. 10, 2014, doi: https://doi.org/10.48550/arXiv.1411.4228.\u003c/li\u003e\n\u003cli\u003eT. J. Ostrand, E. J. Weyuker, and R. M. Bell, \u0026ldquo;Predicting the location and number of faults in large software systems,\u0026rdquo; in \u003cem\u003eIEEE Transactions on Software Engineering\u003c/em\u003e, Apr. 2005, vol. 31, no. 4, pp. 340\u0026ndash;355, doi: 10.1109/TSE.2005.49.\u003c/li\u003e\n\u003cli\u003eT. Ye, L. Zhang, L. Wang, and X. Li, \u0026ldquo;An Empirical Study on Detecting and Fixing Buffer Overflow Bugs,\u0026rdquo; in \u003cem\u003e2016 IEEE International Conference on Software Testing, Verification and Validation (ICST)\u003c/em\u003e, Apr. 2016, pp. 91\u0026ndash;101, doi: 10.1109/ICST.2016.21.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"maintenance, bug reports, machine learning, security, software vulnerabilities","lastPublishedDoi":"10.21203/rs.3.rs-2263306/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-2263306/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eSoftware developers must handle security bug reports (SBRs) before they are widely disclosed, and the system becomes vulnerable to hackers. Bug tracking systems may contain many securities-related reports which are unlabelled as SBRs. Therefore, finding unlabelled SBRs is a challenge to help security engineers identify these security issues fast and accurately. Although many methods have been proposed for classifying SBRs, challenging issues remain due to selecting an accurate and high-performance classification algorithm. This motivates us to tackle the challenges faced by the state-of-the-art SBRs classification methods by selecting a high-performance machine learning algorithm. Therefore, the main goal of this paper is to automate the process of determining which bug report can be labeled as SBR through the use of machine learning techniques. We first extracted 45,940 bug reports from publicly available datasets of five software repositories (e.g., the work of Peters et al. and Shu et al.). Second, we conducted a study on the classification of SBRs using machine learning, where we built a \u003cem\u003efasttext\u003c/em\u003e classifier. We then examined the accuracy of using \u003cem\u003efasttext\u003c/em\u003e in detecting SBRs. Our results show that \u003cem\u003efasttext\u003c/em\u003e can identify SBRs with an average F1 score of 0.81. Furthermore, we investigated the generalizability of identifying SBRs by applying cross-project validation, and our results show that the \u003cem\u003efasttext\u003c/em\u003e classifier achieves an average F1 value of 0.65. Data and results are available at \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://github.com/isultane/fasttext_classifications\u003c/span\u003e\u003cspan address=\"https://github.com/isultane/fasttext_classifications\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/p\u003e","manuscriptTitle":"Improve Classification of Security Bug Reports using fasttext. A Machine Learning Based Approach","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2022-11-15 18:51:44","doi":"10.21203/rs.3.rs-2263306/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"d789af67-9b2b-40c4-a443-cb88f6425583","owner":[],"postedDate":"November 15th, 2022","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2022-12-29T09:29:30+00:00","versionOfRecord":[],"versionCreatedAt":"2022-11-15 18:51:44","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-2263306","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-2263306","identity":"rs-2263306","version":["v1"]},"buildId":"_2-kVJe1T_tPrBINL-cwx","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}