Sentiment Analysis of Imbalanced Dataset through Data Augmentation and Generative Annotation using DistilBERT and Low-Rank Fine-Tuning

Sentiment Analysis of Imbalanced Dataset through Data Augmentation and Generative Annotation using DistilBERT and Low-Rank Fine-Tuning | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Sentiment Analysis of Imbalanced Dataset through Data Augmentation and Generative Annotation using DistilBERT and Low-Rank Fine-Tuning Hossein Nekkouei Nasrabadi, Mohammad Hossein Moattar This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5879286/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract This paper proposes a novel approach to sentiment analysis of imbalanced datasets, focusing on data augmentation and efficient fine-tuning. We address the challenge of limited minority class representation by leveraging GPT-4 to generate synthetic tweets via paraphrasing and back- translation (using Italian as an intermediary language). Furthermore, the main contribution is that we utilize GPT-4 to annotate tweets with positive reasons, derived by inverting the ten predefined negative categories within the dataset. The augmented dataset trains a DistilBERT model for sentence embeddings, and Low-Rank Adaptation (LoRA) enables efficient fine-tuning. A SoftMax layer provides classification into positive, neutral, and negative sentiments. Experiments on the Twitter US Airline Sentiment dataset demonstrate our approach’s efficacy, achieving 100% accuracy with minimal training time, highlighting the importance of data augmentation and efficient fine-tuning for robust sentiment analysis, particularly with imbalanced datasets. Sentiment Analysis Imbalance Classes Annotation DistilBERT Low-Rank Fine-Tuning Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 Figure 9 Figure 10 Figure 11 Figure 12 1 Introduction Sentiment analysis, the automated process of understanding opinions and emotions expressed in text, has evolved significantly over the years. Early approaches relied heavily on lexicon-based methods, matching words to pre-defined sentiment dictionaries. However, these methods struggled with nuances like sarcasm and context-dependent meanings. The advent of machine learning, particularly deep learning models like recurrent neural networks (RNNs) and transformers, revolutionized the field, enabling the capture of complex linguistic patterns and contextual information. More recently, the focus has shifted towards addressing challenges like handling imbalanced datasets, reducing computational costs of large models, and incorporating external knowledge. Techniques like data augmentation, transfer learning, and parameter-efficient fine-tuning methods like LoRA are becoming increasingly prevalent, driving further improvements in accuracy and efficiency. The field continues to evolve rapidly, exploring new frontiers such as multimodal sentiment analysis, incorporating visual and auditory information alongside text. Numerous approaches have been explored for text classification, a core NLP task. These range from traditional machine learning algorithms (like Decision Trees, SVMs, and Naive Bayes) combined with word embeddings (Word2Vec, GloVe) to deep learning methods. While simpler techniques like bag-of-words fail to capture crucial context, CNNs and RNNs (such as LSTMs) offer improvements by considering word order and sequences, although long-range dependencies remain a challenge. Effectively representing context is essential for accurate sentiment analysis, driving the development of more sophisticated models [ 1 , 2 , 3 ]. BERT(Bidirectional Encoder Representation from Transformers), a transformer model introduced in 2018, revolutionized natural language processing with its bidirectional approach. Unlike predecessors, BERT considers both preceding and following words simultaneously when generating contextualized word representations, leading to a deeper understanding of language.[ 4 ] However, BERT’s size can be prohibitive for resource-constrained environments. DistilBERT, a distilled version, addresses this by retaining most of BERT’s performance (97%) while being significantly smaller and faster (40% and 60% respectively), making it a practical choice for various NLP tasks, particularly on-device deployments [ 5 ]. A key challenge in sentiment analysis is the prevalence of imbalanced datasets, where the distribution of sentiment categories is biased. This disproportionate representation can bias models towards the majority class, resulting in inflated accuracy scores that mask poor performance on minority sentiments. Consequently, while overall classification rates may appear high, the model’s ability to identify and interpret less frequent, yet often crucial, sentiments are compromised. Mitigating this issue re- quires strategies like data augmentation or resampling techniques to ensure adequate representation of all sentiment categories during model training. To overcome this challenge, re-sampling techniques can improve classifier performance by under- sampling the majority class and over-sampling the minority class. However, the limited representation of minority class data may fail to reflect the true structure of the dataset, reducing prediction accuracy for the minority class with conventional classification methods [ 6 ]. One approach to under-sampling is to randomly removing samples from the majority class (RUS) until the minority class reaches a specified proportion of the majority. A method suitable when enough data is available. The over-sampling strategy generates synthetic example for the minority class without replacement. Techniques such as repetition, bootstrapping, or the synthetic minority over-sampling Technique (SMOTE), are used to generate new or unique synthetic samples minority class instances [ 7 ]. Generating synthetic data instead of relying on the collection of new datasets to aim at improving the representation of minority class data is a technique that enhances the performance of data-driven models. The significant progress in prompt-based large language models, such as GPT, Gemini and etc., has proven effective for this purpose and has gained substantial interest in recent years [ 8 ]. To improve performance and address class imbalance in datasets, some researchers have employed back- translation. This technique is particularly useful in scenarios with limited data, though its effectiveness may vary depending on the language pair. Back-translation enhances the performance of NLP models by creating synthetic parallel data, where sentences are translated from the target language back into the source language [ 9 ]. Text classification, a core task in natural language processing(NLP), has been extensively explored in previous studies. Researchers have investigated various approaches, including the use of word embedding models like Word2Vec, Doc2Vec, and GloVe, along with multiple classification algorithms such as Decision Tree, Random Forest, SVM, K-Nearest Neighbors, logistic regression, Gaussian Naive Bayes, and AdaBoost. Additionally, advancements have been made by employing convolutional neural network(CNN), recurrent neural network(RNN) with Long short-term memory(LSTM) units which capture long-term dependencies for improved prediction accuracy and better visualization. In our study, we use DistilBERT because of its ability to deliver robust performance while significantly reducing resource consumption The large-scale pre-training followed by task-specific adaptation is common in natural language processing (NLP), but fine-tuning all parameters of massive models like GPT3 175B is highly resource-intensive. Low-Rank Adaptation (LoRA) offers a more efficient solution by freezing pre-trained weights and introducing trainable low-rank matrices, reducing trainable parameters by up to 10,000 times and lowering GPU memory usage by threefold [ 10 ]. The rapidly growing and competitive airline industry has traditionally relied on customer feedback forms, which are often inefficient and time-consuming. Twitter data presents a faster alternative for gathering customer insights. In this analysis, tweets from six major U.S. airlines were used for multi- class sentiment analysis, where tweets were classified into positive, negative, and neutral sentiment categories. To prove with unbalancing dataset, we first employ GPT to balance the dataset by para- phrasing original samples from the minority class, utilizing the API key of GPT-4o. This strategy not only improves representation but also enriches the diversity of expressions within the dataset. In the following approach, back-translation was implemented using the API keys of GPT-4o to back translation samples by translating from English to Italian and then back to English. Italian was chosen after evaluating other languages, including Hochdeutsch and French, both of which demonstrated a close match with English during back-translation. However, we aimed to introduce some variations in adjectives to create new paraphrased sentences. This method allowed us to produce paraphrased sentences while augmenting the dataset, positively impacting the training process of our proposed model. 2 Related works The earlier method employed the Semantic Orientation CALculator (SO-CAL), a lexicon-based approach that utilizes dictionaries annotated with word polarity and strength, incorporating negation to classify text as positive or negative [ 11 ]. Le et al. introduces a new feature selection method using Information Gain, Bigram, and Object- Oriented extraction techniques, combining Naive Bayes and Support Vector Machine (SVM) [ 12 ]. While this method achieves high precision, it struggles with recall, addressed by identifying additional opinionated tweets using lexicon-based results, then training a classifier for sentiment polarity assignment using lexicon-based data instead of manual labeling [ 13 ]. Another approach extracts adjectives with meaningful sentiment from the dataset to form a feature vector through a two-step feature extraction process. Twitter-specific features are first extracted and added to the vector, then removed, followed by extraction on the remaining text. Machine learning algorithms like Naive Bayes, Maximum Entropy, and SVM are applied, alongside the Semantic Orientation-based WordNet to extract synonyms and assess content similarity[ 14 , 15 , 16 ]. Several studies have examined sentiment analysis (SA) on Twitter using machine learning (ML) and deep learning (DL) techniques. Despite the vast body of research on various aspects of sentiment analysis, tackling the issue with imbalanced datasets and models still needs more exploration [ 12 ]. One of the key challenges in sentiment analysis of customer comments on social media is the imbalance in class distribution. To tackle this issue, various techniques can be employed, such as sampling methods like Synthetic Minority Oversampling Technique (SMOTE) and Random Oversampling (ROS), Borderline-SMOTE (BSMOTE), Adaptive Synthetic Sampling (ADASYN), One-Sided Selection (OSS) and Condensed Nearest Neighbor (CNN) can be applied both before and after data splitting [ 17 , 18 , 19 ]. Under sampling achieves dataset balance by decreasing the size of the majority class. This method is employed when there is enough data available. It works by keeping all samples from the minority class and selecting a matching number of samples from the majority class.[ 20 ] Addressing class imbalance in sentiment analysis can also be achieved through generative models, such as using GPT-3 to create synthetic sentences. This method serves as another approach for oversampling in imbalanced datasets [ 21 ]. Back-translation helps the model learn general characteristics of specific topics [ 22 ]. DA methods, while successful in computer vision, have advanced slowly in natural language processing (NLP), with back-translation being a notable exception. The relationship between DA techniques and key factors like lexical diversity and semantic fidelity remains unclear [ 23 , 24 , 25 , 26 ]. Sawai et al. suggest that generative models can efficiently generate data by producing samples, such as those from GPT-2, which offer valuable insights for enhancing translation performance[ 27 ]. The feature extraction phase plays a crucial role in determining the overall effectiveness of sentiment analysis models. For instance, in one study, features were extracted and represented as a feature vector, after which a Bag of Words (BOW) model was constructed to analyze opinions [ 28 ]. Among the most commonly used techniques for word embedding is BERT, an unsupervised and bidirectional language representation model. Given the prominence of Twitter as a social platform, BERT has been employed for sentiment analysis on tweets, enabling the evaluation of emotions and opinions, which provides insights into users’ mental states [ 29 , 30 ]. In this context, robustly optimized BERT (RoBERTa) is utilized alongside Long Short-Term Memory (LSTM) to effectively capture long-distance contextual semantics, specifically designed for sentiment analysis [ 31 ]. The hybrid deep learning model combines predictions using averaging ensemble and majority voting, improving sentiment analysis performance. The ensemble model integrates three hybrid approaches, which combine RoBERTa, LSTM, BiLSTM, and GRU. RoBERTa projects the input text into an embedding space, while LSTM, BiLSTM, and GRU capture long-range dependencies within the data [ 32 ]. The strengths of the Transformer model (RoBERTa) and the Recurrent Neural Network (GRU) are utilized. Texts are projected into a meaningful embedding space using RoBERTa’s attention mechanism, while long-range dependencies are captured, and the vanishing gradients problem is resolved by the GRU.[ 33 ] Fine-tuning large language models (LLMs) using Low-Rank Adaptation (LoRA) enhances efficiency and effectiveness by reducing the number of parameters that need training, lowering computational costs, and allowing adaptation to smaller datasets with limited resources [ 34 , 35 ]. In summary, as shown in Table 1 , there is a clear evolution in sentiment analysis methods, emphasizing significant developments and benefits: Early Approaches [ 11 , 12 , 13 , 14 , 15 , 16 ]: These studies relied on lexicon-based methods, basic n-grams, and traditional classifiers (SVM, Naive Bayes). While simple, these approaches struggled with nuanced language, sarcasm, and context-dependent meanings. WordNet’s introduction [ 13 , 15 ] provided some semantic improvement. Engineering and Ensemble Methods [ 14 , 17 ]: Exploring object-oriented features, Maximum Entropy classifiers, and ensemble methods improved performance but still lacked deep contextual understanding. [ 17 ] begins to address imbalanced datasets using Random Oversampling (ROS).Engineering and Ensemble Methods Addressing Imbalance [ 17 , 18 , 20 ]: A crucial shift occurs as these studies tackle the imbalanced dataset problem. Ref. [ 17 ] uses ROS, while [ 18 ] explores SMOTE, BSMOTE and ADASYN. Ref. [ 20 ] employs RMU (likely Random Majority Under sampling) along with other techniques. Advanced Embeddings and Classifiers [ 20 , 21 , 28 ]: Word2Vec and GloVe embeddings provide richer word representations. More sophisticated classifiers like XGBoost are introduced. Synthetic data generation and back-translation start to emerge as data augmentation techniques. Deep Learning and Word Embeddings [ 22 , 29 , 31 , 32 , 34 , 35 , 30 ]: BERT and RoBERTa create a revolution the field with their contextualized embeddings and attention architecture. LSTM and BiLSTM networks are used to capture sequential information, leading to significant performance improvements. SoftMax is widely adopted for classification. Efficient Fine-tuning [ 30 , 34 , 35 ]: The computational cost of large pre-trained models becomes a challenge. LoRA is introduced for parameter-efficient fine-tuning. Llama-3 and techniques like QLORA are explored for resource-constrained environments. Further Data Augmentation [ 31 , 32 , 34 ]: New data augmentation techniques are investigated, including generating new samples based on GloVe embeddings, random punctuation insertion, and synonym replacement. Papers addressing imbalanced datasets The table explicitly indicates that references [ 17 , 18 , 20 , 21 ], and [ 22 ] directly address imbalanced datasets. Ref [ 30 ] also mentions the use of SMOTE and RUS for this purpose. It’s important to note that while other papers may not explicitly mention imbalance, they might still employ strategies to mitigate its effects. After examining various techniques for sentiment analysis, the current analysis identifies several key issues addressed during implementation: Utilization of a benchmark dataset, specifically the US Airline Sentiment Analysis dataset. Resolving of the imbalanced class issue through the generation of synthetic data. Augmentation of data by incorporating back-translation. Extraction of features using contextualized word embedding. Application of Low-Rank Adaptation (LoRA) for fine-tuning the dataset. Table 1 Summary of related work Ref Dataset Resampling Tech. Embedding Tech. Class. Tech. Results [ 11 ] Amazon’s Mechanical Turk service - Lexicon-based SVM Accuracy > 79% for SO-CAL [ 12 ] Stanford Twitter Sentiment - bigram unigram Object-oriented SVM Naive Bayes Accuracy of 79.58% for Bigram and SVM [ 13 ] Obama, Harry Potter, Tangled iPad, Packers Twitter - unigram WordNet SVM Naive Bayes Accuracy of 89.9% for WordNet + NB [ 14 ] Twitter API - Unigram SVM Naive Bayes Maximum Entropy Accuracy of 90% for ensemble classifier [ 15 ] Twitter API - Unigram WordNet SVM Naive Bayes Accuracy of 89.9% for WordNet + SVM [ 16 ] Twitter API - Lexicon-based Maximum Entropy SVM Naive Bayes Accuracy of 79% for W-WSD + NB [ 17 ] Twitter API ROS SMOTE Lexicon-based NB, SVM K-NN Accuracy of 93% for VADER + SMOTE + SVM Oversampling-Splitting increase performance [ 18 ] Arabic Tweet Covid-19 RO, SMOTE, BSMOTE ADASYN, RU, OSS CountVectorizer Ridge Classifier, LR, SGD, SVM DT, KNN, G-NB F-1 score of 99% RF with SMOTE,BSMOTE or ADASYN [ 20 ] Twitter US Airline Sentiment RMU Word2Vec XGBoost (XGB) Accuracy of 86.5% [ 21 ] Coursera online course review Synthetic Data Generation GloVe SVM, DT, MultinmialNB, AdaBoost Best Accuracy of 75.12% for MultinmialNB [ 22 ] Recovery Back-Translation BERT sigmoid Best F1-Score of 89.1% for real News [ 28 ] Twitter US Airline Sentiment - Bag-of-Words (BOW) SVM, RL, RF, XGB, NB, DT Best Accuracy of 83.13% for SVM [ 29 ] Tweets: the world about COVID-19 Tweets: the India about COVID-19 - Text Blob BERT ML algorithm Accuracy of 94% [ 30 ] Twitter US Airline Sentiment Sentiment140 SMOTE RUS BERT, LSTM GRU, BiLSTM SVM Accuracy of 93.9% for US Airline [ 31 ] IMDb, Sentiment140 Twitter US Airline Sentiment generate new samples Based of GloVe word embedding RoBERTa LSTM SoftMax F-1 Score of 91% for Twitter US Airline [ 32 ] IMDb, Sentiment140 Twitter US Airline Sentiment generate new samples Based on GloVe word embedding RoBERTa, LSTM BiLSTM, GRU SoftMax Best F-1 Score of 91.77% for Twitter US Airline with Ensemble model (majority voting) [ 34 ] SemEval 2017 Task 4 SemEval 2018 Task 1 Randomly inserts punctuation marks Synonym Replacement RoBERTa using LoRA Llama-3 SoftMax MSE of 0.0137 [ 35 ] ReLi (Resenha de Livros) TV - BERT using LoRA OpenCabrita using QLoRA SoftMax Highest F1 Score: 0.846 for BERTimbauBASE Highest F1 Score: 0.615 for BERTimbauBASE 3 The Proposed Model This work investigates a stacked ensemble model for sentiment analysis of imbalanced English text data, focusing on the embedding level. We argue that improved performance comes not from complex models[ 30 ], but from a focus on high-quality data and effective augmentation techniques. As shown in Fig. 1 , our proposed approach uses DistilBERT, a streamlined version of BERT, to generate sentence embeddings from the ”Twitter US Airline Sentiment” dataset. These embeddings are fine-tuned using Low-Rank Adaptation (LoRA), a parameter-efficient method. A SoftMax layer provides the final classification. Subsequent sections will delve into the specifics of the DistilBERT embedding model and the crucial role of data augmentation. The proposed model takes an imbalanced English dataset as input and balances it by Generating Synthetic Tweets(GST) using GPT-based paraphrasing. These new tweets are then annotated with one of ten categorical labels representing their primary meaning, using another large language model like Gemini. BERT extracts feature from the augmented dataset, and the resulting embeddings are fine-tuned with LoRA. Finally, the text is classified into three sentiment categories: positive, neutral, and negative. 3-1- Data Augmentation Data augmentation in natural language processing (NLP) is a crucial method for increasing the variety of training data, which enhances model performance and effectiveness robustness. Techniques such as synonym replacement, back-translation, and contextual word embeddings can generate new variations of existing sentences without altering their meaning. This process mitigates over-fitting by providing more varied examples for the model to learn from and helps to adapt models to different linguistic styles and contexts. As a result, data augmentation is crucial in advancing the capabilities of NLP models, enabling them to handle real-world language complexities more effectively. 3-1-1- Oversampling Dataset Several dataset-balancing techniques are discussed in Section 2 . Here, as shown in Fig. 2 , we employ GPT-4, a large language model, generates synthetic tweets by paraphrasing existing tweets (as seen in Table 2 ). This paraphrasing introduces new vocabulary, enhancing linguistic diversity within the minority class and mitigating potential data bias. Table 2 Original tweet vs paraphrased tweet with GPT4-o Original Tweet Airline Sentiment paraphrased Tweet AI Sentiment @JetBlue I understand but wish you would have announced the delay 2 hours earlier vs sitting for 2 hrs at MCO Negative @JetBlue I understand, but I would have liked you to announce the delay 2 hours earlier instead of waiting 2 hours at MCO Negative 3-1-2- Back translation Back-translation is an effective data augmentation strategy that boosts the resilience of machine learning models, particularly in the realm of natural language processing. This technique entails taking a piece of text from its original language, translating it into a target language, and then translating it back to the source language. In our proposed model, we employ GPT-4o to execute back-translation as a method of data augmentation by first converting English sentences to Italian and subsequently translating them back into English as shown in Fig. 3 The process starts with an English text that is translated into Italian using a reliable translation system. The Italian version is then retranslated into English, yielding a revised iteration of the original sentence. This two-step translation process not only adds variety to the phrasing and structure of the sentences but also enriches the training dataset, enabling the model to learn from a wider array of linguistic expressions. As a result, this technique enhances the model’s ability to generalize and perform more effectively across diverse inputs, showcasing the benefits of back-translation in enhancing language comprehension. 3-1-3- Noise Injection We explored noise injection techniques like random character insertion and punctuation changes to create textual variations. However, unlike image data augmentation, these methods introduced unseen or unknown tokens during tokenization, degrading model performance rather than enhancing it. 3-1-4- Annotating Tweets In three-class sentiment analysis, neutral sentiments are the most challenging to classify and often lead to errors. As shown in Fig. 5 , to mitigate this, we annotate negative instances using ten predefined categories from the dataset, such as [Customer Service Issue, Late Flight, Can’t Tell, Cancelled Flight, Lost Luggage, Bad Flight, Flight Booking Problems, Flight Attendant Complaints, Long Lines, or Damaged Luggage]. For positive instances, we use Gemini to generate corresponding opposite annotations (e.g., ”Excellent Customer Service”) as shown in Fig. 4 . Neutral instances remain unannotated. after that we add the annotations in the first/end of tweets. 3-2- Word Embedding Word embedding is an essential technique in natural language processing (NLP) that converts words into numerical vector forms, which effectively capture their semantic relationships and contextual meanings. These vector representations allow computers to understand and process text more effectively. A significant advancement in word embedding is the introduction of transformer-based models like BERT and DistilBERT to analyze sentiment in social media comments, especially Twitter’s comments. 3-2-1- BERT BERT, a powerful NLP framework based on the Transformer architecture, uses surrounding text to disambiguate language and understand context (Fig. 6 ). Pre-trained on a large text corpus and fine-tunable for tasks like sentiment analysis, BERT leverages an attention mechanism to weigh the importance of different words dynamically. [ 4 ] 3-2-2- DistilBERT DistilBERT, a distilled version of BERT, offers comparable performance while being significantly smaller and faster (40% smaller and 60% faster). Trained through knowledge distillation, it retains much of BERT’s accuracy (97%) while requiring fewer resources, making it ideal for resource- constrained environments and speed-critical applications [ 5 ]. 3-3- LoRA: Low-Rank Adaptation LoRA is a parameter-efficient fine-tuning technique that significantly reduces the computational cost of adapting large language models to specific tasks. Instead of fine-tuning all the model’s weights, LoRA injects trainable rank decomposition matrices into specific layers, effectively adding a small number of parameters while keeping the pre-trained weights frozen as shown in Fig. 7 . This approach reduces the memory footprint and training time, allowing for efficient adaptation of large models on resource- constrained hardware while maintaining comparable performance to full fine-tuning. LoRA simplifies the process of deploying customized models and enables experimentation with various downstream tasks without the overhead of managing numerous large checkpoint files[ 10 ]. 3-4- SoftMax The SoftMax function is a crucial component in multi-class classification, transforming a vector of raw model outputs (logits) into a probability distribution over the predicted classes. It exponentiates each logit and then normalizes the resulting values by dividing by their sum. This ensures that the output values are between 0 and 1 and sum to 1, representing the predicted probability for each class. The class with the highest probability is then selected as the final prediction. SoftMax is widely used in neural networks due to its ability to produce well-calibrated probability estimates, facilitating confident and interpretable predictions in various classification tasks. 4 Results and experiments analysis This section details our experimental setup, including the dataset, evaluation metrics, and results. We demonstrate that our approach yields a lightweight, efficient model with the lowest training and runtime costs. This efficiency is not only beneficial for performance but also contributes to reducing greenhouse gas emissions associated with computationally intensive models. 4-1- Experimental setup Our implementation used Python with the TensorFlow library. The model was configured with a LoRA rank of 16, trained for 3 epochs with a batch size of 32. A learning rate of 5e-5 was employed for Adam optimization. We froze all layers of the pre-trained DistilBERT model and trained only 22,784 parameters for LoRA and 12 parameters for the final dense layer (outputting 3 classes). This resulted in a highly efficient model with low computational cost and fast training times, while still achieving high accuracy. 4-2- Dataset To evaluate our proposed model, we used the real-world Twitter US Airline Sentiment dataset, collected by CrowdFlower in 2017[ 37 ]. This dataset comprises customer feedback on six US airlines: American, US Airways, United, Southwest, Virgin America, and Delta. The data includes 2,363 positive, 9,178 negative, and 3,099 neutral sentiments. Some samples of the annotated dataset with the proposed annotation approach are depicted in Table 3 . 4-3- Evaluation metrics Effective machine learning model evaluation hinges on choosing the right metrics. Metrics like accuracy, precision, recall, and F1-score offer different perspectives on performance, with the best choice depending on the task and the cost of different error types. For multi-class tasks, macro- and weighted- average F1-scores provide a comprehensive view across classes. Ultimately, a thorough evaluation requires selecting metrics aligned with the specific goals of the task. Table 3: Sample Tweets in the annotated dataset Accuracy , the proportion of correctly classified instances, is a straightforward evaluation metric. However, it can be deceptive with imbalanced datasets, where a model might achieve high performance by predicting the dominant class. Therefore, relying solely on accuracy for imbalanced data can be misleading, and supplementary metrics are crucial. The F1-score , balancing precision and recall, is a crucial metric for evaluating classification, especially with imbalanced datasets. Ranging from 0 to 1, it effectively assesses a model’s ability to correctly identify positive instances while minimizing both false positives and false negatives. Precision The proportion of correctly predicted positive instances out of all instances predicted as positive. Recall The proportion of correctly predicted positive instances out of all actual positive instances. Marco-average Macro-averaging calculates overall performance in multi-class classification by averaging per-class metrics (e.g., precision, recall, F1-score), treating all classes equally regardless of their size. While useful for understanding performance across classes, it can be skewed by imbalanced datasets, where smaller classes might disproportionately influence the overall score. Therefore, consider class distribution when choosing an averaging method. ROC-AUC measures a classification model’s ability to distinguish between classes. It visualizes the trade-off between true positives and false positives, with a higher area under the curve (AUC) indicating better performance, especially for imbalanced datasets. 4-4- Experimental findings This section evaluates the performance of our proposed BERT-LoRA model (detailed in Section 3 ) against several baseline models using the ”Twitter US Airline Sentiment” dataset (14,640 English reviews classified as positive, negative, or neutral). As shown in Table 4 , the baselines include models using traditional methods like Bag-of-Words and Word2Vec, as well as more advanced techniques such as RoBERTa-LSTM and BERT-LSTM. These models employ varying resampling strategies and leverage BERT for word embeddings. Our proposed model, trained on an augmented dataset exceeding 54,000 samples (incorporating original tweets, paraphrased versions, and back-translated tweets), achieved a remarkable 100% accuracy in just three epochs. This result, highlighted in Table 4 and Fig. 8 , demonstrates exceptional performance and efficiency compared to the baseline models, which typically require more extensive training and often achieve lower accuracy. Table 5 and Fig. 9 also provide insights into the effectiveness of the resampling techniques and other data augmentation methods employed. Table 4 Comparison results of the proposed model and based models Model Resampling Method Accuracy BOW [ 28 ] - 86% Word2Vec [ 20 ] RMU 86% RoBERTa-LSTM [ 32 ] - 91% BERT-LSTM [ 30 ] SMOTE 93% BERT-LoRA (Propose Model) Imbalanced Dataset 84% BERT-LoRA (Proposed Model) GST 1 & Back-Translation(BT) 92% BERT-LoRA (Proposed Model) GST & BT & Annotated Tweets 100% Table 5 Performance of the proposed model on various augmented datasets, evaluated using 10-fold cross-validation Re-sampling Technique Accuracy F-1 Score Macro-Average AUC Support Imbalanced Dataset 84% 90% 78% 0.95 14640 GST & Back-Translation 92% 94% 92% 0.99 54231 GST & Back-Translation & Annotated Tweet 100% 100% 100% 1.00 54231 1 GST: Generate Synthetic Tweets While common practice suggests that text preprocessing steps like expanding contractions, lower- casing, and removing elements like usernames, emojis, stop words, and URLs improve performance, our study found that keeping these elements is actually beneficial in social network comments. In our experiments, removing them reduced accuracy by 5%. These elements, while often considered noise, contribute to the overall meaning and should be preserved. Generating synthetic samples demonstrably improved classifier performance on the imbalanced dataset. Data augmentation, BERT embeddings, and LoRA fine-tuning resulted in higher accuracy and lower training costs, showcasing the proposed architecture’s effectiveness for English sentiment classification. Annotating tweets with 10 predefined categories further boosted accuracy to 100%. A thorough analysis, including ROC-AUC and the impact of reduced training epochs, is presented in the following figures. Figures 10 – 12 illustrate ROC scores across different dataset configurations (imbalanced, balanced, back-translation augmented, and annotated), emphasizing the superior performance achieved with the annotated data. Our proposed model leverages a data augmentation strategy, generating synthetic samples through paraphrasing and back-translation, to enhance accuracy and reduce both training costs and runtime. This efficiency is achieved through low-rank adaptation fine-tuning. However, it currently relies on external models (like GPT) for annotation and doesn’t address intra-class imbalances. Future work could explore weighted augmentation based on intra-class clustering, and training for independent sentiment classification without external dependencies. 5 Conclusions and future works Our study demonstrates that data augmentation through paraphrasing and back-translation, combined with annotation using GPT-4, DistilBERT embeddings, and LoRA fine-tuning, significantly improves sentiment analysis performance on imbalanced datasets. Achieving 100% accuracy on the Twitter US Airline Sentiment dataset in just three epochs underscores the effectiveness and efficiency of this approach. However, the current model relies on external language models for annotation and does not explicitly address imbalances within sentiment categories. Future work will explore weighted augmentation based on intra-class clustering, and eliminating the need for external language models by training the model to independently classify sentiment categories. These improvements aim to create a more robust and self-contained sentiment analysis system. Declarations Competing interests The authors declare that they have no competing interests or personal relationships that could have appeared to influence the work reported in this paper. Funding The authors declare that no funding was received for this research. Data availability statement The algorithm's code, along with image data, will be made available upon request. Use of experimental animals, and human participants No animals and human participants were involved in this study in any kind. Authors contribution statement HNN performed investigations, software development, visualization, and writing original draft. MHM performed conceptualization, supervision, validation, and writing- reviewing and editing. Ethical and informed consent for data used No ethical concern exists. Acknowledgements Not applicable. Declaration of Generative AI and AI-assisted technologies in the writing process No AI or AI-assisted tool were used in the writing process of the article. References R Monika, S Deivalakshmi, and B Janet. Sentiment analysis of us airlines tweets using lstm/rnn. In 2019 IEEE 9th International Conference on Advanced Computing(IACC) , pages 92–95. IEEE, 2019. Ankita Rane and Anand Kumar. Sentiment classification system of twitter data for us airline service analysis. In 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC) , volume 1, pages 769–773. IEEE, 2018. Mikhail V Koroteev. Bert: a review of applications in natural language processing and under- standing. arXiv preprint arXiv:2103.11943 , 2021. Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018. V Sanh. Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 , 2019. Chuanxia Jian, Jian Gao, and Yinhui Ao. A new sampling method for classifying imbalanced data based on support vector machine ensemble. Neurocomputing , 193:115–122, 2016. Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research , 16:321–357, 2002. Seokjin Oh, Su Ah Lee, and Woohwan Jung. Data augmentation for neural machine translation using generative language model. arXiv preprint arXiv:2307.16833 , 2023. Parul Gupta and Maha Mahmood. Data augmentation for neural language processing. Interna- tional Journal of Data Science and Advanced Analytics , 6(6):352–359, 2024. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021. Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede. Lexicon-based methods for sentiment analysis. Computational linguistics , 37(2):267–307, 2011. Bac Le and Huy Nguyen. Twitter sentiment analysis using machine learning techniques. In Advanced Computational Methods for Knowledge Engineering: Proceedings of 3rd International Conference on Computer Science, Applied Mathematics and Applications-ICCSAMA 2015 , pages 279–289. Springer, 2015. Lei Zhang, Riddhiman Ghosh, Mohamed Dekhil, Meichun Hsu, and Bing Liu. Combining lexicon-based and learning-based methods for twitter sentiment analysis. HP Laboratories, Technical Report HPL-2011 , 89:1–8, 2011. MS Neethu and R Rajasree. Sentiment analysis in twitter using machine learning techniques. In 2013 fourth international conference on computing, communications and networking technologies (ICCCNT) , pages 1–5. IEEE, 2013. Geetika Gautam and Divakar Yadav. Sentiment analysis of twitter data using machine learning approaches and semantic analysis. In 2014 Seventh international conference on contemporary computing (IC3) , pages 437–442. IEEE, 2014. Ali Hasan, Sana Moin, Ahmad Karim, and Shahaboddin Shamshirband. Machine learning-based sentiment analysis for twitter accounts. Mathematical and computational applications , 23(1):11, 2018. Dinar Ajeng Kristiyanti, Samuel Ady Sanjaya, Vinsencius Christio Tjokro, and Jason Suhali. Dealing imbalance dataset problem in sentiment analysis of recession in indonesia. Int J Artif Intell ISSN , 2252(8938):2061, Maisa J Al-Khazaleh, Marwah Alian, and Manar A Jaradat. Sentiment analysis of imbalanced arabic data using sampling techniques and classification algorithms. Bulletin of Electrical Engineering and Informatics , 13(1):607–618, 2024. Mohammad Mohaiminul Islam and Naznin Sultana. Comparative study on machine learning algorithms for sentiment classification. International Journal of Computer Applications , 182(21):1–7, 2018. Saleh Naif Almuayqil, Mamoona Humayun, NZ Jhanjhi, Maram Fahaad Almufareh, and Navid Ali Khan. Enhancing sentiment analysis via random majority under-sampling with reduced time complexity for classifying tweet reviews. Electronics , 11(21):3624, 2022. Cici Suhaeni and Hwan-Seung Yong. Mitigating class imbalance in sentiment analysis through gpt-3-generated synthetic sentences. Applied Sciences , 13(17):9766, 2023. Jiaheng Hua, Xiaodong Cui, Xianghua Li, Keke Tang, and Peican Zhu. Multimodal fake news detection through data augmentation-based contrastive learning. Applied Soft Computing , 136:110125, 2023. Lucas Francisco Amaral Orosco Pellicer, Taynan Maier Ferreira, and Anna Helena Reali Costa. Data augmentation techniques in natural language processing. Applied Soft Computing , 132:109803, 2023. Hugo Queiroz Abonizio, Emerson Cabrera Paraiso, and Sylvio Barbon. Toward text data augmentation for sentiment analysis. IEEE Transactions on Artificial Intelligence , 3(5):657–668, 2021. Yu Li, Xiao Li, Yating Yang, and Rui Dong. A diverse data augmentation strategy for low-resource neural machine translation. Information , 11(5):255, 2020. Hongzheng Li, Jiu Sha, and Can Shi. Revisiting back-translation for low-resource machine translation between chinese and vietnamese. IEEE Access , 8:119931–119939, 2020. Ranto Sawai, Incheon Paik, and Ayato Kuwana. Sentence augmentation for language translation using gpt-2. Electronics , 10(24):3082, 2021. Abdelrahman I Saad. Opinion mining on us airline twitter data using machine learning techniques. In 2020 16th international computer engineering conference (ICENCO) , pages 59–63. IEEE, 2020. Mrityunjay Singh, Amit Kumar Jakhar, and Shivam Pandey. Sentiment analysis on the impact of coronavirus in social life using the bert model. Social Network Analysis and Mining , 11(1):33, 2021. Nassera Habbat, Hicham Nouri, Houda Anoun, and Larbi Hassouni. Sentiment analysis of imbalanced datasets using bert and ensemble stacking for deep learning. Engineering Applications of Artificial Intelligence , 126:106999, 2023. Kian Long Tan, Chin Poo Lee, Kalaiarasi Sonai Muthu Anbananthen, and Kian Ming Lim. Roberta-lstm: a hybrid model for sentiment analysis with transformer and recurrent neural net- work. IEEE Access , 10:21517–21525, 2022. Kian Long Tan, Chin Poo Lee, Kian Ming Lim, and Kalaiarasi Sonai Muthu Anbananthen. Sentiment analysis with ensemble hybrid deep learning model. IEEE Access , 10:103694–103704, 2022. Kian Long Tan, Chin Poo Lee, and Kian Ming Lim. Roberta-gru: A hybrid deep learning model for enhanced sentiment analysis. Applied Sciences , 13(6):3915, 2023. Diefan Lin, Yi Wen, Weishi Wang, and Yan Su. Enhanced sentiment intensity regression through lora fine-tuning on llama 3. IEEE Access , 2024. Jos´e Carlos Ferreira Neto, Denilson Alves Pereira, Bruno Henrique Groenner Barbosa, and Dan- ton Diego Ferreira. Approaches based on language models for aspect extraction for sentiment analysis in the Portuguese language. Neural Computing and Applications , pages 1–11, 2024. Sayanta Paul and Sriparna Saha. Cyberbert: Bert for cyberbullying identification: Bert for cyberbullying identification. Multimedia Systems , 28(6):1897–1904, 2022. CrowdFlower. Twitter US Airline Sentiment. https://www.kaggle.com/datasets/ crowdflower/twitter-airline-sentiment, 2015. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5879286","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":406679495,"identity":"122691ca-4636-4f7e-8e75-c0da0483861c","order_by":0,"name":"Hossein Nekkouei Nasrabadi","email":"","orcid":"","institution":"Islamic Azad University","correspondingAuthor":false,"prefix":"","firstName":"Hossein","middleName":"Nekkouei","lastName":"Nasrabadi","suffix":""},{"id":406679497,"identity":"c8b18c54-d4ef-4ce9-96fc-2c0806795371","order_by":1,"name":"Mohammad Hossein Moattar","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA3UlEQVRIie3RIQvCQBTA8WcxDaxL2ycQPAaH4Jd5w7AyBbEYFg4E17T6JQST+cbBVm5YzyKCwerKMIkn2JSbNsP924P3gwcPwGb74zwAB1x8Tbxhu8X0RvA7CdmTfHVQN51fztfkGG1WZaZOiYBOyltiZiBU5oTxfDraqfGwj7kAVyJk0kQUatJGTRzqYlsAKICMGUlUMX7HiO6lJncBfjOJCcsWiJTH1A0XAnqNRMrJulwi2ak46IfLyCEyZGZSpNvrrEZfH0YOt3rgeYUQlYm85Twf9Quw2Ww224ceo4RZb8YGB9kAAAAASUVORK5CYII=","orcid":"","institution":"Islamic Azad University","correspondingAuthor":true,"prefix":"","firstName":"Mohammad","middleName":"Hossein","lastName":"Moattar","suffix":""}],"badges":[],"createdAt":"2025-01-22 09:08:13","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5879286/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5879286/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":74913536,"identity":"924105ec-dcdd-4b31-b598-2bd2b32bf58b","added_by":"auto","created_at":"2025-01-28 09:31:00","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":56748,"visible":true,"origin":"","legend":"\u003cp\u003eThe proposed framework\u003c/p\u003e","description":"","filename":"1.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5879286/v1/1577f78ced2457572ec9e79a.jpg"},{"id":74912108,"identity":"fc24699b-8e69-410c-ab28-865406079131","added_by":"auto","created_at":"2025-01-28 09:23:00","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":33284,"visible":true,"origin":"","legend":"\u003cp\u003eGenerating new synthetic tweets focusing on the minority positive and neutral classes, utilizing GPT-4o for oversampling to ensure a balanced dataset.\u003c/p\u003e","description":"","filename":"2.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5879286/v1/6a0c3c7ae49c752342bc994a.jpg"},{"id":74913538,"identity":"be81837d-7fb9-45de-b80a-c8b7752202e9","added_by":"auto","created_at":"2025-01-28 09:31:00","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":25752,"visible":true,"origin":"","legend":"\u003cp\u003eBack-translation structure\u003c/p\u003e","description":"","filename":"3.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5879286/v1/7e596dc5ae9add4b52a114d4.jpg"},{"id":74913546,"identity":"31258b8c-17f1-488f-8dab-95fac369f014","added_by":"auto","created_at":"2025-01-28 09:31:00","extension":"jpg","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":18253,"visible":true,"origin":"","legend":"\u003cp\u003eTweets annotation with positive reason using the GPT model\u003c/p\u003e","description":"","filename":"4.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5879286/v1/f85fde8c7c11ed2bf48fb519.jpg"},{"id":74913540,"identity":"b008c351-52f3-4b6f-9bcb-e1dd4326788c","added_by":"auto","created_at":"2025-01-28 09:31:00","extension":"jpg","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":25623,"visible":true,"origin":"","legend":"\u003cp\u003eModified data by combining Tweets from the negative class with negative reasons.\u003c/p\u003e","description":"","filename":"5.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5879286/v1/3cbd3b3966ca760c6fd811a8.jpg"},{"id":74912129,"identity":"2e9f72d4-d6d3-428e-979e-b6817b705ae7","added_by":"auto","created_at":"2025-01-28 09:23:01","extension":"jpg","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":54283,"visible":true,"origin":"","legend":"\u003cp\u003eBERT Architecture [36]\u003c/p\u003e","description":"","filename":"6.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5879286/v1/f721b2e666c0af0ec1d9f4c5.jpg"},{"id":74913549,"identity":"849482c5-7721-40ea-bcab-23387d5bb561","added_by":"auto","created_at":"2025-01-28 09:31:01","extension":"jpg","order_by":7,"title":"Figure 7","display":"","copyAsset":false,"role":"figure","size":39428,"visible":true,"origin":"","legend":"\u003cp\u003eLoRA: Low-Rank Adaptation\u003c/p\u003e","description":"","filename":"7.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5879286/v1/2e4c380825b45a348dd8f29e.jpg"},{"id":74912136,"identity":"18f9daad-3260-4ff4-81de-b578902c8036","added_by":"auto","created_at":"2025-01-28 09:23:01","extension":"jpg","order_by":8,"title":"Figure 8","display":"","copyAsset":false,"role":"figure","size":44021,"visible":true,"origin":"","legend":"\u003cp\u003eComparison results of the proposed model and based models for Twitter US Airline Sentiment\u003c/p\u003e","description":"","filename":"8.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5879286/v1/d67f5086996c4cd1aa42fd17.jpg"},{"id":74912090,"identity":"a53380ba-d197-42da-a881-7a4b2728067a","added_by":"auto","created_at":"2025-01-28 09:23:00","extension":"jpg","order_by":9,"title":"Figure 9","display":"","copyAsset":false,"role":"figure","size":43155,"visible":true,"origin":"","legend":"\u003cp\u003ePerformance of the proposed model on various augmented datasets\u003c/p\u003e","description":"","filename":"9.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5879286/v1/f5fbb09cb0756a96b3301d94.jpg"},{"id":74912096,"identity":"93aa4a59-6dd5-4266-9840-ffc117e971c6","added_by":"auto","created_at":"2025-01-28 09:23:00","extension":"jpg","order_by":10,"title":"Figure 10","display":"","copyAsset":false,"role":"figure","size":46389,"visible":true,"origin":"","legend":"\u003cp\u003eThe ROC curve on the imbalanced dataset\u003c/p\u003e","description":"","filename":"10.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5879286/v1/8568e87a20f1e947010f09df.jpg"},{"id":74912095,"identity":"7364e17e-d863-4c46-a249-28909c68007a","added_by":"auto","created_at":"2025-01-28 09:23:00","extension":"jpg","order_by":11,"title":"Figure 11","display":"","copyAsset":false,"role":"figure","size":47391,"visible":true,"origin":"","legend":"\u003cp\u003eThe ROC curve on the augmented dataset\u003c/p\u003e","description":"","filename":"11.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5879286/v1/174f8feea37cb53b26ff6902.jpg"},{"id":74912139,"identity":"ede7cd08-1f16-43e3-b36d-7e849ae3f23a","added_by":"auto","created_at":"2025-01-28 09:23:01","extension":"jpg","order_by":12,"title":"Figure 12","display":"","copyAsset":false,"role":"figure","size":37740,"visible":true,"origin":"","legend":"\u003cp\u003eThe ROC curve on the annotated tweet\u003c/p\u003e","description":"","filename":"12.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5879286/v1/d13f8a6b40c4e94888ead6d5.jpg"},{"id":75106046,"identity":"09e4ffd8-2dd2-497d-95e2-69f66e58e0b9","added_by":"auto","created_at":"2025-01-30 14:31:55","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1580801,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5879286/v1/0dec4a84-6095-42ea-86b0-582b814b67eb.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Sentiment Analysis of Imbalanced Dataset through Data Augmentation and Generative Annotation using DistilBERT and Low-Rank Fine-Tuning","fulltext":[{"header":"1 Introduction","content":"\u003cp\u003eSentiment analysis, the automated process of understanding opinions and emotions expressed in text, has evolved significantly over the years. Early approaches relied heavily on lexicon-based methods, matching words to pre-defined sentiment dictionaries. However, these methods struggled with nuances like sarcasm and context-dependent meanings. The advent of machine learning, particularly deep learning models like recurrent neural networks (RNNs) and transformers, revolutionized the field, enabling the capture of complex linguistic patterns and contextual information. More recently, the focus has shifted towards addressing challenges like handling imbalanced datasets, reducing computational costs of large models, and incorporating external knowledge. Techniques like data augmentation, transfer learning, and parameter-efficient fine-tuning methods like LoRA are becoming increasingly prevalent, driving further improvements in accuracy and efficiency. The field continues to evolve rapidly, exploring new frontiers such as multimodal sentiment analysis, incorporating visual and auditory information alongside text.\u003c/p\u003e \u003cp\u003eNumerous approaches have been explored for text classification, a core NLP task. These range from traditional machine learning algorithms (like Decision Trees, SVMs, and Naive Bayes) combined with word embeddings (Word2Vec, GloVe) to deep learning methods. While simpler techniques like bag-of-words fail to capture crucial context, CNNs and RNNs (such as LSTMs) offer improvements by considering word order and sequences, although long-range dependencies remain a challenge. Effectively representing context is essential for accurate sentiment analysis, driving the development of more sophisticated models [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eBERT(Bidirectional Encoder Representation from Transformers), a transformer model introduced in 2018, revolutionized natural language processing with its bidirectional approach. Unlike predecessors, BERT considers both preceding and following words simultaneously when generating contextualized word representations, leading to a deeper understanding of language.[\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e] However, BERT\u0026rsquo;s size can be prohibitive for resource-constrained environments. DistilBERT, a distilled version, addresses this by retaining most of BERT\u0026rsquo;s performance (97%) while being significantly smaller and faster (40% and 60% respectively), making it a practical choice for various NLP tasks, particularly on-device deployments [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eA key challenge in sentiment analysis is the prevalence of imbalanced datasets, where the distribution of sentiment categories is biased. This disproportionate representation can bias models towards the majority class, resulting in inflated accuracy scores that mask poor performance on minority sentiments. Consequently, while overall classification rates may appear high, the model\u0026rsquo;s ability to identify and interpret less frequent, yet often crucial, sentiments are compromised. Mitigating this issue re- quires strategies like data augmentation or resampling techniques to ensure adequate representation of all sentiment categories during model training.\u003c/p\u003e \u003cp\u003eTo overcome this challenge, re-sampling techniques can improve classifier performance by under- sampling the majority class and over-sampling the minority class. However, the limited representation of minority class data may fail to reflect the true structure of the dataset, reducing prediction accuracy for the minority class with conventional classification methods [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eOne approach to under-sampling is to randomly removing samples from the majority class (RUS) until the minority class reaches a specified proportion of the majority. A method suitable when enough data is available. The over-sampling strategy generates synthetic example for the minority class without replacement. Techniques such as repetition, bootstrapping, or the synthetic minority over-sampling Technique (SMOTE), are used to generate new or unique synthetic samples minority class instances [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eGenerating synthetic data instead of relying on the collection of new datasets to aim at improving the representation of minority class data is a technique that enhances the performance of data-driven models. The significant progress in prompt-based large language models, such as GPT, Gemini and etc., has proven effective for this purpose and has gained substantial interest in recent years [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. To improve performance and address class imbalance in datasets, some researchers have employed back- translation. This technique is particularly useful in scenarios with limited data, though its effectiveness may vary depending on the language pair. Back-translation enhances the performance of NLP models by creating synthetic parallel data, where sentences are translated from the target language back into the source language [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eText classification, a core task in natural language processing(NLP), has been extensively explored in previous studies. Researchers have investigated various approaches, including the use of word embedding models like Word2Vec, Doc2Vec, and GloVe, along with multiple classification algorithms such as Decision Tree, Random Forest, SVM, K-Nearest Neighbors, logistic regression, Gaussian Naive Bayes, and AdaBoost.\u003c/p\u003e \u003cp\u003eAdditionally, advancements have been made by employing convolutional neural network(CNN), recurrent neural network(RNN) with Long short-term memory(LSTM) units which capture long-term dependencies for improved prediction accuracy and better visualization.\u003c/p\u003e \u003cp\u003eIn our study, we use DistilBERT because of its ability to deliver robust performance while significantly reducing resource consumption The large-scale pre-training followed by task-specific adaptation is common in natural language processing (NLP), but fine-tuning all parameters of massive models like GPT3 175B is highly resource-intensive. Low-Rank Adaptation (LoRA) offers a more efficient solution by freezing pre-trained weights and introducing trainable low-rank matrices, reducing trainable parameters by up to 10,000 times and lowering GPU memory usage by threefold [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe rapidly growing and competitive airline industry has traditionally relied on customer feedback forms, which are often inefficient and time-consuming. Twitter data presents a faster alternative for gathering customer insights. In this analysis, tweets from six major U.S. airlines were used for multi- class sentiment analysis, where tweets were classified into positive, negative, and neutral sentiment categories. To prove with unbalancing dataset, we first employ GPT to balance the dataset by para- phrasing original samples from the minority class, utilizing the API key of GPT-4o. This strategy not only improves representation but also enriches the diversity of expressions within the dataset.\u003c/p\u003e \u003cp\u003eIn the following approach, back-translation was implemented using the API keys of GPT-4o to back translation samples by translating from English to Italian and then back to English. Italian was chosen after evaluating other languages, including Hochdeutsch and French, both of which demonstrated a close match with English during back-translation. However, we aimed to introduce some variations in adjectives to create new paraphrased sentences. This method allowed us to produce paraphrased sentences while augmenting the dataset, positively impacting the training process of our proposed model.\u003c/p\u003e"},{"header":"2 Related works","content":"\u003cp\u003eThe earlier method employed the Semantic Orientation CALculator (SO-CAL), a lexicon-based approach that utilizes dictionaries annotated with word polarity and strength, incorporating negation to classify text as positive or negative [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eLe et al. introduces a new feature selection method using Information Gain, Bigram, and Object- Oriented extraction techniques, combining Naive Bayes and Support Vector Machine (SVM) [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. While this method achieves high precision, it struggles with recall, addressed by identifying additional opinionated tweets using lexicon-based results, then training a classifier for sentiment polarity assignment using lexicon-based data instead of manual labeling [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eAnother approach extracts adjectives with meaningful sentiment from the dataset to form a feature vector through a two-step feature extraction process. Twitter-specific features are first extracted and added to the vector, then removed, followed by extraction on the remaining text. Machine learning algorithms like Naive Bayes, Maximum Entropy, and SVM are applied, alongside the Semantic Orientation-based WordNet to extract synonyms and assess content similarity[\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e, \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eSeveral studies have examined sentiment analysis (SA) on Twitter using machine learning (ML) and deep learning (DL) techniques. Despite the vast body of research on various aspects of sentiment analysis, tackling the issue with imbalanced datasets and models still needs more exploration [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. One of the key challenges in sentiment analysis of customer comments on social media is the imbalance in class distribution. To tackle this issue, various techniques can be employed, such as sampling methods like Synthetic Minority Oversampling Technique (SMOTE) and Random Oversampling (ROS), Borderline-SMOTE (BSMOTE), Adaptive Synthetic Sampling (ADASYN), One-Sided Selection (OSS) and Condensed Nearest Neighbor (CNN) can be applied both before and after data splitting [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e, \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e, \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eUnder sampling achieves dataset balance by decreasing the size of the majority class. This method is employed when there is enough data available. It works by keeping all samples from the minority class and selecting a matching number of samples from the majority class.[\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e] Addressing class imbalance in sentiment analysis can also be achieved through generative models, such as using GPT-3 to create synthetic sentences. This method serves as another approach for oversampling in imbalanced datasets [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eBack-translation helps the model learn general characteristics of specific topics [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]. DA methods, while successful in computer vision, have advanced slowly in natural language processing (NLP), with back-translation being a notable exception. The relationship between DA techniques and key factors like lexical diversity and semantic fidelity remains unclear [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e, \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e, \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eSawai et al. suggest that generative models can efficiently generate data by producing samples, such as those from GPT-2, which offer valuable insights for enhancing translation performance[\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e]. The feature extraction phase plays a crucial role in determining the overall effectiveness of sentiment analysis models. For instance, in one study, features were extracted and represented as a feature vector, after which a Bag of Words (BOW) model was constructed to analyze opinions [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eAmong the most commonly used techniques for word embedding is BERT, an unsupervised and bidirectional language representation model. Given the prominence of Twitter as a social platform, BERT has been employed for sentiment analysis on tweets, enabling the evaluation of emotions and opinions, which provides insights into users\u0026rsquo; mental states [\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eIn this context, robustly optimized BERT (RoBERTa) is utilized alongside Long Short-Term Memory (LSTM) to effectively capture long-distance contextual semantics, specifically designed for sentiment analysis [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e]. The hybrid deep learning model combines predictions using averaging ensemble and majority voting, improving sentiment analysis performance. The ensemble model integrates three hybrid approaches, which combine RoBERTa, LSTM, BiLSTM, and GRU. RoBERTa projects the input text into an embedding space, while LSTM, BiLSTM, and GRU capture long-range dependencies within the data [\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe strengths of the Transformer model (RoBERTa) and the Recurrent Neural Network (GRU) are utilized. Texts are projected into a meaningful embedding space using RoBERTa\u0026rsquo;s attention mechanism, while long-range dependencies are captured, and the vanishing gradients problem is resolved by the GRU.[\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e] Fine-tuning large language models (LLMs) using Low-Rank Adaptation (LoRA) enhances efficiency and effectiveness by reducing the number of parameters that need training, lowering computational costs, and allowing adaptation to smaller datasets with limited resources [\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e, \u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eIn summary, as shown in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e, there is a clear evolution in sentiment analysis methods, emphasizing significant developments and benefits:\u003c/p\u003e \u003cp\u003e \u003cb\u003eEarly Approaches\u003c/b\u003e [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e, \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e, \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e, \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]: These studies relied on lexicon-based methods, basic n-grams, and traditional classifiers (SVM, Naive Bayes). While simple, these approaches struggled with nuanced language, sarcasm, and context-dependent meanings. WordNet\u0026rsquo;s introduction [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e] provided some semantic improvement.\u003c/p\u003e \u003cp\u003e \u003cb\u003eEngineering and Ensemble Methods\u003c/b\u003e [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]: Exploring object-oriented features, Maximum Entropy classifiers, and ensemble methods improved performance but still lacked deep contextual understanding. [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e] begins to address imbalanced datasets using Random Oversampling (ROS).Engineering and Ensemble Methods\u003c/p\u003e \u003cp\u003e \u003cb\u003eAddressing Imbalance\u003c/b\u003e [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e, \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e, \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]: A crucial shift occurs as these studies tackle the imbalanced dataset problem. Ref. [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e] uses ROS, while [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e] explores SMOTE, BSMOTE and ADASYN. Ref. [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e] employs RMU (likely Random Majority Under sampling) along with other techniques.\u003c/p\u003e \u003cp\u003e \u003cb\u003eAdvanced Embeddings and Classifiers\u003c/b\u003e [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]: Word2Vec and GloVe embeddings provide richer word representations. More sophisticated classifiers like XGBoost are introduced. Synthetic data generation and back-translation start to emerge as data augmentation techniques.\u003c/p\u003e \u003cp\u003e \u003cb\u003eDeep Learning and Word Embeddings\u003c/b\u003e [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e, \u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e, \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e, \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e, \u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e, \u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e]: BERT and RoBERTa create a revolution the field with their contextualized embeddings and attention architecture. LSTM and BiLSTM networks are used to capture sequential information, leading to significant performance improvements. SoftMax is widely adopted for classification.\u003c/p\u003e \u003cp\u003e \u003cb\u003eEfficient Fine-tuning\u003c/b\u003e [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e, \u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e, \u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e]: The computational cost of large pre-trained models becomes a challenge. LoRA is introduced for parameter-efficient fine-tuning. Llama-3 and techniques like QLORA are explored for resource-constrained environments.\u003c/p\u003e \u003cp\u003e \u003cb\u003eFurther Data Augmentation\u003c/b\u003e [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e, \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e, \u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e]: New data augmentation techniques are investigated, including generating new samples based on GloVe embeddings, random punctuation insertion, and synonym replacement.\u003c/p\u003e \u003cp\u003e \u003cstrong\u003ePapers addressing imbalanced datasets\u003c/strong\u003e \u003cp\u003eThe table explicitly indicates that references [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e, \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e, \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e], and [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e] directly address imbalanced datasets. Ref [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e] also mentions the use of SMOTE and RUS for this purpose. It\u0026rsquo;s important to note that while other papers may not explicitly mention imbalance, they might still employ strategies to mitigate its effects.\u003c/p\u003e \u003c/p\u003e \u003cp\u003eAfter examining various techniques for sentiment analysis, the current analysis identifies several key issues addressed during implementation:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eUtilization of a benchmark dataset, specifically the US Airline Sentiment Analysis dataset.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eResolving of the imbalanced class issue through the generation of synthetic data.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eAugmentation of data by incorporating back-translation.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eExtraction of features using contextualized word embedding.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eApplication of Low-Rank Adaptation (LoRA) for fine-tuning the dataset.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eSummary of related work\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"6\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRef\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDataset\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eResampling Tech.\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eEmbedding Tech.\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eClass. Tech.\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eResults\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAmazon\u0026rsquo;s Mechanical Turk service\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eLexicon-based\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSVM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAccuracy\u0026thinsp;\u003cem\u003e\u0026gt;\u003c/em\u003e\u0026thinsp;79% for SO-CAL\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStanford Twitter Sentiment\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ebigram\u003c/p\u003e \u003cp\u003eunigram\u003c/p\u003e \u003cp\u003eObject-oriented\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSVM\u003c/p\u003e \u003cp\u003eNaive Bayes\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAccuracy of 79.58% for Bigram and SVM\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eObama, Harry Potter, Tangled\u003c/p\u003e \u003cp\u003eiPad, Packers Twitter\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eunigram\u003c/p\u003e \u003cp\u003eWordNet\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSVM\u003c/p\u003e \u003cp\u003eNaive Bayes\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAccuracy of 89.9% for WordNet\u0026thinsp;+\u0026thinsp;NB\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTwitter API\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eUnigram\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSVM\u003c/p\u003e \u003cp\u003eNaive Bayes\u003c/p\u003e \u003cp\u003eMaximum Entropy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAccuracy of 90% for ensemble classifier\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTwitter API\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eUnigram\u003c/p\u003e \u003cp\u003eWordNet\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSVM\u003c/p\u003e \u003cp\u003eNaive Bayes\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAccuracy of 89.9% for WordNet\u0026thinsp;+\u0026thinsp;SVM\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTwitter API\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eLexicon-based\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eMaximum Entropy\u003c/p\u003e \u003cp\u003eSVM\u003c/p\u003e \u003cp\u003eNaive Bayes\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAccuracy of 79% for W-WSD\u0026thinsp;+\u0026thinsp;NB\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTwitter API\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eROS\u003c/p\u003e \u003cp\u003eSMOTE\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eLexicon-based\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eNB, SVM\u003c/p\u003e \u003cp\u003eK-NN\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAccuracy of 93% for VADER\u0026thinsp;+\u0026thinsp;SMOTE\u0026thinsp;+\u0026thinsp;SVM\u003c/p\u003e \u003cp\u003eOversampling-Splitting increase performance\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eArabic Tweet Covid-19\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eRO, SMOTE, BSMOTE\u003c/p\u003e \u003cp\u003eADASYN, RU, OSS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eCountVectorizer\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eRidge Classifier, LR, SGD, SVM\u003c/p\u003e \u003cp\u003eDT, KNN, G-NB\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eF-1 score of 99% RF with\u003c/p\u003e \u003cp\u003eSMOTE,BSMOTE or ADASYN\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTwitter US Airline Sentiment\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eRMU\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eWord2Vec\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eXGBoost (XGB)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAccuracy of 86.5%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCoursera online course review\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSynthetic Data Generation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eGloVe\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSVM, DT, MultinmialNB, AdaBoost\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eBest Accuracy of 75.12% for MultinmialNB\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eRecovery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eBack-Translation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eBERT\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003esigmoid\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eBest F1-Score of 89.1% for real News\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTwitter US Airline Sentiment\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eBag-of-Words (BOW)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSVM, RL, RF, XGB, NB, DT\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eBest Accuracy of 83.13% for SVM\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTweets: the world about COVID-19\u003c/p\u003e \u003cp\u003eTweets: the India about COVID-19\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eText Blob\u003c/p\u003e \u003cp\u003eBERT\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eML algorithm\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAccuracy of 94%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTwitter US Airline Sentiment\u003c/p\u003e \u003cp\u003eSentiment140\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSMOTE\u003c/p\u003e \u003cp\u003eRUS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eBERT, LSTM\u003c/p\u003e \u003cp\u003eGRU, BiLSTM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSVM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAccuracy of 93.9% for US Airline\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eIMDb, Sentiment140\u003c/p\u003e \u003cp\u003eTwitter US Airline Sentiment\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003egenerate new samples\u003c/p\u003e \u003cp\u003eBased of GloVe word embedding\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eRoBERTa\u003c/p\u003e \u003cp\u003eLSTM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSoftMax\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eF-1 Score of 91% for Twitter US Airline\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eIMDb, Sentiment140\u003c/p\u003e \u003cp\u003eTwitter US Airline Sentiment\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003egenerate new samples\u003c/p\u003e \u003cp\u003eBased on GloVe word embedding\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eRoBERTa, LSTM\u003c/p\u003e \u003cp\u003eBiLSTM, GRU\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSoftMax\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eBest F-1 Score of 91.77% for Twitter US Airline\u003c/p\u003e \u003cp\u003ewith Ensemble model (majority voting)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSemEval 2017 Task 4\u003c/p\u003e \u003cp\u003eSemEval 2018 Task 1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eRandomly inserts punctuation marks\u003c/p\u003e \u003cp\u003eSynonym Replacement\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eRoBERTa using LoRA\u003c/p\u003e \u003cp\u003eLlama-3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSoftMax\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eMSE of 0.0137\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eReLi (Resenha de Livros)\u003c/p\u003e \u003cp\u003eTV\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eBERT using LoRA\u003c/p\u003e \u003cp\u003eOpenCabrita using QLoRA\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSoftMax\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eHighest F1 Score: 0.846 for BERTimbauBASE\u003c/p\u003e \u003cp\u003eHighest F1 Score: 0.615 for BERTimbauBASE\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e"},{"header":"3 The Proposed Model","content":"\u003cp\u003eThis work investigates a stacked ensemble model for sentiment analysis of imbalanced English text data, focusing on the embedding level. We argue that improved performance comes not from complex models[\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e], but from a focus on high-quality data and effective augmentation techniques. As shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e, our proposed approach uses DistilBERT, a streamlined version of BERT, to generate sentence embeddings from the \u0026rdquo;Twitter US Airline Sentiment\u0026rdquo; dataset. These embeddings are fine-tuned using Low-Rank Adaptation (LoRA), a parameter-efficient method. A SoftMax layer provides the final classification. Subsequent sections will delve into the specifics of the DistilBERT embedding model and the crucial role of data augmentation.\u003c/p\u003e \u003cp\u003eThe proposed model takes an imbalanced English dataset as input and balances it by Generating Synthetic Tweets(GST) using GPT-based paraphrasing. These new tweets are then annotated with one of ten categorical labels representing their primary meaning, using another large language model like Gemini. BERT extracts feature from the augmented dataset, and the resulting embeddings are fine-tuned with LoRA. Finally, the text is classified into three sentiment categories: positive, neutral, and negative.\u003c/p\u003e\n\u003ch3\u003e3-1- Data Augmentation\u003c/h3\u003e\n\u003cp\u003eData augmentation in natural language processing (NLP) is a crucial method for increasing the variety of training data, which enhances model performance and effectiveness robustness. Techniques such as synonym replacement, back-translation, and contextual word embeddings can generate new variations of existing sentences without altering their meaning. This process mitigates over-fitting by providing more varied examples for the model to learn from and helps to adapt models to different linguistic styles and contexts. As a result, data augmentation is crucial in advancing the capabilities of NLP models, enabling them to handle real-world language complexities more effectively.\u003c/p\u003e\n\u003ch3\u003e3-1-1- Oversampling Dataset\u003c/h3\u003e\n\u003cp\u003eSeveral dataset-balancing techniques are discussed in Section \u003cspan refid=\"Sec2\" class=\"InternalRef\"\u003e2\u003c/span\u003e. Here, as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e, we employ GPT-4, a large language model, generates synthetic tweets by paraphrasing existing tweets (as seen in Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). This paraphrasing introduces new vocabulary, enhancing linguistic diversity within the minority class and mitigating potential data bias.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eOriginal tweet vs paraphrased tweet with GPT4-o\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOriginal Tweet\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAirline Sentiment\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eparaphrased Tweet\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAI Sentiment\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e@JetBlue I understand but wish you would have announced the delay 2 hours earlier vs sitting for 2 hrs at MCO\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNegative\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e@JetBlue I understand, but I\u003c/p\u003e \u003cp\u003ewould have liked you to announce the delay 2 hours earlier instead of waiting 2 hours at MCO\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eNegative\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e\n\u003ch3\u003e3-1-2- Back translation\u003c/h3\u003e\n\u003cp\u003eBack-translation is an effective data augmentation strategy that boosts the resilience of machine learning models, particularly in the realm of natural language processing. This technique entails taking a piece of text from its original language, translating it into a target language, and then translating it back to the source language.\u003c/p\u003e \u003cp\u003eIn our proposed model, we employ GPT-4o to execute back-translation as a method of data augmentation by first converting English sentences to Italian and subsequently translating them back into English as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e The process starts with an English text that is translated into Italian using a reliable translation system. The Italian version is then retranslated into English, yielding a revised iteration of the original sentence. This two-step translation process not only adds variety to the phrasing and structure of the sentences but also enriches the training dataset, enabling the model to learn from a wider array of linguistic expressions. As a result, this technique enhances the model\u0026rsquo;s ability to generalize and perform more effectively across diverse inputs, showcasing the benefits of back-translation in enhancing language comprehension.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\n\u003ch3\u003e3-1-3- Noise Injection\u003c/h3\u003e\n\u003cp\u003eWe explored noise injection techniques like random character insertion and punctuation changes to create textual variations. However, unlike image data augmentation, these methods introduced unseen or unknown tokens during tokenization, degrading model performance rather than enhancing it.\u003c/p\u003e\n\u003ch3\u003e3-1-4- Annotating Tweets\u003c/h3\u003e\n\u003cp\u003eIn three-class sentiment analysis, neutral sentiments are the most challenging to classify and often lead to errors. As shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e, to mitigate this, we annotate negative instances using ten predefined categories from the dataset, such as [Customer Service Issue, Late Flight, Can\u0026rsquo;t Tell, Cancelled Flight, Lost Luggage, Bad Flight, Flight Booking Problems, Flight Attendant Complaints, Long Lines, or Damaged Luggage]. For positive instances, we use Gemini to generate corresponding opposite annotations (e.g., \u0026rdquo;Excellent Customer Service\u0026rdquo;) as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e. Neutral instances remain unannotated. after that we add the annotations in the first/end of tweets.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e\n\u003ch3\u003e3-2- Word Embedding\u003c/h3\u003e\n\u003cp\u003eWord embedding is an essential technique in natural language processing (NLP) that converts words into numerical vector forms, which effectively capture their semantic relationships and contextual meanings. These vector representations allow computers to understand and process text more effectively. A significant advancement in word embedding is the introduction of transformer-based models like BERT and DistilBERT to analyze sentiment in social media comments, especially Twitter\u0026rsquo;s comments.\u003c/p\u003e\n\u003ch3\u003e3-2-1- BERT\u003c/h3\u003e\n\u003cp\u003eBERT, a powerful NLP framework based on the Transformer architecture, uses surrounding text to disambiguate language and understand context (Fig.\u0026nbsp;\u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e6\u003c/span\u003e). Pre-trained on a large text corpus and fine-tunable for tasks like sentiment analysis, BERT leverages an attention mechanism to weigh the importance of different words dynamically. [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\n\u003ch3\u003e3-2-2- DistilBERT\u003c/h3\u003e\n\u003cp\u003eDistilBERT, a distilled version of BERT, offers comparable performance while being significantly smaller and faster (40% smaller and 60% faster). Trained through knowledge distillation, it retains much of BERT\u0026rsquo;s accuracy (97%) while requiring fewer resources, making it ideal for resource- constrained environments and speed-critical applications [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e].\u003c/p\u003e\n\u003ch3\u003e3-3- LoRA: Low-Rank Adaptation\u003c/h3\u003e\n\u003cp\u003eLoRA is a parameter-efficient fine-tuning technique that significantly reduces the computational cost of adapting large language models to specific tasks. Instead of fine-tuning all the model\u0026rsquo;s weights, LoRA injects trainable rank decomposition matrices into specific layers, effectively adding a small number of parameters while keeping the pre-trained weights frozen as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig7\" class=\"InternalRef\"\u003e7\u003c/span\u003e. This approach reduces the memory footprint and training time, allowing for efficient adaptation of large models on resource- constrained hardware while maintaining comparable performance to full fine-tuning. LoRA simplifies the process of deploying customized models and enables experimentation with various downstream tasks without the overhead of managing numerous large checkpoint files[\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e].\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\n\u003ch3\u003e3-4- SoftMax\u003c/h3\u003e\n\u003cp\u003eThe SoftMax function is a crucial component in multi-class classification, transforming a vector of raw model outputs (logits) into a probability distribution over the predicted classes. It exponentiates each logit and then normalizes the resulting values by dividing by their sum. This ensures that the output values are between 0 and 1 and sum to 1, representing the predicted probability for each class. The class with the highest probability is then selected as the final prediction. SoftMax is widely used in neural networks due to its ability to produce well-calibrated probability estimates, facilitating confident and interpretable predictions in various classification tasks.\u003c/p\u003e"},{"header":"4 Results and experiments analysis","content":"\u003cp\u003eThis section details our experimental setup, including the dataset, evaluation metrics, and results. We demonstrate that our approach yields a lightweight, efficient model with the lowest training and runtime costs. This efficiency is not only beneficial for performance but also contributes to reducing greenhouse gas emissions associated with computationally intensive models.\u003c/p\u003e\n\u003ch3\u003e4-1- Experimental setup\u003c/h3\u003e\n\u003cp\u003eOur implementation used Python with the TensorFlow library. The model was configured with a LoRA rank of 16, trained for 3 epochs with a batch size of 32. A learning rate of 5e-5 was employed for Adam optimization. We froze all layers of the pre-trained DistilBERT model and trained only 22,784 parameters for LoRA and 12 parameters for the final dense layer (outputting 3 classes). This resulted in a highly efficient model with low computational cost and fast training times, while still achieving high accuracy.\u003c/p\u003e\n\u003ch3\u003e4-2- Dataset\u003c/h3\u003e\n\u003cp\u003eTo evaluate our proposed model, we used the real-world Twitter US Airline Sentiment dataset, collected by CrowdFlower in 2017[\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e]. This dataset comprises customer feedback on six US airlines: American, US Airways, United, Southwest, Virgin America, and Delta. The data includes 2,363 positive, 9,178 negative, and 3,099 neutral sentiments. Some samples of the annotated dataset with the proposed annotation approach are depicted in Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e.\u003c/p\u003e\n\u003ch3\u003e4-3- Evaluation metrics\u003c/h3\u003e\n\u003cp\u003eEffective machine learning model evaluation hinges on choosing the right metrics. Metrics like accuracy, precision, recall, and F1-score offer different perspectives on performance, with the best choice depending on the task and the cost of different error types. For multi-class tasks, macro- and weighted- average F1-scores provide a comprehensive view across classes. Ultimately, a thorough evaluation requires selecting metrics aligned with the specific goals of the task.\u003c/p\u003e\n\u003cp\u003eTable 3: Sample Tweets in the annotated dataset\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAocAAAFICAYAAAAiQlQeAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAE1pSURBVHhe7d1bjB1bft/31XNmrJmxLpFHI0cMTdBs2oIdXxDAJmSGZGDCZhNxBCcAA5A8LwQC02gGsSDggHzhwwAmEDTDPMzDkABfwoeIzYPQRgAHCUkjjMImKIBJZMWRbcHDZmiaJiCNpMxoZI2tOed06rtYv+K/16nau/bufenL7wPU2XVZtWrVWquq/nvVbp6FjUoyMzMzM6t8of40MzMzM3NwaGZmZmbvOTg0MzMzs4aDQzMzMzNrODg0MzMzs4aDQzMzMzNrODg0MzMzs8ZI/87hwsJCPWdmZmZmO02fsG/k4HCE5GZmc+P7lY3D/WY+XO+z0bee/VrZzMzMzBoODs3MzMys4eDQzMzMzBoODs3MzMysMdHg8Pr16/nHjsOmly9f1nuYmc3H6dOnW+9Pmthu1iX2H/WVS5cu5c9p0TH9DH3n8OHDOe7YS+hjs7hHTXzkcGlpKf8lTPxrmOXl5by8vr5er5ktLqR79+7VS2ZmKT148CCtra3VSynfn+K96+HDhw4QrRX94sWLF01/uXr16lSCtjLw4Zh4+/Zt/pyX7fBMffr0ab5mb9++Xa+Zn3HqY9w6vHnzZo6ppm3iwSEF73Lo0KG0srJSL83OtL/NmdnuovsUAeKkH/i2s9Ef6BcXL16s16R07NixtLq6Wi9NBse5cuVKvfSOAlKON0/b4Zl69+7d/EmASKA4T+PUx3aPSyYaHF6+fDkHgIP0STNJNAAXspnZOGZ5v7Kdg8AtjvycPXt2on1lcXGxnttetuMz9dq1a/Xc7I1THzsiLqm+hfQ2YvKcnml5eble887a2lqzjalcJn31LSzPVxdIvdfGxtLSUpOG7RHptI38QD5ap8nM9oY+13u896yvr9dr39+7yvuM7ktM3I8irdcU911ZWem9TfcvyqN13N9iurJcNjnU7zDxWcSkNivFNOpf8bkU+1N81mmdJvaJx+R4cV+2x3zZHvsL81Hcxn6I1wLHKvNDXKdpUkbJi3OnTLE88fqF1jPFulJd9DlfifFFbKeu+ojHi+sxqA7jtrLNYnqlK+9BfbBfH/1bo9I3U4knUlLlqaLjjTB2RFHjkE77Kp32gzo96Zi0H/uY2d6he8Ig8QERp7b7he4tumkzr3uUbtbsF/OE7m3Kk/3jPNuUp/LRvY10ykv7MR8fUDZZ1G8fZYAY20RtrnVqV9FzSevic4p9yz4kWqf+oXyZ2Ff9iYn5tnxiX1U51f/i/jE/BSHTfKaSZ1+cA+L56RwkbiN9W10MO1/ofDl3MK927aoPlpmg+laZu/ZRf6KcKqu2Kz3rwTzTNIPDuf1TNgzBVyfW/F7g+fPnzY8snz17ln9vcfDgwbysH55WFZSH7ffv35/X85sDDeuTFw4cOJA/79+/79dBZtab7jE4d+7c535rqB++Hz16NH+S9tatW3me31pX99N8XyvpPkSe3K/4aY3SlXnqnle+JuNY7GfbB3/QRJ8R5vnrWfD8walTp/Kn2rX8A4Tq4V/PvTfOc4tnZ+x7g/qL+uyRI0eaY5V/1MHzdFBfnqcYG/DbS12zg/4wZdDfQqDrfMvYA8yzjrbsqg/uBUxt2vbR71jBOe3bty/P37lzZ1MZ9FtTxUrTNLfgECdPnsyfBIOvX79O58+fz8s0MsHimTNn8vKbN2/yJxXEX4QdP348L1Oh7Acqlm3cgPHq1av8aWbW16NHj+q5z/9gnPsPuP9wr9FyDCL561Ldn6BtelBwf2JfUR4l/VWqbT88rBXk8aCnbRXk0Z5s1/OHQIz21h+W6Hk1L7GvEmyoL3b1w+3oxo0buT4pe7wOVfeTpNijTZ+25MuCgvFY96X41+eckwJe7gODyjBNcw0OFfwRDDLip28BNPKTJ0+aCFsjhVBEzsQ3N40Usl/cNuybgplZiXuOHvR84Yz/lIhu2GyP9xr24aHETZ2gYC388zgRabVNI0xdhm23+eKfrokYdVoJ/xKHRrYYkYp9Zd6jv3HUir4Yy7ZTEGTFcsfAVn/BPAuKPdrwxZL7ASN/fUb5NFIIzkfnRnCo+Cee5yzMNTikoyoYZIgb+ucBdHEh/tm+btZ0EBpA+5GHvs1xo277hzHLkQAzsxIPet3QGaHQaIReEcbAQP8O4oULF/LniRMnWr/p86AA97L4sFBAoX004lQGH7a98LwpA3gGOXie0cb6mQBfMNR/eD6Vr5X74FnX9jwbl36CFX+6MO6/5znrZyrlLK8N4ghdU4zSTXL0kHuBYhRhnnVtr6GpD46v0UL6wqARQ7CPYiEwMiqcb4x/dG7xDcfUVNFpb32Txx95xqmq1DrFe6StOmu99P5HpG1pY15VRdZr3//4V1PMjx90ar1+zGlmux/X/CDcJ3Rv0FQ9ZOqtm/9oQPcUtsf0uqfE9XFe96lyv3h/K++XyrPtvhaXY1ltcqjbQWgf2mzQcwd6lmlSe5V9oWxX9Q2tVx8q08VnG9OwfGP52o5ZlrdMwzljWs9U8hskXo+x75flZCrrpq0u+p4v4rHVHtJWH1pmiu0yrA7jcZhk0L0glrMP9uljgf9UiXvh2+8Iyc3M5sb3KxuH+818uN5no289z/W1spmZmZltLw4OzczMzKzh4NDMzMzMGg4OzczMzKzh4NDMzMzMGg4OzczMzKzh4NDMzMzMGg4OzczMzKzh4NDMzMzMGg4OzczMzKzh4NDMzMzMGg4OzczMzKzh4NDMzMzMGg4OzczMzKzh4NDMzMzMGg4OzczMzKzh4NDMzMzMGg4OzczMzKzh4NDMzMzMGgsblXp+qIWFhXrOzMzMzHaaPmHfyMHhCMnNzObG9ysbh/vNfLjeZ6NvPfu1spmZmZk1HByamZmZWcPBoZmZmZk1phocnj59Or/f1nT48OF6yzsvX77ctP369ev1lpQuXbrUrBfyu3fvXr00GvKL+fdB+rLM0zZOOdtQbtVfn3MgHe2xG1Gf1KtZFK+ROE3rmidvXWNPnz6d+b1lUuJ9neuKcxn3vmxm29TGCPomX1tby2n5jNbX11vXLy4ubqysrNRL7y0tLdVz88Hx28q7UywvL+dpL1tdXc1tuNfrYS/qc7+iX5T3nmn0Fe5xlId7oO6DrNuKefTp8lrS9cXnOLi3jrvvtAzrN2rLcppUe9AfyW8QyjCP9p+mYeccqd9pUqygOlEdtk2iOGXQtN365iRwXn1MfOSQb5HXrl3j6OnYsWN51IZvmBoNqxotb49Yfvz4cb303oULF+q52eM8OH7V6dLdu3frtbbTnD17Nvc5s75u3rxZz03Oixcv6rmUDh06lKoHU700HkbqZj3Szz28Cko21Q/XF+fy+vXres1oymfBTkBbVkFDnq8C/fysY/nWrVt5VHWrLl++nPMchDJMo5/uBIxWnzt3Lvc76onp6tWrOc549OhRThPrkPu/0kEj+G/evElVMNls41nPpGX2G7df7wYTDw652B88eJDnuZncvn27aRRuLGfOnEknT57cdGPjBvPw4cMckAk3vyNHjtRL7/AaRmn0qpCG1usZBaJx0nHiK2k+tRz3jwgIKRcBojpcpOPrmOTHscrjIq6PNw/21zlofSwnWNa+OveYH1M81jiUXxTzj2J5OHcZVqdd+4G02sZEWihPbUdsY6WD2kPHidukrOuYF3Wgemjb1/YG+oioLzFx7cW+AvUnpnjNxvXxnjaIrhumeH2wrH6p/EjLw5F7prbPAvfyixcv1kvvMQjAw1jXE5+xzMJ1zET5OQ/ql3PgXHTNxXqI9xDWk0bbtU1pYz3HPFSXKo/apm+79MWzgsCC85l0e5Bf7Jd7Ge1JEE5gSL8T5okzYp9po6D+/v37+fOjjz7Kn23o03taVaG9DUvO6xKGc6EhW2F4VkO/zJM2Yjg4DpMrH2Ff8iNf9mc+5q9XNcKwO2lBvmxjP5WLScdjvixPPD7blRfi8UmnY3NMUNZyf+WvbUxxH8Ryaln58Kn6i+vjfIltTMNwTCaJdRfPhU+VDSrrsDrt2g9s0z7ko/qIeca2icuaJy+lVbmZj8fQMumZF9bFOtK52s4X27kLbU86TfF6BNvjdS2x39KHlKbsz7EMzKvvln09Hpd0WqdygXzVPzlmLM8sUI5h10csYzx/yhuvP7aBc2A92BbPibxYVj5Mqlstg3Va31WXSq9jDaJ8ByEf0qk9wXnreJNEHej8drM+9U6aUfo96WOfVf9rq0/yHSXvnapPPWOiI4fPnz9PR48ezfOMvFWNkufx5MmTPGKItqHa8+fP528EaPvmxWhkddHneb0qrBo4L4NXNYj7MmwMht+rRs/zfMOoLuy8HIfl3759W8+9e6V85cqV5tsn4qvlePz47UIjjCoL9A2VspMX3yxfvXqV92P/+E08lhPkx0grSK8RWdKxzLck1dlWVBdMPfe+/vStjGPqHBk5iKO51CPtOqxOu/bDgQMHNrWZyhLzjPVZ9dmcr9oFsT1UbpZ1DLCNdKJjso56ZpmJ8tjeQl+hXzGdOnWqXvsO/ZnrjP6m6w/8DIZ7FuhDem3Meq5J0h8/fjyvi/27zbNnz3K/Zx/1a+5dyjNen9w75kX333Fw/TNCeOPGjVzPuk4jRnP0jAB1z/2y6/5SBWH58+DBg/kTXXWpOoz3gElT2+zbty9/QuVgKkf/1K8Y6YzbmNcIGJ/UgfqUnifMM4rGsvJnUl9T3ho5jaOpTDuRzj32kVHpmRT7jLWb+GtlXRg8cBUo0mHp3Ap0uIjiAx/cALj46czcJJR2FNxAFIRxkx/3RsANRg8LJvLdShBGmWJ+8QY3TAxahTriHKnjGCBPSnwYldrK00fXfmojzoeHqW74Xbjp6Tetk0KATp9jmubDw7a/tmuTexXXcPlw15fPEveLeL2X97o2XMdxn+3YD7mn8kWvxP192GtP6oDzOnHiRL7WCVbaTCL4nUddErjwjIhfZjlP7meUgXsq21VP3MN1H+NLrII6tsdnDV8QyFPnxHOSfIVl1pOGfqdjsx/H5os9dc0xSMdEXgo+d5Ku6w0KhjUpkCzp7xjGiS/2mokGh/v372+CgBhgcGNlouPS+fWNu0TDcfNpCx77uHPnTnMBjBKARW3f8nVz6bqhDcLFS13oGxyG3UiFm3H8wbbyoI648Mepo2HIk7aK5dV8WR5uOF1tGQ3ajzql3dVu1FcXXfBxBCfSyC2ooz5lAzdQRornOSpj2wd9Un2e37nxuyQetvQvXbuMXvAjeFF61jNCJqxvu6dEfIkmIFD/5nOce820cU/lXqbfBwr3i3i/1XXEl2zot4LUBfdSgpg4qi+qB9UX9TDql99Z1yXnTjDCF1vmdW/iXAnYdD/jvqpBBp2f+gl1F98KDTtn7pMl7qGxTjlnHZvnIsdV4MQ87TisX+4kXJ/ULzi3+ByJbwHZNq1n565TdbTe+iSvOnbzSXqmqjGa+aoBm+1tYpqoutCaPL75zW828yvh9wRap6m6WPP6WJZvfetbzTzrSaPlmG8sQ9yfcrBNyxxf80wxrfKoOuSmNCzH/WKdaB11hlg+nWvcV9vLOov7qR66lOnK8qosiO2gY8b2LeuUvNC2H5jXek2cX5mnlOfFJ9gnHkN1FfOP9cYUcYyyDm1nK9u4FPtSOUHXI/0i9iP1x9jfYh/VfnF9eU1onjwQ8yct4j6x75bXR7w+Z6GstyiWS/UDzi/Wl+4Lqiudc6wH1c2g+0us67ZrXvmWy4OQbhgdQ+dRolwqv+g81F4qExP5ic5RyId1EfvofIV1Kk/cRl6z7iPjoPyD6LrpasO2Nmmrpy7Uc9lmu9GwepZ+qWp9MtWNa5BpNUDZCegsXRevbQ9tbdT3Yo7Yp7yBjmIr+9r21PcmaBb16Td9gsMyHwWH5T4EO6zXc3Pc4JA0TOQTn8HsP849ddbK+mqjem07HweH/fSpZ0z8N4e8ouMVYvnqAQx1M7QbX8dMSturWobZPXy8vcVXcNhNrzrMbG/SP5ESn4P8UWMV3ORnEq+d9bqbV6JVMJjnB+HeOOgVOT+j4ZUxx4mvVXnlzKtVvWpHn587bEe8dq8CuHw+fX+eZWOqg8ReRkmub0lxKr/9TFp5vPjtybYnfdvT1PXKYJCYxyjfkGMftd3H7WrjGNZvNNKnqeu5xghWVzruUxoFY9J9K67TvZBtcVnbmcr7HWna7oHKo2u/7YBy9dUWXzDpmV+eL1OXtrx2c+wwqC6iBf5TJe6FUb8RkpuZzY3vVzYO95v5cL3PRt96nvhrZTMzMzPbuRwcmpmZmVnDwaGZmZmZNRwcmpmZmVnDwaGZmZmZNRwcmpmZmVnDwaGZmZmZNRwcmpmZmVnDwaGZmZmZNRwcmpmZmVnDwaGZmZmZNRwcmpmZmVljYWOE/9O1/8fYu8vGZ9X028/Txif/6t2KD/alL3ztSFr44kK1oVquPsx2Kt+vbBzuN/Phep+NvvXskcM96rPv/1b65Ff/TvXt4H9OH3ztS1VQ+EfSF77wD9Mnv/YL6bPvVsGiA0MzM7M9ySOHe9DGJ5+lT3/lb6Uv/uUfpE+//B+kD9KX8vpP0x+mDz75p+mTX/k0ffBz/11a+NIX83qzncj3KxuH+818uN5no289Ozjcgz55+6vpD/7+z6cvfPan0idP/k364n/xlZT+SLX+5g/SF49/OX32xW+nr/7N/yF98U8crfcw23l8v7JxuN/Mh+t9NvrWs18r70E//N1fTf/if3qbfvMX/kn6wd/7jfT7v7Ge/vU/X68Cxn+WvvOL/zS9+h/fpj/87ed1ajMzM9tLHBzuQT/43qfp3z2e0m/91U/Tb/ylT9PXjn05/dTPfSV9++c20tsTP0w/81e/kH7w3U/r1Ga719OnT/M3aaZ79+6lS5cu1Vsmg/yuX79eL5mZ7Qx+rbwHfffb/2f6t/f/cvrxn/2T6Q82fiR97d9799vC33n7w/SVjR+mf/3iZfrg55+kP/ZnqwjSbIfqc78izfr6ejp06FAO5G7duuV73B7n59x8uN5no289Ozjcgz77LKVXt/7j9Aev/5f0v776sWoFA8gbaWNhI/31xe+nL//x/ygd/K8epw8+8MCy7VzD7lcvX75Mi4uLm9IQIN68ebNesr3Iz7n5cL3PRt969tN/D/pC1eo/c/5O+sq+v5Z+7qe+n35q4XvV9Hvpr/z099OPfP14+pkP/3sHhrbrMVpIcMjNktfLiIEhwSPbmE6fPl2vTenw4cM5iGT9hx9+2KQhD+2j9HzyulrYV+lJi67jmJnNiyOAvaj60vCVn/zpdOgX/mH6s39nLZ26+N+mU3/rv0l/5r/839KhX3ySvvr1/XVCs93txYsXaWlpKR0/fnxTwAYCR14561s2vx0keGMdWP9Lv/RLaW1tLS0vL6djx47lgJPlBw8e5ADy4cOHOS3Y99q1a3k/0t+4cSOvbzuOmdk8+bWyme1Ko96vGNUjSCO4AwFjREDHyCLp7ty5k4NBYR2BJhgpPHv2bJ4nILxw4UJebisPo41dx7H58HNuPlzvs9G3nj1yaGZ7EqOEjO4JwR2B2bNnz/Kyfo+oaVDAdurUqRzoMR05cqRe+3lxZFJGOY6Z2Sw4ODSzPYu/TtbvDfHo0aN04MCBPCrIKGJ8xRsDydL58+fzK+M3b97kV8tteH2tV8kg71GPY2Y2C36tbGa70rD7FaN49+/fT48fP25+G7iyspIuX76c59nOqJ4QxMXfEbIcA0FeLV+8eLHZn7QEn+BVNYEgZZLV1dX8urntOF0Bpk2fn3Pz4Xqfjb717ODQzHYl369sHO438+F6n42+9ezXymZmZmbWcHBoZmZmZg0Hh2ZmZmbWcHBoZmZmZg0Hh2ZmZmbWcHBoZmZmZg0Hh2ZmZmbWcHBoZmZmZg0Hh2ZmZmbWcHBoZmZmZg0Hh2ZmZmbWcHBoZmZmZg0Hh2ZmZmbWcHBoZmZmZg0Hh2ZmZmbWcHBoZmZmZg0Hh2ZmZmbWWNio1PNDLSws1HNmZmZmttP0CftGDg5HSG5mNje+X9k43G/mw/U+G33r2a+VzczMzKzh4NDMzMzMGg4OzczMzKwx1eDw9OnT+f22psOHD9dbto68nj59Wi+N7tKlS3myrbl3715u51nbavtPwvXr12fah6hrXUsceytoM/KzlPtRvE+9fPlypu3K8eO9kWOP2r6Uu+16KM8tTkof+wLnzrZhZt33zWzGNkbQN/na2lpOy2e0vr7eun6ryHdlZaVeGm51dTWXY3l5uV5jo9bhrMXyLS0tTaUfDTPPOtK1A/rv4uJinrduqq9BaE/SUb9C3c6qftWuOp7KM0o/63M9lOdIWvoR90C2Md/XsPsn22d9bU4S59ZF7aWpbCfVjaZR6nUS6AtMOxH1NQjXSKzbWPfzrvedhPrpo1+qWp9MuSnEzhlvdlxYfE6683KTKi/SYdjHweF749ThLJXl40Yx6wfQPOuIc51VwLJb9LlfdT1IZvmALdt2nH7GeQy6Htgeg8OIcx31YUr5uu6f87g2J6lPvwHpuup9pwZo89Sn3ul3g9K53ofr278n/lr52rVr6cGDB3meVw+3b9+mJHm5ummkM2fOpJMnTzavbniFwSsNvf6Ir1dY1msOpvjaRNvI49atW+nKlSvNvl2v3+Irli7kof2YlJ5jKV+9gomvzdle0jbS61yZoLx1TjqutoN9tJ9e3cZ8Yn1EbOuqt7g/81pX1mEU89KkeuVT+WibllU/qpuuc1Qecd+IdV3lU35qEygvpq46in0k5sl6yq08tK2tDKxTPSAel3lROm2P20ran0llp0zHjx9P1cM9r4/HhNJDx1AanSef8bicg/IfVL5YT5ra6pR1MQ3Ub1Rf6g/kCY6jY6p/z5LKcfbs2fwZ6R4Gyq/z0j6cr+pQ26K4T7w3aJ3qpA/VEVOs+7b26mNQep2X0C90bE3xfMr2Y1/6Kf111HLtNFUgkp9pnOu8zOO6macDBw7UczZ172LEfoYl18gg+DYV0/PNVFE98/oWG7+18qlv0Hyyv/KI36a1TXnEbeW3cNKxjrSxPMwP+uarvClTTKeyxmPGcpfiMUmvfchf3zhjHbCdZT7ZN+bLOpWlrF9R3WhbWU7Ng7RajulKZflUBtZxHC0jlpdzVD0OOkcmtpVtF5Xl03kilinOd9URZeL4QnqWlV7lQVcd8Uk6LZNe89B+Sqc8B50jZVB9qSxxuWs/0pBWVLeah84RfJKePIeVL5aB/VQvpbgPeWq5zI951RV5x22TRv6DqAyDUFbVm+ZV30zaxnlwroj1RL2rP8Y0cX1ZR6xX+/EZ06m8cT3HYr3ybqPyaopUXp2XylL2q1h+1V3cT2K6naisny6xL6gfSLk8DbRB7Dc7XZ96V1/vMot63+n69u+Jjhw+f/48HT16NM/fvXs3VZ03z+PJkyd5xBCvX7/On4O8ePEif1YXX/7Eq1ev8qe2tXn27FkzwsKEN2/epPv376fqZpqXEedLp06dyulBWRkxAqMGR44cyfOPHj3Ko0gc49y5c5vKGVWdtRltoPyMpIL8jx07lucfPnyY8yAv8uT8Ll++nMt48eLFnAaPHz/OZSGdvq3Gb/EYVG937txp2gcsk+cwhw4d2tRmOiYjLrGNQd1pdIP+wL4YdI7kodGbrnpsUz2A6rn3ZepTR9S9+iJu3ryZy0d7VDef3GYqDyPhbXWksktX3fY9R8pInai+KAv7UYdbQX7UxUcffdSMhvFZPVTy/LDyke7t27f1Uvu1S/+m3YU8UdZ7pOPG/j1rfUYhqBPqi1Ew+i+oU/oedRNHGLnPgL6kuqRvMVEX1Ct9kvagjw6qH+nqz9xHzp8/n5d1rGE4fnXP39RvI52XqC/GcuocQT7x2H3OZ7ehjrhn0ObliH4UR2DLUWONvpYT4sg0k0YK2Yf7KG3Keh2bvDViG/fTM0jHUj60WUwXR6Z3unhunFdsH+ojnreovtVeo4zw7zYTf628b9++/EnwpIcljcQNjlfKIFjRjWcauGlxE9TU9+Yp3HRjQKAAjwdjLDc30nicNhcuXMiBMXVw4sSJvK7tAtSNm2lQ8MuNKB5z1HqMN/e+eLgpEOYzPhBL1B1fDNr0Pcet6lNHCpgnaZy6jaifSaPtqAOCt3FudAS5CmigwK+0EwMDfdHTgzPiIco56WHBtUu/6qurPmK/HHQdRVu95kv0ib44NkEwdUAfGvVeuhdQJzxzuDe23dsVlKj9uA4VnLGNZ6O28ayhvpkHzw++uLHM84YglGPQhqxXWq5L8o33ENaTJgbx7Ec+9D36KG1KOibammt9J17LbbiG9cyhHoXrnWeyzpv6oe6oV30B4/nAtmk+p7a7iQaH+/fvb0YZYielAzNxU6PB9I1XNBpBg7GfvvmMg4CUi00XKZ90BkYJ4rd1gleW2x4MjNromyABrQK8iIsqdriuMvMA4jiMVnGBMlLC6GbEDSHu35UXo10avQDlG+VCZv+rV6/WS+9GWeM5dKE8usiYBqHuqFvqVQ9f9D3HrepTR+ojWk8f4QYhtL0QHPWpo3HrVrg2uEb0IAH1OMrDWOdD+fWgUkCom9wo/QWMnKrdu4IZ2lkPLXAMjqsgRvcC1jNP+4xajmmgfDw8KY/KDtrg4MGDebvegIzSDvT1Gzdu1Evv8lP7jnoNdPVn6pe2gcquB9sk0f/V/qMElXsNdUP7trUBo7z6cs3EtaJ7DEEI/UW4Z8RnJ9ctgR/302HtS1rKEPH84j4i8brjmcSxVC71s62+rdhOdI2pHkGf5v6v82aeemBgSyPnvGnZ86qLvrc+yauHbPNJeqaqwpv56ttJsx0sa1t1kTS/oeBT66ub86b5uA1xO2Keyg+xTBxLZWjDNtII+3AeEduVH2XoEo9VdcKcns8onhPlj+ekOkM8h7byD6o3xP1jmWPaUtxHE3UR67nMK9a7DDrHON+2b9xe1rvmVR/D6gix7LGdWR/zj/t3lVftE4/LdnTt03aO0HYm9ZFYVqayHyKWmXkdv60uYtrqodXMt5UvtpmmNvEaL89NefDJVLa96m/SyLuPsn5VdyjPi88PP/ywWUddxvpU22iZKZ5fXE9a3Q+Y1G5a1n5tbQit0/Hb+kUsP1NZ1zHvjz/+uJknT2hZk9pPy7G8TFB5lMdOo/MYpjw/tSXr4zbWlfd7UftoO+0R91Vds175q52pe9ojYjn2EZCf2j32bdLF5Xnj3IZRfXTVZ6y7su9rH+qo7VpB2R67EefXR79UtT6Z0tm6Kl5iA9r213YD2U43lUni5uP++V7Zzlzbw67v7aLvTdC6le3P9bGbH5zo22/a7hMKXmLQpoC6SwzQ47HLYLBcpm36BIccW+liezK/ne518dy7cO6xDiLqpzx3qN6UP+dc9mtR/ru5j/epZ0z8N4cM3fK6o+1P7BkaZxg3vn6z7Y1XVvojGqEd4x9f2O7EK8zyt5m8ZtUfUtnu1vbam5/X6OcC9nn6/WHET4nKn1KobrmX8hOG6lncTCX9lnnQ6974c5QSZaqCnfxM1u/+wTyvt+O+3O8p03bFvacK7j73ip26rQLgTT9Z43xZT3/l/IVX9+XvQ/VzDQuIEPsaJbki8Di1RfW2/ZXfbHdrO8Y+6776DiMOse0HjYBsN5TXtia2PVPbiM1uM6zfMOo0rE7KEbnyHqrrqMxLk+4/cb84z/7xfsVIV7xWy/sXy22jhOVzep4jiRy/r1gXmsrRPuo21klsp7LeWaZO47rd2tc5tz4W+E+VuBdG/UZIbmY2N75f2Thm2W8YpeMPusrRWEayuv5lgN3K1+ts9K3nib9WNjMzs8F4jRn/El1YP41/astsFB45NLNdyfcrG8esRw7LAJHfzu3Ff1/P1+ts9K1nB4dmtiv5fmXjcL+ZD9f7bPStZ79WNjMzM7OGg0MzMzMzazg4NDMzM7OGg0MzMzMzazg4NDMzM7OGg0MzMzMzazg4NDMzM7OGg0MzMzMzazg4NDMzM7OGg0MzMzMzazg4NDMzM7OGg0MzMzMzazg4NDMzM7OGg0MzMzMza0w0OFxYWGim06dP12tTOnz4cLP+6dOn9dp2bCf9VnQdJ5ajnF6+fFmnmh6OP+j8r1+/3lq2WJfj4NzIp49Lly7lcuxmqo+26d69ezkNdUBdSN96oa2UR19d/XLUfNoM6nN7oa3NzGwMGyPok3xpaWljZWWlXnpveXm5nhsNea2vr9dLw3F8yrm2tlaveY9tQhrlS9lGOcY4BpUrYvvi4mK99A7L49YfOG6fttuNBtUb9Rrbgz5Af1tdXc31NWqdk5792L/NoPzYFq8bytWnvwzSt8/tVnu1z9vWuN/Mh+t9NvrW87Z/rXzlypV6rp8HDx7Uc5/Xte3mzZvp0KFD9dJ0cOwqGKmXRnPt2rX06NGjeml0VdBTz+0tjJiNUm/0gcuXL6ezZ8+mKlCr1/ZHP6oCsnppM0Yrb926VS8Nd+zYsZzXs2fP6jWj20qfMzOzvWsuwSGvy/T6jddn8TWytoFt4AGn11+8CtNrt/i6TOvjq8BRkR9l4fjkxQOd4+p4KlcsA3Qew8o4rqtXr6ZTp07VS5tfP6tMUJ2q/G2vDFVWprZX6coDykv7xHaKyte0bfkqjeqGeon7xfOI6+Mx2UfrYzuzHPchHdPx48dzYKx1g1BXbeWWWC+gXDoeUyw/tF35KkhjXcynC+V9+PBhOnr0aL2mu90Ry9OWf6wfbY/nRN7UqY4R84/1XuZhZma7UD2C2Euf5LzKGvRaWa/LmLSOeV7paRt5iLaBfMt8ENfrleCwV2kxXzDPunLfmC7Oc8z4+lD7dJUR5WvMNtonTmV9sq4sE8dUevKgbKpHnRtUtjbKg31jObQP8zpuxHaVMc5HyivWWcwv9pu4njpjn3gO0Ho+lTfi8TkHtneJ+zLFcyMPnTefbFfZ47byGJyH8iJ9Wxu00THiFMvDcXRMkG8sH9uh42iZsjFPWbQOOh7rmXRMzq0sq/IA+6l+t7t4DmZ9ud/Mh+t9NvrW88RHDoe9nuV1WfUwyq/MeA0nb9++bbZ1efz4cX41x8gFo0JgROT27dvp/PnzeZlXguOg3NUDMI/wUA6p6iiXjWNGjOg8efIkz8cRp64yjoIycFxNnF8cQWsrk15pUn8q/4sXL/KnMDJ04sSJeunz4mvRQe1UYjuvYylj16vTKuDIn2ofjeRxrpwHo2SvXr3K6zmm+hHnwD7Pnz/Py6RlIr/Xr18356j8QT590ebUZxXw1Gs+L9YLDhw4sKlN47FBvcXy90UZ1Obqi4zk4e7du5vajjLplTl1rjbnuOQTX0ezL3019ut4TnqNXgV+uR1F50i7vnnzJs9jlPo1M7OdZ8f9UzY8ePUAZRoWjG4VD0Z+88exIh60ejgTuMQH76TLeOfOnRyAKKDqKtMwlFcB7STpVST5E2D0VQbBCkK7AiqCmZg+BjJbRV5920kBLufMFwACuUmjP1GXfDEQguEug76A0C7qq+OgXc6dO9e0sdrJzMx2p4kHh4xu8Eck8WE1qd8nnTx5Mj+khKCE4yhYggIojdpthfLq+kOWixcvNiM70lXGrdC5ETAMK9MgCrom/XsxAhgCpFGCYM6FgDfWHyObWh/LSJojR47k0UWtp07Lup8VynDhwoUmSI1fDCaF82NEUL811XUlfCGh/4Gg+caNG3kejF6fOXOmXnr35YI+FH+nOQr207mOMhJqZmY7VHXD761vcn6TRFpN1cOr3rL5N3X8fmkx/O7r448/3rQN2q5lPss00DqOxSfHaROPz7Ra/46sCkiadbG8sXyaF+1THqutjCoXE/u1KestTnGftjLFY8Z84nFJy/lqWecuMY9vfetbzTzr4zHL8sfjKV2Zd9wusc6ZlG/ZRl3rEcsVy8F8zD+WpzwuaaNYR2yL9UIZ4nZNZbpYFvWnuC2K5xCn2A8R8+RYUcxD/TG2faw79o1l/cY3vtHMx2MwlfloUptsZyq/2Sjcb+bD9T4bfet5gf9UiXvhtdIIyc12JUYOGcmMI6WMYk7yNfd2Up4bo5qMXI77+95Z8f3KxuF+Mx+u99noW8877jeHZvMWfzaArf5sYDsjEOY1dXT//v0cHJuZ2e7k4NBsRKurq81fWTPxu8DdOmoY//hGE6b9h2BmZjY/fq1sZruS71c2Dveb+XC9z0bfevbIoZmZmZk1HByamZmZWcPBoZmZmZk1HByamZmZWcPBoZmZmZk1HByamZmZWcPBoZmZmZk1HByamZmZWcPBoZmZmZk1HByamZmZWcPBoZmZmZk1HByamZmZWcPBoZmZmZk1HByamZmZWcPBoZmZmZk1HByamZmZWcPBoZmZmZk1HByamZmZWWOiweHhw4fTwsLCpun06dP11tFcv349Xbp0qV7a7N69e2Pn29fLly9z+fns6+nTpyPvMwrql2OMYpx9dO7TNqiN7T36On1+Gugb9BEzMzOZaHD44sWLtLy8nFZWVtLGxkZaX19PDx8+HDkA4EF45cqVemkzHmbnzp2rl6bv7du39dxwd+/eTYuLi+n+/fv1mskhQKA+RzHOPjh06FBuv2m7fPlyunnzZr1kXR48eJDOnj1bL20d1xfXEV8Cjh8/Xq81MzN7Z6qvlQkyCBYfPXpUr+mHByEBZptjx46l1dXVemn69u3bV8/1c/HixXT79u16aXIIEAg8RzHOPrb7Xb16NX9yfa6treV5MzMzmfpvDm/dupVOnTqV5xlBZIqvm3m1yHJcF7Gua5vEPPT6Ta/L+NQ26NV3fE2nY5BW63lwxs9h2O/8+fPpzJkzebSufLWsV6gqq85H5dGytsfyldheplG9MnEebXR8HZPlNqo7UT221Z2Udci8zkmvqTlePKbKA/YhvfaNx9f+cdJ+pZgm1gPLHI/PtvKLyki6sk3KfZVu0HrmoTz4jOcjzOvcmWLZVf8gP/LQMZQ/4v5t+YC86J+MGHbtG6nN43qdC3nHPNrSmpnZDrQxgj7Jl5eXczpNi4uLef3KysqmZaytreX0srS01CwrPWnA/Orqap7nk7TlfPXQy+l++Zd/OX8yaRvHZZk05K315M8y4jEQyzqM8gB5x2XyVHlYr3IK6XWebItliHQOIJ9YV5onn5g3++gcVQaOoTLouKL1OnfKFveLxyrpWCCd6pj0Oi/ygcrDp8rMpLx1PJCP6mTQ8UmnfZQny6o39h1EZdK5g+PqPMhLecZykEb7MB+Pw3otk17zKh9UPh2HNMqPdWwjPeuZZ+I4rItlZX1bnZXYh32hcuhc4raYB8dW+eKn9utKO0+cl9mo3G/mw/U+G33reaTW6JNpfACWym0sxweYHuQgnR48iMvso4cP6yhXnHi4lQ9O5nWsuD/Yp+1hNsoDrixDPDZi+WOQAcqqY3XVHeKDm3Tah8/y+Mo77sPxY/7lsrTVXdtxS13pqG/KFNsa8fhtbdJW7q7jx74j7KdjxjoZpK2OVKeaKBdlUPmicr3SIubNevIS5mP54nJXG5Z5xHQcM55HFNPxGeuN+VhncVI68o7HRVfaeSrLaNaH+818uN5no289z/2fsnn9+nU9N9zBgwfruc2qhyBn20z8LnEU7MPvsHgdxiszGeWVcvUgb47PPFP5Sq8L5eWPecpX0aOoHujN8Zn6ln0W+A0pZXry5Emu41HPkz9a4Q+U2JdPfkvZhjqfhq7+9ebNm/xZ6lo/bXfu3Mmvi/Valz/42arYr+mjoP5Z5jh69Y62tGZmtvPMNTg8ceLEpr9Kfv78ef6Djjak4/d8JeWhgINArW9QBv1uigf+2tpaevz4cb3lXVDSB0FPDMaYX1payn+93Ne1a9fybzMPHDhQr+nv5MmTm/6CW79raxPPj9+DttXpVigw4jj8pbp+H0c9U5/Ly8u5nUfB/jHwaEOdLy4ubgru+UOorf6Vb1f/os71hx3Qccv1z549SxcuXKiXUnr16lX+ZD1icKV64Rj0n1EDfPqQ6qgrgB4FZYi/KdS8fg/KcWhjdKU1M7MdqLrB9zYsOa+SSMPEq6eIV1zaFl8vxvW8MovIQ9via7AyfcyDfapAYtNyzId94jz5DTqOltvE48Syc35azxSXY1mZonI5imVsq7PyvBD3oaysj+tiO8iguhvUVijbgQnxuHrdGNN+4xvfaOZJG/uRyq1lTV3tEtOwL2J+g8QydfVRnRNi3cT6iOVlX1GfYtK+EtfF9XFdFaQ287FMqtN4npraKM+YnnXxWKrfmEZ1EtPF82tLO0+Uw2xU7jfz4Xqfjb71vMB/qsS98BpphOQ2IkZbpvnv/pE/r+Yn8bpxlhiVK8vctm4n49qqgtkt/RygrBONoI/6M4vdwvcrG4f7zXy43mejbz3P/TeH9g6vEvmncGwzApzy342kro4ePVovGQgM9cpa+FnDXg0MzcxsfA4O54zRPCJ5frc4zQc5wQO/MeT3cwRXOwV1wm8xqSNN066rWdNv+Ba38A+WM2LIbyxjPfF7STMzs1H5tbKZ7Uq+X9k43G/mw/U+G33r2SOHZmZmZtZwcGhmZmZmDQeHZmZmZtZwcGhmZmZmDQeHZmZmZtZwcGhmZmZmDQeHZmZmZtZwcGhmZmZmDQeHZmZmZtZwcGhmZmZmDQeHZmZmZtZwcGhmZmZmDQeHZmZmZtZwcGhmZmZmDQeHZmZmZtZwcGhmZmZmDQeHZmZmZtZwcGhmZmZmjakEh9evX08LCwt5unfvXr12fIcPHx6aj47H9PTp03ptu5i2nMD+HHNSyPfly5f10nCnT5/OdWhmZmY2axMPDgmsbt++nTY2NtLKykq6c+dOvWU8ly5dSuvr6/VSO4IpjqdjXrhwod7yeQRpy8vLOa3y1b5LS0t5+/Hjx/P6SRg3yHz16lU9Z2ZmZjY7Ew8O37x50wREly9fTg8ePMjz47p582ZaXFysl9odOnSoGVk8c+bMwGCStOTZhrKyfW1trV6zdS9evKjnRnPw4MF6zszMzGx2Jhoc8ir03Llz6eHDh/lVKiN6fDL6B7261StWgki9wmV9fHWsV9PkEZFGaZUvwd7Zs2fz/PPnz/PI4CToWEwS18VRQcrJNp1z22thzlX7Mq9l0upcQIB64MCBesnMzMxsdiYaHDJSuLq6ml/P8pqWkThe80oc0SOIYplXuI8ePcrpnjx5krcRZD1+/DjncfXq1U378Zqa9aRt+x0f6btGBkfBMTkGx2LkkkAO5M/IIuvBegI7AmICY7ZTB5S/9OzZs5wv+x47dixdu3atyYc6iPbv31/PmZmZmc3OVP4gpQ8CR4IuAi1GyqBgjwBQvxskiIqvlUnLaNtHH330uVfWBJXjvsYtccwYZPK6HOS/b9++XAYFraQjICYopLwoy3Hjxo38qXPFyZMncz5Hjx7dlN6vlM3MzGxe5hYcjotAjNG2U6dOjf3HHlvBiCfH1ohiX4wMlqOJjLSSDyOIBIkRAaiZmZnZrM0kONSIIL8HBMHVIIyu8XoW7MsIHa9smVdAqJG2+GpZvzucFo7F6+NxRicJDhkpjL9FJNAkT0ZAGXnUq2uCxjjCaGZmZjYzGyMYlnxlZSWn0VQFdXnSchUANes1zxT3W15eznktLi7mZT6ZVldX83q2l2mxtraW1/HZB/kpn7hfWd5YTtLEZZXxww8/bNbFcyGt0jCpjEyko/zaTlphXTw3Mxsd15XZqNxv5sP1Pht963mB/1SJe+HV5wjJzczmxvcrG4f7zXy43mejbz3vuN8cmpmZmdn0ODg0MzMzs4aDQzMzMzNrODg0MzMzs4aDQzMzMzNrODg0MzMzs4aDQzMzMzNrODg0MzMzs4aDQzMzMzNrODg0MzMzs4aDQzMzMzNrODg0MzMzs4aDQzMzMzNrODg0MzMzs4aDQzMzMzNrODg0MzMzs4aDQzMzMzNrLGxU6vmhFhYW6jkzMzMz22n6hH0jB4cjJDczmxvfr2wc7jfz4Xqfjb717NfKZmZmZtZwcGhmZmZmDQeHZmZmZtaYanB4+vTp/H5b0+HDh+stm2n7pUuX0r1799LTp0/rLVtHXl3HnaWXL1/mc+w6N7aRZq+jrVRHqrNZoN/RX6PYf9vahrJqO333+vXr9ZZ3+5LnMPF8R8Gx4rGjuK1E2q79IsqudPG8zMxs95tKcMjDjofK1atX8w8fNT169Civjw9DHqKrq6t5+4kTJ9K5c+fqLePjYcbDnOn48eP12tnoCm7v37+fFhcX0927d+s1m3H+hw4dqpd2nz5BP31hfX29Xkq5vmbl7Nmz6cGDB/XSuz5Ee9AuS0tLuf0iAquLFy/m7ZTz1q1b9ZZ3yIs8BynPt9QVvFGPV65cycdm4tiqW+r58ePHef3a2lo+hnA9cI1pv5s3b9ZbNiMd1yFpuDZv375db9k9OEcFv20BcAyOmVg224n9hvvIdhgg2Qpfr3NQPQB665O8eiBtVA/TemljY2VlJe/HZ/UgzJ9xe/VgzftI9TDatDwOjsexQF4cY1bK85Hl5eV8biNW+a7RVS+lmI42nFd90V701S70YdpThqXv0lUvg/ptuS3mUZar3EZ9xuuvzaBj7yR9+w7pmFRP0bC6st1nN/Ub7kuUcSdcz7up3rezvvU88ZHDa9euNSMwRPeMOlTHyctVB01nzpxJJ0+ezN8EcOrUqTy6p28CjLYcO3Ysz6PtNRhpWeYzfqOAPjlW/HYRv1lAefCNKuahfZjXqEvcN+bJvkxsZ+SGeUaCOJ846sM2Rmw0klSOoOn4Qj4673g8ieWJx2E/rdc3J5WLT20rxW3aT3lJW97UD8vsrzLFkSph37JeVP8xv1GwP3kpH44b21H9C32OpfxAXozGMTpX5gXO5+HDh3l0je1tVOeI5dLE8SKtp3zsR11RZ6xTPqLrgzzYxjWkdZRr//79eR6U482bN3mekUJdi6xvw/HjsTkG66gT9mGd0qnMMS+l1XZtU9ryXLaD6oGS7xect1lfs+o3bffUvrjmqwCxXtodfL3OSA4RexqWnJEejZ4Q1cf0jGYoqmeetMI+pGWKox7Mx9GYqkM0y3GkpjwW88pf20gP8tA3DvaP+cc8VA7Sso+QhnVs13loHWL+Eo9BOVQWYX8dmzpivkwjsTw6Np+UJdYv89rOpG1t5QNp43mWZdY+g86XfXScUkwXyxrPATGdtpXYX+fFMZVO5SdvlX/QsUT5xTpnPtZBiTzZT2J6tpGfziOmJU08DmUmLeI29tX5dGE/nZuU51eWU7rWIx6befKM+fIZj0uZWY5pdR5aBuviuU+bjjsMZVffiOeFctl2v+3Ub7gnDLsPDMM1t9U8ZsHX62z0reeJjhw+f/48HT16NM/z27qqY+d5PHnyJI8Y4vXr1/lTLl++TGnzNxxGYzSqcufOnSY/sMzvqUZVXRibfl+lkRTy1u+pGNEgHSMeUBmfPXvWjKIwgf2PHDmSy3rjxo1c9jjaWeIY2p8RqfL3aeQvjLpSjvPnz9drNqM8/NYN+k0cn/rNHKNf+t0m66sHds4v/p5O5x+RlhEonX9EeXV+pKNdKce46AuMcFEflA1v377Nn31wrpSB/kLfEX7TCsoofY6l/CZFbSiUJ/b5ciSSNpJyWxfambZH1yjgIPweuLwO29DuVRCZv62rXvn9pa5lcG1RxzFtvN50fgcPHsyf2xHnRtk5j3JUN4ojpkxxJJT9aIs4UqzRaGEUqGub7Tx9+w3b1O66XuM60Je0zDz9g7cXev6QXn2r3Df2K6btOEI/Sb5ep2/ir5X37duXP3lQK7Cj8gkweKWMV69eNQ+aOGTOA4UGjwFgWyAzKQp4KB/BDq/ECUDpQDEoJQjhQayJYILyM8/rYjpOW1AF8iLPuD+60vdB/ZU4DuWgPNThOAhI284f1NEkEYzFOhkUXG/VLI/Vhn6tV9R8xkB9HPQdtQd5cYPTDZLAjC9p8uLFi/xFps2BAwfqudG19cGdjuuaa502ig8Rod7jH9nRr3i1RVrqXw9yvmSxneuQ+57aintd3J9tgx5stjMM6zdqY7U71yt9gS+28V7NfSkOFHDfoI/xRZP9eH7qS6f+6IyJfkmQpGXuATzLdjtfr9M10eCQ3zppVCZ2cjo0EwEV0XccFePhFSuckR4FjoxO0DhCAHfhwoV66f0DSqNYMdDsi46hv0Sls3GRkZ8CCIIkOoU6H590OibKzT50KsrdhhHUMhihQxOEjUPlUQemPEwaqaU841I5qeNYZm42jJAKNyYF+lAAz/ph3+RAAMuFqXOgLlW/kzbLY3Whz3M96CYzCdRzpECPtlPf4lx5EOl6ikg3bl9p64P06d2ABzL3qrbfM3Evig9dHu56EDOvBzn3NMTfflJXtBn5aiQC47wJse1nUL/hzRH3ILU7/aC8fvvgOtZzNT4XuY51X+F6HyfvncrX6xRVnaq3Psmrh0TzSXqm+FukKpBqtuuzaqRme9V4eb3EfEgnMU/24VOqBs/LH3744aY0SsfE/lA+1UWXlzmeyiaUWfuRt9bF/LS/1sVtzEssNxP7aZ682/IsxfpSeWK+o5x/ifMqzx/Ks9y3bLt4rpGOre1tba40TGW9RLE9Yj5Msb+or7UdKyrzi+Uoj41YF6SN+bf1i1gmTdRhTBfzIH08f51HFPMs2ytuk5gfU1ffinXBVJZLYjrVaeyDpI31FMtEnrPAsfpQ+UV1xfq4jXVlW+g8wXnF/qL6ID/m2/qSbT/T6DfDrjdRHrrHln2q3C661thOn4zHj310O4v1MMgo9e7r9fP61nO/VLU+mVLhZcctlY1r2wcX07D2s9G0BUNt62yy+t4E2+5HemjHBwTLPFyi+CAe9LDRA4xP294m3W+YL4MUUXpRP9E9uOxT5Xa0BYPlcsxju5p0vbPs6/Xz+tbzxH9zyHAtw7Ztr3h5zcXwbBwSt+2F1+Pla3AbH69cy39Emuug/E2nbS+8qqseJPXSO9XDJL9Op/2E5T73M14JVg+l/DOWaK/9yH23a+s3/AEhfySon2GgbHdt0893eJ1Z/vxl0M919GoU+sO8vcTX6xTUQWIvoyRXFB6nMoq37UNtFL+R2mTQ730dzB51PQgjB7Fd2vp+OUrRtU/ZxhrJ0KR0jFTE9btxZGKno10GGafftPUPiX1CeStPPuM+mtcyNMqlScci33jcOEq2HVHGQXy9Tgbn0ccC/6kS98Ko3wjJzczmxvcrG4f7zXy43mejbz1P/LWymZmZme1cDg7NzMzMrOHg0MzMzMwaDg7NzMzMrOHg0MzMzMwaDg7NzMzMrOHg0MzMzMwaDg7NzMzMrOHg0MzMzMwaDg7NzMzMrOHg0MzMzMwaDg7NzMzMrOHg0MzMzMwaDg7NzMzMrOHg0MzMzMwaDg7NzMzMrOHg0MzMzMwaEw0ODx8+nBYWFvJnm5cvX+btTNevX2/my+nSpUv1Hpv3OX36dHr69OmmtMI2ltuOfe/evU37xCnmzxSPLcqbqevc9jrqiLpErE+ti2h7UNdKp3Ul0tB+II3mu5CmrQ0nLfbDQccjXewzpO0613nRNTCq8tppa2uJba20ui6HtamZmc3Yxgj6JF9eXs7p1tbW6jXvtW1bWlraWF1drZc28jbSsF7W19c3FhcX66X3aUrk34U8yn04rspSlqO0srIyMP8u5D8o392AttE5xnajvmK7gXYgrSaoPdUWovWx/siftpi2Ye1GuTgXlbGN+pzqgHKzPIvybwXnXbZFm7Kt43Ib8i37A6ZVH13tYjaI+818uN5no289T+W1cvWgSNeuXauXRnPs2LFUPZjSw4cPmxGFQ4cO5dEXjUyQpnoQbRpxYNvBgwfrpX7Onj2b85qmcethp2AU7OLFi7ku8eDBg/yJjz76KFUBUr30zv3793NaTVB7vnnzJi/L3bt38/qI/G/fvj1wlGoSBrWbjk2/pOzVdZSXS2ynL8vly5fztbHdXb16tZ4bjpFRnD9/Pr148SLPj4KRQ+rFzMy2j6kEhwQFBHd6cIBAjgdIHwoWnjx5Uq9J6cKFCzmwEB68cfvz58/TmTNn6qXhZvHqkVet1MO5c+fyPAim9GpN60D9sMwn2/QqUmnLutT6+MpSebCObWBer/T0OlPbmdgH7McyaWP+HJeJ+bbXoVeuXBlY74uLi/XccPv376/n3tUT/agNgduNGzfqpc3ia1uCOJ1TV3u3tQefZbsJdaFzYh/VpdJpObZXX6rnmB+Y1/ry3GJbKWjVtnI9YttrPcdlPfgkoD9+/HjrcbWOfQnU9eXq2bNn+UvCqMrAWnnruJRN/berDc3MbMLy+GFPfZLr1SufmofmySO+suJVVNvrO9YzCa/o4jLzsTzxWG30ik9T+Xqrqxwy7mvlmC/nHfNgG8usV7liPTGBdVpf1gPrWY55kAasj3mCedW/6kTLygd8xuO01Q1p2l4TCvvE/UjfVcfxWJRLrxpZX+7D9q7Xk5yP9uVcScv+rG8T6yvOtx1XSBPzI10sP9tiPcayUqa216hlnsoj7h/n2c6ktuVTZYjHiPPxnFjPpOPGMjKv8iOWi/1VR8JyPP8u7BuPU2KbzguUj3n2ozyxHH2Mmt4M7jfz4Xqfjb71PLW/VmaU8NatW3kUgG//J06cqLf0x+igMM9rK+XHq6/qgdSMfPV9pVydc55OnTpVr5kdXpPGerh582Z69OhRHn2pHoD5fFgn1QMxf8ZzY/T05MmT9dK7PBjlinmo3hjZqR64m0ZsaRON9pCuChzyqA/IlzKCdbG+46ie8BpYI05tGNnVq2OQZ1wWRoTi62hGBYe9aqwCknruvfK1LXXN+YM2b8P6t2/f5pGpeWLkO5adctFOTLQDI3eM5onOX/1Fo/VgHXVB29DewnbVP2mY6APqZ13oIxo5fP369abrEpQ9tt+49Fo6ti11Qpn37dtXrzEzs2mbWnDIQ41AhQd9V1DQhWCEB1kZUPLaigcRAQz5E8wQgBAgHj16tE71+b9ObqOH6qzxcN2qV69e1XPjIeBrw+thglUhgCYYpf0UUPZFYNKnjgk6YjtTNgIatZ1e747zSpE+R5BFHyGvtvOmnLym7goeZ6mtfKxTHx4WxAl1yj60ZQw40dX2g9Av9DvPAwcO1GvNzGy3mlpwCEb34shFHwSGjJAQWJYBJQHgnTt3mpE0HlrkT8AVgxcFBZrGRUAy6GFKEKqRyz4IgviNnhDojvo7LeqAc1a5qK8yABhEAbs8fvy4+c2gRoQILjgOI45s78JoYtsfIfC7NAWZlJP8KGcM4KG6UztT35Qhth3lZUS0DDQ1IjgI+XFc9qWOqO+IbZjkqJfyjKN8fRw5ciQHwtqfeqN+CM4pe9dIqtIrHQjkCCTLEb6y7TUaOAz58EWBiXKWRvniZ2ZmO0D1AO5tWPL4myF+I6R1/CaJZW1j4ndQcTlO/MaoS7m9euC1/oYrIn3MX2WTuC1O5A0+27YzDVM9sHM66gHxvNmGWDesi/Wo/Zl0nvF8VMYyD8Ry0wbS1k7CMVRWMD+oPdge81a+cWK7yiTxvDS1tSPnUB6f5TI/xLpVGtVBPKco1oXmofKV+5X9WGXTso5HOs47ri/LV2KdtjMh5qHykZfWt51fPI72KcupdWUZoTy1DM4nLovKTD6DlO3dll5lZYrnEPct22MQ0puNyv1mPlzvs9G3nhf4T5W4F15VjZDc9gCNPs3ynyNhZFK/l9urGFmsAqWZXY+MUPIb0500Suj7lY3D/WY+XO+z0beep/pa2XY/gkJePY/yen0reFXMb033cmA4D/zO16+Pzcz2BgeHtmX8Zo/fgg76feYkMErJ7zZnOUq5XTFqiEF/LT4J5M83zfKPw8zMbPfya2Uz25V8v7JxuN/Mh+t9NvrWs0cOzczMzKzh4NDMzMzMGg4OzczMzKzh4NDMzMzMGg4OzczMzKzh4NDMzMzMGg4OzczMzKzh4NDMzMzMGg4OzczMzKzh4NDMzMzMGg4OzczMzKzh4NDMzMzMGg4OzczMzKzh4NDMzMzMGg4OzczMzKzh4NDMzMzMGg4OzczMzKyxsFGp54daWFhIIyQ3s0G+/3+n9Lv/IKXf+8fvln/8L6T0x34+pR/7i++WbUt8v7JxuN/Mh+t9NvrW88RHDjnwy5cv66Wtu379erp06VK9NBmUj3IynT59ul47HTqWPH36NB0+fLheGmyUtML53Lt3r17abFJ1Ocv622lor6763+TNnZS+91+n9GkVIP7hv3k3Mc86tpmZmc3JRIPDUQOZPi5fvpxu3rxZL03GqVOn0traWlpfX08PHz6caDBbWlxcrOfeOXbsWHrx4kW9lD4XrGmZMh0/fjzP98W+nE+Xsi7HDRRnUX8EstNsl2l6/fp1Pdfh9389pS8+TG9//bfSv/pn30npg+oyrCbmWce2nMbMzGwOJhocxqBnOyOo2bdvXzp06FAeXuVzWjhWF4KfW7du1UspjzgpIKJMBGCjIPBbWlqqlwaLxxrVLOrvypUr9dzOc+DAgXqu3cZv/730f/zv/yQ9/b/+3/QpV+CXqv9UE/OsY9vG7/z9d4nNzMxmbKLBobx9+7Z57cirUeEVpNYzMiSMYLHMyCPb4oiWtkGvWflUPkKgo3Wa4jGgNGBET69EOYb2icdmf21rGxVlu/br+3qVoIy0lEWjitr/3LlzeSSO5bbAjX11vPLcSm3pVJfkM+xYbW3VVX99qG21f6y72J46R3AM0qndVSbKEffvKgdpyvYbtF8sI+UQ0qnd2EYeKmfZL1jev39/vdRu43d+Lf3ot//99LN/9E+lf/T/fCf95r/8bvrO2++lX/v176Q/Xa370W//ubTxnX9UpzYzM5uxjRH0SU6a6qGe55eXlzdWVlbyPJ+rq6t5HqRjmfXMM62vr+eJ+bW1tWYbn1rPtLS0lPPgOKQD65Q/6Tl2Fx0LKoOQJ8us1/G6xHzifKRyg7Iyr/LHbeCY2gbSqy7jPNhP5x6xf3k8jhPrEuWxIp2/sF+53HauXcq20XFZp7zKsmi96ogpnm8sQ5wX5c0k7B/7BcfUMp/KX8dUeuXDsvLVflov5FmW5XP+wcWNjV+8s/G7F29sfHf1/MbvPTu58S8f/ocb37t3fuN3//aNvC2nsS2hbcxG5X4zH6732ehbz1MZOXz06FE9l9KrV6/y5+3bt9ORI0fyPKqHbHry5En+HVz1oE1V0JBfTzKx/OzZs2Yb9Jq1CpDSgwcP8jq8efMmf7I9/tarbTSszZ07d9LRo0frpXfLjx8/TmfPns1l0vHbVPXXjJL2we8NOe9xUB9V0NGMbEHnXtIxOF4VrKTnz59vqsthutpqXIxQMlJJuXldrJ8fqI5p06tXr+Z1pdjunI8Mq/u29rt79246ceJEvfTuNbz6Kq/3lT/HZF/qXK/qqQNtpyz67SbzEfsO9fU/ndK/88P0k1/7cvqJn0npx378q+kHP/gkPf/n6+knv/7VlH7yhyn99M/Wic3MzGZrKsFhFx7m08LDmsBDAUgMIIfpCrKG4RXitWvXcqAyCwQ6HEsTAVAfw15ztpl0WxHYqtzxt6l8eeC8Lly4UK/pZ9y6H/THIn2/UAwzNED88389pf/saUp/Yy2ln/heSp98kH7/e5+l/+8Lv5nSf1Kt/0+r6c/9tTqxmZnZbM0sOOQvXHmYCyNR58+fr5dSHq0TRnHOnDlTL/XDb8tiANLXyZMnN41aMVrUJ1DRbylHCUK3gtFN6kXH5TP+Lq4NaQjE4ohbH8PaalSMvMXfcmqe3+4xkkdgzzHL3+91GbfuOVb8QxdGVC9evJjnKeONGzfyPOiPo/ZBaERxoK/+hZR+6nhKB/6wmv+RxEvrH/3ql9Kf+OM/UUXy/zalr1fbSGNmZjYPVSDV27Dki4uLOQ3TSv0bN82D32Np3Wr4Ddvy8nLrtpjHN7/5zWaetDF9+duwuD7Sb8k0qVxxX62jDOW6UjxfzZfidsqjZY4Zt3OMuP3jjz9u5jlXxDKRXxelYZJYl+QTj1XWE9rao6v++oh1RX46F+oh5qvzUvoPP/yw2aZ6QMxP81Gsq1jOWA9qA4l5qk7KvqF5yhLTcw5Qul6+92sbG6/+7sZn//g/3/j+r/zNjY1/cW1j4/eqdTYRtIXZqNxv5sP1Pht963lb/B9SGEk6ePBg/l3cuBiFKvdvW2dme8O07le2u7nfzIfrfTb61vNMf3M4Lbxm5I8oIl65xj80MTMzM7Ph5h4cMrrHb+n4Ldiw39B14Td1/GaNiFgTv5Mb9bd2ZmZmZnvdtnitbGY2ab5f2Tjcb+bD9T4bfet5V7xWNjMzM7PJcHBoZmZmZg0Hh2ZmZmbWcHBoZmZmZg0Hh2ZmZmbWcHBoZmZmZg0Hh2ZmZmbWcHBoZmZmZg0Hh2ZmZmbWcHBoZmZmZg0Hh2ZmZmbWcHBoZmZmZg0Hh2ZmZmbWWNio1PNDLSws1HNmZmZmttP0CftGCg7NzMzMbHfza2UzMzMzazg4NDMzM7OGg0MzMzMzq6X0/wN04++do30P8AAAAABJRU5ErkJggg==\" width=\"647\" height=\"328\"\u003e\u003c/p\u003e \u003cp\u003e \u003cb\u003eAccuracy\u003c/b\u003e, the proportion of correctly classified instances, is a straightforward evaluation metric. However, it can be deceptive with imbalanced datasets, where a model might achieve high performance by predicting the dominant class. Therefore, relying solely on accuracy for imbalanced data can be misleading, and supplementary metrics are crucial.\u003c/p\u003e \u003cp\u003eThe \u003cb\u003eF1-score\u003c/b\u003e, balancing precision and recall, is a crucial metric for evaluating classification, especially with imbalanced datasets. Ranging from 0 to 1, it effectively assesses a model\u0026rsquo;s ability to correctly identify positive instances while minimizing both false positives and false negatives.\u003c/p\u003e \u003cp\u003e \u003cstrong\u003ePrecision\u003c/strong\u003e \u003cp\u003eThe proportion of correctly predicted positive instances out of all instances predicted as positive.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eRecall\u003c/strong\u003e \u003cp\u003eThe proportion of correctly predicted positive instances out of all actual positive instances.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eMarco-average\u003c/strong\u003e \u003cp\u003eMacro-averaging calculates overall performance in multi-class classification by averaging per-class metrics (e.g., precision, recall, F1-score), treating all classes equally regardless of their size. While useful for understanding performance across classes, it can be skewed by imbalanced datasets, where smaller classes might disproportionately influence the overall score. Therefore, consider class distribution when choosing an averaging method.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003eROC-AUC\u003c/b\u003e measures a classification model\u0026rsquo;s ability to distinguish between classes. It visualizes the trade-off between true positives and false positives, with a higher area under the curve (AUC) indicating better performance, especially for imbalanced datasets.\u003c/p\u003e\n\u003ch3\u003e4-4- Experimental findings\u003c/h3\u003e\n\u003cp\u003eThis section evaluates the performance of our proposed BERT-LoRA model (detailed in Section \u003cspan refid=\"Sec4\" class=\"InternalRef\"\u003e3\u003c/span\u003e) against several baseline models using the \u0026rdquo;Twitter US Airline Sentiment\u0026rdquo; dataset (14,640 English reviews classified as positive, negative, or neutral). As shown in Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e, the baselines include models using traditional methods like Bag-of-Words and Word2Vec, as well as more advanced techniques such as RoBERTa-LSTM and BERT-LSTM. These models employ varying resampling strategies and leverage BERT for word embeddings. Our proposed model, trained on an augmented dataset exceeding 54,000 samples (incorporating original tweets, paraphrased versions, and back-translated tweets), achieved a remarkable 100% accuracy in just three epochs. This result, highlighted in Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e and Fig.\u0026nbsp;\u003cspan refid=\"Fig8\" class=\"InternalRef\"\u003e8\u003c/span\u003e, demonstrates exceptional performance and efficiency compared to the baseline models, which typically require more extensive training and often achieve lower accuracy. Table\u0026nbsp;\u003cspan refid=\"Tab5\" class=\"InternalRef\"\u003e5\u003c/span\u003e and Fig.\u0026nbsp;\u003cspan refid=\"Fig9\" class=\"InternalRef\"\u003e9\u003c/span\u003e also provide insights into the effectiveness of the resampling techniques and other data augmentation methods employed.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eComparison results of the proposed model and based models\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eResampling Method\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBOW [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e86%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eWord2Vec [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eRMU\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e86%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRoBERTa-LSTM [\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e91%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBERT-LSTM [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSMOTE\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e93%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBERT-LoRA (Propose Model)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eImbalanced Dataset\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e84%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBERT-LoRA (Proposed Model)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eGST \u003csup\u003e1\u003c/sup\u003e \u0026amp; Back-Translation(BT)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e92%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eBERT-LoRA (Proposed Model)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eGST \u0026amp; BT \u0026amp; Annotated Tweets\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e100%\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab5\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 5\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003ePerformance of the proposed model on various augmented datasets, evaluated using 10-fold cross-validation\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"6\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRe-sampling Technique\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eF-1 Score\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eMacro-Average\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eAUC\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eSupport\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eImbalanced Dataset\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e84%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e90%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e78%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.95\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e14640\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGST \u0026amp; Back-Translation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e92%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e94%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e92%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.99\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e54231\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGST \u0026amp; Back-Translation \u0026amp; Annotated Tweet\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e100%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e100%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e100%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e1.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e54231\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"6\" colname=\"c1\"\u003e \u003cp\u003e\u003csup\u003e1\u003c/sup\u003eGST: Generate Synthetic Tweets\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eWhile common practice suggests that text preprocessing steps like expanding contractions, lower- casing, and removing elements like usernames, emojis, stop words, and URLs improve performance, our study found that keeping these elements is actually beneficial in social network comments. In our experiments, removing them reduced accuracy by 5%. These elements, while often considered noise, contribute to the overall meaning and should be preserved.\u003c/p\u003e \u003cp\u003eGenerating synthetic samples demonstrably improved classifier performance on the imbalanced dataset. Data augmentation, BERT embeddings, and LoRA fine-tuning resulted in higher accuracy and lower training costs, showcasing the proposed architecture\u0026rsquo;s effectiveness for English sentiment classification. Annotating tweets with 10 predefined categories further boosted accuracy to 100%. A thorough analysis, including ROC-AUC and the impact of reduced training epochs, is presented in the following figures. Figures\u0026nbsp;\u003cspan refid=\"Fig10\" class=\"InternalRef\"\u003e10\u003c/span\u003e\u0026ndash;\u003cspan refid=\"Fig12\" class=\"InternalRef\"\u003e12\u003c/span\u003e illustrate ROC scores across different dataset configurations (imbalanced, balanced, back-translation augmented, and annotated), emphasizing the superior performance achieved with the annotated data.\u003c/p\u003e \u003cp\u003eOur proposed model leverages a data augmentation strategy, generating synthetic samples through paraphrasing and back-translation, to enhance accuracy and reduce both training costs and runtime. This efficiency is achieved through low-rank adaptation fine-tuning. However, it currently relies on external models (like GPT) for annotation and doesn\u0026rsquo;t address intra-class imbalances. Future work could explore weighted augmentation based on intra-class clustering, and training for independent sentiment classification without external dependencies.\u003c/p\u003e"},{"header":"5 Conclusions and future works","content":"\u003cp\u003eOur study demonstrates that data augmentation through paraphrasing and back-translation, combined with annotation using GPT-4, DistilBERT embeddings, and LoRA fine-tuning, significantly improves sentiment analysis performance on imbalanced datasets. Achieving 100% accuracy on the Twitter US Airline Sentiment dataset in just three epochs underscores the effectiveness and efficiency of this approach. However, the current model relies on external language models for annotation and does not explicitly address imbalances within sentiment categories. Future work will explore weighted augmentation based on intra-class clustering, and eliminating the need for external language models by training the model to independently classify sentiment categories. These improvements aim to create a more robust and self-contained sentiment analysis system.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eCompeting interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors declare that they have no competing interests or personal relationships that could have appeared to influence the work reported in this paper.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors declare that no funding was received for this research.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData availability statement\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe algorithm\u0026apos;s code, along with image data, will be made available upon request.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eUse of experimental animals, and human participants\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNo animals and human participants were involved in this study in any kind.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthors contribution statement\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eHNN performed investigations, software development, visualization, and writing original draft. MHM performed conceptualization, supervision, validation, and writing- reviewing and editing.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEthical and informed consent for data used\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNo ethical concern exists.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDeclaration of Generative AI and AI-assisted technologies in the writing process\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNo AI or AI-assisted tool were used in the writing process of the article.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eR Monika, S Deivalakshmi, and B Janet. Sentiment analysis of us airlines tweets using lstm/rnn. In \u003cem\u003e2019 IEEE 9th International Conference on Advanced Computing(IACC)\u003c/em\u003e, pages 92\u0026ndash;95. IEEE, 2019.\u003c/li\u003e\n\u003cli\u003eAnkita Rane and Anand Kumar. Sentiment classification system of twitter data for us airline service analysis. In \u003cem\u003e2018\u003c/em\u003e\u003cem\u003e IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC)\u003c/em\u003e, volume 1, pages 769\u0026ndash;773. IEEE, 2018.\u003c/li\u003e\n\u003cli\u003eMikhail V Koroteev. Bert: a review of applications in natural language processing and under- standing. \u003cem\u003earXiv preprint arXiv:2103.11943\u003c/em\u003e, 2021.\u003c/li\u003e\n\u003cli\u003eJacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. \u003cem\u003earXiv\u003c/em\u003e\u003cem\u003epreprint\u003c/em\u003e\u003cem\u003earXiv:1810.04805\u003c/em\u003e, 2018.\u003c/li\u003e\n\u003cli\u003eV Sanh. Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter. \u003cem\u003earXiv preprint arXiv:1910.01108\u003c/em\u003e, 2019.\u003c/li\u003e\n\u003cli\u003eChuanxia Jian, Jian Gao, and Yinhui Ao. A new sampling method for classifying imbalanced data based on support vector machine ensemble. \u003cem\u003eNeurocomputing\u003c/em\u003e, 193:115\u0026ndash;122, 2016.\u003c/li\u003e\n\u003cli\u003eNitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. \u003cem\u003eJournal\u003c/em\u003e\u003cem\u003eof\u003c/em\u003e\u003cem\u003eartificial\u003c/em\u003e\u003cem\u003eintelligence\u003c/em\u003e\u003cem\u003eresearch\u003c/em\u003e, 16:321\u0026ndash;357, 2002.\u003c/li\u003e\n\u003cli\u003eSeokjin Oh, Su Ah Lee, and Woohwan Jung. Data augmentation for neural machine translation using generative language model. \u003cem\u003earXiv preprint arXiv:2307.16833\u003c/em\u003e, 2023.\u003c/li\u003e\n\u003cli\u003eParul Gupta and Maha Mahmood. Data augmentation for neural language processing. \u003cem\u003eInterna- tional Journal of Data Science and Advanced Analytics\u003c/em\u003e, 6(6):352\u0026ndash;359, 2024.\u003c/li\u003e\n\u003cli\u003eEdward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. \u003cem\u003earXiv preprint arXiv:2106.09685\u003c/em\u003e, 2021.\u003c/li\u003e\n\u003cli\u003eMaite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede. Lexicon-based methods for sentiment analysis. \u003cem\u003eComputational linguistics\u003c/em\u003e, 37(2):267\u0026ndash;307, 2011.\u003c/li\u003e\n\u003cli\u003eBac Le and Huy Nguyen. Twitter sentiment analysis using machine learning techniques. In \u003cem\u003eAdvanced Computational Methods for Knowledge Engineering: Proceedings of 3rd International Conference on Computer Science, Applied Mathematics and Applications-ICCSAMA 2015\u003c/em\u003e, pages 279\u0026ndash;289. Springer, 2015.\u003c/li\u003e\n\u003cli\u003eLei Zhang, Riddhiman Ghosh, Mohamed Dekhil, Meichun Hsu, and Bing Liu. Combining lexicon-based and learning-based methods for twitter sentiment analysis. \u003cem\u003eHP Laboratories, Technical Report HPL-2011\u003c/em\u003e, 89:1\u0026ndash;8, 2011.\u003c/li\u003e\n\u003cli\u003eMS Neethu and R Rajasree. Sentiment analysis in twitter using machine learning techniques. In \u003cem\u003e2013 fourth international conference on computing, communications and networking technologies (ICCCNT)\u003c/em\u003e, pages 1\u0026ndash;5. IEEE, 2013.\u003c/li\u003e\n\u003cli\u003eGeetika Gautam and Divakar Yadav. Sentiment analysis of twitter data using machine learning approaches and semantic analysis. In \u003cem\u003e2014 Seventh international conference on contemporary computing (IC3)\u003c/em\u003e, pages 437\u0026ndash;442. IEEE, 2014.\u003c/li\u003e\n\u003cli\u003eAli Hasan, Sana Moin, Ahmad Karim, and Shahaboddin Shamshirband. Machine learning-based sentiment analysis for twitter accounts. \u003cem\u003eMathematical and computational applications\u003c/em\u003e, 23(1):11, 2018.\u003c/li\u003e\n\u003cli\u003eDinar Ajeng Kristiyanti, Samuel Ady Sanjaya, Vinsencius Christio Tjokro, and Jason Suhali. Dealing imbalance dataset problem in sentiment analysis of recession in indonesia. \u003cem\u003eInt J Artif Intell ISSN\u003c/em\u003e, 2252(8938):2061, \u003c/li\u003e\n\u003cli\u003eMaisa J Al-Khazaleh, Marwah Alian, and Manar A Jaradat. Sentiment analysis of imbalanced arabic data using sampling techniques and classification algorithms. \u003cem\u003eBulletin of Electrical Engineering and Informatics\u003c/em\u003e, 13(1):607\u0026ndash;618, 2024.\u003c/li\u003e\n\u003cli\u003eMohammad Mohaiminul Islam and Naznin Sultana. Comparative study on machine learning algorithms for sentiment classification. \u003cem\u003eInternational Journal of Computer Applications\u003c/em\u003e, 182(21):1\u0026ndash;7, 2018.\u003c/li\u003e\n\u003cli\u003eSaleh Naif Almuayqil, Mamoona Humayun, NZ Jhanjhi, Maram Fahaad Almufareh, and Navid Ali Khan. Enhancing sentiment analysis via random majority under-sampling with reduced time complexity for classifying tweet reviews. \u003cem\u003eElectronics\u003c/em\u003e, 11(21):3624, 2022.\u003c/li\u003e\n\u003cli\u003eCici Suhaeni and Hwan-Seung Yong. Mitigating class imbalance in sentiment analysis through gpt-3-generated synthetic sentences. \u003cem\u003eApplied Sciences\u003c/em\u003e, 13(17):9766, 2023.\u003c/li\u003e\n\u003cli\u003eJiaheng Hua, Xiaodong Cui, Xianghua Li, Keke Tang, and Peican Zhu. Multimodal fake news detection through data augmentation-based contrastive learning. \u003cem\u003eApplied Soft Computing\u003c/em\u003e, 136:110125, 2023.\u003c/li\u003e\n\u003cli\u003eLucas Francisco Amaral Orosco Pellicer, Taynan Maier Ferreira, and Anna Helena Reali Costa. Data augmentation techniques in natural language processing. \u003cem\u003eApplied Soft Computing\u003c/em\u003e, 132:109803, 2023.\u003c/li\u003e\n\u003cli\u003eHugo Queiroz Abonizio, Emerson Cabrera Paraiso, and Sylvio Barbon. Toward text data augmentation for sentiment analysis. \u003cem\u003eIEEE Transactions on Artificial Intelligence\u003c/em\u003e, 3(5):657\u0026ndash;668, 2021.\u003c/li\u003e\n\u003cli\u003eYu Li, Xiao Li, Yating Yang, and Rui Dong. A diverse data augmentation strategy for low-resource neural machine translation. \u003cem\u003eInformation\u003c/em\u003e, 11(5):255, 2020.\u003c/li\u003e\n\u003cli\u003eHongzheng Li, Jiu Sha, and Can Shi. Revisiting back-translation for low-resource machine translation between chinese and vietnamese. \u003cem\u003eIEEE Access\u003c/em\u003e, 8:119931\u0026ndash;119939, 2020.\u003c/li\u003e\n\u003cli\u003eRanto Sawai, Incheon Paik, and Ayato Kuwana. Sentence augmentation for language translation using gpt-2. \u003cem\u003eElectronics\u003c/em\u003e, 10(24):3082, 2021.\u003c/li\u003e\n\u003cli\u003eAbdelrahman I Saad. Opinion mining on us airline twitter data using machine learning techniques. In \u003cem\u003e2020 16th international computer engineering conference (ICENCO)\u003c/em\u003e, pages 59\u0026ndash;63. IEEE, 2020.\u003c/li\u003e\n\u003cli\u003eMrityunjay Singh, Amit Kumar Jakhar, and Shivam Pandey. Sentiment analysis on the impact of coronavirus in social life using the bert model. \u003cem\u003eSocial Network Analysis and Mining\u003c/em\u003e, 11(1):33, 2021.\u003c/li\u003e\n\u003cli\u003eNassera Habbat, Hicham Nouri, Houda Anoun, and Larbi Hassouni. Sentiment analysis of imbalanced datasets using bert and ensemble stacking for deep learning. \u003cem\u003eEngineering Applications of Artificial Intelligence\u003c/em\u003e, 126:106999, 2023.\u003c/li\u003e\n\u003cli\u003eKian Long Tan, Chin Poo Lee, Kalaiarasi Sonai Muthu Anbananthen, and Kian Ming Lim. Roberta-lstm: a hybrid model for sentiment analysis with transformer and recurrent neural net- work. \u003cem\u003eIEEE Access\u003c/em\u003e, 10:21517\u0026ndash;21525, 2022.\u003c/li\u003e\n\u003cli\u003eKian Long Tan, Chin Poo Lee, Kian Ming Lim, and Kalaiarasi Sonai Muthu Anbananthen. Sentiment analysis with ensemble hybrid deep learning model. \u003cem\u003eIEEE Access\u003c/em\u003e, 10:103694\u0026ndash;103704, 2022.\u003c/li\u003e\n\u003cli\u003eKian Long Tan, Chin Poo Lee, and Kian Ming Lim. Roberta-gru: A hybrid deep learning model for enhanced sentiment analysis. \u003cem\u003eApplied Sciences\u003c/em\u003e, 13(6):3915, 2023.\u003c/li\u003e\n\u003cli\u003eDiefan Lin, Yi Wen, Weishi Wang, and Yan Su. Enhanced sentiment intensity regression through lora fine-tuning on llama 3. \u003cem\u003eIEEE Access\u003c/em\u003e, 2024.\u003c/li\u003e\n\u003cli\u003eJos\u0026acute;e Carlos Ferreira Neto, Denilson Alves Pereira, Bruno Henrique Groenner Barbosa, and Dan- ton Diego Ferreira. Approaches based on language models for aspect extraction for sentiment analysis in the Portuguese language. \u003cem\u003eNeural Computing and Applications\u003c/em\u003e, pages 1\u0026ndash;11, 2024.\u003c/li\u003e\n\u003cli\u003eSayanta Paul and Sriparna Saha. Cyberbert: Bert for cyberbullying identification: Bert for cyberbullying identification. \u003cem\u003eMultimedia Systems\u003c/em\u003e, 28(6):1897\u0026ndash;1904, 2022.\u003c/li\u003e\n\u003cli\u003eCrowdFlower. Twitter US Airline Sentiment. https://www.kaggle.com/datasets/ crowdflower/twitter-airline-sentiment, 2015.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Sentiment Analysis, Imbalance Classes, Annotation, DistilBERT, Low-Rank Fine-Tuning","lastPublishedDoi":"10.21203/rs.3.rs-5879286/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5879286/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThis paper proposes a novel approach to sentiment analysis of imbalanced datasets, focusing on data augmentation and efficient fine-tuning. We address the challenge of limited minority class representation by leveraging GPT-4 to generate synthetic tweets via paraphrasing and back- translation (using Italian as an intermediary language). Furthermore, the main contribution is that we utilize GPT-4 to annotate tweets with positive reasons, derived by inverting the ten predefined negative categories within the dataset. The augmented dataset trains a DistilBERT model for sentence embeddings, and Low-Rank Adaptation (LoRA) enables efficient fine-tuning. A SoftMax layer provides classification into positive, neutral, and negative sentiments. Experiments on the Twitter US Airline Sentiment dataset demonstrate our approach\u0026rsquo;s efficacy, achieving 100% accuracy with minimal training time, highlighting the importance of data augmentation and efficient fine-tuning for robust sentiment analysis, particularly with imbalanced datasets.\u003c/p\u003e","manuscriptTitle":"Sentiment Analysis of Imbalanced Dataset through Data Augmentation and Generative Annotation using DistilBERT and Low-Rank Fine-Tuning","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-01-28 09:22:55","doi":"10.21203/rs.3.rs-5879286/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"8c72b3f0-1c27-4eee-b5b4-3b711c88e3d6","owner":[],"postedDate":"January 28th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-03-08T13:38:21+00:00","versionOfRecord":[],"versionCreatedAt":"2025-01-28 09:22:55","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5879286","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5879286","identity":"rs-5879286","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}