An Evaluation Framework for Dialectal Sentiment Classification and Linguistic Phenomena in Large Language Models

An Evaluation Framework for Dialectal Sentiment Classification and Linguistic Phenomena in Large Language Models | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article An Evaluation Framework for Dialectal Sentiment Classification and Linguistic Phenomena in Large Language Models Tarek Rashed, Ramadan Alfared, Abduelbaset Goweder, Husien Alhammi, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8419385/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Social media platforms provide individuals with a seamless way to share their opinions and interests using informal wording, creative spellings, local idioms, and frequent code-switching. This informal nature adds a layer of significant complexity to sentiment classification tasks. Recently, Large Language Models (LLMs) have shown promising capabilities in this area; however, previous research still lacks replicable and consistent evaluation protocols for assessing how these models reach their inferences, handle non-literal language, or explain the reasoning behind their decisions. To bridge this gap, this study introduces the Dialectal Sentiment Classification and Linguistic Phenomena (DSCLP) Framework. DSCLP is a four-phase protocol designed to analyze sentiment classification outcomes across one or more LLMs. The study applied DSCLP to a dataset of 1,469 Libyan-dialect social media posts, incorporating "Sarcastic" and "Ambiguous" as auxiliary labels alongside traditional sentiment categories. Two Generative Pre-trained Transformer (GPT) models, GPT-4o-mini and Gemini-1.5-flash, were examined under two prompting instructions: Model Default Inference (MDI) and Dialect-Aware Inference (DAI). Through the models’ APIs, each model produced a sentiment label and a rational explanation for every sentence in the dataset. The experiments showed that both models achieved moderate performance under the two defined prompting conditions. When the LLMs was instructed under MDI, the models showed biases toward Modern Standard Arabic (MSA) interpretations and still struggle with idioms, figurative language, and dual-sentiment expressions. Analysis of the generated rationale sentences revealed that LLMs frequently relied on literal understanding rather than cultural and contextual meanings in Libyan Arabic. By integrating performance metrics (Macro-F1, per-class F1, weighted-F1, and Cohen’s κ), rationale evaluation, and bias analysis, the DSCLP protocol demonstrated that it is a practical and reproducible method for studying LLM behavior in low-resource dialect settings. Future work may apply the protocol to additional dialects and explore model fine-tuning. sentiment classification low-resource dialects Arabic dialects large language models linguistic phenomena Libyan dialect bias detection Figures Figure 1 1. Introduction On social media platforms, individuals commonly use their dialectal language to express their feelings and thoughts. While dialectal content is quite widespread on social media platforms, formal or official language is more common in professional and institutional domains (Habberrih & Abuzaraida, 2024 ; Pham et al., 2024 ). Arabic dialects add further challenges to sentiment classification approaches, as they are written in an informal way with rich morphology, frequent code-switching, and culturally grounded idioms. Moreover, Arabic dialects’ vocabulary and sentence structure are greatly different from those of Modern Standard Arabic (MSA). Thus, sentiment classifiers that are trained mainly on MSA datasets often fail to recognize dialectal expressions, which leads to misclassifications and linguistic biases (Abugharsa, 2021; Alakrot et al., 2024 ; Rokaya & Ghiduk, 2019 ). In practice, sentiment misclassifications can affect the quality of decision-making systems that depend on sentiment analysis (Aguilar-Moreno et al., 2024 ; Kauffmann et al., 2019 ). For this reason, accurate sentiment classification for dialects such as Libyan Arabic is important for reliable interpretation in areas like marketing and decision-making. Large Language Models (LLMs) based on Generative Pre-trained Transformer architectures (GPT) are one of the most powerful technological innovations in Natural Language Processing (NLP). In the sentiment classification aspect, the models that are implemented based on the GPT architecture have not only far surpassed earlier lexicon-based and traditional machine-learning approaches, but they have also outperformed earlier deep learning models (Al-Thubaity et al., 2023 ; Hannani et al., 2024 ; Shi & Agrawal, 2025 ). Although GPT-based LLMs are not designed to classify textual contents as traditional classifiers do, its GPT can simulate classification behavior because it has been trained on massive data that encode semantic, pragmatic, and affective patterns. Through the use of LLMs’ APIs, these models can be utilized to perform sentiment classification inference. LLM can be instructed textually to activate the model’s reasoning and inference and generate textual comprehensive explanations for each inference it produces. Within the context of sentiment classification using LLMs, this approach is mainly employed to explain in more details how the model constructs its sentiment classifications. In this study, the ChatGPT-4.mini (OpenAI) and Gemini-2.5-flash (Google DeepMind) were instructed to classify the Libyan sentiment expressions into five categories using two types of prompt instructions; Default Model Inference (DMI) and Dialect-Aware Inference (DAI), and to produce a rationale sentence for each sentiment prediction. The following sections describe the proposed methodology. 1.2 Problem and Research Gap The binary method used in earlier classification approaches only allowed sentiment expressions to be classified as either positive or negative. This often leads to incorrectly classifying factual or neutral sentences as positive or negative sentiment (Mäntylä et al., 2018 ; Pang et al., 2002 ). Even with modern sentiment classification models, which rely on three classes (Positive, Negative, Neutral), current models still fail to handle linguistic phenomena inherent in human communication, where individuals usually express emotions in indirect or non-literal ways, such as sarcasm and ambiguity (Bhargava et al., 2025 ). Sarcasm occurs when individuals prefer to express their opinions in the opposite way of what they truly mean. This is often expressed in the form of humor, criticism, or irony, where sarcastic expressions frequently combine appreciation and irony within the same sentence. In the context of social media, sarcastic sentences are often expressed through cultural idioms, exaggeration, punctuation, and emojis (Alakrot et al., 2024 ; Kaseb & Farouk, 2022 ). Ambiguity occurs when individuals, within the same sentence, use words that have more than one possible meaning or have mixed polarities, making the intended message uncertain. This phenomenon is common in low-resource languages and arises from overlapping meanings, code-switching between languages, and context-specific expressions, particularly in dialectal and informal language use (Abeysiriwardana & Sumanathilaka, 2024 ; Hamed et al., 2025 ; Khalilia et al., 2024 ; Kim et al., 2023 ; Liu et al., 2023 ). In natural languages, sarcasm and ambiguity are key features of linguistic phenomena, as they add richness and complexity to communication. Sentences that fall under sarcasm and ambiguity remain among the major causes of misclassification in sentiment analysis tasks. Although these linguistic phenomena appear naturally and frequently, particularly on social media, most sentiment analysis approaches still treat sarcastic and ambiguous sentences as noise rather than meaningful linguistic features. This practice leads to frequent misclassification and impurity of traditional labels, especially in dialectal and informal Arabic. Recent studies that use LLMs for sentiment classification often lack clear and replicable evaluation procedures. Most studies focused on overall accuracy without considering sarcasm and ambiguity features or enabling LLMs to generate rationale sentences for the sake of understanding model predictions. These studies emphasize overall results without examining how models handle sarcastic or ambiguous sentences. Although several studies (Fleisig et al., 2024 ; Hovy & Prabhumoye, 2021 ) have stated that linguistic bias is one of the major sources of misclassification, no research has identified exactly the source of bias, whether a particular word is not present at all in a dataset during training or it is present but in a different context and meaning. Identifying whether a sentence is biased toward a particular language or dialect requires comparative analysis (e.g., a formal language versus its dialect, or dialect A versus dialect B). There is a need for a structured framework that examines classification accuracy, rationale quality, and dialectal linguistic bias, which are issues this study directly addresses. Understanding how LLMs classify dialectal text and explain their predictions is important for improving sentiment interpretation and reducing misclassification, particularly for sarcasm, ambiguity, and linguistic bias. 1.3 Research Questions and Objectives Addressing the existing knowledge gap, this study proposes the Dialectal Sentiment Classification and Linguistic Phenomena (DSCLP) Protocol to classify, evaluate LLM performance, and analyze LLM inferences on Libyan Arabic as a representative case. However, the proposed four-phase protocol is designed as a generalized methodology that can be applied to other dialects and languages. The protocol extends sentiment analysis beyond traditional sentiment labels to include Sarcastic and Ambiguous labels as related linguistic phenomena influencing sentiment interpretation. According to the above, the study addresses the following research questions: RQ1. How accurately do ChatGPT-4-mini and Gemini-2.5-flash classify sentiment in Libyan Arabic across Positive, Negative, Neutral, Sarcastic, and Ambiguous in Libyan Arabic? RQ2. How do Model Default Inference (DMI) and Dialect-Aware Inference (DAI) Rules, affect sentiment classification results, and the quality of models’ rationales? RQ3. Do LLMs show dialectal or cultural bias when interpreting sentiment in Libyan Arabic, and can dialect-aware instructions help reveal such bias? To answer the study’s research questions, a 4-phase protocol combining guided zero-shot inference, human validation, and comprehensive evaluation metrics (Macro-F1, Per-class precision, recall, and F1, Cohen’s κ, and confusion analysis) is implemented. This study developed a clear and reproducible methodology for evaluating LLM inferences in dialectal sentiment tasks. In addition, it supports fair and reliable evaluation across different dialects. The framework incorporates four phases to evaluate classification accuracy, review rationale quality, and analyze linguistic bias under DMI and DAI. 2. Related work Over the past few years, studies on Arabic sentiment and its dialect analysis have evolved across several approaches, starting with lexicon-based methods, followed by machine learning and deep learning approaches, and most recently LLMs and fine-tuned transformers. The following sections provide an overview of these approaches. 2.1 Lexicon-Based Sentiment Analysis in Arabic Lexicon-based sentiment analysis was an early approach used to analyze sentiment in Arabic and its dialect texts. Most studies on lexicon-based sentiment analysis were conducted in comparison with other techniques, such as machine learning algorithms. Frequent misclassifications were the most common issue of this approach due to the use of predefined sentiment lexicons to infer polarity, where each word is assigned a fixed sentiment score or label (e.g., positive, negative, neutral). For example, For example, Abdulla et al. ( 2013 ) developed a sentiment lexicon for MSA and incorporated some Jordanian and Gulf dialects. The study reported challenges in achieving reliable sentiment classification for Arabic due to its morphological richness and dialectal diversity. Similarly, Al-Ayyoub et al. ( 2015 ) constructed a large Arabic sentiment lexicon for analyzing Arabic tweets. The study confirmed that customized lexicons are an effective technique in informal and online social media environments, where textual content mainly appears as short and messy, is rich in idioms, and is created by everyday people. The method showed when dealing with sarcasm and in cases where feeling were not stated directly. Rokaya and Ghiduk ( 2019 ) addressed the limitations of lexicon-based sentiment misclassification by proposing an adaptive Arabic lexicon learning approach. The method employed genetic algorithms to automatically build and optimize Arabic sentiment lexicons. The study stated that the learning approach can handle two major dialectal challenges, namely bias and ambiguity. 3.1 Traditional ML and Deep Learning Approaches Many researchers transitioned from lexicon-based methods to explore the potential of traditional machine learning and deep learning algorithms. Abu Kwaik et al. ( 2019 ) introduced a hybrid LSTM-CNN model for dialectal Arabic sentiment analysis. The model was trained on the ASTD dataset and a dialectal dataset built by the authors. Hybrid deep learning worked better than single models for sentiment analysis of dialectal Arabic. The study showed promising accuracy in handling noisy, informal language. It also identified challenges related to variations in wording across dialects and difficulty with ambiguous or sarcastic sentiments, where sentences with unclear or sarcastic meaning often confused the model. In parallel, Farha and Magdy ( 2020 ) compared SVM, CNN, and BiLSTM on three sentiment datasets: ASTD (tweets in MSA and dialects), LABR (book reviews in MSA), and ArSenTD-Lev (tweets in the Levantine dialect), where the datasets cover MSA and dialect-focused content (Egyptian, Gulf, Levantine, and North African). The comparison identified that BiLSTM achieved the best performance due to its ability to capture context from both directions, which is a key factor in sentiment understanding. In a related work, Farha and Magdy ( 2020 ) directly addressed the issue of sarcasm to cover sarcasm detection using a sarcasm-labeled Twitter dataset. Their results showed that sarcasm reduced accuracy and made sentiment classification more difficult across all models, especially in dialects, confirming that sarcasm is a key source of sentiment ambiguity. In the context of Libyan Arabic dialects, Abugharsa (2021) examined sentiment in Libyan–Misurata dialect poetry using traditional ML and CNNs. The study stated that metaphor, symbolism, and idiomatic expressions often lead to ambiguity and misclassification. The study demonstrated that culturally embedded and non-literal expressions were the key issues causing deep learning (CNN) to fail to achieve competitive accuracy in comparison with traditional ML. The study primarily concerned with the role of dialects in introducing bias and sentiment ambiguity. This reinforces the need to test advanced models on their ability to handle dialectal nuance, sarcasm, and cultural bias. Similarly, focusing on Libyan Arabic sarcasm, Alakrot et al. ( 2024 ) focused on sarcasm detection in Libyan dialects using multiple approaches, including traditional machine learning classifiers (SVM, Naïve Bayes), a BiLSTM network with pre-trained Arabic embedding and a fine-tuned transformer model (AraBERT). The authors built a Libyan Arabic dialect sarcasm-labeled corpus from Twitter and Facebook data. AraBERT outperformed both BiLSTM and traditional models in detecting sarcasm in Libyan dialects. The study reached a conclusion consistent with prior research that sarcasm expressions a major obstacle to achieving high accuracy in sentiment analysis, as sarcasm is often expressed through cultural idioms, exaggeration, and punctuation/emojis rather than straightforward textual statements. 2.3 LLMs and Fine-Tuned Transformers More recently, researchers have turned to Large Language Models (LLMs) and fine-tuned transformers to address the limitations of earlier approaches in sentiment analysis. Multiple studies assessed the performance of LLMs, when applied to Arabic sentiment analysisAl-Thubaity et al. ( 2023 ) evaluated s GPT-3.5, GPT-4, and Bard powered by PaLM 2 on Arabic sentiment analysis across MSA and multiple dialects, including Levantine, Egyptian, and Gulf Arabic. The study found that while both models achieved reasonable accuracy on MSA, their performance dropped significantly when applied to dialectal texts, especially if the text sarcastic expressions. The study revealed that the LLMs used had biases toward standard or formal Arabic, which negatively affected the sentiments classification process. In a related evaluation, Khondaker et al. ( 2023 ) introduced GPTAraEval, a benchmark framework for evaluating ChatGPT (GPT-3.5-turbo version) on Arabic NLP tasks, including sentiment analysis, text classification, named entity recognition, and question answering. The authors found that GPT-3.5 performed competitively on MSA tasks, while it failed in dialectal settings and struggled with ambiguous sentiment cases, especially when sarcasm or context-dependent expressions were involved, where both require cultural knowledge or contextual understanding. Moreover, the GPT-3.5 showed biases toward MSA, where for a given sentence the responses were unstable across different prompts. In contrast to using general-purpose LLMs, AraBERT and other specialized models perform better on Arabic dialects. Hannani et al. ( 2024 ) compared ChatGPT-4 with fine-tuned AraBERT and traditional ML models on Moroccan Arabic. AraBERT outperformed ChatGPT-4, particularly in handling idiomatic expressions and code-switching. The study reported that ChatGPT often misclassified idiomatic expressions as negative sentiment rather than recognizing their neutral or figurative meaning. In addition, ChatGPT struggled with sentences that mixed Moroccan Arabic with French or English. Similarly, Haddad et al. ( 2023 ) introduced TunBERT, the first BERT-based transformer model specifically intended for the Tunisian dialect. For the sake of tackling the lack of sufficient datasets for the Tunisian dialect, the authors constructed 500K social media and blog sentences so the model can learn better, show less bias, and handle sarcasm and ambiguity more effectively. TunBERT-P (PyTorch) and TunBERT-T (TensorFlow) were two model variations implemented. These models were fine-tuned and evaluated on real-world NLP applications (downstream tasks), including sentiment analysis, dialect identification, and reading comprehension. TunBERT showed competitive results for Tunisian dialect sentiment analysis and dialect identification in comparison with AraBERT, mBERT, and MARBERT. Importantly, the study confirmed that specialized models (trained only on one dialect) can surpass larger multilingual LLMs, particularly in capturing sarcasm sentences in dialectal expressions. These results directly the need for dialect-aware models in Libyan Arabic sentiment analysis, where resource scarcity, ambiguity, and sarcasm similarly challenge the reliability of general-purpose LLMs. In a related attempt to enhance LLM performance, recent research has explored how model fine-tuning, data augmentation, and sarcasm mitigation techniques can enhance LLM performance in sentiment analysis, Bhargava et al. ( 2025 ) conducted several experiments to examine how sarcasm, paraphrasing, and emoji-related sentences affects the performance of LLMs. For each target LLM, two fine-tuning versions were developed using datasets containing both sarcastic and non-sarcastic tweet sentences. The first version was trained on the complete dataset, while the second version was trained on the same dataset after removing or paraphrasing the sarcastic text. The results indicated noticeable improvements in performance when sarcastic sentences were removed or paraphrased. In similar direction, Alsharbi ( 2025 ) proposed a strategy to optimize existing state-of-the-art pre-trained LLMs through data augmentation and fine-tuning. The approach was evaluated on Saudi dialect data using the Multilingual Arabic Dialect (MADAR) dataset on classification tasks. The MADAR includes Tunisian, Egyptian, Levantine, and Gulf. The study demonstrated that the fine-tuned model with data augmentation outperformed MSA-trained baselines. However, the study reported several challenges that complicate sentiment analysis, including biases toward MSA. In addition, the study noted that the model struggled with sarcastic expressions, since sarcasm often relies on cultural idioms, exaggeration, and emojis that confuse language models. Ambiguity is also prevalent, as dialectal expressions may carry multiple meanings depending on social or contextual cues. In an another approach, Khaled et al. ( 2024 ) proposed the Retrieval-Augmented Generation (RAG) approach to improve the performance of generative LLMs for Arabic sentiment analysis. The study evaluated LLaMA-3, Gemini-1.0-pro, and Ace-GPT on three datasets including ASAD, ArSarcasm-v2, and SemEval. The results showed that LLMs on their own struggled with dataset imbalance and neutral sentiment labels, leading to frequent misclassifications, while RAG mitigated hallucinations by incorporating external knowledge. However, the study emphasized that dataset imbalance and resource scarcity remain unresolved obstacles. Likewise, Bellaouar et al. ( 2025 ) introduced Dhati+, which is pre-trained LLMs models, consisting of XLM-RoBERTa, AraBERT, and ArabianGPT and improved by ensemble approach. The authors addressed data scarcity and imbalance datasets by constructing AraDhati+, a balanced dataset of approximately 78K instances derived from ASTD, LABR, HARD, and SANAD. The results showed that the ensemble approach significantly outperformed LLMs on their own. However, the study noted persistent challenges in handling mixed tweets, where personal opinion and factual expressions appear together, as well as short posts lacking sufficient context. 3. Research Methodology In this study, a 4-phase protocol for analyzing dialectal sentiment and related phenomena classification on LLMs is implemented, aiming at evaluating how LLM classify textual sentiment and related phenomena in dialectal expressions. In the case of comparison between two more LLMs, the 4-phase protocol is applied on each LLM. Unlike traditional sentiment classification labels, which are concepts inherently recognized by LLMs, this methodology adopts a guided zero-shot technique. This approach provides concise definitions for only two specific linguistic phenomena—Sarcastic and Ambiguous—allowing the LLM to interpret the linguistic context across all five defined labels. The primary focus of this study is on Libyan Arabic. The protocol can be generalized and applied to other dialects and languages. The four phases together provide methodological process to assess LLM robustness in handling sarcastic and ambiguous sentiment, and linguistic bias within dialectal contexts. The next subsections provide detailed descriptions of the methodology’s components. 3.1 The Framework for Dialectal Sentiment Classification and Linguistic Phenomena Give human- annotators labeled sentiment dialectal dataset, which is a benchmark gold standard dataset. Only sentiment sentences (no label) are processed through the selected LLM standardized API-based prompts. The protocol integrates guided zero-shot classification (presenting few definition for the related-phenomenon classification), human validation, evaluation metrics, and comparative analysis to ensure a comprehensive assessment of LLM inferences. The model produce a structured format consisting of: - Labels : (Positive, Negative, Neutral, Sarcastic, and Ambiguous) - Rationale : (1–2 sentences justifying classification cues). Phase 1: Guided Zero-Shot LLM Classification under Default Model Inference Behavior The LLM is instructed to generate classification label for sentiment sentences based on its default interpretation behavior, without mention to any language or dialect, leaving LLM rely on its latent knowledge in inferring classification label. Phase 2: Extracting Human– LLM Classification Disagreement Cases Based on the Phase 1 output, all samples showing label mismatch between human annotations and model-generated labels are isolated for further analysis. Phase 3: Guided Zero-Shot LLM Classification under Dialect-Aware Inference Instructions Using the Phase 2 isolated sentences, the LLM is instructed to generate classification label for expressions based on the dialectal type, intended meaning, and interpretation, including, idioms, figurative language, and cultural semantics. Phase 4: LLM Accuracy, Performance Evaluation and Comparative Analysis The evaluation applies the widely accepted classification metrics used in evaluate machine learning classifiers, including per-class precision, recall, F1-based metrics which include Macro-F1 and Weighted-F1, Cohen’s Kappa. The comparative analysis examines how the two rule conditions impact the LLM’s decisions in detecting Positive, Negative, Neutral, Sarcastic, and Ambiguous sentences, as well as in detecting expressions that contain linguistic biases. Special attention is directed toward NLP challenging such as idiomatic expressions, and code-switching. The Four-Phase Protocol is illustrated in Fig. 1, which summarizes the phases workflow applied in this study. 4. Materials and Experimental Setup The methodology integrates dataset description, model prompting rules, computational environment, and the four-phase evaluation protocol. The following subsections provide detailed information about the methodology’s elements. 4.1 Preliminary Study A preliminary pilot study was conducted to test the procedures, instructions, dataset reliability, and feasibility, as well as to identify potential methodological issues prior to conducting the main experiments. First, several versions of LLM instruction were tested and refined, until reach the final refined version, which is described in section 4.4 . Second, it was observed that many users’ social-media comments frequently included extensive use of emojis across all sentiment labels. In many cases, these emojis did not align with the textual content and sometimes conveyed an opposite emotional signal or introduced misleading cues that were not intended by the writer. The pilot study also revealed that the presence of emojis reduced the performance of the LLMs, as the models tended to interpret emojis as textual content. To prevent this distortion and ensure that the evaluation focused entirely on textual understanding, all emojis were removed, allowing the LLMs to classify the textual content only, without interference from emoji-based emotional cues. 4.2 Human Annotators and arrival to agreements The study recruited two linguistic experts to label 1,469 sentences. Out of the total dataset, the two annotators reached label agreement on 1,416 samples. The remaining 53 samples constituted the disagreement subset. Disagreement cases were resolved by consulting a third linguistic expert, who assigned the final labels. 4.3 Dataset Description In this study, the primary source of the dataset is that of Alhammi et al. ( 2018 ). Additional social media expressions were collected from Facebook and Twitter, and then incorporated into the dataset, particularly sentences that might be considered (after review by the annotators) as related phenomena of classification, such as sarcastic or ambiguous sentences. During collecting data, the text data were preserved (as-is), including idioms, figurative expressions, semantics, code-switching, and both short and long sentences, and typos, etc., in their original form except for the removal of emoji characters, where this study focuses on classifying textual content. As a result, the final dataset consists of 1469 sentences, reflecting the diversity of sentences across both traditional sentiment and related phenomena labels, and it covers users’ sentiments across several domains such as Phones, Cars, Shopping, Communications, News, General Life, Religion, and Sports. Although the number of sentences of each label is not balanced and might produce skewed or unreal accuracy, macro and weighted F1 will be calculated to address the issue of unbalanced datasets. Also, the size of dataset is relatively small, the purpose is not to use the sentences and their associated labels as normally done in training procedure or fine-tuning with LLMs. Instead, the purpose is to hide the labels and provide only the sentences to the LLM, allowing the pre-trained model to generate the classification labels directly. Table 1 presents the distribution of sentiment labels in the dataset. Table 1 Sentiment Label Distribution Label Name Count Positive 233 Negative 374 Neutral 324 Sarcastic 313 Ambiguous 225 Total 1469 4.4 LLM Instructions and Rules In order to conduct comprehensive analysis of traditional sentiment and related-phenomena classification on LLMs, the LLM was accessed through its official API to classify sentiments using two distinct sets of system instructions. Firstly, the MDI instructed LLM to classify each sentence using its default interpretation behavior without indication to any specific language or dialect. The next text shows the instructions given to the LLM:- SYSTEM_INSTRUCTION = ( "Your task is to classify the sentiment of the given text according to your model's default interpretation. Classify each text into exactly one of these labels: "Positive, Negative, Neutral, Sarcastic, Ambiguous" "Notes: - 'Sarcastic' covers irony; mocking; ridicule; or a satirical tone; Deliberate ironic appreciation; or the intentional use of positive words to express criticism. - 'Ambiguous' covers hesitant or uncertain emotion; vague or indirect expression without mockery; mixed polarity without intent; or unclear meaning from code-switching." Secondly, the DAI instructed LLM to classify each Arabic sentence according to the Libyan Arabic dialect. The following instructions were given to the LLM: SYSTEM_INSTRUCTION = ( "Your task is to classify the sentiment of the given text according to your model's default interpretation. Classify each text into exactly one of these labels: "Positive, Negative, Neutral, Sarcastic, Ambiguous" "Notes: - 'Sarcastic' covers irony; mocking; ridicule; or a satirical tone; Deliberate ironic appreciation; or the intentional use of positive words to express criticism. - 'Ambiguous' covers hesitant or uncertain emotion; vague or indirect expression without mockery; mixed polarity without intent; or unclear meaning from code-switching." "DIALECT-SPECIFIC RULES\n" "- Recognize Libyan tone markers and pragmatic cues.\n" "- Contrastive particles or laughter may signal sarcasm or implicit criticism.\n" "- Code-switching can intensify, mock, or blur sentiment.\n" "- Indirect politeness or understatement may imply negativity.\n" "- Humor, exaggeration, repetition, or exclamation indicate emotional emphasis.\n" "- Prioritize pragmatic and cultural cues over literal polarity.\n\n" The outputs of both defined rules were saved in CSV format, including the assigned label (Positive, Negative, Neutral, Sarcastic, or Ambiguous), and one or two sentences providing a rationale explanation for the assigned label. The following instructions were given: "OUTPUT FORMAT\n" "Return valid JSON:\n" "{ \"label\": \"Positive|Negative|Neutral|Sarcastic|Ambiguous\", " "\"rationale\": \"1–2 sentences explaining the Libyan-specific cues that justify the label.\" }" 4.5 Experimental Setup The experimental evaluation used GPT-4o-mini and Gemini-2.5-flash. Both models were accessed through their official APIs between May and Dec 2025 to maintain stable conditions and allow the researchers to repeat the experiments under the same instructions given and obtain the same results. Each model was tested on the Libyan dataset under both DMI and DAI Rules. All experiments were conducted in Python 3.9 on a Windows 11 workstation, using API standard libraries. In all experiments, the temperature sat to 0. 5. Results The study followed 4-phase Protocol to perform text classification task on Libyan dialect using GPT-4o-mini and Gemini-2.5-flash. Each model classified Libyan-dialect textual sentiment sentences under two distinct prompting conditions. Under DMI, the LLM operates according to its internal reasoning without mention to any dialect-specific guidance, and under DAI rules, the LLM is explicitly instructed to classify the Arabic text with specific awareness of Libyan dialect and consider idioms, and cultural expressions. Four experiments were conducted, one for each combination of model and instruction condition. 1- Sample Results of LLM Classifications under Different Prompting Rules This section presents the classification results for the LLMs evaluated in this study. Table 2 displays the classification of two distinct texts using GPT-4o-mini and Gemini-2.5-flash under MDI rules, while Table 3 provides the classification for the same sentences using Libyan DAI rules. Each representative sentence is presented alongside its human-annotated label, the model’s classification inference, and the associated rationale. The tables below use the following abbreviations: Text : the original Libyan Arabic sentence Ano_Lbl : Annotator Label LLM_Lbl : LLM Inference Label Rationale Sentence : Rationale Sentence produced by the LLM Sentiment categories are abbreviated as: Pos = Positive, Neg = Negative, Neu = Neutral, Sar = Sarcastic, Amb = Ambiguous. Table 2 Sample LLM Classification Outputs under MDI Text Ano_Lbl LLM_Lbl Rationale Sentence ChatGPT: Model Default Inference Mode السعر مشيط و الناس تشري. Neg Pos The phrase indicates that the price is appealing and that people are buying, suggesting a positive sentiment towards the pricing and consumer behavior. يا سلام يا راجل<< عالمي لكن يعطّلك يا لطيف أكثر مما يساعدك Sar Amb The text expresses a mix of admiration and frustration, making it unclear whether the overall sentiment is positive or negative. The use of 'يا سلام يا راجل' suggests appreciation, but the following phrase indicates a sense of annoyance. Gemini: Model Default Inference Mode السعر مشيط و الناس تشري. Neg Pos The text states that the price is 'good' or 'attractive' ('مشيط') and people are buying, which are both positive indicators of market activity. يا سلام يا راجل<< عالمي لكن يعطّلك يا لطيف أكثر مما يساعدك Sar Neg The text starts with an exclamation and a positive descriptor ('world-class'), but immediately follows with 'but it hinders you' and 'more than it helps you', clearly indicating a negative overall assessment. Table 3 Sample LLM Classification Outputs under Libyan DAI ChatGPT: Libyan Dialect Aware Inference Mode السعر مشيط و الناس تشري. Neg Pos The phrase 'السعر مشيط' suggests that the price is good or favorable, and 'الناس تشري' indicates that people are buying, which implies a positive sentiment about the market situation. يا سلام يا راجل<< عالمي لكن يعطّلك يا لطيف أكثر مما يساعدك Sar Sar The phrase 'يا سلام يا راجل' is often used sarcastically to express disbelief or mock admiration, while 'يعطّلك أكثر مما يساعدك' indicates that the person is actually causing more trouble than help, reinforcing the ironic tone. Gemini: Libyan Dialect Aware Inference Mode السعر مشيط و الناس تشري. Neg Neg The term ""مشيط"" (mashīt) in Libyan Arabic unequivocally means ""expensive"" or ""exorbitant,"" carrying a strong negative connotation regarding cost. The phrase ""و الناس تشري"" (and people are buying) highlights a situation where consumers are forced to purchase despite high prices, implying economic hardship or a lack of alternatives, thus reinforcing the negative sentiment about the market conditions.", يا سلام يا راجل<< عالمي لكن يعطّلك يا لطيف أكثر مما يساعدك Sar Neg The text starts with an appreciative exclamation 'يا سلام يا راجل' and a positive descriptor 'عالمي' (world-class), but immediately shifts to a strong negative with the contrastive particle 'لكن' (but). The subsequent phrase 'يعطّلك يا لطيف أكثر مما يساعدك بوه' explicitly states that it hinders more than it helps, with 'يا لطيف' expressing distress and 'بوه' intensifying the negative impact in Libyan colloquialism.", 2. Confusion Matrixes and associated metrics The results of these experiments are summarized in two separate tables that report the confusion matrix across the five sentiment categories (Positive, Negative, Neutral, Sarcastic, and Ambiguous). Each table presents key evaluation metrics, including Precision, Recall, F1-score, Macro- F1 Average, Weighted F1, and Cohen’s Kappa, to provide a comprehensive view of model accuracy and reliability. Table 4 shows confusion matrix and related metrics for ChatGPT and Gemini under DMI while Table 5 shows confusion matrix and related metrics for ChatGPT and Gemini under Libyan DAI Mode. Table 4 Confusion matrix for ChatGPT and Gemini under Default Inference Mode ChatGPT: Default Inference Mode Ano_Lbl↓ / LLM_Lbl → Pos Neg Neu Sar Amb Precision Recall F1-Score Support Pos 140 8 29 14 42 0.62 0.60 0.61 233 Neg 9 210 54 33 68 0.73 0.56 0.63 374 Neu 45 15 178 9 77 0.54 0.54 0.54 324 Sar 12 31 3 195 72 0.60 0.62 0.61 313 Amb 17 20 25 17 146 0.42 0.64 0.51 225 Total 1469 Metric Accuracy 0.59 Macro F1 0.59 Weighted F1 0.60 Cohen’s Kappa 0.49 Gemini: Default Inference Mode Pos 160 9 18 15 31 0.68 0.68 0.68 233 Neg 17 230 41 29 57 0.73 0.61 0.66 374 Neu 27 13 201 6 77 0.59 0.62 0.60 324 Sar 11 26 2 271 3 0.86 0.86 0.73 313 Amb 16 23 17 17 152 0.64 0.31 0.42 225 1469 Metric Accuracy 0.63 Macro F1 0.60 Weighted F1 0.61 Cohen’s Kappa 0.50 Table 5 Confusion matrix for ChatGPT and Gemini under Libyan Dialect Aware Inference Mode ChatGPT: Libyan Dialect Aware Inference Mode Ano_Lbl↓ / LLM_Lbl → Pos Neg Neu Sar Amb Precision Recall F1-Score Support Pos 26 9 13 36 9 0.32 0.27 0.30 93 Neg 6 57 28 63 10 0.49 0.34 0.40 164 Neu 36 21 28 37 24 0.33 0.19 0.24 146 Sar 1 14 0 102 1 0.39 0.86 0.54 118 Amb 11 14 14 19 21 0.32 0.26 0.29 79 Total . 600 Metric Accuracy 0.39 Macro F1 0.35 Weighted F1 0.36 Cohen’s Kappa 0.23 Gemini: Libyan Dialect Aware Inference Mode Pos 27 6 8 14 1 0.23 0.48 0.31 56 Neg 6 47 8 30 3 0.28 0.50 0.36 94 Neu 63 56 31 36 4 0.52 0.16 0.24 190 Sar 0 4 0 38 0 0.23 0.90 0.37 42 Amb 21 53 12 44 24 0.75 0.15 0.25 154 536 Metric Accuracy 0.31 Macro F1 0.31 Weighted F1 0.28 Cohen’s Kappa 0.18 6. Discussion The results of this study provide a detailed comparative understanding of how GPT-4o-mini and Gemini-2.5-flash classify textual dialectal sentiments under two instruction modes: DMI (Phase 1, applied to the full dataset) and DAI (Phase 3, applied only to disagreement cases). 6.1 Comparative Performance Metrics of GPT-4o-mini and Gemini-2.5-flash Under the DMI (Phase 1), both models achieved moderate performance. Their accuracy and macro-F1 results were consistent with previous evaluations of LLMs on Arabic sentiment datasets(Al-Thubaity et al., 2023 ; Khondaker et al., 2023 ). Gemini-2.5.flash achieved higher overall scores, including better accuracy (0.63 vs. 0.59), a higher Cohen’s κ (0.50 vs. 0.49), and slightly stronger macro and weighted-F1 values. A closer look at the confusion matrices showed clearer differences between the two models. Gemini’s stronger overall performance came mainly from big performance gain in categories that have clear sentiment cues, especially sarcasm where the LLM was instructed with an abstract definition. Under abstract sarcasm definition. Gemini correctly classified 271 sarcastic cases compared to ChatGPT’s 195, achieving an F1-score of 0.73 versus 0.61. In contrast, GPT-4o-mini showed more similar Precision, Recall, and F1 values across multiple classes, indicating a more balanced distribution of predictions across categories. Gemini performed quite well when sentiment cues were explicit, but its performance dropped significantly for unclear or uncertain expressions. For example, its F1-score for the ambiguous class fell to 0.42, compared to 0.51 for GPT-4o-mini. On the other hand, GPT-4o-mini showed more consistent behavior across sentiment categories, suggesting a more uniform distribution of semantic reasoning. Similar behavior has been reported in earlier research on Arabic and dialectal sentiment analysis, where models tend to perform best when sentiment is clearly expressed but struggle when meaning is implied through indirectly figurative language or cultural nuance (Alakrot et al., 2024 ; Hannani et al., 2024 ) At the class level, GPT-4o-mini’s F1 scores stayed relatively close, ranging from 0.54 to 0.63. Gemini, however, showed a greater variation, with a high of 0.73 for sarcastic labels and a low of 0.42 for ambiguous labels. This indicated that Gemini’s inference are more sensitive to the presence or absence of clear sentiment cues. Overall, the results indicated various strengths. Gemini performed better when sentiment markers are clear, whereas GPT-4o-mini remained more stable when cues are weak, mixed, or indirect. Although Gemini achieved slightly higher overall metrics, the two models differ in how they interpret dialectal sentiment. The DAI instruction used in Phase 3 is not intended to improve model performance. Instead, it is used to examine how each model adjusts its reasoning when it is guided by Libyan DAI instructions and to identify linguistic patterns that lead to misclassification. Phase 3 focuses only on the subset of sentences for which human annotators and the LLMs initially disagreed in Phase 1. These disagreement cases can be considered the hardest samples in the dataset, as they are involved idiomatic expressions, figurative language, cultural pragmatics, or dialect-specific ambiguity. Under the DAI instructions, ChatGPT showed more stable inference behavior than Gemini. ChatGPT correctly classified 234 out of 600 samples (39%), while Gemini classified 167 out of 500 (33.4%). ChatGPT also achieved higher accuracy (0.39 vs. 0.31), macro-F1 (0.35 vs. 0.31), weighted-F1 (0.36 vs. 0.28), and Cohen’s κ (0.23 vs. 0.18). These results indicated that ChatGPT benefited more from dialect guidance when handling difficult cases. The confusion matrix supported this difference. ChatGPT’s showed strongest improvement in the sarcastic category, where it achieved an F1-score of 0.54 with 102 correct predictions. In comparison, Gemini achieved an F1-score of 0.37 with 38 correct predictions. In contrast, even with DAI instructions, Gemini showed greater variation across sentiment classes. Gemini’s recall for sarcastic category reached 0.90 and its precision was 0.23. This indicated excessive use of the sarcastic label, where many positive, negative, neutral, and ambiguous samples were incorrectly labeled as sarcastic label. Different behavior was also clear in the ambiguous label. Gemini achieved high precision and low recall compared to the more balanced performance of ChatGPT. This means that Gemini overlooked many ambiguous samples and assigned them to other sentiment categories. Gemini also struggled with the neutral class, where its recall fell to 0.16, despite the large number of neutral samples (190). This indicated ongoing challenges with subtle, mixed, or weak affective indicators. According to previous research, LLMs often struggle understanding indirect expressions, idiomatic wording, exaggeration, and pragmatic tone, especially when emotive indications are not clear (Bellaouar et al., 2025 ; Hannani et al., 2024 ). Overall, The results showed that ChatGPT adopts a more stable and adaptable inference strategy when DAI rule is applied, whereas Gemini performed better when sentiment cues are very clear but remains more sensitive to ambiguity and cultural complexity. While DAI prompting helped reduce some sources of biases, it can not fully address the limited exposure most LLMs have to Libyan Arabic during pre-training, particularly for models that rely heavily on explicit affective indicators. 6.2 Sentence-Level Insights into LLM Successes and Failures under Default Inference Behavior Examining the confusion matrices from Phase 1 along with examples of correct and incorrect predications helped clarify where the models performed well and where they struggled. In most correct cases, sentiment was expressed clearly and directly, without the use of idioms, sarcasm, exaggeration, metaphor, or culturally specific language. For example, consider the idiomatic positive expression “this coffee opens the soul,” which in Libyan Arabic is “القهوة هذه تفتح النفس.” Both ChatGPT and Gemini produced accurate rationale sentences that corresponded to the intended meaning, ChatGPT stated: “The phrase 'تفتح النفس' suggests that the coffee is uplifting or refreshing, indicating a positive sentiment toward it,”, while Gemini generated: “The phrase 'تفتح النفس' (taftah al-nafs) is an idiom meaning 'to cheer one up' or 'to stimulate the appetite/soul,' indicating a positive and pleasant effect of the coffee.” In contrast, most misclassified samples contained Libyan dialect words, reflecting the lack of Libyan pre-training data, or expressed their meaning in an indirect way. One example is a negative sentiment that was classified as Ambiguous by ChatGPT and as neutral by Gemini: “The place is so far,” expressed in the Libyan dialect as “المكان تيرو”. The word “تيرو” is an Italian term written in Arabic characters, refereeing to so far location. Both models produced rationale sentences that reveal their misunderstanding of this dialectal term. ChatGPT stated: “The text is vague and does not provide clear sentiment or context. It simply mentions a place without any emotional cues.” Similarly, Gemini explained: “The text 'المكان تيرو' (The place is Tero) is a purely descriptive statement. It identifies a location without expressing any explicit positive, negative, or emotional sentiment.” From these rationale sentences, it is evident that the models did not understand the meaning of “تيرو,” which indicates the absence of Libyan dialect vocabulary in their pre-training. This behavior is typically behavior of LLMs, which are primarily trained on large collections of publicly available text such as books and web pages that are mostly written in formal languages and non-dialectal language (Liu et al., 2023 ; Meyer, 2023). Even though the abstract definition of the sarcastic category was provided to both models, the two models showed clear differences in performance. Gemini performed particularly well on sarcastic sentences, achieving strong precision, recall, and F1 scores. For instance, compared with ChatGPT, the sentence “النت سريععععع زي مشية جدّاي” was correctly classified by Gemini, which also produced an accurate rationale: “The text uses 'fast' with exaggerated emphasis, but then compares it to 'my grandmother's walk,' which implies slowness. This ironic contrast indicates sarcasm.” On the other hand, ChatGPT failed to infer the correct prediction for the full sentence and stated: “The text expresses a positive sentiment about the speed of the internet, using enthusiastic language that suggests satisfaction.” It is clear from ChatGPT’s rationale that it focused only on the positive indicator “fast سريععععع” and interpreted the repeated letters “ععععع” as enthusiasm, while ignoring the rest of the sentence. These results indicated ongoing challenges for ChatGPT in interpreting sarcastic expressions. While the abstract definition of sarcastic sentiment helped the LLMs in inferring sarcastic sentences to some extent, it did not enhance their ability to detect ambiguous tones, as the ambiguous category remained the lowest-performing class. Many ambiguous expressions were interpreted as neutral or negative because the model focused too heavily on small clues, such as strong words or contrastive particles, while overlooking the overall uncertainty of the sentence. In general, the review of misclassified sentences indicated that these sentences did not contain any obvious clues, in other words, their examination did not reveal any consistent pattern or rule that could explain why the LLM failed to understand ambiguous expressions. However, a deeper analysis of these sentences showed that the ambiguous category actually provides an important function for the overall classification system, as it contained the majority of false-negative cases. Some sentences do not express a single, clear sentiment and instead contain mixed cues. For instance, “شنو منو تمام التمام. في الكلام، أما الواقع ما يمشيش”. This expression combines a positive expression ( “تمام التمام” ) with a negative one ( “الواقع ما يمشيش” ). When only three sentiment labels are used, sentences like this are easily misclassified, as they do not belong clearly to any single category. By including related linguistic phenomena, this issue is reduced and preserved the clarity of the traditional sentiment labels. This suggests that including related-phenomena labels keeps the traditional sentiment categories clearer and more semantically consistent. 6.3 Influence of Dialectal and Cultural Features on Model Behavior The DSCLP Protocol provides a reliable method for detecting linguistic bias by comparing each sentence’s inference label in Phase 3 with its corresponding inference in Phase 1. During sentence-level analysis under DMI, the study reached to a clear understanding that many Phase 1 misinterpretations errors were closely linked to the models’ limited familiarity with Libyan dialect usage. In particular, dialect-specific words, frequent code-switching, and local idiomatic. Words such as “مشيط”, “يعطّلك”, “الجو مليح”, “حشية”, and “تيرو” carry sentiment meanings that culturally understood in the Libyan dialect and cannot be accurately interpreted through literal pattern matching. Several errors showed that both ChatGPT-4 and Gemini-2.5 mapped dialectal expressions to their meanings in MSA or to unrelated dialects, showing a form of linguistic bias. For example, in the sentence “السعر مشيط و الناس تشري”, the accurate Libyan meaning is “the price is very expensive and people still buy, “However, both models classified it as Positive. Their rationales “the price is appealing” and “the price is 'good' or 'attractive' ('مشيط')” explain that both models relied on the MSA meaning of mashīt, which refers to something slender, stretched, or smoothly arranged. Once the models were instructed to interpret Libyan linguistic markers, idiomatic contrasts, and pragmatic tone, Gemini correctly inferred the negative sentiment. In another example, ChatGPT showed signs of linguistic bias during DAI instructions. For instance, ChatGPT labeled the sentence “مش يقولو طنجره و لقت مغطتها ولا حاجه زي هك لمهم” as ambiguous. In everyday Libyan usage, this expression is a familiar idiom used to describe negative behavior rather than uncertainty. After applying the DAI instructions, the model was able to revise its interpretation and assign a label that better reflected the intended cultural meaning. By reviewing some other sentences, the study revealed that in some cases, both models did not infer the correct label even with DAI instruction due to LLMs' linguistic bias. As previously shown in the sentence “السعر مشيط و الناس تشري”, ChatGPT has only one meaning for the word “mashīt,” which shows a lack of Libyan training data, while under Gemini, it shows linguistic bias toward dominant training data. These observations addressed the source of bias, which is due to the absence of dialect words or the fact that LLMs rely by default on dominant languages. Similar difficulties have been noted in the studies (Abugharsa, 2021; Haddad et al., 2023 ; Hannani et al., 2024 ), which report that figurative and idiomatic expressions in Arabic dialects often confuse models trained. By contrast, DAI instructions mitigated many of these errors by guiding the models toward culturally grounded tone markers (e.g., “حشية”, “تيرو”, “الجو حلو”), which often signal sarcasm or implicit criticism. This supports the conclusion that explicit dialectal grounding enhances model reasoning precisely in the categories where LLMs tend to struggle most. These results align with the findings reported by Haddad et al. ( 2023 ). 6.4 Rationale Quality and Model Interpretability An important component of this study was reviewing the rationale sentences generated by each model. Under DMI mode, the models rationale sentence were generally understandable but largely focused on the literal meaning of the words. With limited attention to cultural context or dialect clues, which is similarly to what Khondaker et al. ( 2023 ) observed. When the models were guided by DAI instructions, their explanations were more context-sensitive and culturally informed. This behavior was driven by the abstract definitions given to the LLM in Section 4.2 , where some terms related to dialect-specific rules were defined.While ChatGPT-4-mini was more consistent, Gemini-2.5-flash sometimes introduced unsupported detail or misunderstood idioms. This matches the observation presented in Bellaouar et al. ( 2025 ) who noted that LLMs can provide strong explanations when they understand the context even if it made some mistakes. Despite these limitations, the rationale outputs were still useful. They helped explain why the models changed their decisions in Phase 3, especially in how they recognized tone, emphasis, and ironic intent. 6.5 Annotators agreement To better understand the moderate performance of the models, this study reviewed the subset of sentences where the annotators disagreed and each assigned a different label. The analysis showed that about 36 out of 53 disagreement cases matched the label assigned by one annotator and also matched the LLM’s inference label, but conflicted with the label assigned by the third annotator. This observation supports previous research (Barz et al., 2025 ; Lindahl, 2024 ), indicating that sentiment identification is a challenging task even for experts, as interpretations often vary and reflect subjective judgment. 6.6 Limitations of the Study This study has several limitations. The dataset is imbalanced. The Libyan sentiment dataset is relatively small and lacks fully accurate labels. Although reproducibility was considered by using standardized instructions and consistent API parameters, achieving broader generalization requires larger and more balanced datasets that are specifically designed for dialectal language. 7. Conclusions and Future Work Using Libyan Arabic as a representative dialect, this study proposed DSCLP, a 4-Phase protocol for evaluating the capabilities of LLMs in classifying sentiments in social-media users’ textual content. The DSCLP integrates guided zero-shot inference, human validation, and both model-default and dialect-aware prompting instructions. DSCLP explored how LLMs interpret sentiment under different reasoning conditions. The results showed that both models performed moderately under MDI, but they struggled with dialect-specific slang, idiomatic expressions, and sentences containing sarcasm or ambiguity that rely on cultural and pragmatic understanding. Applying phase 3 of DAI helped correct a number of complex misclassifications from Phase 1, particularly those sentences involving phrases that imply deep cultural nuances. However, this phase does not ensure increased overall accuracy or resolve all misclassified samples from the first phase. The results indicated that LLMs still lack a deep structural understanding of the Libyan dialect; instructing the models to focus on the dialect tends to highlight these underlying gaps rather than bridge them. Across both models, Sarcastic and ambiguous sentences were the dominant sources of misclassification in Libyan dialect sentiment. Ambiguous sentences were also often confused with the neutral or negative categories, which indicates that detecting whether a sentence carries a dual meaning or uncertain sentiment remains a challenging aspect of sentiment classification. Many rationale sentences under DMI showed a misinterpretation of Libyan texts where the LLM relied on MSA meanings instead of dialectal meanings, revealing a Linguistic bias toward Standard Arabic in both models. Overall, the study showed that LLMs are sensitive to instruction design, and prompting alone is insufficient to overcome dialectal bias, figurative complexity, or ambiguity challenges. Future work could focus on developing a Libyan-dialect transformer or fine-tuning existing models by incorporating retrieval-augmented dialect knowledge and expanding the current dataset with more balanced label distributions. In addition, future datasets should consist of carefully annotated samples and adopt an iterative annotation technique. Further research may explore whether LLM-generated rationales can assist human annotators in refining the definitions of sarcasm and ambiguity, particularly in dialectal contexts. Declarations Author Contribution Tarek Rashed introduced the Evaluation Framework for Dialectal Sentiment Classification and Linguistic Phenomena. He presented the related phenomena as auxiliary labels to make traditional labels more precise and consistent with the intended sentiment. Furthermore, he prepared the research methodology and research gap; the Python code for utilizing the LLM APIs was also developed by Tarek Rashed. Ramadan Alfared collected the Libyan dataset and assisted in the analysis of the confusion matrices. Abduelbaset Goweder reviewed the manuscript and refined the presentation of the research gap in academic form. Husien Alhammi and Abubaker Kashada prepared the related work. References Abeysiriwardana, M., & Sumanathilaka, D. (2024). A survey on lexical ambiguity detection and word sense disambiguation. arXiv. https://arxiv.org/abs/2403.16129 Abdulla, N. A., Ahmed, N. A., Shehab, M. A., & Al-Ayyoub, M. (2013). Arabic sentiment analysis: Lexicon-based and corpus-based. In 2013 IEEE Jordan Conference on Applied Electrical Engineering and Computing Technologies (AEECT) (pp. 1–6). IEEE. https://doi.org/10.1109/AEECT.2013.6716448 Abu Kwaik, K., Saad, M., Chatzikyriakidis, S., & Dobnik, S. (2019). LSTM-CNN deep learning model for sentiment analysis of dialectal Arabic. In International Conference on Arabic Language Processing (pp. 108–121). Springer. https://doi.org/10.1007/978-3-030-32959-4_8 Aguilar-Moreno, J. A., Palos-Sanchez, P. R., & Pozo-Barajas, R. D. (2024). Sentiment analysis to support business decision-making: A bibliometric study. AIMS Mathematics , 9(2), 4337–4375. https://doi.org/10.3934/math.2024215 Al-Ayyoub, M., Essa, S. B., & Alsmadi, I. (2015). Lexicon-based sentiment analysis of Arabic tweets. International Journal of Social Network Mining, 2 (2), 101–114. https://doi.org/10.1504/IJSNM.2015.072280 Al-Thubaity, A., Alkhereyf, S., Murayshid, H., Alshalawi, N., Omirah, M., Alateeq, R., et al. (2023). Evaluating ChatGPT and Bard on Arabic sentiment analysis. In Proceedings of ArabicNLP 2023 (pp. 335–349). https://doi.org/10.18653/v1/2023.arabicnlp-1.27 Alakrot, A., Dogman, A., & Ammer, F. (2024). Sarcasm Detection in Libyan Arabic Dialects Using Natural Language Processing Techniques | IEEE Conference Publication | IEEE Xplore. In 2024 IEEE 4th International Maghreb Meeting of the Conference on Sciences and Techniques of Automatic Control and Computer Engineering (MI-STA) (pp. 761–767). IEEE. https://ieeexplore.ieee.org/document/10599695 Alhammi, H. A., Alfard, R. A., & Ramadan, A. (2018). Building a Twitter social media network corpus for Libyan dialect. International Journal of Computer Electrical Engineering, 10 (1). Alsharbi, B. M. (2025). Optimizing large language models for low-resource languages: A case study on Saudi dialects. International Journal of Advanced Computer Science & Applications , 16(3). https://doi.org/10.14569/IJACSA.2025.0160384 Barz, C., Siegel, M., Hanss, D., & Wiegand, M. (2025). Understanding disagreement: An annotation study of sentiment and emotional language in environmental communication. In Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX) (pp. 1–20). https://doi.org/10.18653/v1/2025.law-1.1 Bellaouar, S., Nehar, A., Souffi, S., & Bouameur, M. (2025). Dhati+: Fine-tuned large language models for Arabic subjectivity evaluation. arXiv. https://arxiv.org/abs/2508.19966 Bhargava, N., Radaideh, M. I., Kwon, O. H., & Verma, A. (2025). On the impact of language nuances on sentiment analysis with large language models: Paraphrasing, sarcasm, and emojis. arXiv. https://arxiv.org/abs/2504.05603 Farha, I. A., & Magdy, W. (2020). From Arabic sentiment analysis to sarcasm detection: The ArSarcasm dataset. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools (pp. 32–39). European Language Resources Association. https://aclanthology.org/2020.osact-1.5/ Farha, I. A., & Magdy, W. (2021). A comparative study of effective approaches for Arabic sentiment analysis. Information Processing & Management, 58 (2), 102438. https://doi.org/10.1016/j.ipm.2020.102438 Fleisig, E., Smith, G., Bossi, M., Rustagi, I., Yin, X., & Klein, D. (2024). Linguistic bias in ChatGPT: Language models reinforce dialect discrimination. arXiv. https://arxiv.org/abs/2406.08818 Habberrih, A., & Abuzaraida, M. A. (2024). A review of available Arabic dialect datasets for sentiment analysis. Journal of Sustainable Research in Applied Sciences, 1 (2), 30–37. Haddad, H., Rouhou, A. C., Messaoudi, A., Korched, A., Fourati, C., Sellami, A., et al. (2023). TunBERT: Pretraining BERT for Tunisian dialect understanding. SN Computer Science , 4(2), 194. https://doi.org/10.1007/s42979-022-01541-y Hamed, I., Sabty, C., Abdennadher, S., Vu, N. T., Solorio, T., & Habash, N. (2025). A survey of code-switched Arabic NLP: Progress, challenges, and future directions. arXiv. https://arxiv.org/abs/2501.13419 Hannani, M., Soudi, A., & Van Laerhoven, K. (2024). Assessing the performance of ChatGPT-4, fine-tuned BERT and traditional machine learning models on Moroccan Arabic sentiment analysis. In Proceedings of the 4th International Conference on Natural Language Processing for Digital Humanities (pp. 489–498). https://doi.org/10.18653/v1/2024.nlp4dh-1.47 Hovy, D., & Prabhumoye, S. (2021). Five sources of bias in natural language processing. Language and Linguistics Compass, 15 (8), e12432. https://doi.org/10.1111/lnc3.12432 Kaseb, A., & Farouk, M. (2022). SAIDS: A novel approach for sentiment analysis informed by dialect and sarcasm. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP) (pp. 22–30). https://doi.org/10.18653/v1/2022.wanlp-1.3 Kauffmann, E., Peral, J., Gil, D., Ferrández, A., Sellers, R., & Mora, H. (2019). Managing marketing decision-making with sentiment analysis: An evaluation of the main product features using text data mining. Sustainability , 11(15), 4235. https://doi.org/10.3390/su11154235 Khaled, S., Mohamed, E. H., & Medhat, W. (2024). Evaluating large language models for Arabic sentiment analysis: A comparative study using retrieval-augmented generation. Procedia Computer Science , 244, 363–370. https://doi.org/10.1016/j.procs.2024.10.210 Khalilia, M., Malaysha, S., Suwaileh, R., Jarrar, M., Aljabari, A., Elsayed, T., & Zitouni, I. (2024). ArabicNLU 2024: The first Arabic natural language understanding shared task. arXiv. https://arxiv.org/abs/2407.20663 Khondaker, M. T. I., Waheed, A., & Abdul-Mageed, M. (2023). GPTAraEval: A comprehensive evaluation of ChatGPT on Arabic NLP. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 220–247). https://doi.org/10.18653/v1/2023.emnlp-main.16 Kim, J. M., Lee, Y. J., Jung, S., & Choi, H. J. (2023). Semantic ambiguity detection in sentence classification using task-specific embeddings. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track) (pp. 425–437). https://doi.org/10.18653/v1/2023.acl-industry.41 Lindahl, A. (2024). Disagreement in argumentation annotation. In Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) at LREC-COLING 2024 (pp. 56–66). Liu, A., Wu, Z., Michael, J., Suhr, A., West, P., Koller, A., et al. (2023). We’re afraid language models aren’t modeling ambiguity. arXiv. https://arxiv.org/abs/2304.14399 Mäntylä, M. V., Graziotin, D., & Kuutila, M. (2018). The evolution of sentiment analysis: A review of research topics, venues, and top cited papers. Computer Science Review, 27 , 16–32. Pang, B., Lee, L., & Vaithyanathan, S. (2002). Thumbs up? Sentiment classification using machine learning techniques. arXiv. https://arxiv.org/abs/cs/0205070 Pham, N., Pham, L., & Meyers, A. (2024). Towards better inclusivity: A diverse tweet corpus of English varieties. In Proceedings of the 18th Linguistic Annotation Workshop (LAW-XVIII) (pp. 61–70). Rokaya, M. B., & Ghiduk, A. S. (2019). Arabic lexicon learning to analyze sentiment in microblogs. International Journal of Advanced Computer Science and Applications, 10 (8). Shi, Z., & Agrawal, R. (2025). A comprehensive survey of contemporary Arabic sentiment analysis: Methods, challenges, and future directions. arXiv. https://arxiv.org/abs/2502.03827 Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8419385","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":563725546,"identity":"7dcb0e1b-519e-4fb3-97a1-961d6d244e2c","order_by":0,"name":"Tarek Rashed","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA70lEQVRIiWNgGAWjYJACxgYQKQFEH4A0GzsB5TzIWiRngLQwk6JFGsRjIKTFnv3sw4czau7Jy89uPnjb5tc2eT5mBsYPH3Pw2MKTbmy44Vix4YY7x5Ktc/tuG7YxMzBLztyGz2FpbJIP2BIYN0jkmEnn9txmBGphY+bFp4X/GfvPB/8S7OfPyP8mbdlz256wFok0NsaNbQmJDTdy2KQZftxOJKzlxjOgy/sSkjfcSDO27G24ndzGzNiM1y/s/WmMH3u+JdjOn5H88MaPP7dt57c3H/zwEY8WVMDYBiYbiFUPAn9IUTwKRsEoGAUjBQAAaglPMGaegPUAAAAASUVORK5CYII=","orcid":"","institution":"Higher Institute of Science and Technology, Al-Zahra-Libya","correspondingAuthor":true,"prefix":"","firstName":"Tarek","middleName":"","lastName":"Rashed","suffix":""},{"id":563725548,"identity":"ca7e2d2b-b818-4d09-9bec-b4c086269575","order_by":1,"name":"Ramadan Alfared","email":"","orcid":"","institution":"University of Zawia, Libya","correspondingAuthor":false,"prefix":"","firstName":"Ramadan","middleName":"","lastName":"Alfared","suffix":""},{"id":563725550,"identity":"5288772c-5b13-4be3-9aca-285c02df5041","order_by":2,"name":"Abduelbaset Goweder","email":"","orcid":"","institution":"The Libyan Academy of Graduate Studies","correspondingAuthor":false,"prefix":"","firstName":"Abduelbaset","middleName":"","lastName":"Goweder","suffix":""},{"id":563725551,"identity":"221ca964-47d2-43c4-861c-977e8127cc57","order_by":3,"name":"Husien Alhammi","email":"","orcid":"","institution":"Higher Institute of Science and Technology, Al-Zawia","correspondingAuthor":false,"prefix":"","firstName":"Husien","middleName":"","lastName":"Alhammi","suffix":""},{"id":563725552,"identity":"b67f3481-bd87-48df-a704-d3e8b8d88d1b","order_by":4,"name":"Abubaker Kashada","email":"","orcid":"","institution":"Surman College of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Abubaker","middleName":"","lastName":"Kashada","suffix":""}],"badges":[],"createdAt":"2025-12-21 20:38:13","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8419385/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8419385/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":99311061,"identity":"e4969a80-c710-421e-842a-be6296bdc6ad","added_by":"auto","created_at":"2025-12-31 16:13:48","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":80032,"visible":true,"origin":"","legend":"","description":"","filename":"AnEvaluationFrameworkforDialectalSentimentClassificationandLinguisticPhenomenainLargeLanguageModel1.docx","url":"https://assets-eu.researchsquare.com/files/rs-8419385/v1/6d27e312b3e85c00f79e31c8.docx"},{"id":99311420,"identity":"0f3a6d68-0f18-40f4-a521-9848bf87becf","added_by":"auto","created_at":"2025-12-31 16:15:01","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":7557,"visible":true,"origin":"","legend":"","description":"","filename":"b7fb649f54b8442a9374a7cd40fcfb7e.json","url":"https://assets-eu.researchsquare.com/files/rs-8419385/v1/cf3b5959a91025cbacc8d5b8.json"},{"id":99311298,"identity":"f1702631-17ee-48f0-8b6d-bf3f53319c53","added_by":"auto","created_at":"2025-12-31 16:14:20","extension":"xml","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":147153,"visible":true,"origin":"","legend":"","description":"","filename":"b7fb649f54b8442a9374a7cd40fcfb7e1enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-8419385/v1/58b00c69c3bc9e3f33097b23.xml"},{"id":99311890,"identity":"8dc09179-63ae-40ef-8b2a-dfddfb095189","added_by":"auto","created_at":"2025-12-31 16:17:15","extension":"jpeg","order_by":3,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":59378,"visible":true,"origin":"","legend":"","description":"","filename":"groupimage1.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-8419385/v1/af2cef372b0af536bddf8312.jpeg"},{"id":99310950,"identity":"01530a7b-b754-44fb-93a9-1c186ae598a5","added_by":"auto","created_at":"2025-12-31 16:13:35","extension":"png","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":22882,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinegroupimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-8419385/v1/832a4e20677178248296e1c7.png"},{"id":99311390,"identity":"866b43f9-3e3c-42c0-94a6-336fde5650bf","added_by":"auto","created_at":"2025-12-31 16:14:47","extension":"xml","order_by":5,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":145493,"visible":true,"origin":"","legend":"","description":"","filename":"b7fb649f54b8442a9374a7cd40fcfb7e1structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-8419385/v1/fcc1f15481fd465254f0b7b4.xml"},{"id":98958782,"identity":"3edb9161-049a-4e78-abab-18cfe4d7186d","added_by":"auto","created_at":"2025-12-24 16:25:09","extension":"html","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":152587,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-8419385/v1/1253993b7cb7724d397e8631.html"},{"id":98958775,"identity":"68c9f767-847b-4d79-b7e8-0a9672002fe3","added_by":"auto","created_at":"2025-12-24 16:25:07","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":73929,"visible":true,"origin":"","legend":"\u003cp\u003eThe Four-Phase Protocol for Dialectal Sentiment Classification and Linguistic Phenomena (DSCLP).\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-8419385/v1/6fd2b483782e62a0d3de5f92.png"},{"id":106892414,"identity":"8be97a07-39ad-4b8b-b932-012cafa458fb","added_by":"auto","created_at":"2026-04-14 13:43:08","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1952699,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8419385/v1/b626fc2a-3ef4-4c8c-8478-0b1c78d6d58a.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"An Evaluation Framework for Dialectal Sentiment Classification and Linguistic Phenomena in Large Language Models","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eOn social media platforms, individuals commonly use their dialectal language to express their feelings and thoughts. While dialectal content is quite widespread on social media platforms, formal or official language is more common in professional and institutional domains (Habberrih \u0026amp; Abuzaraida, \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Pham et al., \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). Arabic dialects add further challenges to sentiment classification approaches, as they are written in an informal way with rich morphology, frequent code-switching, and culturally grounded idioms. Moreover, Arabic dialects\u0026rsquo; vocabulary and sentence structure are greatly different from those of Modern Standard Arabic (MSA). Thus, sentiment classifiers that are trained mainly on MSA datasets often fail to recognize dialectal expressions, which leads to misclassifications and linguistic biases (Abugharsa, 2021; Alakrot et al., \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Rokaya \u0026amp; Ghiduk, \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e2019\u003c/span\u003e). In practice, sentiment misclassifications can affect the quality of decision-making systems that depend on sentiment analysis (Aguilar-Moreno et al., \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Kauffmann et al., \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2019\u003c/span\u003e). For this reason, accurate sentiment classification for dialects such as Libyan Arabic is important for reliable interpretation in areas like marketing and decision-making.\u003c/p\u003e \u003cp\u003eLarge Language Models (LLMs) based on Generative Pre-trained Transformer architectures (GPT) are one of the most powerful technological innovations in Natural Language Processing (NLP). In the sentiment classification aspect, the models that are implemented based on the GPT architecture have not only far surpassed earlier lexicon-based and traditional machine-learning approaches, but they have also outperformed earlier deep learning models (Al-Thubaity et al., \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Hannani et al., \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Shi \u0026amp; Agrawal, \u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). Although GPT-based LLMs are not designed to classify textual contents as traditional classifiers do, its GPT can simulate classification behavior because it has been trained on massive data that encode semantic, pragmatic, and affective patterns. Through the use of LLMs\u0026rsquo; APIs, these models can be utilized to perform sentiment classification inference. LLM can be instructed textually to activate the model\u0026rsquo;s reasoning and inference and generate textual comprehensive explanations for each inference it produces. Within the context of sentiment classification using LLMs, this approach is mainly employed to explain in more details how the model constructs its sentiment classifications.\u003c/p\u003e \u003cp\u003eIn this study, the ChatGPT-4.mini (OpenAI) and Gemini-2.5-flash (Google DeepMind) were instructed to classify the Libyan sentiment expressions into five categories using two types of prompt instructions; Default Model Inference (DMI) and Dialect-Aware Inference (DAI), and to produce a rationale sentence for each sentiment prediction. The following sections describe the proposed methodology.\u003c/p\u003e \u003cdiv id=\"Sec2\" class=\"Section2\"\u003e \u003ch2\u003e1.2 Problem and Research Gap\u003c/h2\u003e \u003cp\u003eThe binary method used in earlier classification approaches only allowed sentiment expressions to be classified as either positive or negative. This often leads to incorrectly classifying factual or neutral sentences as positive or negative sentiment (M\u0026auml;ntyl\u0026auml; et al., \u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e2018\u003c/span\u003e; Pang et al., \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e2002\u003c/span\u003e). Even with modern sentiment classification models, which rely on three classes (Positive, Negative, Neutral), current models still fail to handle linguistic phenomena inherent in human communication, where individuals usually express emotions in indirect or non-literal ways, such as sarcasm and ambiguity (Bhargava et al., \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). Sarcasm occurs when individuals prefer to express their opinions in the opposite way of what they truly mean. This is often expressed in the form of humor, criticism, or irony, where sarcastic expressions frequently combine appreciation and irony within the same sentence. In the context of social media, sarcastic sentences are often expressed through cultural idioms, exaggeration, punctuation, and emojis (Alakrot et al., \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Kaseb \u0026amp; Farouk, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2022\u003c/span\u003e). Ambiguity occurs when individuals, within the same sentence, use words that have more than one possible meaning or have mixed polarities, making the intended message uncertain. This phenomenon is common in low-resource languages and arises from overlapping meanings, code-switching between languages, and context-specific expressions, particularly in dialectal and informal language use (Abeysiriwardana \u0026amp; Sumanathilaka, \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Hamed et al., \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Khalilia et al., \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Kim et al., \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Liu et al., \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). In natural languages, sarcasm and ambiguity are key features of linguistic phenomena, as they add richness and complexity to communication. Sentences that fall under sarcasm and ambiguity remain among the major causes of misclassification in sentiment analysis tasks.\u003c/p\u003e \u003cp\u003eAlthough these linguistic phenomena appear naturally and frequently, particularly on social media, most sentiment analysis approaches still treat sarcastic and ambiguous sentences as noise rather than meaningful linguistic features. This practice leads to frequent misclassification and impurity of traditional labels, especially in dialectal and informal Arabic. Recent studies that use LLMs for sentiment classification often lack clear and replicable evaluation procedures. Most studies focused on overall accuracy without considering sarcasm and ambiguity features or enabling LLMs to generate rationale sentences for the sake of understanding model predictions. These studies emphasize overall results without examining how models handle sarcastic or ambiguous sentences. Although several studies (Fleisig et al., \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Hovy \u0026amp; Prabhumoye, \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e2021\u003c/span\u003e) have stated that linguistic bias is one of the major sources of misclassification, no research has identified exactly the source of bias, whether a particular word is not present at all in a dataset during training or it is present but in a different context and meaning. Identifying whether a sentence is biased toward a particular language or dialect requires comparative analysis (e.g., a formal language versus its dialect, or dialect A versus dialect B). There is a need for a structured framework that examines classification accuracy, rationale quality, and dialectal linguistic bias, which are issues this study directly addresses. Understanding how LLMs classify dialectal text and explain their predictions is important for improving sentiment interpretation and reducing misclassification, particularly for sarcasm, ambiguity, and linguistic bias.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e1.3 Research Questions and Objectives\u003c/h2\u003e \u003cp\u003eAddressing the existing knowledge gap, this study proposes the Dialectal Sentiment Classification and Linguistic Phenomena (DSCLP) Protocol to classify, evaluate LLM performance, and analyze LLM inferences on Libyan Arabic as a representative case. However, the proposed four-phase protocol is designed as a generalized methodology that can be applied to other dialects and languages. The protocol extends sentiment analysis beyond traditional sentiment labels to include Sarcastic and Ambiguous labels as related linguistic phenomena influencing sentiment interpretation. According to the above, the study addresses the following research questions:\u003c/p\u003e \u003cp\u003e \u003cb\u003eRQ1.\u003c/b\u003e How accurately do ChatGPT-4-mini and Gemini-2.5-flash classify sentiment in Libyan Arabic across Positive, Negative, Neutral, Sarcastic, and Ambiguous in Libyan Arabic?\u003c/p\u003e \u003cp\u003e \u003cb\u003eRQ2.\u003c/b\u003e How do Model Default Inference (DMI) and Dialect-Aware Inference (DAI) Rules, affect sentiment classification results, and the quality of models\u0026rsquo; rationales?\u003c/p\u003e \u003cp\u003e \u003cb\u003eRQ3.\u003c/b\u003e Do LLMs show dialectal or cultural bias when interpreting sentiment in Libyan Arabic, and can dialect-aware instructions help reveal such bias?\u003c/p\u003e \u003cp\u003eTo answer the study\u0026rsquo;s research questions, a 4-phase protocol combining guided zero-shot inference, human validation, and comprehensive evaluation metrics (Macro-F1, Per-class precision, recall, and F1, Cohen\u0026rsquo;s κ, and confusion analysis) is implemented.\u003c/p\u003e \u003cp\u003eThis study developed a clear and reproducible methodology for evaluating LLM inferences in dialectal sentiment tasks. In addition, it supports fair and reliable evaluation across different dialects. The framework incorporates four phases to evaluate classification accuracy, review rationale quality, and analyze linguistic bias under DMI and DAI.\u003c/p\u003e \u003c/div\u003e"},{"header":"2. Related work","content":"\u003cp\u003eOver the past few years, studies on Arabic sentiment and its dialect analysis have evolved across several approaches, starting with lexicon-based methods, followed by machine learning and deep learning approaches, and most recently LLMs and fine-tuned transformers. The following sections provide an overview of these approaches.\u003c/p\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Lexicon-Based Sentiment Analysis in Arabic\u003c/h2\u003e \u003cp\u003eLexicon-based sentiment analysis was an early approach used to analyze sentiment in Arabic and its dialect texts. Most studies on lexicon-based sentiment analysis were conducted in comparison with other techniques, such as machine learning algorithms. Frequent misclassifications were the most common issue of this approach due to the use of predefined sentiment lexicons to infer polarity, where each word is assigned a fixed sentiment score or label (e.g., positive, negative, neutral). For example, For example, Abdulla et al. (\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2013\u003c/span\u003e) developed a sentiment lexicon for MSA and incorporated some Jordanian and Gulf dialects. The study reported challenges in achieving reliable sentiment classification for Arabic due to its morphological richness and dialectal diversity. Similarly, Al-Ayyoub et al. (\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2015\u003c/span\u003e) constructed a large Arabic sentiment lexicon for analyzing Arabic tweets. The study confirmed that customized lexicons are an effective technique in informal and online social media environments, where textual content mainly appears as short and messy, is rich in idioms, and is created by everyday people. The method showed when dealing with sarcasm and in cases where feeling were not stated directly. Rokaya and Ghiduk (\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e2019\u003c/span\u003e) addressed the limitations of lexicon-based sentiment misclassification by proposing an adaptive Arabic lexicon learning approach. The method employed genetic algorithms to automatically build and optimize Arabic sentiment lexicons. The study stated that the learning approach can handle two major dialectal challenges, namely bias and ambiguity.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Traditional ML and Deep Learning Approaches\u003c/h2\u003e \u003cp\u003eMany researchers transitioned from lexicon-based methods to explore the potential of traditional machine learning and deep learning algorithms. Abu Kwaik et al. (\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2019\u003c/span\u003e) introduced a hybrid LSTM-CNN model for dialectal Arabic sentiment analysis. The model was trained on the ASTD dataset and a dialectal dataset built by the authors. Hybrid deep learning worked better than single models for sentiment analysis of dialectal Arabic. The study showed promising accuracy in handling noisy, informal language. It also identified challenges related to variations in wording across dialects and difficulty with ambiguous or sarcastic sentiments, where sentences with unclear or sarcastic meaning often confused the model. In parallel, Farha and Magdy (\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e2020\u003c/span\u003e) compared SVM, CNN, and BiLSTM on three sentiment datasets: ASTD (tweets in MSA and dialects), LABR (book reviews in MSA), and ArSenTD-Lev (tweets in the Levantine dialect), where the datasets cover MSA and dialect-focused content (Egyptian, Gulf, Levantine, and North African). The comparison identified that BiLSTM achieved the best performance due to its ability to capture context from both directions, which is a key factor in sentiment understanding. In a related work, Farha and Magdy (\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e2020\u003c/span\u003e) directly addressed the issue of sarcasm to cover sarcasm detection using a sarcasm-labeled Twitter dataset. Their results showed that sarcasm reduced accuracy and made sentiment classification more difficult across all models, especially in dialects, confirming that sarcasm is a key source of sentiment ambiguity. In the context of Libyan Arabic dialects, Abugharsa (2021) examined sentiment in Libyan\u0026ndash;Misurata dialect poetry using traditional ML and CNNs. The study stated that metaphor, symbolism, and idiomatic expressions often lead to ambiguity and misclassification. The study demonstrated that culturally embedded and non-literal expressions were the key issues causing deep learning (CNN) to fail to achieve competitive accuracy in comparison with traditional ML. The study primarily concerned with the role of dialects in introducing bias and sentiment ambiguity. This reinforces the need to test advanced models on their ability to handle dialectal nuance, sarcasm, and cultural bias. Similarly, focusing on Libyan Arabic sarcasm, Alakrot et al. (\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) focused on sarcasm detection in Libyan dialects using multiple approaches, including traditional machine learning classifiers (SVM, Na\u0026iuml;ve Bayes), a BiLSTM network with pre-trained Arabic embedding and a fine-tuned transformer model (AraBERT). The authors built a Libyan Arabic dialect sarcasm-labeled corpus from Twitter and Facebook data. AraBERT outperformed both BiLSTM and traditional models in detecting sarcasm in Libyan dialects. The study reached a conclusion consistent with prior research that sarcasm expressions a major obstacle to achieving high accuracy in sentiment analysis, as sarcasm is often expressed through cultural idioms, exaggeration, and punctuation/emojis rather than straightforward textual statements.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e2.3 LLMs and Fine-Tuned Transformers\u003c/h2\u003e \u003cp\u003eMore recently, researchers have turned to Large Language Models (LLMs) and fine-tuned transformers to address the limitations of earlier approaches in sentiment analysis. Multiple studies assessed the performance of LLMs, when applied to Arabic sentiment analysisAl-Thubaity et al. (\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2023\u003c/span\u003e) evaluated s GPT-3.5, GPT-4, and Bard powered by PaLM 2 on Arabic sentiment analysis across MSA and multiple dialects, including Levantine, Egyptian, and Gulf Arabic. The study found that while both models achieved reasonable accuracy on MSA, their performance dropped significantly when applied to dialectal texts, especially if the text sarcastic expressions. The study revealed that the LLMs used had biases toward standard or formal Arabic, which negatively affected the sentiments classification process. In a related evaluation, Khondaker et al. (\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2023\u003c/span\u003e) introduced GPTAraEval, a benchmark framework for evaluating ChatGPT (GPT-3.5-turbo version) on Arabic NLP tasks, including sentiment analysis, text classification, named entity recognition, and question answering. The authors found that GPT-3.5 performed competitively on MSA tasks, while it failed in dialectal settings and struggled with ambiguous sentiment cases, especially when sarcasm or context-dependent expressions were involved, where both require cultural knowledge or contextual understanding. Moreover, the GPT-3.5 showed biases toward MSA, where for a given sentence the responses were unstable across different prompts.\u003c/p\u003e \u003cp\u003eIn contrast to using general-purpose LLMs, AraBERT and other specialized models perform better on Arabic dialects. Hannani et al. (\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) compared ChatGPT-4 with fine-tuned AraBERT and traditional ML models on Moroccan Arabic. AraBERT outperformed ChatGPT-4, particularly in handling idiomatic expressions and code-switching. The study reported that ChatGPT often misclassified idiomatic expressions as negative sentiment rather than recognizing their neutral or figurative meaning. In addition, ChatGPT struggled with sentences that mixed Moroccan Arabic with French or English. Similarly, Haddad et al. (\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2023\u003c/span\u003e) introduced TunBERT, the first BERT-based transformer model specifically intended for the Tunisian dialect. For the sake of tackling the lack of sufficient datasets for the Tunisian dialect, the authors constructed 500K social media and blog sentences so the model can learn better, show less bias, and handle sarcasm and ambiguity more effectively. TunBERT-P (PyTorch) and TunBERT-T (TensorFlow) were two model variations implemented. These models were fine-tuned and evaluated on real-world NLP applications (downstream tasks), including sentiment analysis, dialect identification, and reading comprehension. TunBERT showed competitive results for Tunisian dialect sentiment analysis and dialect identification in comparison with AraBERT, mBERT, and MARBERT. Importantly, the study confirmed that specialized models (trained only on one dialect) can surpass larger multilingual LLMs, particularly in capturing sarcasm sentences in dialectal expressions. These results directly the need for dialect-aware models in Libyan Arabic sentiment analysis, where resource scarcity, ambiguity, and sarcasm similarly challenge the reliability of general-purpose LLMs.\u003c/p\u003e \u003cp\u003eIn a related attempt to enhance LLM performance, recent research has explored how model fine-tuning, data augmentation, and sarcasm mitigation techniques can enhance LLM performance in sentiment analysis, Bhargava et al. (\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) conducted several experiments to examine how sarcasm, paraphrasing, and emoji-related sentences affects the performance of LLMs. For each target LLM, two fine-tuning versions were developed using datasets containing both sarcastic and non-sarcastic tweet sentences. The first version was trained on the complete dataset, while the second version was trained on the same dataset after removing or paraphrasing the sarcastic text. The results indicated noticeable improvements in performance when sarcastic sentences were removed or paraphrased. In similar direction, Alsharbi (\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) proposed a strategy to optimize existing state-of-the-art pre-trained LLMs through data augmentation and fine-tuning. The approach was evaluated on Saudi dialect data using the Multilingual Arabic Dialect (MADAR) dataset on classification tasks. The MADAR includes Tunisian, Egyptian, Levantine, and Gulf. The study demonstrated that the fine-tuned model with data augmentation outperformed MSA-trained baselines. However, the study reported several challenges that complicate sentiment analysis, including biases toward MSA. In addition, the study noted that the model struggled with sarcastic expressions, since sarcasm often relies on cultural idioms, exaggeration, and emojis that confuse language models. Ambiguity is also prevalent, as dialectal expressions may carry multiple meanings depending on social or contextual cues. In an another approach, Khaled et al. (\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) proposed the Retrieval-Augmented Generation (RAG) approach to improve the performance of generative LLMs for Arabic sentiment analysis. The study evaluated LLaMA-3, Gemini-1.0-pro, and Ace-GPT on three datasets including ASAD, ArSarcasm-v2, and SemEval. The results showed that LLMs on their own struggled with dataset imbalance and neutral sentiment labels, leading to frequent misclassifications, while RAG mitigated hallucinations by incorporating external knowledge. However, the study emphasized that dataset imbalance and resource scarcity remain unresolved obstacles. Likewise, Bellaouar et al. (\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) introduced Dhati+, which is pre-trained LLMs models, consisting of XLM-RoBERTa, AraBERT, and ArabianGPT and improved by ensemble approach. The authors addressed data scarcity and imbalance datasets by constructing AraDhati+, a balanced dataset of approximately 78K instances derived from ASTD, LABR, HARD, and SANAD. The results showed that the ensemble approach significantly outperformed LLMs on their own. However, the study noted persistent challenges in handling mixed tweets, where personal opinion and factual expressions appear together, as well as short posts lacking sufficient context.\u003c/p\u003e \u003c/div\u003e"},{"header":"3. Research Methodology","content":"\u003cp\u003eIn this study, a 4-phase protocol for analyzing dialectal sentiment and related phenomena classification on LLMs is implemented, aiming at evaluating how LLM classify textual sentiment and related phenomena in dialectal expressions. In the case of comparison between two more LLMs, the 4-phase protocol is applied on each LLM. Unlike traditional sentiment classification labels, which are concepts inherently recognized by LLMs, this methodology adopts a guided zero-shot technique. This approach provides concise definitions for only two specific linguistic phenomena—Sarcastic and Ambiguous—allowing the LLM to interpret the linguistic context across all five defined labels.\u003c/p\u003e\n\u003cp\u003eThe primary focus of this study is on Libyan Arabic. The protocol can be generalized and applied to other dialects and languages. The four phases together provide methodological process to assess LLM robustness in handling sarcastic and ambiguous sentiment, and linguistic bias within dialectal contexts. The next subsections provide detailed descriptions of the methodology’s components.\u003c/p\u003e\n\u003cdiv id=\"Sec9\"\u003e\n \u003ch2\u003e3.1 The Framework for Dialectal Sentiment Classification and Linguistic Phenomena\u003c/h2\u003e\n \u003cp\u003eGive human- annotators labeled sentiment dialectal dataset, which is a benchmark gold standard dataset. Only sentiment sentences (no label) are processed through the selected LLM standardized API-based prompts. The protocol integrates guided zero-shot classification (presenting few definition for the related-phenomenon classification), human validation, evaluation metrics, and comparative analysis to ensure a comprehensive assessment of LLM inferences. The model produce a structured format consisting of:\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003e- Labels\u003c/strong\u003e: (Positive, Negative, Neutral, Sarcastic, and Ambiguous)\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003e- Rationale\u003c/strong\u003e: (1–2 sentences justifying classification cues).\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003ePhase 1: Guided Zero-Shot LLM Classification under Default Model Inference Behavior\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003eThe LLM is instructed to generate classification label for sentiment sentences based on its default interpretation behavior, without mention to any language or dialect, leaving LLM rely on its latent knowledge in inferring classification label.\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003ePhase 2: Extracting Human– LLM Classification Disagreement Cases\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003eBased on the Phase 1 output, all samples showing label mismatch between human annotations and model-generated labels are isolated for further analysis.\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003ePhase 3: Guided Zero-Shot LLM Classification under Dialect-Aware Inference Instructions\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003eUsing the Phase 2 isolated sentences, the LLM is instructed to generate classification label for expressions based on the dialectal type, intended meaning, and interpretation, including, idioms, figurative language, and cultural semantics.\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003ePhase 4: LLM Accuracy, Performance Evaluation and Comparative Analysis\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003eThe evaluation applies the widely accepted classification metrics used in evaluate machine learning classifiers, including per-class precision, recall, F1-based metrics which include Macro-F1 and Weighted-F1, Cohen’s Kappa. The comparative analysis examines how the two rule conditions impact the LLM’s decisions in detecting Positive, Negative, Neutral, Sarcastic, and Ambiguous sentences, as well as in detecting expressions that contain linguistic biases. Special attention is directed toward NLP challenging such as idiomatic expressions, and code-switching. The Four-Phase Protocol is illustrated in Fig. 1, which summarizes the phases workflow applied in this study.\u003c/p\u003e\n\u003c/div\u003e"},{"header":"4. Materials and Experimental Setup","content":"\u003cp\u003eThe methodology integrates dataset description, model prompting rules, computational environment, and the four-phase evaluation protocol. The following subsections provide detailed information about the methodology\u0026rsquo;s elements.\u003c/p\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003e4.1 Preliminary Study\u003c/h2\u003e \u003cp\u003eA preliminary pilot study was conducted to test the procedures, instructions, dataset reliability, and feasibility, as well as to identify potential methodological issues prior to conducting the main experiments. First, several versions of LLM instruction were tested and refined, until reach the final refined version, which is described in section \u003cspan refid=\"Sec14\" class=\"InternalRef\"\u003e4.4\u003c/span\u003e. Second, it was observed that many users\u0026rsquo; social-media comments frequently included extensive use of emojis across all sentiment labels. In many cases, these emojis did not align with the textual content and sometimes conveyed an opposite emotional signal or introduced misleading cues that were not intended by the writer. The pilot study also revealed that the presence of emojis reduced the performance of the LLMs, as the models tended to interpret emojis as textual content. To prevent this distortion and ensure that the evaluation focused entirely on textual understanding, all emojis were removed, allowing the LLMs to classify the textual content only, without interference from emoji-based emotional cues.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003e4.2 Human Annotators and arrival to agreements\u003c/h2\u003e \u003cp\u003eThe study recruited two linguistic experts to label 1,469 sentences. Out of the total dataset, the two annotators reached label agreement on 1,416 samples. The remaining 53 samples constituted the disagreement subset. Disagreement cases were resolved by consulting a third linguistic expert, who assigned the final labels.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003e4.3 Dataset Description\u003c/h2\u003e \u003cp\u003eIn this study, the primary source of the dataset is that of Alhammi et al. (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2018\u003c/span\u003e). Additional social media expressions were collected from Facebook and Twitter, and then incorporated into the dataset, particularly sentences that might be considered (after review by the annotators) as related phenomena of classification, such as sarcastic or ambiguous sentences. During collecting data, the text data were preserved (as-is), including idioms, figurative expressions, semantics, code-switching, and both short and long sentences, and typos, etc., in their original form except for the removal of emoji characters, where this study focuses on classifying textual content. As a result, the final dataset consists of 1469 sentences, reflecting the diversity of sentences across both traditional sentiment and related phenomena labels, and it covers users\u0026rsquo; sentiments across several domains such as Phones, Cars, Shopping, Communications, News, General Life, Religion, and Sports. Although the number of sentences of each label is not balanced and might produce skewed or unreal accuracy, macro and weighted F1 will be calculated to address the issue of unbalanced datasets. Also, the size of dataset is relatively small, the purpose is not to use the sentences and their associated labels as normally done in training procedure or fine-tuning with LLMs. Instead, the purpose is to hide the labels and provide only the sentences to the LLM, allowing the pre-trained model to generate the classification labels directly. Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e presents the distribution of sentiment labels in the dataset.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eSentiment Label Distribution\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLabel Name\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCount\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePositive\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e233\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNegative\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e374\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNeutral\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e324\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSarcastic\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e313\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAmbiguous\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e225\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eTotal\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e1469\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003e4.4 LLM Instructions and Rules\u003c/h2\u003e \u003cp\u003eIn order to conduct comprehensive analysis of traditional sentiment and related-phenomena classification on LLMs, the LLM was accessed through its official API to classify sentiments using two distinct sets of system instructions. Firstly, the MDI instructed LLM to classify each sentence using its default interpretation behavior without indication to any specific language or dialect. The next text shows the instructions given to the LLM:-\u003c/p\u003e \u003cp\u003e \u003cem\u003eSYSTEM_INSTRUCTION = (\u003c/em\u003e \u003cdiv class=\"BlockQuote\"\u003e \u003cp\u003e \u003cem\u003e\"Your task is to classify the sentiment of the given text according to your model's default interpretation. Classify each text into exactly one of these labels: \"Positive, Negative, Neutral, Sarcastic, Ambiguous\"\u003c/em\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003e\"Notes: -\u003c/em\u003e \u003cb\u003e'Sarcastic'\u003c/b\u003e \u003cem\u003ecovers irony; mocking; ridicule; or a satirical tone; Deliberate ironic appreciation; or the intentional use of positive words to express criticism. -\u003c/em\u003e \u003cb\u003e'Ambiguous'\u003c/b\u003e \u003cem\u003ecovers hesitant or uncertain emotion; vague or indirect expression without mockery; mixed polarity without intent; or unclear meaning from code-switching.\"\u003c/em\u003e\u003c/p\u003e \u003c/div\u003e \u003c/p\u003e \u003cp\u003eSecondly, the DAI instructed LLM to classify each Arabic sentence according to the Libyan Arabic dialect. The following instructions were given to the LLM:\u003c/p\u003e \u003cp\u003e \u003cem\u003eSYSTEM_INSTRUCTION = (\u003c/em\u003e \u003cdiv class=\"BlockQuote\"\u003e \u003cp\u003e \u003cem\u003e\"Your task is to classify the sentiment of the given text according to your model's default interpretation. Classify each text into exactly one of these labels: \"Positive, Negative, Neutral, Sarcastic, Ambiguous\"\u003c/em\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003e\"Notes: -\u003c/em\u003e \u003cb\u003e'Sarcastic'\u003c/b\u003e \u003cem\u003ecovers irony; mocking; ridicule; or a satirical tone; Deliberate ironic appreciation; or the intentional use of positive words to express criticism. -\u003c/em\u003e \u003cb\u003e'Ambiguous'\u003c/b\u003e \u003cem\u003ecovers hesitant or uncertain emotion; vague or indirect expression without mockery; mixed polarity without intent; or unclear meaning from code-switching.\"\u003c/em\u003e\u003c/p\u003e \u003cp\u003e \u003cem\u003e\"DIALECT-SPECIFIC RULES\\n\"\u003c/em\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003e\"- Recognize Libyan tone markers and pragmatic cues.\\n\"\u003c/em\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003e\"- Contrastive particles or laughter may signal sarcasm or implicit criticism.\\n\"\u003c/em\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003e\"- Code-switching can intensify, mock, or blur sentiment.\\n\"\u003c/em\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003e\"- Indirect politeness or understatement may imply negativity.\\n\"\u003c/em\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003e\"- Humor, exaggeration, repetition, or exclamation indicate emotional emphasis.\\n\"\u003c/em\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003e\"- Prioritize pragmatic and cultural cues over literal polarity.\\n\\n\"\u003c/em\u003e \u003c/p\u003e \u003c/div\u003e \u003c/p\u003e \u003cp\u003eThe outputs of both defined rules were saved in CSV format, including the assigned label (Positive, Negative, Neutral, Sarcastic, or Ambiguous), and one or two sentences providing a rationale explanation for the assigned label. The following instructions were given:\u003c/p\u003e \u003cp\u003e \u003cem\u003e\"OUTPUT FORMAT\\n\"\u003c/em\u003e \u003cdiv class=\"BlockQuote\"\u003e \u003cp\u003e \u003cem\u003e\"Return valid JSON:\\n\"\u003c/em\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003e\"{ \\\"label\\\": \\\"Positive|Negative|Neutral|Sarcastic|Ambiguous\\\", \"\u003c/em\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003e\"\\\"rationale\\\": \\\"1\u0026ndash;2 sentences explaining the Libyan-specific cues that justify the label.\\\" }\"\u003c/em\u003e \u003c/p\u003e \u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003e4.5 Experimental Setup\u003c/h2\u003e \u003cp\u003eThe experimental evaluation used GPT-4o-mini and Gemini-2.5-flash. Both models were accessed through their official APIs between May and Dec 2025 to maintain stable conditions and allow the researchers to repeat the experiments under the same instructions given and obtain the same results. Each model was tested on the Libyan dataset under both DMI and DAI Rules. All experiments were conducted in Python 3.9 on a Windows 11 workstation, using API standard libraries. In all experiments, the temperature sat to 0.\u003c/p\u003e \u003c/div\u003e"},{"header":"5. Results","content":"\u003cp\u003eThe study followed 4-phase Protocol to perform text classification task on Libyan dialect using GPT-4o-mini and Gemini-2.5-flash. Each model classified Libyan-dialect textual sentiment sentences under two distinct prompting conditions. Under DMI, the LLM operates according to its internal reasoning without mention to any dialect-specific guidance, and under DAI rules, the LLM is explicitly instructed to classify the Arabic text with specific awareness of Libyan dialect and consider idioms, and cultural expressions. Four experiments were conducted, one for each combination of model and instruction condition.\u003c/p\u003e\n\u003ch3\u003e1- Sample Results of LLM Classifications under Different Prompting Rules\u003c/h3\u003e\n\u003cp\u003eThis section presents the classification results for the LLMs evaluated in this study. Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e displays the classification of two distinct texts using GPT-4o-mini and Gemini-2.5-flash under MDI rules, while Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e provides the classification for the same sentences using Libyan DAI rules. Each representative sentence is presented alongside its human-annotated label, the model\u0026rsquo;s classification inference, and the associated rationale.\u003c/p\u003e \u003cp\u003eThe tables below use the following abbreviations:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eText\u003c/b\u003e: the original Libyan Arabic sentence\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eAno_Lbl\u003c/b\u003e: Annotator Label\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eLLM_Lbl\u003c/b\u003e: LLM Inference Label\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eRationale Sentence\u003c/b\u003e: Rationale Sentence produced by the LLM\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003eSentiment categories are abbreviated as:\u003c/p\u003e \u003cp\u003e \u003cb\u003ePos\u003c/b\u003e\u0026thinsp;=\u0026thinsp;Positive, \u003cb\u003eNeg\u003c/b\u003e\u0026thinsp;=\u0026thinsp;Negative, \u003cb\u003eNeu\u003c/b\u003e\u0026thinsp;=\u0026thinsp;Neutral, \u003cb\u003eSar\u003c/b\u003e\u0026thinsp;=\u0026thinsp;Sarcastic, \u003cb\u003eAmb\u003c/b\u003e\u0026thinsp;=\u0026thinsp;Ambiguous.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eSample LLM Classification Outputs under MDI\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eText\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAno_Lbl\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eLLM_Lbl\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eRationale Sentence\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colspan=\"4\" nameend=\"c4\" namest=\"c1\"\u003e \u003cp\u003eChatGPT: Model Default Inference Mode\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eالسعر مشيط و الناس تشري.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNeg\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003ePos\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eThe phrase indicates that the price is appealing and that people are buying, suggesting a positive sentiment towards the pricing and consumer behavior.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eيا سلام يا راجل\u0026lt;\u0026lt; عالمي لكن يعطّلك يا لطيف أكثر مما يساعدك\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSar\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAmb\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eThe text expresses a mix of admiration and frustration, making it unclear whether the overall sentiment is positive or negative. The use of 'يا سلام يا راجل' suggests appreciation, but the following phrase indicates a sense of annoyance.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"4\" nameend=\"c4\" namest=\"c1\"\u003e \u003cp\u003e\u003cb\u003eGemini: Model Default Inference Mode\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eالسعر مشيط و الناس تشري.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNeg\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003ePos\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eThe text states that the price is 'good' or 'attractive' ('مشيط') and people are buying, which are both positive indicators of market activity.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eيا سلام يا راجل\u0026lt;\u0026lt; عالمي لكن يعطّلك يا لطيف أكثر مما يساعدك\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSar\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eNeg\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eThe text starts with an exclamation and a positive descriptor ('world-class'), but immediately follows with 'but it hinders you' and 'more than it helps you', clearly indicating a negative overall assessment.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eSample LLM Classification Outputs under Libyan DAI\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colspan=\"4\" nameend=\"c4\" namest=\"c1\"\u003e \u003cp\u003eChatGPT: Libyan Dialect Aware Inference Mode\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eالسعر مشيط و الناس تشري.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNeg\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003ePos\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eThe phrase 'السعر مشيط' suggests that the price is good or favorable, and 'الناس تشري' indicates that people are buying, which implies a positive sentiment about the market situation.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eيا سلام يا راجل\u0026lt;\u0026lt; عالمي لكن يعطّلك يا لطيف أكثر مما يساعدك\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSar\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSar\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eThe phrase 'يا سلام يا راجل' is often used sarcastically to express disbelief or mock admiration, while 'يعطّلك أكثر مما يساعدك' indicates that the person is actually causing more trouble than help, reinforcing the ironic tone.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"4\" nameend=\"c4\" namest=\"c1\"\u003e \u003cp\u003e\u003cb\u003eGemini: Libyan Dialect Aware Inference Mode\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eالسعر مشيط و الناس تشري.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNeg\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eNeg\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eThe term \"\"مشيط\"\" (mashīt) in Libyan Arabic unequivocally means \"\"expensive\"\" or \"\"exorbitant,\"\" carrying a strong negative connotation regarding cost. The phrase \"\"و الناس تشري\"\" (and people are buying) highlights a situation where consumers are forced to purchase despite high prices, implying economic hardship or a lack of alternatives, thus reinforcing the negative sentiment about the market conditions.\",\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eيا سلام يا راجل\u0026lt;\u0026lt; عالمي لكن يعطّلك يا لطيف أكثر مما يساعدك\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSar\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eNeg\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eThe text starts with an appreciative exclamation 'يا سلام يا راجل' and a positive descriptor 'عالمي' (world-class), but immediately shifts to a strong negative with the contrastive particle 'لكن' (but). The subsequent phrase 'يعطّلك يا لطيف أكثر مما يساعدك بوه' explicitly states that it hinders more than it helps, with 'يا لطيف' expressing distress and 'بوه' intensifying the negative impact in Libyan colloquialism.\",\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e\n\u003ch3\u003e2. Confusion Matrixes and associated metrics\u003c/h3\u003e\n\u003cp\u003eThe results of these experiments are summarized in two separate tables that report the confusion matrix across the five sentiment categories (Positive, Negative, Neutral, Sarcastic, and Ambiguous). Each table presents key evaluation metrics, including Precision, Recall, F1-score, Macro- F1 Average, Weighted F1, and Cohen\u0026rsquo;s Kappa, to provide a comprehensive view of model accuracy and reliability. Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e shows confusion matrix and related metrics for ChatGPT and Gemini under DMI while Table \u003cspan refid=\"Tab5\" class=\"InternalRef\"\u003e5\u003c/span\u003e shows confusion matrix and related metrics for ChatGPT and Gemini under Libyan DAI Mode.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eConfusion matrix for ChatGPT and Gemini under Default Inference Mode\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"10\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c10\" colnum=\"10\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colspan=\"10\" nameend=\"c10\" namest=\"c1\"\u003e \u003cp\u003eChatGPT: Default Inference Mode\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAno_Lbl\u0026darr; / LLM_Lbl \u0026rarr;\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePos\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eNeg\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eNeu\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSar\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAmb\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003ePrecision\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e \u003cp\u003eRecall\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c9\"\u003e \u003cp\u003eF1-Score\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c10\"\u003e \u003cp\u003eSupport\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003ePos\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e140\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e29\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e14\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e42\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.62\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.60\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.61\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e233\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eNeg\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e210\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e54\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e33\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e68\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.73\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.56\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.63\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e374\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eNeu\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e45\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e178\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e77\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.54\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.54\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.54\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e324\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eSar\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e31\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e195\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e72\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.60\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.62\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.61\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e313\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eAmb\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e20\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e25\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e146\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.42\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.64\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.51\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e225\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eTotal\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"8\" nameend=\"c9\" namest=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e1469\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"10\" nameend=\"c10\" namest=\"c1\"\u003e \u003cp\u003e\u003cb\u003eMetric\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eAccuracy\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"9\" nameend=\"c10\" namest=\"c2\"\u003e \u003cp\u003e\u003cb\u003e0.59\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eMacro F1\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"9\" nameend=\"c10\" namest=\"c2\"\u003e \u003cp\u003e\u003cb\u003e0.59\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eWeighted F1\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"9\" nameend=\"c10\" namest=\"c2\"\u003e \u003cp\u003e\u003cb\u003e0.60\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eCohen\u0026rsquo;s Kappa\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"9\" nameend=\"c10\" namest=\"c2\"\u003e \u003cp\u003e\u003cb\u003e0.49\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"10\" nameend=\"c10\" namest=\"c1\"\u003e \u003cp\u003e\u003cb\u003eGemini: Default Inference Mode\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003ePos\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e160\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e31\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.68\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.68\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.68\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e233\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eNeg\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e230\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e41\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e29\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e57\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.73\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.61\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.66\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e374\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eNeu\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e27\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e13\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e201\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e77\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.59\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.62\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.60\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e324\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eSar\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e26\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e271\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.86\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.86\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.73\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e313\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eAmb\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e16\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e23\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e152\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.64\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.31\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.42\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e225\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colspan=\"8\" nameend=\"c9\" namest=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e1469\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"10\" nameend=\"c10\" namest=\"c1\"\u003e \u003cp\u003e\u003cb\u003eMetric\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eAccuracy\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"9\" nameend=\"c10\" namest=\"c2\"\u003e \u003cp\u003e\u003cb\u003e0.63\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eMacro F1\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"9\" nameend=\"c10\" namest=\"c2\"\u003e \u003cp\u003e\u003cb\u003e0.60\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eWeighted F1\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"9\" nameend=\"c10\" namest=\"c2\"\u003e \u003cp\u003e\u003cb\u003e0.61\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eCohen\u0026rsquo;s Kappa\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"9\" nameend=\"c10\" namest=\"c2\"\u003e \u003cp\u003e\u003cb\u003e0.50\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab5\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 5\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eConfusion matrix for ChatGPT and Gemini under Libyan Dialect Aware Inference Mode\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"10\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c10\" colnum=\"10\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colspan=\"10\" nameend=\"c10\" namest=\"c1\"\u003e \u003cp\u003eChatGPT: Libyan Dialect Aware Inference Mode\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAno_Lbl\u0026darr; / LLM_Lbl \u0026rarr;\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePos\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eNeg\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eNeu\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSar\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAmb\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003ePrecision\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e \u003cp\u003eRecall\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c9\"\u003e \u003cp\u003eF1-Score\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c10\"\u003e \u003cp\u003eSupport\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003ePos\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e26\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e13\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e36\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.32\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.27\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.30\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e93\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eNeg\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e57\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e28\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e63\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e10\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.49\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.34\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e164\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eNeu\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e36\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e21\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e28\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e37\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e24\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.33\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.19\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.24\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e146\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eSar\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e14\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e102\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.39\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.86\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.54\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e118\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eAmb\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e14\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e14\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e19\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e21\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.32\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.26\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.29\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e79\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eTotal\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"8\" nameend=\"c9\" namest=\"c2\"\u003e \u003cp\u003e.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e600\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"10\" nameend=\"c10\" namest=\"c1\"\u003e \u003cp\u003e\u003cb\u003eMetric\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eAccuracy\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"9\" nameend=\"c10\" namest=\"c2\"\u003e \u003cp\u003e\u003cb\u003e0.39\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eMacro F1\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"9\" nameend=\"c10\" namest=\"c2\"\u003e \u003cp\u003e\u003cb\u003e0.35\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eWeighted F1\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"9\" nameend=\"c10\" namest=\"c2\"\u003e \u003cp\u003e\u003cb\u003e0.36\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eCohen\u0026rsquo;s Kappa\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"9\" nameend=\"c10\" namest=\"c2\"\u003e \u003cp\u003e\u003cb\u003e0.23\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"10\" nameend=\"c10\" namest=\"c1\"\u003e \u003cp\u003e\u003cb\u003eGemini: Libyan Dialect Aware Inference Mode\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003ePos\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e27\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e14\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.23\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.48\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.31\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e56\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eNeg\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e47\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e30\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.28\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.50\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.36\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e94\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eNeu\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e63\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e56\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e31\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e36\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.52\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.16\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.24\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e190\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eSar\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e38\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.23\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.90\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.37\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e42\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eAmb\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e21\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e53\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e44\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e24\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.75\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.25\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e154\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colspan=\"8\" nameend=\"c9\" namest=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e536\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"10\" nameend=\"c10\" namest=\"c1\"\u003e \u003cp\u003e\u003cb\u003eMetric\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eAccuracy\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"9\" nameend=\"c10\" namest=\"c2\"\u003e \u003cp\u003e\u003cb\u003e0.31\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eMacro F1\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"9\" nameend=\"c10\" namest=\"c2\"\u003e \u003cp\u003e\u003cb\u003e0.31\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eWeighted F1\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"9\" nameend=\"c10\" namest=\"c2\"\u003e \u003cp\u003e\u003cb\u003e0.28\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eCohen\u0026rsquo;s Kappa\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"9\" nameend=\"c10\" namest=\"c2\"\u003e \u003cp\u003e\u003cb\u003e0.18\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e"},{"header":"6. Discussion","content":"\u003cp\u003eThe results of this study provide a detailed comparative understanding of how GPT-4o-mini and Gemini-2.5-flash classify textual dialectal sentiments under two instruction modes: DMI (Phase 1, applied to the full dataset) and DAI (Phase 3, applied only to disagreement cases).\u003c/p\u003e \u003cdiv id=\"Sec20\" class=\"Section2\"\u003e \u003ch2\u003e6.1 Comparative Performance Metrics of GPT-4o-mini and Gemini-2.5-flash\u003c/h2\u003e \u003cp\u003eUnder the DMI (Phase 1), both models achieved moderate performance. Their accuracy and macro-F1 results were consistent with previous evaluations of LLMs on Arabic sentiment datasets(Al-Thubaity et al., \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Khondaker et al., \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). Gemini-2.5.flash achieved higher overall scores, including better accuracy (0.63 vs. 0.59), a higher Cohen\u0026rsquo;s κ (0.50 vs. 0.49), and slightly stronger macro and weighted-F1 values. A closer look at the confusion matrices showed clearer differences between the two models. Gemini\u0026rsquo;s stronger overall performance came mainly from big performance gain in categories that have clear sentiment cues, especially sarcasm where the LLM was instructed with an abstract definition. Under abstract sarcasm definition. Gemini correctly classified 271 sarcastic cases compared to ChatGPT\u0026rsquo;s 195, achieving an F1-score of 0.73 versus 0.61. In contrast, GPT-4o-mini showed more similar Precision, Recall, and F1 values across multiple classes, indicating a more balanced distribution of predictions across categories.\u003c/p\u003e \u003cp\u003eGemini performed quite well when sentiment cues were explicit, but its performance dropped significantly for unclear or uncertain expressions. For example, its F1-score for the ambiguous class fell to 0.42, compared to 0.51 for GPT-4o-mini. On the other hand, GPT-4o-mini showed more consistent behavior across sentiment categories, suggesting a more uniform distribution of semantic reasoning. Similar behavior has been reported in earlier research on Arabic and dialectal sentiment analysis, where models tend to perform best when sentiment is clearly expressed but struggle when meaning is implied through indirectly figurative language or cultural nuance (Alakrot et al., \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Hannani et al., \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e2024\u003c/span\u003e)\u003c/p\u003e \u003cp\u003eAt the class level, GPT-4o-mini\u0026rsquo;s F1 scores stayed relatively close, ranging from 0.54 to 0.63. Gemini, however, showed a greater variation, with a high of 0.73 for sarcastic labels and a low of 0.42 for ambiguous labels. This indicated that Gemini\u0026rsquo;s inference are more sensitive to the presence or absence of clear sentiment cues. Overall, the results indicated various strengths. Gemini performed better when sentiment markers are clear, whereas GPT-4o-mini remained more stable when cues are weak, mixed, or indirect. Although Gemini achieved slightly higher overall metrics, the two models differ in how they interpret dialectal sentiment.\u003c/p\u003e \u003cp\u003eThe DAI instruction used in Phase 3 is not intended to improve model performance. Instead, it is used to examine how each model adjusts its reasoning when it is guided by Libyan DAI instructions and to identify linguistic patterns that lead to misclassification. Phase 3 focuses only on the subset of sentences for which human annotators and the LLMs initially disagreed in Phase 1. These disagreement cases can be considered the hardest samples in the dataset, as they are involved idiomatic expressions, figurative language, cultural pragmatics, or dialect-specific ambiguity.\u003c/p\u003e \u003cp\u003eUnder the DAI instructions, ChatGPT showed more stable inference behavior than Gemini. ChatGPT correctly classified 234 out of 600 samples (39%), while Gemini classified 167 out of 500 (33.4%). ChatGPT also achieved higher accuracy (0.39 vs. 0.31), macro-F1 (0.35 vs. 0.31), weighted-F1 (0.36 vs. 0.28), and Cohen\u0026rsquo;s κ (0.23 vs. 0.18). These results indicated that ChatGPT benefited more from dialect guidance when handling difficult cases. The confusion matrix supported this difference. ChatGPT\u0026rsquo;s showed strongest improvement in the sarcastic category, where it achieved an F1-score of 0.54 with 102 correct predictions. In comparison, Gemini achieved an F1-score of 0.37 with 38 correct predictions. In contrast, even with DAI instructions, Gemini showed greater variation across sentiment classes. Gemini\u0026rsquo;s recall for sarcastic category reached 0.90 and its precision was 0.23. This indicated excessive use of the sarcastic label, where many positive, negative, neutral, and ambiguous samples were incorrectly labeled as sarcastic label. Different behavior was also clear in the ambiguous label. Gemini achieved high precision and low recall compared to the more balanced performance of ChatGPT. This means that Gemini overlooked many ambiguous samples and assigned them to other sentiment categories. Gemini also struggled with the neutral class, where its recall fell to 0.16, despite the large number of neutral samples (190). This indicated ongoing challenges with subtle, mixed, or weak affective indicators. According to previous research, LLMs often struggle understanding indirect expressions, idiomatic wording, exaggeration, and pragmatic tone, especially when emotive indications are not clear (Bellaouar et al., \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Hannani et al., \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e2024\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eOverall, The results showed that ChatGPT adopts a more stable and adaptable inference strategy when DAI rule is applied, whereas Gemini performed better when sentiment cues are very clear but remains more sensitive to ambiguity and cultural complexity. While DAI prompting helped reduce some sources of biases, it can not fully address the limited exposure most LLMs have to Libyan Arabic during pre-training, particularly for models that rely heavily on explicit affective indicators.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec21\" class=\"Section2\"\u003e \u003ch2\u003e6.2 Sentence-Level Insights into LLM Successes and Failures under Default Inference Behavior\u003c/h2\u003e \u003cp\u003eExamining the confusion matrices from Phase 1 along with examples of correct and incorrect predications helped clarify where the models performed well and where they struggled. In most correct cases, sentiment was expressed clearly and directly, without the use of idioms, sarcasm, exaggeration, metaphor, or culturally specific language. For example, consider the idiomatic positive expression \u0026ldquo;this coffee opens the soul,\u0026rdquo; which in Libyan Arabic is \u0026ldquo;القهوة هذه تفتح النفس.\u0026rdquo; Both ChatGPT and Gemini produced accurate rationale sentences that corresponded to the intended meaning, ChatGPT stated: \u0026ldquo;The phrase 'تفتح النفس' suggests that the coffee is uplifting or refreshing, indicating a positive sentiment toward it,\u0026rdquo;, while Gemini generated: \u0026ldquo;The phrase 'تفتح النفس' (taftah al-nafs) is an idiom meaning 'to cheer one up' or 'to stimulate the appetite/soul,' indicating a positive and pleasant effect of the coffee.\u0026rdquo;\u003c/p\u003e \u003cp\u003eIn contrast, most misclassified samples contained Libyan dialect words, reflecting the lack of Libyan pre-training data, or expressed their meaning in an indirect way. One example is a negative sentiment that was classified as Ambiguous by ChatGPT and as neutral by Gemini: \u0026ldquo;The place is so far,\u0026rdquo; expressed in the Libyan dialect as \u0026ldquo;المكان تيرو\u0026rdquo;. The word \u0026ldquo;تيرو\u0026rdquo; is an Italian term written in Arabic characters, refereeing to so far location. Both models produced rationale sentences that reveal their misunderstanding of this dialectal term. ChatGPT stated: \u0026ldquo;The text is vague and does not provide clear sentiment or context. It simply mentions a place without any emotional cues.\u0026rdquo; Similarly, Gemini explained: \u0026ldquo;The text 'المكان تيرو' (The place is Tero) is a purely descriptive statement. It identifies a location without expressing any explicit positive, negative, or emotional sentiment.\u0026rdquo; From these rationale sentences, it is evident that the models did not understand the meaning of \u0026ldquo;تيرو,\u0026rdquo; which indicates the absence of Libyan dialect vocabulary in their pre-training. This behavior is typically behavior of LLMs, which are primarily trained on large collections of publicly available text such as books and web pages that are mostly written in formal languages and non-dialectal language (Liu et al., \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Meyer, 2023).\u003c/p\u003e \u003cp\u003eEven though the abstract definition of the sarcastic category was provided to both models, the two models showed clear differences in performance. Gemini performed particularly well on sarcastic sentences, achieving strong precision, recall, and F1 scores. For instance, compared with ChatGPT, the sentence \u0026ldquo;النت سريععععع زي مشية جدّاي\u0026rdquo; was correctly classified by Gemini, which also produced an accurate rationale: \u0026ldquo;The text uses 'fast' with exaggerated emphasis, but then compares it to 'my grandmother's walk,' which implies slowness. This ironic contrast indicates sarcasm.\u0026rdquo; On the other hand, ChatGPT failed to infer the correct prediction for the full sentence and stated: \u0026ldquo;The text expresses a positive sentiment about the speed of the internet, using enthusiastic language that suggests satisfaction.\u0026rdquo; It is clear from ChatGPT\u0026rsquo;s rationale that it focused only on the positive indicator \u0026ldquo;fast سريععععع\u0026rdquo; and interpreted the repeated letters \u0026ldquo;ععععع\u0026rdquo; as enthusiasm, while ignoring the rest of the sentence. These results indicated ongoing challenges for ChatGPT in interpreting sarcastic expressions.\u003c/p\u003e \u003cp\u003eWhile the abstract definition of sarcastic sentiment helped the LLMs in inferring sarcastic sentences to some extent, it did not enhance their ability to detect ambiguous tones, as the ambiguous category remained the lowest-performing class. Many ambiguous expressions were interpreted as neutral or negative because the model focused too heavily on small clues, such as strong words or contrastive particles, while overlooking the overall uncertainty of the sentence. In general, the review of misclassified sentences indicated that these sentences did not contain any obvious clues, in other words, their examination did not reveal any consistent pattern or rule that could explain why the LLM failed to understand ambiguous expressions. However, a deeper analysis of these sentences showed that the ambiguous category actually provides an important function for the overall classification system, as it contained the majority of false-negative cases.\u003c/p\u003e \u003cp\u003eSome sentences do not express a single, clear sentiment and instead contain mixed cues. For instance, \u0026ldquo;شنو منو تمام التمام. في الكلام، أما الواقع ما يمشيش\u0026rdquo;. This expression combines a positive expression (\u003cem\u003e\u0026ldquo;تمام التمام\u0026rdquo;\u003c/em\u003e) with a negative one (\u003cem\u003e\u0026ldquo;الواقع ما يمشيش\u0026rdquo;\u003c/em\u003e). When only three sentiment labels are used, sentences like this are easily misclassified, as they do not belong clearly to any single category. By including related linguistic phenomena, this issue is reduced and preserved the clarity of the traditional sentiment labels. This suggests that including related-phenomena labels keeps the traditional sentiment categories clearer and more semantically consistent.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec22\" class=\"Section2\"\u003e \u003ch2\u003e6.3 Influence of Dialectal and Cultural Features on Model Behavior\u003c/h2\u003e \u003cp\u003eThe DSCLP Protocol provides a reliable method for detecting linguistic bias by comparing each sentence\u0026rsquo;s inference label in Phase 3 with its corresponding inference in Phase 1. During sentence-level analysis under DMI, the study reached to a clear understanding that many Phase 1 misinterpretations errors were closely linked to the models\u0026rsquo; limited familiarity with Libyan dialect usage. In particular, dialect-specific words, frequent code-switching, and local idiomatic. Words such as \u0026ldquo;مشيط\u0026rdquo;, \u0026ldquo;يعطّلك\u0026rdquo;, \u0026ldquo;الجو مليح\u0026rdquo;, \u0026ldquo;حشية\u0026rdquo;, and \u0026ldquo;تيرو\u0026rdquo; carry sentiment meanings that culturally understood in the Libyan dialect and cannot be accurately interpreted through literal pattern matching. Several errors showed that both ChatGPT-4 and Gemini-2.5 mapped dialectal expressions to their meanings in MSA or to unrelated dialects, showing a form of linguistic bias.\u003c/p\u003e \u003cp\u003eFor example, in the sentence \u0026ldquo;السعر مشيط و الناس تشري\u0026rdquo;, the accurate Libyan meaning is \u0026ldquo;the price is very expensive and people still buy, \u0026ldquo;However, both models classified it as Positive. Their rationales \u0026ldquo;the price is appealing\u0026rdquo; and \u0026ldquo;the price is 'good' or 'attractive' ('مشيط')\u0026rdquo; explain that both models relied on the MSA meaning of mashīt, which refers to something slender, stretched, or smoothly arranged. Once the models were instructed to interpret Libyan linguistic markers, idiomatic contrasts, and pragmatic tone, Gemini correctly inferred the negative sentiment.\u003c/p\u003e \u003cp\u003eIn another example, ChatGPT showed signs of linguistic bias during DAI instructions. For instance, ChatGPT labeled the sentence \u0026ldquo;مش يقولو طنجره و لقت مغطتها ولا حاجه زي هك لمهم\u0026rdquo; as ambiguous. In everyday Libyan usage, this expression is a familiar idiom used to describe negative behavior rather than uncertainty. After applying the DAI instructions, the model was able to revise its interpretation and assign a label that better reflected the intended cultural meaning. By reviewing some other sentences, the study revealed that in some cases, both models did not infer the correct label even with DAI instruction due to LLMs' linguistic bias. As previously shown in the sentence \u0026ldquo;السعر مشيط و الناس تشري\u0026rdquo;, ChatGPT has only one meaning for the word \u0026ldquo;mashīt,\u0026rdquo; which shows a lack of Libyan training data, while under Gemini, it shows linguistic bias toward dominant training data. These observations addressed the source of bias, which is due to the absence of dialect words or the fact that LLMs rely by default on dominant languages. Similar difficulties have been noted in the studies (Abugharsa, 2021; Haddad et al., \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Hannani et al., \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e2024\u003c/span\u003e), which report that figurative and idiomatic expressions in Arabic dialects often confuse models trained.\u003c/p\u003e \u003cp\u003eBy contrast, DAI instructions mitigated many of these errors by guiding the models toward culturally grounded tone markers (e.g., \u0026ldquo;حشية\u0026rdquo;, \u0026ldquo;تيرو\u0026rdquo;, \u0026ldquo;الجو حلو\u0026rdquo;), which often signal sarcasm or implicit criticism. This supports the conclusion that explicit dialectal grounding enhances model reasoning precisely in the categories where LLMs tend to struggle most. These results align with the findings reported by Haddad et al. (\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2023\u003c/span\u003e).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec23\" class=\"Section2\"\u003e \u003ch2\u003e6.4 Rationale Quality and Model Interpretability\u003c/h2\u003e \u003cp\u003eAn important component of this study was reviewing the rationale sentences generated by each model. Under DMI mode, the models rationale sentence were generally understandable but largely focused on the literal meaning of the words. With limited attention to cultural context or dialect clues, which is similarly to what Khondaker et al. (\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2023\u003c/span\u003e) observed. When the models were guided by DAI instructions, their explanations were more context-sensitive and culturally informed. This behavior was driven by the abstract definitions given to the LLM in Section \u003cspan refid=\"Sec12\" class=\"InternalRef\"\u003e4.2\u003c/span\u003e, where some terms related to dialect-specific rules were defined.While ChatGPT-4-mini was more consistent, Gemini-2.5-flash sometimes introduced unsupported detail or misunderstood idioms. This matches the observation presented in Bellaouar et al. (\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2025\u003c/span\u003e) who noted that LLMs can provide strong explanations when they understand the context even if it made some mistakes.\u003c/p\u003e \u003cp\u003eDespite these limitations, the rationale outputs were still useful. They helped explain why the models changed their decisions in Phase 3, especially in how they recognized tone, emphasis, and ironic intent.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec24\" class=\"Section2\"\u003e \u003ch2\u003e6.5 Annotators agreement\u003c/h2\u003e \u003cp\u003eTo better understand the moderate performance of the models, this study reviewed the subset of sentences where the annotators disagreed and each assigned a different label. The analysis showed that about 36 out of 53 disagreement cases matched the label assigned by one annotator and also matched the LLM\u0026rsquo;s inference label, but conflicted with the label assigned by the third annotator. This observation supports previous research (Barz et al., \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Lindahl, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e2024\u003c/span\u003e), indicating that sentiment identification is a challenging task even for experts, as interpretations often vary and reflect subjective judgment.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec25\" class=\"Section2\"\u003e \u003ch2\u003e6.6 Limitations of the Study\u003c/h2\u003e \u003cp\u003eThis study has several limitations. The dataset is imbalanced. The Libyan sentiment dataset is relatively small and lacks fully accurate labels. Although reproducibility was considered by using standardized instructions and consistent API parameters, achieving broader generalization requires larger and more balanced datasets that are specifically designed for dialectal language.\u003c/p\u003e \u003c/div\u003e"},{"header":"7. Conclusions and Future Work","content":"\u003cp\u003eUsing Libyan Arabic as a representative dialect, this study proposed DSCLP, a 4-Phase protocol for evaluating the capabilities of LLMs in classifying sentiments in social-media users\u0026rsquo; textual content. The DSCLP integrates guided zero-shot inference, human validation, and both model-default and dialect-aware prompting instructions. DSCLP explored how LLMs interpret sentiment under different reasoning conditions. The results showed that both models performed moderately under MDI, but they struggled with dialect-specific slang, idiomatic expressions, and sentences containing sarcasm or ambiguity that rely on cultural and pragmatic understanding.\u003c/p\u003e \u003cp\u003eApplying phase 3 of DAI helped correct a number of complex misclassifications from Phase 1, particularly those sentences involving phrases that imply deep cultural nuances. However, this phase does not ensure increased overall accuracy or resolve all misclassified samples from the first phase. The results indicated that LLMs still lack a deep structural understanding of the Libyan dialect; instructing the models to focus on the dialect tends to highlight these underlying gaps rather than bridge them. Across both models, Sarcastic and ambiguous sentences were the dominant sources of misclassification in Libyan dialect sentiment. Ambiguous sentences were also often confused with the neutral or negative categories, which indicates that detecting whether a sentence carries a dual meaning or uncertain sentiment remains a challenging aspect of sentiment classification. Many rationale sentences under DMI showed a misinterpretation of Libyan texts where the LLM relied on MSA meanings instead of dialectal meanings, revealing a Linguistic bias toward Standard Arabic in both models. Overall, the study showed that LLMs are sensitive to instruction design, and prompting alone is insufficient to overcome dialectal bias, figurative complexity, or ambiguity challenges.\u003c/p\u003e \u003cp\u003eFuture work could focus on developing a Libyan-dialect transformer or fine-tuning existing models by incorporating retrieval-augmented dialect knowledge and expanding the current dataset with more balanced label distributions. In addition, future datasets should consist of carefully annotated samples and adopt an iterative annotation technique. Further research may explore whether LLM-generated rationales can assist human annotators in refining the definitions of sarcasm and ambiguity, particularly in dialectal contexts.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eTarek Rashed introduced the Evaluation Framework for Dialectal Sentiment Classification and Linguistic Phenomena. He presented the related phenomena as auxiliary labels to make traditional labels more precise and consistent with the intended sentiment. Furthermore, he prepared the research methodology and research gap; the Python code for utilizing the LLM APIs was also developed by Tarek Rashed. Ramadan Alfared collected the Libyan dataset and assisted in the analysis of the confusion matrices. Abduelbaset Goweder reviewed the manuscript and refined the presentation of the research gap in academic form. Husien Alhammi and Abubaker Kashada prepared the related work.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eAbeysiriwardana, M., \u0026amp; Sumanathilaka, D. (2024). A survey on lexical ambiguity detection and word sense disambiguation. \u003cem\u003earXiv.\u003c/em\u003e https://arxiv.org/abs/2403.16129\u003c/li\u003e\n\u003cli\u003eAbdulla, N. A., Ahmed, N. A., Shehab, M. A., \u0026amp; Al-Ayyoub, M. (2013). Arabic sentiment analysis: Lexicon-based and corpus-based. In \u003cem\u003e2013 IEEE Jordan Conference on Applied Electrical Engineering and Computing Technologies (AEECT)\u003c/em\u003e (pp. 1\u0026ndash;6). IEEE. \u003cstrong\u003ehttps://doi.org/10.1109/AEECT.2013.6716448\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003eAbu Kwaik, K., Saad, M., Chatzikyriakidis, S., \u0026amp; Dobnik, S. (2019). LSTM-CNN deep learning model for sentiment analysis of dialectal Arabic. In \u003cem\u003eInternational Conference on Arabic Language Processing\u003c/em\u003e (pp. 108\u0026ndash;121). Springer. \u003cstrong\u003ehttps://doi.org/10.1007/978-3-030-32959-4_8\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003eAguilar-Moreno, J. A., Palos-Sanchez, P. R., \u0026amp; Pozo-Barajas, R. D. (2024). Sentiment analysis to support business decision-making: A bibliometric study. \u003cem\u003eAIMS Mathematics\u003c/em\u003e, 9(2), 4337\u0026ndash;4375. \u003cstrong\u003ehttps://doi.org/10.3934/math.2024215\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003eAl-Ayyoub, M., Essa, S. B., \u0026amp; Alsmadi, I. (2015). Lexicon-based sentiment analysis of Arabic tweets. \u003cem\u003eInternational Journal of Social Network Mining, 2\u003c/em\u003e(2), 101\u0026ndash;114. https://doi.org/10.1504/IJSNM.2015.072280\u003c/li\u003e\n\u003cli\u003eAl-Thubaity, A., Alkhereyf, S., Murayshid, H., Alshalawi, N., Omirah, M., Alateeq, R., et al. (2023). Evaluating ChatGPT and Bard on Arabic sentiment analysis. In \u003cem\u003eProceedings of ArabicNLP 2023\u003c/em\u003e (pp. 335\u0026ndash;349). \u003cstrong\u003ehttps://doi.org/10.18653/v1/2023.arabicnlp-1.27\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003eAlakrot, A., Dogman, A., \u0026amp; Ammer, F. (2024). Sarcasm Detection in Libyan Arabic Dialects Using Natural Language Processing Techniques | IEEE Conference Publication | IEEE Xplore. In \u003cem\u003e2024 IEEE 4th International Maghreb Meeting of the Conference on Sciences and Techniques of Automatic Control and Computer Engineering (MI-STA)\u003c/em\u003e (pp. 761\u0026ndash;767). IEEE. \u003cstrong\u003ehttps://ieeexplore.ieee.org/document/10599695\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003eAlhammi, H. A., Alfard, R. A., \u0026amp; Ramadan, A. (2018). Building a Twitter social media network corpus for Libyan dialect. \u003cem\u003eInternational Journal of Computer Electrical Engineering, 10\u003c/em\u003e(1).\u003c/li\u003e\n\u003cli\u003eAlsharbi, B. M. (2025). Optimizing large language models for low-resource languages: A case study on Saudi dialects. \u003cem\u003eInternational Journal of Advanced Computer Science \u0026amp; Applications\u003c/em\u003e, 16(3). \u003cstrong\u003ehttps://doi.org/10.14569/IJACSA.2025.0160384\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003eBarz, C., Siegel, M., Hanss, D., \u0026amp; Wiegand, M. (2025). Understanding disagreement: An annotation study of sentiment and emotional language in environmental communication. In \u003cem\u003eProceedings of the 19th Linguistic Annotation Workshop (LAW-XIX)\u003c/em\u003e (pp. 1\u0026ndash;20). \u003cstrong\u003ehttps://doi.org/10.18653/v1/2025.law-1.1\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003eBellaouar, S., Nehar, A., Souffi, S., \u0026amp; Bouameur, M. (2025). Dhati+: Fine-tuned large language models for Arabic subjectivity evaluation. \u003cem\u003earXiv.\u003c/em\u003e https://arxiv.org/abs/2508.19966\u003c/li\u003e\n\u003cli\u003eBhargava, N., Radaideh, M. I., Kwon, O. H., \u0026amp; Verma, A. (2025). On the impact of language nuances on sentiment analysis with large language models: Paraphrasing, sarcasm, and emojis. \u003cem\u003earXiv.\u003c/em\u003e https://arxiv.org/abs/2504.05603\u003c/li\u003e\n\u003cli\u003eFarha, I. A., \u0026amp; Magdy, W. (2020). From Arabic sentiment analysis to sarcasm detection: The ArSarcasm dataset. In \u003cem\u003eProceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools\u003c/em\u003e (pp. 32\u0026ndash;39). European Language Resources Association. https://aclanthology.org/2020.osact-1.5/\u003c/li\u003e\n\u003cli\u003eFarha, I. A., \u0026amp; Magdy, W. (2021). A comparative study of effective approaches for Arabic sentiment analysis. \u003cem\u003eInformation Processing \u0026amp; Management, 58\u003c/em\u003e(2), 102438. https://doi.org/10.1016/j.ipm.2020.102438\u003c/li\u003e\n\u003cli\u003eFleisig, E., Smith, G., Bossi, M., Rustagi, I., Yin, X., \u0026amp; Klein, D. (2024). Linguistic bias in ChatGPT: Language models reinforce dialect discrimination. \u003cem\u003earXiv.\u003c/em\u003e https://arxiv.org/abs/2406.08818\u003c/li\u003e\n\u003cli\u003eHabberrih, A., \u0026amp; Abuzaraida, M. A. (2024). A review of available Arabic dialect datasets for sentiment analysis. \u003cem\u003eJournal of Sustainable Research in Applied Sciences, 1\u003c/em\u003e(2), 30\u0026ndash;37.\u003c/li\u003e\n\u003cli\u003eHaddad, H., Rouhou, A. C., Messaoudi, A., Korched, A., Fourati, C., Sellami, A., et al. (2023). TunBERT: Pretraining BERT for Tunisian dialect understanding. \u003cem\u003eSN Computer Science\u003c/em\u003e, 4(2), 194. \u003cstrong\u003ehttps://doi.org/10.1007/s42979-022-01541-y\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003eHamed, I., Sabty, C., Abdennadher, S., Vu, N. T., Solorio, T., \u0026amp; Habash, N. (2025). A survey of code-switched Arabic NLP: Progress, challenges, and future directions. \u003cem\u003earXiv.\u003c/em\u003e https://arxiv.org/abs/2501.13419\u003c/li\u003e\n\u003cli\u003eHannani, M., Soudi, A., \u0026amp; Van Laerhoven, K. (2024). Assessing the performance of ChatGPT-4, fine-tuned BERT and traditional machine learning models on Moroccan Arabic sentiment analysis. In \u003cem\u003eProceedings of the 4th International Conference on Natural Language Processing for Digital Humanities\u003c/em\u003e (pp. 489\u0026ndash;498). \u003cstrong\u003ehttps://doi.org/10.18653/v1/2024.nlp4dh-1.47\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003eHovy, D., \u0026amp; Prabhumoye, S. (2021). Five sources of bias in natural language processing. \u003cem\u003eLanguage and Linguistics Compass, 15\u003c/em\u003e(8), e12432. \u003cstrong\u003ehttps://doi.org/10.1111/lnc3.12432\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003eKaseb, A., \u0026amp; Farouk, M. (2022). SAIDS: A novel approach for sentiment analysis informed by dialect and sarcasm. In \u003cem\u003eProceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)\u003c/em\u003e (pp. 22\u0026ndash;30). \u003cstrong\u003ehttps://doi.org/10.18653/v1/2022.wanlp-1.3\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003eKauffmann, E., Peral, J., Gil, D., Ferr\u0026aacute;ndez, A., Sellers, R., \u0026amp; Mora, H. (2019). Managing marketing decision-making with sentiment analysis: An evaluation of the main product features using text data mining. \u003cem\u003eSustainability\u003c/em\u003e, 11(15), 4235. \u003cstrong\u003ehttps://doi.org/10.3390/su11154235\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003eKhaled, S., Mohamed, E. H., \u0026amp; Medhat, W. (2024). Evaluating large language models for Arabic sentiment analysis: A comparative study using retrieval-augmented generation. \u003cem\u003eProcedia Computer Science\u003c/em\u003e, 244, 363\u0026ndash;370. \u003cstrong\u003ehttps://doi.org/10.1016/j.procs.2024.10.210\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003eKhalilia, M., Malaysha, S., Suwaileh, R., Jarrar, M., Aljabari, A., Elsayed, T., \u0026amp; Zitouni, I. (2024). ArabicNLU 2024: The first Arabic natural language understanding shared task. \u003cem\u003earXiv.\u003c/em\u003e https://arxiv.org/abs/2407.20663\u003c/li\u003e\n\u003cli\u003eKhondaker, M. T. I., Waheed, A., \u0026amp; Abdul-Mageed, M. (2023). GPTAraEval: A comprehensive evaluation of ChatGPT on Arabic NLP. In \u003cem\u003eProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP)\u003c/em\u003e (pp. 220\u0026ndash;247). \u003cstrong\u003ehttps://doi.org/10.18653/v1/2023.emnlp-main.16\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003eKim, J. M., Lee, Y. J., Jung, S., \u0026amp; Choi, H. J. (2023). Semantic ambiguity detection in sentence classification using task-specific embeddings. In \u003cem\u003eProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track)\u003c/em\u003e (pp. 425\u0026ndash;437). \u003cstrong\u003ehttps://doi.org/10.18653/v1/2023.acl-industry.41\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003eLindahl, A. (2024). Disagreement in argumentation annotation. In \u003cem\u003eProceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) at LREC-COLING 2024\u003c/em\u003e (pp. 56\u0026ndash;66).\u003c/li\u003e\n\u003cli\u003eLiu, A., Wu, Z., Michael, J., Suhr, A., West, P., Koller, A., et al. (2023). We\u0026rsquo;re afraid language models aren\u0026rsquo;t modeling ambiguity. \u003cem\u003earXiv.\u003c/em\u003e https://arxiv.org/abs/2304.14399\u003c/li\u003e\n\u003cli\u003eM\u0026auml;ntyl\u0026auml;, M. V., Graziotin, D., \u0026amp; Kuutila, M. (2018). The evolution of sentiment analysis: A review of research topics, venues, and top cited papers. \u003cem\u003eComputer Science Review, 27\u003c/em\u003e, 16\u0026ndash;32.\u003c/li\u003e\n\u003cli\u003ePang, B., Lee, L., \u0026amp; Vaithyanathan, S. (2002). Thumbs up? Sentiment classification using machine learning techniques. \u003cem\u003earXiv.\u003c/em\u003e https://arxiv.org/abs/cs/0205070\u003c/li\u003e\n\u003cli\u003ePham, N., Pham, L., \u0026amp; Meyers, A. (2024). Towards better inclusivity: A diverse tweet corpus of English varieties. In \u003cem\u003eProceedings of the 18th Linguistic Annotation Workshop (LAW-XVIII)\u003c/em\u003e (pp. 61\u0026ndash;70).\u003c/li\u003e\n\u003cli\u003eRokaya, M. B., \u0026amp; Ghiduk, A. S. (2019). Arabic lexicon learning to analyze sentiment in microblogs. \u003cem\u003eInternational Journal of Advanced Computer Science and Applications, 10\u003c/em\u003e(8).\u003c/li\u003e\n\u003cli\u003eShi, Z., \u0026amp; Agrawal, R. (2025). A comprehensive survey of contemporary Arabic sentiment analysis: Methods, challenges, and future directions. \u003cem\u003earXiv.\u003c/em\u003e https://arxiv.org/abs/2502.03827\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"sentiment classification, low-resource dialects, Arabic dialects, large language models, linguistic phenomena, Libyan dialect, bias detection","lastPublishedDoi":"10.21203/rs.3.rs-8419385/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8419385/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eSocial media platforms provide individuals with a seamless way to share their opinions and interests using informal wording, creative spellings, local idioms, and frequent code-switching. This informal nature adds a layer of significant complexity to sentiment classification tasks. Recently, Large Language Models (LLMs) have shown promising capabilities in this area; however, previous research still lacks replicable and consistent evaluation protocols for assessing how these models reach their inferences, handle non-literal language, or explain the reasoning behind their decisions.\u003c/p\u003e \u003cp\u003eTo bridge this gap, this study introduces the Dialectal Sentiment Classification and Linguistic Phenomena (DSCLP) Framework. DSCLP is a four-phase protocol designed to analyze sentiment classification outcomes across one or more LLMs. The study applied DSCLP to a dataset of 1,469 Libyan-dialect social media posts, incorporating \"Sarcastic\" and \"Ambiguous\" as auxiliary labels alongside traditional sentiment categories. Two Generative Pre-trained Transformer (GPT) models, GPT-4o-mini and Gemini-1.5-flash, were examined under two prompting instructions: Model Default Inference (MDI) and Dialect-Aware Inference (DAI). Through the models\u0026rsquo; APIs, each model produced a sentiment label and a rational explanation for every sentence in the dataset.\u003c/p\u003e \u003cp\u003eThe experiments showed that both models achieved moderate performance under the two defined prompting conditions. When the LLMs was instructed under MDI, the models showed biases toward Modern Standard Arabic (MSA) interpretations and still struggle with idioms, figurative language, and dual-sentiment expressions. Analysis of the generated rationale sentences revealed that LLMs frequently relied on literal understanding rather than cultural and contextual meanings in Libyan Arabic. By integrating performance metrics (Macro-F1, per-class F1, weighted-F1, and Cohen\u0026rsquo;s κ), rationale evaluation, and bias analysis, the DSCLP protocol demonstrated that it is a practical and reproducible method for studying LLM behavior in low-resource dialect settings. Future work may apply the protocol to additional dialects and explore model fine-tuning.\u003c/p\u003e","manuscriptTitle":"An Evaluation Framework for Dialectal Sentiment Classification and Linguistic Phenomena in Large Language Models","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-12-24 16:25:03","doi":"10.21203/rs.3.rs-8419385/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"c6913c67-4969-48ae-a2e1-cac8bb6c6612","owner":[],"postedDate":"December 24th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-04-14T13:41:09+00:00","versionOfRecord":[],"versionCreatedAt":"2025-12-24 16:25:03","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8419385","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8419385","identity":"rs-8419385","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}