Fusion-Based Hybrid Model for SMS Spam Detection Integrating Local, Sequential, and Contextual Features

Fusion-Based Hybrid Model for SMS Spam Detection Integrating Local, Sequential, and Contextual Features | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Fusion-Based Hybrid Model for SMS Spam Detection Integrating Local, Sequential, and Contextual Features Hussein Al-Kaabi, Fuqdan Al-Ibraheemi, Ali Kadhim Jasim, Mohammed AL-Rekabi, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5865706/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract In the era of pervasive digital communication, SMS spam poses significant threats, including financial fraud and phishing attacks, necessitating robust detection mechanisms. This paper introduces a novel hybrid model for SMS spam detection, leveraging advanced deep-learning techniques to capture diverse textual features comprehensively. The model integrates Convolutional Neural Networks (CNN) for local feature extraction, Bidirectional Long Short-Term Memory (Bi-LSTM) networks for sequential dependencies, and Bidirectional Encoder Representations from Transformers (BERT) for contextual embeddings. A parallel architecture combines these components to achieve a holistic representation of SMS content. Fused feature vectors undergo attention-based selection to enhance computational efficiency while preserving critical information. Evaluated on the UCI SMS Spam Collection dataset using a 10-fold cross-validation strategy, the proposed model achieves a remarkable accuracy of 99.68%, outperforming state-of-the-art techniques. This work addresses the limitations of traditional and hybrid methods, offering a highly reliable and adaptable solution to the evolving challenges of SMS spam detection. Future directions include real-time adaptability, multimodal integration, and resource-efficient deployment. Feature Fusion Hybrid Deep Learning SMS Spam Detection Text Classification Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 1. Introduction In the digital age, SMS spam has emerged as a pervasive issue affecting millions of users globally. These unsolicited and irrelevant messages diminish communication quality and present significant security risks, including phishing attacks and financial fraud. While attempts at fraud, such as phishing emails and unwanted messages, have proliferated across various platforms, SMS remains particularly prevalent due to its widespread accessibility and the fact that it is readily available on devices without additional software installations. As mobile communication continues to expand, the need for effective SMS spam detection has become increasingly critical to safeguarding user trust and ensuring the integrity of messaging systems. According to Truecaller, around 15–20 billion spam messages are sent globally each month, translating to approximately 600 million daily [ 1 ]. That means the communication system needed for effective teamwork can filter the SMS. Artificial Intelligence (AI) has significantly transformed numerous fields, such as security [ 2 ], healthcare [ 3 ], and transportation [ 4 ], offering advanced solutions to complex problems. In the domain of SMS spam detection, AI has not just played a role but has revolutionized the entire process. AI has demonstrated its power by introducing sophisticated methods beyond traditional rule-based approaches. Machine learning algorithms, especially those employing natural language processing (NLP) and deep learning techniques, have shown remarkable efficacy in identifying and filtering spam messages [ 5 ]. By leveraging large datasets and learning from patterns in text, AI models can detect nuanced spam characteristics and adapt to evolving tactics used by spammers, instilling confidence in the effectiveness of AI-driven spam detection. The integration of AI in SMS spam detection exemplifies its broader impact across various domains, showcasing its potential to address and mitigate emerging challenges with unprecedented precision and adaptability [ 6 ]. Many research papers have addressed detecting unwanted messages, starting from traditional rule-based approaches and progressing through machine learning algorithms to deep learning. Hybrid methods have been developed to increase detection accuracy. However, these approaches often fell short of the desired performance, relying on improving classifiers or extracting only one feature type. This paper presents a model that extracts and fuses three features: Contextual Features using BERT, Local Features using CNN, and Temporal Features using Bi-LSTM. This approach ensures the inclusion of all feature types, making the system more effective and sensitive to words, sentences, and phrases. The contribution of this paper can be outlined as follows: Novel Hybrid Model : The paper introduces a new hybrid model that integrates three distinct features (contextual, local, and temporal) using advanced techniques like BERT for contextual features, CNN for local features, and Bi-LSTM for temporal features. This comprehensive feature extraction process is designed to enhance the accuracy of SMS spam detection. Comprehensive Feature Fusion : Unlike existing approaches that focus on improving classifiers or extracting a single feature type, this model combines multiple feature types, providing a more holistic understanding of the message. This ensures the model captures a wide range of spam characteristics. Improved Detection Performance : By fusing contextual, local, and temporal features, the proposed model is expected to outperform traditional and hybrid methods focusing on limited features. Including various features improves detection accuracy and sensitivity to subtle spam indicators in SMS messages. These contributions collectively address gaps in existing methods and offer an advanced solution to SMS spam detection. The paper is organized as follows: Section 2 reviews related work on SMS spam detection, while Section 3 discusses the background of feature types. Section 4 presents the proposed hybrid model, Section 5 covers the evaluation methodology, Section 6 discusses the results, and Section 7 concludes with findings and future directions. 2. Related work This section will present methods for detecting unwanted messages that utilize the most commonly used dataset, UCI SMS Spam Collection V.1[ 7 ]. Some papers have employed specialized datasets [ 8 ] or datasets in less common languages, such as [ 9 ][ 10 ]. Spam detection methods started with traditional rule-based methods, such as black-and-white lists, and then were developed into machine learning methods. T. Shahi et al. [ 11 ] used the TF-IDF and Naive Bayes (NB). Its method Uses TF-IDF for feature extraction and NB for classification, and its model achieved 92.67%. N. N. A. Sjarif et al. [ 12 ] used the Support Vector Machine (SVM) to classify messages (Spam and Non-spam). This method achieved 98.91%. Saeed Vaman [ 13 ] compared machine learning classifiers (J48, KNN, and Decision Tree). The evaluation focused on key metrics such as accuracy, recall, and precision to assess the performance of each classifier. The results indicated that the Decision Tree (DT) classifier achieved the highest accuracy at 97.06%, outperforming KNN with 92.76% and J48 with 87.33%. The machine learning method has a limitation in performance because it can deal with large datasets and needs feature engineering. The deep learning method performs better in the text classification challenges. M. Jehad et al. [ 14 ] present a deep learning model to detect unwanted SMS using RNN to capture sequences, and LSTM handles long-term dependencies. This model achieved an accuracy of 98%. However, this model focuses only on the temporal features. Mai A. Shaaban et al. [ 15 ] suggest a hybrid model using convolutional and pooling layers to extract the local features. The Extracts feature employs ensemble learning with boosting and bagging. This model achieved an accuracy of 98.38%. Also, this model focused on one type of feature. Some papers combined CNN and LSTM in different architectures to extract the local and temporal features. A. Ghourabi et al. [ 16 ] proposed a model based on Sequential CNN-LSTM; this model achieved an accuracy of 98.37% for the UCI SMS spam collection V.1 and private Arabic SMS datasets. MRF Derakhshi et al. [ 17 ] suggest another CNN-LSTM combination based on the parallel structure to improve the feature representation. This model achieved 99.28%. X. Liu et al. [ 18 ] introduced a modified Transformer model for SMS spam detection, leveraging the UCI SMS Spam Collection V.1 and UtkMl’s Twitter dataset [ 19 ]. Their approach achieved an accuracy of 98.92%, and this study emphasizes the potential of transformer-based architectures in addressing the evolving challenges of SMS and social media spam detection. The hybrid method combines the power of the machine and deep learning methods to enhance the performance of spam detection models; Sarab M. Hameed et al. [ 20 ] proposed a fuzzy rule-based classification method combined with Binary Particle Swarm Optimization (PSO) for SMS spam detection Their approach optimized rule selection using binary PSO, achieving a high accuracy of 98.5% while minimizing the number of rules required for classification. E. Larijani et al. [ 21 ] introduced a hybrid classical-quantum transfer learning approach combined with BERT for SMS spam detection. This method integrates classical machine learning with quantum computing techniques to enhance text classification capabilities. The model achieved an accuracy of 95%. M. Gupta et al. [ 22 ] employed a combination of TF-IDF and CNN for SMS spam detection using an English and Hindi dataset. The TF-IDF method was used for feature extraction, while CNN was applied for pattern recognition. Their approach achieved an accuracy of 99.10%, combining the traditional feature extraction technique TF-IDF for global feature extraction with a deep learning technique CNN for local feature extraction for spam detection. Existing methods for SMS spam detection often rely on rigid rule-based systems or machine learning models requiring extensive feature engineering, limiting their adaptability to evolving spam tactics. While deep learning approaches show promise, many focus on a single feature type, leading to incomplete representations. Hybrid models improve performance but often lack effective feature fusion mechanisms, resulting in suboptimal accuracy, particularly on imbalanced datasets. 3. Background 3.1 Contextual Features Contextual features insights into the broader context of a message. These features are derived from the semantics and syntactic structure of the text, capturing the meaning and relevance of the words within their surrounding context. Advanced models such as BERT [ 23 ], RoBERTa [ 24 ], and GPT 3 [ 25 ] excel at generating contextual embeddings by understanding the relationships between words in a sentence. This deep contextualization allows for the identification of subtle patterns and nuances that are indicative of spam. For instance, contextual features can help distinguish between benign and malicious messages by analyzing the intent and sentiment expressed, which simpler, keyword-based approaches often miss [ 26 ]. 3.2 Local Features Local features refer to the specific attributes of a message that can be directly extracted from the text itself, such as the presence of specific keywords, phrases, or particular patterns [ 27 ]. These features often involve basic text processing techniques, including tokenization, term frequency, and n-grams. Local features are handy for capturing direct spam indicators, such as commonly used spammy phrases or suspicious links. By focusing on localized text elements, these features enable models to identify known spam signatures and anomalies quickly [ 28 ]. However, it's important to note that while effective for detecting straightforward spam, local features alone may not account for the more sophisticated and adaptive spam strategies, highlighting the need for a comprehensive approach to spam detection. 3.3 Temporal Features Temporal features add an important dimension to spam detection by incorporating messages' timing and frequency patterns. These features analyze when messages are sent and how often, providing insights into the behavioral patterns of spam campaigns [ 29 ]. For instance, spam messages might exhibit specific temporal characteristics, such as being sent in bulk during particular times of the day or at regular intervals. By incorporating temporal data, models can better differentiate between legitimate and spam messages based on their sending patterns. This temporal analysis is instrumental in identifying spam campaigns that leverage time-based strategies to evade detection, thus enhancing the overall robustness of spam classification systems [ 30 ]. 4. Proposed Method The proposed method shown in Fig. 1 consists of five main steps: pre-processing, synchronous feature extraction, feature fusion, feature selection, and classification. 4.1 preprocessing The pre-processing step prepares SMS data for effective feature extraction by cleaning and standardizing the text. Key steps include: Text Normalization : Convert all text to lowercase to ensure uniformity. Special Character Removal : Eliminate non-alphanumeric characters, such as symbols and punctuation [ 31 ]. Tokenization : Break down text into individual words or tokens for easier processing. Stopword Removal : Exclude common words (e.g., "and," "the") that do not contribute to meaning. Lemmatization/Stemming : Reduce words to their base or root forms to group similar terms [ 32 ]. These steps help minimize noise and optimize the text for processing using the CNN-BiLSTM-BERT model. 4.2 Parallel Feature Extraction The proposed model adopts a parallel architecture that integrates Convolutional Neural Networks (CNN), Bidirectional Long Short-Term Memory (Bi-LSTM), and Bidirectional Encoder Representations from Transformers (BERT) to capture diverse features from SMS messages comprehensively. Each component contributes uniquely to the feature extraction process, leveraging its strengths in processing textual data. The CNN component captures localized features by identifying n-gram patterns and short-term dependencies inherent within the text [ 33 ]. CNNs are adept at recognizing spatial hierarchies in data, effectively detecting word patterns frequently observed in spam messages. For example, in a spam SMS like "Congrats! You won a $ 1000 gift card. Claim now!" CNN identifies the recurring n-grams like "Congrats," "won," and "Claim now," which are frequently used in spam content to capture immediate user attention. The Bi-LSTM network complements this by focusing on extracting temporal and sequential features. Its architecture, which processes information bidirectionally, allows the model to capture long-term dependencies and the temporal ordering of words [ 34 ]. In the same message, the Bi-LSTM captures the sequence of actions implied by the text, such as the logical progression from the congratulatory statement to the imperative call to action. Recognizing this sequence helps the model understand how spam messages are structured to manipulate recipients. For instance, the temporal flow from "won" to "Claim now!" reflects a typical tactic used in fraudulent messaging to create urgency. The BERT model is pivotal in extracting deep contextual features through its transformer-based architecture. BERT's pre-trained embeddings and bidirectional attention mechanism enable the model to capture the semantic meaning of each word in its entire context [ 35 ]. In the example message, BERT understands the deeper context of words like "gift card" and "Claim now!" recognizing them as standard terms in spam aimed at misleading recipients. BERT's ability to consider both preceding and succeeding words helps differentiate a legitimate message offering a promotion from a deceptively constructed one. In summary, this parallel feature extraction strategy combines CNN for local patterns (e.g., "Congrats!" and "gift card"), Bi-LSTM for sequential dynamics (e.g., flow from "won" to "Claim now!"), and BERT for contextual understanding (e.g., the overall deceptive nature of the message). This enables a holistic representation of SMS messages, significantly improving the model’s performance in distinguishing spam from legitimate communications. The early papers in SMS spam filtering used one type of feature, like [ 12 ], as shown in Fig. 2 (a). Then, the advanced paper-like [ 17 ] used different deep learning methods to extract two types of features and fuse them to enhance feature representation and get better classification results, as shown in Fig. 2 (b). Our model improves feature representation by using three tools to extract temporal, local, and contextual features to enrich feature representation, which helps the model make a better classification decision. 4.3 Feature Fusion : Once the local, temporal, and contextual features are extracted from CNN, Bi-LSTM, and BERT, respectively, the outputs are concatenated into a single unified feature vector using early data fusion [ 36 ]. Let the feature vectors from CNN, Bi-LSTM, and BERT be represented as F CNN ​, F Bi−LSTM ​, and F BERT ​, respectively. The fusion process can be mathematically expressed as. F fused ​ = [F CNN ​⊕ F Bi−LSTM ​ ⊕ F BERT ​] Where ⊕ denotes the concatenation operation. This fused representation F fused aggregates the rich, diverse information captured from local patterns (CNN), sequential dependencies (Bi-LSTM), and deep contextual meanings (BERT). The model can leverage complementary information by merging these different features into a single vector, improving its overall detection performance and robustness in distinguishing spam from legitimate messages. 4.4 Feature Selection : After the fusion of local, temporal, and contextual features, the next crucial step is feature selection. This step is vital for reducing the dimensionality of the combined feature vector and retaining only the most relevant features for accurate classification. Without proper selection, the fused feature vector may include redundant or irrelevant information, leading to increased computational complexity, inefficiency, and the risk of overfitting. Our proposed model employs an attention mechanism to select features [ 37 ]. The attention layer dynamically assigns different weights to the features based on their importance for the classification task. Let the fused feature vector be represented as Ff used , and the attention mechanism produces a set of attention scores α, which are learned during training. The attention mechanism can be formulated as: F selected = α⋅F fused Here, α represents the attention weights and ⋅ denotes element-wise multiplication. The attention layer effectively learns which features from the fused representation contribute most to identifying spam and non-spam messages. Features more relevant to the task receive higher attention scores, while less important features are down-weighted. This results in a reduced feature vector, F selected ​, that retains the most critical information, thus improving the model's efficiency without sacrificing accuracy. 4.5 Classification : The final step involves passing the selected feature vector through fully connected layers, followed by a softmax classifier, which predicts whether the SMS message is spam or legitimate. This classification layer is optimized to ensure high accuracy and low false positive rates. The proposed method integrates CNN, Bi-LSTM, and BERT in a parallel structure, ensuring a comprehensive feature extraction process and enhancing the effectiveness of SMS spam detection. 5. Evaluation This section evaluates the proposed model using the UCI SMS Spam Collection dataset, focusing on key metrics such as accuracy, precision, recall, and F1-score to validate its effectiveness. 5.1 The dataset We evaluated our method using the UCI SMS Spam Collection dataset [ 7 ] hosted by the University of California, Irvine (UCI). This dataset is a well-known resource in SMS spam detection, comprising over 5,000 text messages categorized into two groups: legitimate (or "ham") messages and spam messages. Figure 3 shows the distribution of these labels within the dataset. Table 1 shows examples from the UCI SMS Spam Collection dataset. Spam messages contain unsolicited offers or urgent calls to action, while ham messages are regular, legitimate communication. This dataset helps evaluate spam detection models by providing real-world message examples. Table 1 Examples of SMS messages from the UCI SMS Spam Collection dataset Message Class 1 "Congratulations! You've won a $ 1000 gift card." Spam 2 "Hey, are we still meeting at 5 PM today?" Ham 3 "Claim your free prize now by clicking this link." Spam 4 "Don't forget to submit the report by tomorrow." Ham 5 "Win a free vacation! Call now to claim." Spam 5.2 Data Splitting Data splitting is a crucial phase in developing and evaluating SMS spam detection models, as it ensures that models are trained and tested on separate datasets to prevent overfitting and provide an unbiased performance assessment. We employed the K-Fold Cross-Validation , which partitions the dataset into K equal-sized folds. The model is trained and validated K times, each using a different fold as the validation set and the remaining K-1 folds as the training set [ 38 ]. The results from each fold are averaged to provide a more comprehensive evaluation of model performance. This method is especially beneficial for small and imbalanced datasets, as it maximizes the utilization of available data. For this study, we will employ 10-fold cross-validation to rigorously assess the performance of our SMS spam detection model, ensuring reliable and balanced results across multiple iterations. 5.3 Performance Metrics To effectively evaluate the performance of deep learning models used in text classification tasks, such as spam detection, it is essential to consider several key metrics derived from the confusion matrix. These metrics comprehensively understand the model's predictive accuracy and reliability, especially when distinguishing between spam and legitimate (ham) messages. The confusion matrix outlines the possible outcomes of the model’s predictions, including true positives, false positives, and negatives, which form the basis for calculating the metrics [ 39 ]. Table 2 Performance Metrics for Spam Detection Metric Formula Description Accuracy \(\:\text{A}\text{c}\text{c}\text{u}\text{r}\text{a}\text{c}\text{y}=\frac{\left(TP+TN\right)}{\left(TP+FP+TN+FN\right)}\ast\:100\%\:\) Overall correctness of the model in classifying both spam and non-spam messages. Precision \(\:\text{P}\text{r}\text{e}\text{c}\text{i}\text{s}\text{i}\text{o}\text{n}=\frac{\left(TP\right)}{\left(TP+FP\right)}\) The proportion of messages predicted as spam (minimizes false positives). Recall \(\:\text{R}\text{e}\text{c}\text{a}\text{l}\text{l}=\frac{\left(TP\right)}{\left(TP+TN\right)}\) The proportion of actual spam messages correctly identified by the model (minimizes false negatives). F1-score \(\:\text{F}1\:\text{s}\text{c}\text{o}\text{r}\text{e}=\frac{2\ast\:(Recall\ast\:Precision)}{\left(Recall+Precision\right)}\:\:\:\) Harmonic means of precision and recall are balanced in the evaluation. These performance metrics are critical in determining the efficacy of models that classify SMS or social media content, such as tweets, as spam or non-spam. By carefully analyzing these metrics, researchers and developers can fine-tune their models to optimize accuracy and reliability in real-world applications [ 40 ]. For instance, models that achieve high accuracy might underperform if precision or recall is low, indicating that a balance must be achieved between correctly detecting spam and avoiding the misclassification of legitimate messages. 5.4 Model Configuration The proposed model is structured to maximize the effectiveness of feature extraction and fusion using a combination of CNN, Bi-LSTM, and BERT. The configuration is designed to handle the nuances of short text data, particularly SMS messages, while efficiently learning patterns that distinguish spam from legitimate communication. Input Layer : This layer processes the tokenized and padded text sequences, ensuring uniform input for the following layers. Textual data is converted into word embeddings to capture semantic meaning. CNN Layer : A 1D Convolutional layer with 128 filters of size 3 is used to extract local patterns from the input text. CNN is responsible for identifying short-term dependencies and word n-grams commonly found in spam messages. The convolution operation helps capture spatial hierarchies and n-gram structures in the text. Max Pooling Layer : Following the CNN layer, a Global Max Pooling layer is applied to reduce the dimensionality of the feature maps. This layer highlights the most important local features while minimizing computational complexity. Bi-LSTM Layer : A Bidirectional LSTM layer with 64 units is employed to capture sequential dependencies and the temporal flow of words. By processing the text forward and backward, the Bi-LSTM extracts long-term dependencies, which are critical in understanding the structure and intent behind spam messages. BERT Layer : BERT, pre-trained on a large corpus, is integrated to extract contextual features from the input text. Its transformer architecture, equipped with multi-head attention, enables the model to capture deep contextual relationships by considering the surrounding words in both directions. This helps in understanding the semantic meaning of the message and distinguishing deceptive language. Feature Fusion Layer : As described in the previous section, the outputs of the CNN, Bi-LSTM, and BERT layers are concatenated into a single feature vector. This fused vector aggregates the input message's local, temporal, and contextual features, providing a comprehensive representation. Attention Mechanism : To perform feature selection, an attention layer is applied to the fused feature vector. The attention mechanism assigns weights to each feature based on its importance for the classification task, allowing the model to focus on the most relevant information. Fully Connected Layers : Two fully connected layers, with 256 and 128 units, respectively, follow the feature extraction process. Dropout is applied at a rate of 0.5 to prevent overfitting and ensure robustness during training. Output Layer : The final layer uses softmax activation to classify the SMS as either spam or non-spam. The softmax function outputs the probabilities for each class, allowing the model to provide a confident prediction. This configuration ensures that the model captures different feature types (local, temporal, and contextual) and integrates them in a way that optimally supports the classification of spam and non-spam messages. Attention mechanisms further enhance the model’s ability to focus on the most critical features, improving its accuracy and reducing false classifications. 6. Results Using a 10-fold cross-validation approach, the model demonstrated robust performance across all folds, with accuracy results illustrated in Fig. 4 . The average accuracy across folds was 99.68%, indicating consistent performance. In addition to accuracy, the model achieved the following average metrics across the folds: Precision: 99.65%, Recall: 99.70%, and F1-Score: 99.67%. These metrics reflect the model's effectiveness in correctly identifying spam messages while minimizing false positives and negatives. The consistently high values across all metrics suggest that the proposed model excels in overall accuracy and maintains a strong balance between precision and recall, making it highly reliable for practical spam detection applications. The comparative analysis of various methodologies for SMS spam detection highlights significant advancements in the field. Traditional approaches, such as T. Shahi et al.’s use of TF-IDF with Naive Bayes, achieved an accuracy of 92.67%. However, more sophisticated techniques, including SVM and hybrid models like CNN with LSTM, demonstrated improved performance, with accuracies reaching up to 99.28%. MRF Derakhshi et al.’s parallel CNN-LSTM model and the proposed dual-attention CNN-BiLSTM with the BERT approach outperformed previous methods, achieving an accuracy of 99.68%. This indicates the effectiveness of integrating multiple feature extraction techniques and leveraging deep learning architectures to enhance spam detection capabilities, ultimately leading to more robust and reliable models distinguishing between spam and legitimate messages. Table 2 compares SMS spam detection methods and their accuracy rates. Table 3 Comparison of SMS spam detection methods and their accuracy rates. Reference Methodology Accuracy (%) Shahi[ 11 ] TF-IDF + Naive Bayes 92.67 Jehad [230] RNN + LSTM 98.00 . Shaaban [234] Hybrid model (CNN + Pooling + Ensemble Learning) 98.38 Ghourabi [227] Sequential CNN-LSTM 98.37 Derakhshi [228] Parallel CNN-LSTM 99.28 Liu [206] Modified Transformer Model 98.92 Hameed [65] Fuzzy Rule-Based + Binary PSO 98.50 Larijani [204] Hybrid Classical-Quantum Transfer Learning + BERT 95.00 Gupta [221] TF-IDF + CNN 99.10 Proposed Method CNN + BiLSTM + BERT 99.68 Figure 5 compares the proposed model's performance with existing SMS spam detection methods. They highlight the superior accuracy and robustness of the proposed CNN-BiLSTM-BERT hybrid approach, demonstrating its effectiveness in capturing diverse feature types and outperforming traditional and hybrid models. While the proposed model demonstrates high accuracy and robustness, it has limitations. Relating to computationally intensive techniques like BERT and Bi-LSTM may make it less suitable for real-time applications or deployment in resource-constrained environments. Additionally, the model's performance has been validated on a single dataset, which may only partially capture the diversity of SMS spam in different languages or contexts. 7. Conclusion This paper introduced a parallel feature extraction method for SMS spam detection, combining CNN, Bi-LSTM, and BERT to capture diverse features: CNN for local patterns, Bi-LSTM for sequential dependencies, and BERT for contextual understanding. The model achieves a comprehensive representation by fusing these feature types into a unified vector, enhancing its accuracy and robustness against complex spam tactics. An attention-based feature selection step further refines this vector, retaining only the most relevant information, reducing computational complexity, and minimizing overfitting. Future research could focus on making the model adaptable to evolving spam tactics through real-time learning and adaptive updates. Expanding this approach to multimodal data, such as combining message content with metadata or user behavior, may also improve spam detection. Additionally, experimenting with other transformer models like RoBERTa or T5 and optimizing the model for deployment in resource-limited environments (e.g., mobile devices) would make it more versatile and practical for real-world applications. Declarations Author Contribution Hussein Al-Kaabi conceptualized the study, developed the hybrid model, and conducted the primary analysis. Mohammed Al-Rekabi contributed to the methodology design, conducted experiments, and prepared the initial draft of the manuscript. Fuqdan Al-Ibraheemi reviewed the results, validated the findings, and assisted with the interpretation of the data. Ali Kadhim Jasim prepared figures and tables and provided editorial revisions to the manuscript. All authors contributed to the review and finalization of the manuscript and approved the submitted version. References Aprinia D (2022) Role Of Truecaller Application. Preventing Phone Call Text Message Scams Jurnal Mantik 6(2):1475–1483 Liu X et al (2020) Privacy and security issues in deep learning: A survey. IEEE Access 9:4566–4593 Kamber AN (2020) A Variety of Electrocardiogram (ECG) Signal Processing. Solid State Technol 63(3):5370–5377 Al-kabi, Hussein, Sayyed Majid Mazinani (2021) DNCS: New UAV navigation with considering the no-fly zone and efficient selection of the charging station. Ain Shams Eng J 12(4):3669–3676 Al Saidat M, Rasol, Suleiman Y, Yerima (2024) Advancements of SMS Spam Detection: A Comprehensive Survey of NLP and ML Techniques. Procedia Comput Sci 244:248–259 Al-Kaabi H, Darroudi AD, Jasim AK (2024) Survey of SMS Spam Detection Techniques: A Taxonomy, KJCS , vol. 2, no. 4, pp. 23–34, Dec Almeida TA, José María G, Hidalgo, Yamakami A (2011) Contributions to the study of SMS spam filtering: new collection and results. Proceedings of the 11th ACM symposium on Document engineering Adewole K, Sakariyah et al (2019) SMSAD: a framework for spam message and spam account detection. Multimedia Tools Appl 78:3925–3960 Al Saidat M, Rasol, Suleiman Y, Yerima, Shaalan K (2024) A Novel Approach for Arabic SMS Spam Detection Using Hybrid Deep Learning Techniques. Procedia Comput Sci 244:260–267 Theodorus A et al (2021) Short message service (SMS) spam filtering using machine learning in Bahasa Indonesia. 2021 3rd East Indonesia Conference on Computer and Information Technology (EIConCIT) . IEEE Shahi TB, Yadav A (2013) Mobile SMS spam filtering for Nepali text using naïve bayesian and support vector machine. Int J Intell Sci 4(1):24–28 Sjarif NN, Amir et al (2020) Support vector machine algorithm for SMS spam classification in the telecommunication industry. Int J Adv Sci Eng Inf Technol 10(2):635–639 Saeed VA (2023) A Method for SMS Spam Message Detection Using Machine Learning. Artif Intell Robot Dev J 3:214–228 Jehad M, Nassif AB, AlShabi M (2023) Spam SMS detection based on long short-term memory and recurrent neural network. Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications V, vol 12538. SPIE Shaaban MA, Yasser F, Hassan, Shawkat K (2022) Guirguis. Deep convolutional forest: a dynamic deep ensemble approach for spam detection in text. Complex Intell Syst 8(6):4897–4909 Ghourabi A, Mahmood MA, Qusay M (2020) Alzubi. A hybrid CNN-LSTM model for SMS spam detection in arabic and english messages. Future Internet 12:156 Derakhshi MR, Feizi et al (2024) Pclf: parallel cnn-lstm fusion model for sms spam filtering. BIO Web of Conferences . Vol. 97. EDP Sciences Liu X, Lu H, Nayak A (2021) A spam transformer model for SMS spam detection. IEEE Access 9:80253–80263 Manasa P, Malik A, Batra I (2024) Detection of twitter spam using GLoVe vocabulary features, bidirectional LSTM and convolution neural network. SN Comput Sci 5(2):206 Hameed SM, Zuhair Hussein A (2021) SMS Spam Detection Based on Fuzzy Rules and Binary Particle Swarm Optimization. Int J Intell Eng Syst 14.2 Ardeshir-Larijani (2024) Ebrahim, and Mohammad Mahdi Nasiri Fatmehsari. Hybrid classical-quantum transfer learning for text classification. Quantum Mach Intell 6(1):19 Gupta M et al (2018) A comparative study of spam SMS detection using machine learning classifiers. eleventh international conference on contemporary computing (IC3) . IEEE, 2018 Aftan S, Shah H. A survey on bert and its applications. (2023) 20th Learning and Technology Conference (L&T) . IEEE, 2023 Al-Kabbi H, Alaa M-R Feizi-Derakhshi, and, Pashazadeh S (2024) A Hierarchical Two-Level Feature Fusion Approach for SMS Spam Filtering. Intell Autom Soft Comput 39:4 Kalyan K, Subramanyam (2024) A survey of GPT-3 family large language models including ChatGPT and GPT-4. Nat Lang Process J 6:100048 Adam, Edriss Eisa Babikir (2020) Deep learning based NLP techniques in text to speech synthesis for communication recognition. J Soft Comput Paradigm (JSCP) 2(04):209–215 Al-Kabbi H, Alaa M-R (2023) Feizi-Derakhshi, and Saeid Pashazadeh. Multi-type feature extraction and early fusion framework for SMS spam detection. IEEE Access Almarashy AH, Jawad (2023) Mohammad-Reza Feizi-Derakhshi, and Pedram Salehpour. Enhancing fake news detection by multi-feature classification. IEEE Access Derakhshi MR, Feizi et al (2024) Pclf: parallel cnn-lstm fusion model for sms spam filtering. BIO Web of Conferences . Vol. 97. EDP Sciences Zhang W et al (2023) Sentiment Text Spatio-Temporal Feature Analysis based on CharCNN and BiLSTM. 4th International Conference on Big Data & Artificial Intelligence & Software Engineering (ICBASE) . IEEE, 2023 Salman Al-Tameemi IK et al (2023) An Efficient Sentiment Classification Method with the Help of Neighbors and a Hybrid of RNN Models. Complexity 1 (2023): 1896556 Ramezani M, Mohammad-Reza Feizi-Derakhshi, and, Mohammad-Ali B (2022) Research Article Knowledge Graph-Enabled Text-Based Automatic Personality Prediction Chen X, Cong P, Lv S (2022) A long-text classification method of Chinese news based on BERT and CNN. IEEE Access 10:34046–34057 Busst MM, Azman et al (2024) Ensemble BiLSTM: A Novel Approach for Aspect Extraction From Online Text. IEEE Access Choi H et al (2020) Evaluation of bert and albert sentence embedding performance on downstream nlp tasks. 25th International conference on pattern recognition (ICPR) . IEEE, 2021 Pawłowski M (2023) Anna Wróblewska, and Sylwia Sysko-Romańczuk. Effective techniques for multimodal data fusion: A comparative analysis. Sensors 23:2381 Farhan BI, Ammar D, Jasim (2023) Improving Detection for Intrusion Using Deep LSTM With Hybrid Feature Selection Method. Iraqi J Inform Communication Technol 6(1):40–50 Muhammed S, Mahdi (2023) Ghassan Abdul-Majeed, and Mahmoud Shuker Mahmoud. Prediction of heart diseases by using supervised machine learning algorithms. Wasit J Pure Sci 2(1):231–243 Jabbooree A, Issa et al (2023) A novel facial expression recognition algorithm using geometry β–skeleton in fusion based on deep CNN. Image Vis Comput 134:104677 Alabid NN (2024) A Review of Sentiment Analysis in Social Media Perspectives. J Kufa Math Comput 112:1–11 Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5865706","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":405057838,"identity":"3e578307-54d0-4848-b61c-9abd41057719","order_by":0,"name":"Hussein Al-Kaabi","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA8UlEQVRIiWNgGAWjYBACAwR58PmPD0CKjZ14LYcNJGeAtDATpQUMmA2kecA0AS3mEumPP3wouMNgzniYwdjm1zZ5PmYGxg8fc3BrsZyRYyY5w+AZg2XDYYbk3L7bhm3MDMySM7fhcdiNHDZmHoPDDAYHzh84nNtzmxGohY2ZF6+W9Mef/4C1HGZstuy5bU+ElgQDaQaIFmZmhh+3EwlrOfPGTLLH4DAPUAsbY2/D7eQ2ZsZm/H45DgyxH38OyxncOMzG8OPPbdv57c0HP3zEowUGeBgkDjAwMLaB2IwNhNWDAT9I4R8iFY+CUTAKRsGIAgBOFVMPTjPEGgAAAABJRU5ErkJggg==","orcid":"","institution":"Ministry of Education Iraq, General Direction Of Vocational Education, Al-Najaf, Iraq","correspondingAuthor":true,"prefix":"","firstName":"Hussein","middleName":"","lastName":"Al-Kaabi","suffix":""},{"id":405057839,"identity":"7b156bea-04cc-4f9b-8e86-8ccca3a43ba1","order_by":1,"name":"Fuqdan Al-Ibraheemi","email":"","orcid":"","institution":"College of Dentistry University of Al-Ameed","correspondingAuthor":false,"prefix":"","firstName":"Fuqdan","middleName":"","lastName":"Al-Ibraheemi","suffix":""},{"id":405057840,"identity":"fb8b2694-5d16-4476-af47-8e4f76cb7f12","order_by":2,"name":"Ali Kadhim Jasim","email":"","orcid":"","institution":"University of Tabriz","correspondingAuthor":false,"prefix":"","firstName":"Ali","middleName":"Kadhim","lastName":"Jasim","suffix":""},{"id":405057841,"identity":"4c334617-44fb-4eaa-ad9c-065a5ab0a3ae","order_by":3,"name":"Mohammed AL-Rekabi","email":"","orcid":"","institution":"Department of Computer Engineering, College of Engineering, University of Al-Shatra","correspondingAuthor":false,"prefix":"","firstName":"Mohammed","middleName":"","lastName":"AL-Rekabi","suffix":""},{"id":405057842,"identity":"a83ba7a3-ee98-422e-8adb-313f4af80109","order_by":4,"name":"jaber parchami","email":"","orcid":"","institution":"Sadjad University of Technology","correspondingAuthor":false,"prefix":"","firstName":"jaber","middleName":"","lastName":"parchami","suffix":""}],"badges":[],"createdAt":"2025-01-20 12:08:16","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5865706/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5865706/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":74521686,"identity":"1f4dd1b1-3279-45bf-95e3-dd95a01905bc","added_by":"auto","created_at":"2025-01-23 06:05:19","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":134182,"visible":true,"origin":"","legend":"\u003cp\u003eFusion-based hybrid proposed method architecture\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-5865706/v1/540c2c5e22da7b9ded0ed535.png"},{"id":74521729,"identity":"a49210d9-1d6e-4352-b1de-530a2bc9d485","added_by":"auto","created_at":"2025-01-23 06:05:19","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":52336,"visible":true,"origin":"","legend":"\u003cp\u003eFeature extraction techniques\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-5865706/v1/ff8d2b8585a1fef61e835881.png"},{"id":74521679,"identity":"49a17a56-c528-43aa-86b0-5d3f0988809a","added_by":"auto","created_at":"2025-01-23 06:05:18","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":53612,"visible":true,"origin":"","legend":"\u003cp\u003eUCI SMS collection v.1 statistic\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-5865706/v1/edda546152fab953136f6112.png"},{"id":74521702,"identity":"65b07376-8727-40bc-87e0-eeaded3c5d6d","added_by":"auto","created_at":"2025-01-23 06:05:19","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":77333,"visible":true,"origin":"","legend":"\u003cp\u003eAccuracy via 10 fold cross validation\u003c/p\u003e","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-5865706/v1/e395f39e43b805ac9bc4bd19.png"},{"id":74521730,"identity":"abb3f706-9e42-48b7-83d8-20df13834dbf","added_by":"auto","created_at":"2025-01-23 06:05:19","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":148542,"visible":true,"origin":"","legend":"\u003cp\u003ePerformance comparison of the proposed model with existing SMS spam detection methods.\u003c/p\u003e","description":"","filename":"floatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-5865706/v1/1f1b654430175f6bd8f2d9f7.png"},{"id":75760776,"identity":"741aadae-056d-4636-9df6-2225a660c50a","added_by":"auto","created_at":"2025-02-08 02:16:35","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1232276,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5865706/v1/2bf13d6d-a1bf-4271-ad0b-1b16c853c240.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Fusion-Based Hybrid Model for SMS Spam Detection Integrating Local, Sequential, and Contextual Features","fulltext":[{"header":"1. Introduction ","content":"\u003cp\u003eIn the digital age, SMS spam has emerged as a pervasive issue affecting millions of users globally. These unsolicited and irrelevant messages diminish communication quality and present significant security risks, including phishing attacks and financial fraud. While attempts at fraud, such as phishing emails and unwanted messages, have proliferated across various platforms, SMS remains particularly prevalent due to its widespread accessibility and the fact that it is readily available on devices without additional software installations. As mobile communication continues to expand, the need for effective SMS spam detection has become increasingly critical to safeguarding user trust and ensuring the integrity of messaging systems. According to Truecaller, around 15–20\u0026nbsp;billion spam messages are sent globally each month, translating to approximately 600\u0026nbsp;million daily [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. That means the communication system needed for effective teamwork can filter the SMS.\u003c/p\u003e\u003cp\u003eArtificial Intelligence (AI) has significantly transformed numerous fields, such as security [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e], healthcare [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e], and transportation [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e], offering advanced solutions to complex problems. In the domain of SMS spam detection, AI has not just played a role but has revolutionized the entire process. AI has demonstrated its power by introducing sophisticated methods beyond traditional rule-based approaches. Machine learning algorithms, especially those employing natural language processing (NLP) and deep learning techniques, have shown remarkable efficacy in identifying and filtering spam messages [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. By leveraging large datasets and learning from patterns in text, AI models can detect nuanced spam characteristics and adapt to evolving tactics used by spammers, instilling confidence in the effectiveness of AI-driven spam detection. The integration of AI in SMS spam detection exemplifies its broader impact across various domains, showcasing its potential to address and mitigate emerging challenges with unprecedented precision and adaptability [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. Many research papers have addressed detecting unwanted messages, starting from traditional rule-based approaches and progressing through machine learning algorithms to deep learning. Hybrid methods have been developed to increase detection accuracy. However, these approaches often fell short of the desired performance, relying on improving classifiers or extracting only one feature type. This paper presents a model that extracts and fuses three features: Contextual Features using BERT, Local Features using CNN, and Temporal Features using Bi-LSTM. This approach ensures the inclusion of all feature types, making the system more effective and sensitive to words, sentences, and phrases. The contribution of this paper can be outlined as follows:\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eNovel Hybrid Model\u003c/b\u003e: The paper introduces a new hybrid model that integrates three distinct features (contextual, local, and temporal) using advanced techniques like BERT for contextual features, CNN for local features, and Bi-LSTM for temporal features. This comprehensive feature extraction process is designed to enhance the accuracy of SMS spam detection.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eComprehensive Feature Fusion\u003c/b\u003e: Unlike existing approaches that focus on improving classifiers or extracting a single feature type, this model combines multiple feature types, providing a more holistic understanding of the message. This ensures the model captures a wide range of spam characteristics.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eImproved Detection Performance\u003c/b\u003e: By fusing contextual, local, and temporal features, the proposed model is expected to outperform traditional and hybrid methods focusing on limited features. Including various features improves detection accuracy and sensitivity to subtle spam indicators in SMS messages.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eThese contributions collectively address gaps in existing methods and offer an advanced solution to SMS spam detection.\u003c/p\u003e\u003cp\u003eThe paper is organized as follows: Section 2 reviews related work on SMS spam detection, while Section \u003cspan refid=\"Sec1\" class=\"InternalRef\"\u003e3\u003c/span\u003e discusses the background of feature types. Section \u003cspan refid=\"Sec5\" class=\"InternalRef\"\u003e4\u003c/span\u003e presents the proposed hybrid model, Section \u003cspan refid=\"Sec8\" class=\"InternalRef\"\u003e5\u003c/span\u003e covers the evaluation methodology, Section \u003cspan refid=\"Sec13\" class=\"InternalRef\"\u003e6\u003c/span\u003e discusses the results, and Section \u003cspan refid=\"Sec14\" class=\"InternalRef\"\u003e7\u003c/span\u003e concludes with findings and future directions.\u003c/p\u003e"},{"header":"2. Related work","content":"\u003cp\u003eThis section will present methods for detecting unwanted messages that utilize the most commonly used dataset, UCI SMS Spam Collection V.1[\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. Some papers have employed specialized datasets [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e] or datasets in less common languages, such as [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e][\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. Spam detection methods started with traditional rule-based methods, such as black-and-white lists, and then were developed into machine learning methods. T. Shahi et al. [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e] used the TF-IDF and Naive Bayes (NB). Its method Uses TF-IDF for feature extraction and NB for classification, and its model achieved 92.67%. N. N. A. Sjarif et al. [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e] used the Support Vector Machine (SVM) to classify messages (Spam and Non-spam). This method achieved 98.91%. Saeed Vaman [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e] compared machine learning classifiers (J48, KNN, and Decision Tree). The evaluation focused on key metrics such as accuracy, recall, and precision to assess the performance of each classifier. The results indicated that the Decision Tree (DT) classifier achieved the highest accuracy at 97.06%, outperforming KNN with 92.76% and J48 with 87.33%. The machine learning method has a limitation in performance because it can deal with large datasets and needs feature engineering. The deep learning method performs better in the text classification challenges. M. Jehad et al. [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e] present a deep learning model to detect unwanted SMS using RNN to capture sequences, and LSTM handles long-term dependencies. This model achieved an accuracy of 98%. However, this model focuses only on the temporal features. Mai A. Shaaban et al. [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e] suggest a hybrid model using convolutional and pooling layers to extract the local features. The Extracts feature employs ensemble learning with boosting and bagging. This model achieved an accuracy of 98.38%. Also, this model focused on one type of feature.\u003c/p\u003e\u003cp\u003eSome papers combined CNN and LSTM in different architectures to extract the local and temporal features. A. Ghourabi et al. [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e] proposed a model based on Sequential CNN-LSTM; this model achieved an accuracy of 98.37% for the UCI SMS spam collection V.1 and private Arabic SMS datasets. MRF Derakhshi et al. [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e] suggest another CNN-LSTM combination based on the parallel structure to improve the feature representation. This model achieved 99.28%. X. Liu et al. [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e] introduced a modified Transformer model for SMS spam detection, leveraging the UCI SMS Spam Collection V.1 and UtkMl’s Twitter dataset [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. Their approach achieved an accuracy of 98.92%, and this study emphasizes the potential of transformer-based architectures in addressing the evolving challenges of SMS and social media spam detection. The hybrid method combines the power of the machine and deep learning methods to enhance the performance of spam detection models; Sarab M. Hameed et al. [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e] proposed a fuzzy rule-based classification method combined with Binary Particle Swarm Optimization (PSO) for SMS spam detection Their approach optimized rule selection using binary PSO, achieving a high accuracy of 98.5% while minimizing the number of rules required for classification. E. Larijani et al. [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e] introduced a hybrid classical-quantum transfer learning approach combined with BERT for SMS spam detection. This method integrates classical machine learning with quantum computing techniques to enhance text classification capabilities. The model achieved an accuracy of 95%. M. Gupta et al. [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e] employed a combination of TF-IDF and CNN for SMS spam detection using an English and Hindi dataset. The TF-IDF method was used for feature extraction, while CNN was applied for pattern recognition. Their approach achieved an accuracy of 99.10%, combining the traditional feature extraction technique TF-IDF for global feature extraction with a deep learning technique CNN for local feature extraction for spam detection.\u003c/p\u003e\u003cp\u003eExisting methods for SMS spam detection often rely on rigid rule-based systems or machine learning models requiring extensive feature engineering, limiting their adaptability to evolving spam tactics. While deep learning approaches show promise, many focus on a single feature type, leading to incomplete representations. Hybrid models improve performance but often lack effective feature fusion mechanisms, resulting in suboptimal accuracy, particularly on imbalanced datasets.\u003c/p\u003e"},{"header":"3. Background","content":"\u003cdiv id=\"Sec2\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Contextual Features\u003c/h2\u003e \u003cp\u003eContextual features insights into the broader context of a message. These features are derived from the semantics and syntactic structure of the text, capturing the meaning and relevance of the words within their surrounding context. Advanced models such as BERT [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e], RoBERTa [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e], and GPT 3 [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e] excel at generating contextual embeddings by understanding the relationships between words in a sentence. This deep contextualization allows for the identification of subtle patterns and nuances that are indicative of spam. For instance, contextual features can help distinguish between benign and malicious messages by analyzing the intent and sentiment expressed, which simpler, keyword-based approaches often miss [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e].\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e3.2 Local Features\u003c/h2\u003e \u003cp\u003eLocal features refer to the specific attributes of a message that can be directly extracted from the text itself, such as the presence of specific keywords, phrases, or particular patterns [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e]. These features often involve basic text processing techniques, including tokenization, term frequency, and n-grams. Local features are handy for capturing direct spam indicators, such as commonly used spammy phrases or suspicious links. By focusing on localized text elements, these features enable models to identify known spam signatures and anomalies quickly [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]. However, it's important to note that while effective for detecting straightforward spam, local features alone may not account for the more sophisticated and adaptive spam strategies, highlighting the need for a comprehensive approach to spam detection.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e3.3 Temporal Features\u003c/h2\u003e \u003cp\u003eTemporal features add an important dimension to spam detection by incorporating messages' timing and frequency patterns. These features analyze when messages are sent and how often, providing insights into the behavioral patterns of spam campaigns [\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e]. For instance, spam messages might exhibit specific temporal characteristics, such as being sent in bulk during particular times of the day or at regular intervals. By incorporating temporal data, models can better differentiate between legitimate and spam messages based on their sending patterns. This temporal analysis is instrumental in identifying spam campaigns that leverage time-based strategies to evade detection, thus enhancing the overall robustness of spam classification systems [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e].\u003c/p\u003e \u003c/div\u003e"},{"header":"4. Proposed Method","content":"\u003cp\u003eThe proposed method shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e consists of five main steps: pre-processing, synchronous feature extraction, feature fusion, feature selection, and classification.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e4.1 preprocessing\u003c/h2\u003e \u003cp\u003eThe pre-processing step prepares SMS data for effective feature extraction by cleaning and standardizing the text. Key steps include:\u003c/p\u003e \u003cp\u003e \u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eText Normalization\u003c/b\u003e: Convert all text to lowercase to ensure uniformity.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eSpecial Character Removal\u003c/b\u003e: Eliminate non-alphanumeric characters, such as symbols and punctuation [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e].\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eTokenization\u003c/b\u003e: Break down text into individual words or tokens for easier processing.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eStopword Removal\u003c/b\u003e: Exclude common words (e.g., \"and,\" \"the\") that do not contribute to meaning.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eLemmatization/Stemming\u003c/b\u003e: Reduce words to their base or root forms to group similar terms [\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e].\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e \u003c/p\u003e \u003cp\u003eThese steps help minimize noise and optimize the text for processing using the CNN-BiLSTM-BERT model.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e4.2 Parallel Feature Extraction\u003c/h2\u003e \u003cp\u003eThe proposed model adopts a parallel architecture that integrates Convolutional Neural Networks (CNN), Bidirectional Long Short-Term Memory (Bi-LSTM), and Bidirectional Encoder Representations from Transformers (BERT) to capture diverse features from SMS messages comprehensively. Each component contributes uniquely to the feature extraction process, leveraging its strengths in processing textual data.\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eThe CNN component captures localized features by identifying n-gram patterns and short-term dependencies inherent within the text [\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e]. CNNs are adept at recognizing spatial hierarchies in data, effectively detecting word patterns frequently observed in spam messages. For example, in a spam SMS like \"Congrats! You won a \u003cspan\u003e$\u003c/span\u003e1000 gift card. Claim now!\" CNN identifies the recurring n-grams like \"Congrats,\" \"won,\" and \"Claim now,\" which are frequently used in spam content to capture immediate user attention.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eThe Bi-LSTM network complements this by focusing on extracting temporal and sequential features. Its architecture, which processes information bidirectionally, allows the model to capture long-term dependencies and the temporal ordering of words [\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e]. In the same message, the Bi-LSTM captures the sequence of actions implied by the text, such as the logical progression from the congratulatory statement to the imperative call to action. Recognizing this sequence helps the model understand how spam messages are structured to manipulate recipients. For instance, the temporal flow from \"won\" to \"Claim now!\" reflects a typical tactic used in fraudulent messaging to create urgency.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eThe BERT model is pivotal in extracting deep contextual features through its transformer-based architecture. BERT's pre-trained embeddings and bidirectional attention mechanism enable the model to capture the semantic meaning of each word in its entire context [\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e]. In the example message, BERT understands the deeper context of words like \"gift card\" and \"Claim now!\" recognizing them as standard terms in spam aimed at misleading recipients. BERT's ability to consider both preceding and succeeding words helps differentiate a legitimate message offering a promotion from a deceptively constructed one.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003eIn summary, this parallel feature extraction strategy combines CNN for local patterns (e.g., \"Congrats!\" and \"gift card\"), Bi-LSTM for sequential dynamics (e.g., flow from \"won\" to \"Claim now!\"), and BERT for contextual understanding (e.g., the overall deceptive nature of the message). This enables a holistic representation of SMS messages, significantly improving the model\u0026rsquo;s performance in distinguishing spam from legitimate communications.\u003c/p\u003e \u003cp\u003eThe early papers in SMS spam filtering used one type of feature, like [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e], as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e (a). Then, the advanced paper-like [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e] used different deep learning methods to extract two types of features and fuse them to enhance feature representation and get better classification results, as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e (b). Our model improves feature representation by using three tools to extract temporal, local, and contextual features to enrich feature representation, which helps the model make a better classification decision.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003e4.3 Feature Fusion\u003c/b\u003e: Once the local, temporal, and contextual features are extracted from CNN, Bi-LSTM, and BERT, respectively, the outputs are concatenated into a single unified feature vector using early data fusion [\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e]. Let the feature vectors from CNN, Bi-LSTM, and BERT be represented as F\u003csub\u003e\u003cem\u003eCNN\u003c/em\u003e\u003c/sub\u003e ​, F\u003csub\u003e\u003cem\u003eBi\u0026minus;LSTM\u003c/em\u003e\u003c/sub\u003e​, and F\u003csub\u003e\u003cem\u003eBERT\u003c/em\u003e\u003c/sub\u003e​, respectively. The fusion process can be mathematically expressed as.\u003c/p\u003e \u003cp\u003eF\u003csub\u003e\u003cem\u003efused\u003c/em\u003e\u003c/sub\u003e\u003cem\u003e​\u003c/em\u003e = [F\u003csub\u003e\u003cem\u003eCNN\u003c/em\u003e\u003c/sub\u003e ​\u0026oplus; F\u003csub\u003e\u003cem\u003eBi\u0026minus;LSTM\u003c/em\u003e\u003c/sub\u003e​ \u0026oplus; F\u003csub\u003e\u003cem\u003eBERT\u003c/em\u003e\u003c/sub\u003e​]\u003c/p\u003e \u003cp\u003eWhere \u0026oplus; denotes the concatenation operation. This fused representation F\u003csub\u003e\u003cem\u003efused\u003c/em\u003e\u003c/sub\u003e aggregates the rich, diverse information captured from local patterns (CNN), sequential dependencies (Bi-LSTM), and deep contextual meanings (BERT). The model can leverage complementary information by merging these different features into a single vector, improving its overall detection performance and robustness in distinguishing spam from legitimate messages.\u003c/p\u003e \u003cp\u003e \u003cb\u003e4.4 Feature Selection\u003c/b\u003e: After the fusion of local, temporal, and contextual features, the next crucial step is feature selection. This step is vital for reducing the dimensionality of the combined feature vector and retaining only the most relevant features for accurate classification. Without proper selection, the fused feature vector may include redundant or irrelevant information, leading to increased computational complexity, inefficiency, and the risk of overfitting.\u003c/p\u003e \u003cp\u003eOur proposed model employs an attention mechanism to select features [\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e]. The attention layer dynamically assigns different weights to the features based on their importance for the classification task. Let the fused feature vector be represented as Ff\u003csub\u003e\u003cem\u003eused\u003c/em\u003e\u003c/sub\u003e, and the attention mechanism produces a set of attention scores α, which are learned during training. The attention mechanism can be formulated as:\u003c/p\u003e \u003cp\u003eF\u003csub\u003e\u003cem\u003eselected\u003c/em\u003e\u003c/sub\u003e\u0026thinsp;=\u0026thinsp;α\u0026sdot;F\u003csub\u003e\u003cem\u003efused\u003c/em\u003e\u003c/sub\u003e\u003c/p\u003e \u003cp\u003eHere, α represents the attention weights and \u0026sdot; denotes element-wise multiplication. The attention layer effectively learns which features from the fused representation contribute most to identifying spam and non-spam messages. Features more relevant to the task receive higher attention scores, while less important features are down-weighted. This results in a reduced feature vector, F\u003csub\u003e\u003cem\u003eselected\u003c/em\u003e\u003c/sub\u003e​, that retains the most critical information, thus improving the model's efficiency without sacrificing accuracy.\u003c/p\u003e \u003cp\u003e \u003cb\u003e4.5 Classification\u003c/b\u003e: The final step involves passing the selected feature vector through fully connected layers, followed by a softmax classifier, which predicts whether the SMS message is spam or legitimate. This classification layer is optimized to ensure high accuracy and low false positive rates.\u003c/p\u003e \u003cp\u003eThe proposed method integrates CNN, Bi-LSTM, and BERT in a parallel structure, ensuring a comprehensive feature extraction process and enhancing the effectiveness of SMS spam detection.\u003c/p\u003e \u003c/div\u003e"},{"header":"5. Evaluation","content":"\u003cp\u003eThis section evaluates the proposed model using the UCI SMS Spam Collection dataset, focusing on key metrics such as accuracy, precision, recall, and F1-score to validate its effectiveness.\u003c/p\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003e5.1 The dataset\u003c/h2\u003e \u003cp\u003eWe evaluated our method using the UCI SMS Spam Collection dataset [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e] hosted by the University of California, Irvine (UCI). This dataset is a well-known resource in SMS spam detection, comprising over 5,000 text messages categorized into two groups: legitimate (or \"ham\") messages and spam messages. Figure\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e shows the distribution of these labels within the dataset.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e shows examples from the UCI SMS Spam Collection dataset. Spam messages contain unsolicited offers or urgent calls to action, while ham messages are regular, legitimate communication. This dataset helps evaluate spam detection models by providing real-world message examples.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eExamples of SMS messages from the UCI SMS Spam Collection dataset\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMessage\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eClass\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\"Congratulations! You've won a \u003cspan\u003e$\u003c/span\u003e1000 gift card.\"\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSpam\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\"Hey, are we still meeting at 5 PM today?\"\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHam\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\"Claim your free prize now by clicking this link.\"\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSpam\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\"Don't forget to submit the report by tomorrow.\"\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHam\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\"Win a free vacation! Call now to claim.\"\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSpam\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003e5.2 Data Splitting\u003c/h2\u003e \u003cp\u003eData splitting is a crucial phase in developing and evaluating SMS spam detection models, as it ensures that models are trained and tested on separate datasets to prevent overfitting and provide an unbiased performance assessment. We employed the \u003cb\u003eK-Fold Cross-Validation\u003c/b\u003e, which partitions the dataset into K equal-sized folds. The model is trained and validated K times, each using a different fold as the validation set and the remaining K-1 folds as the training set [\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e]. The results from each fold are averaged to provide a more comprehensive evaluation of model performance. This method is especially beneficial for small and imbalanced datasets, as it maximizes the utilization of available data. For this study, we will employ 10-fold cross-validation to rigorously assess the performance of our SMS spam detection model, ensuring reliable and balanced results across multiple iterations.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003e5.3 Performance Metrics\u003c/h2\u003e \u003cp\u003eTo effectively evaluate the performance of deep learning models used in text classification tasks, such as spam detection, it is essential to consider several key metrics derived from the confusion matrix. These metrics comprehensively understand the model's predictive accuracy and reliability, especially when distinguishing between spam and legitimate (ham) messages. The confusion matrix outlines the possible outcomes of the model\u0026rsquo;s predictions, including true positives, false positives, and negatives, which form the basis for calculating the metrics [\u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e39\u003c/span\u003e].\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003ePerformance Metrics for Spam Detection\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMetric\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFormula\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eDescription\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eAccuracy\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\text{A}\\text{c}\\text{c}\\text{u}\\text{r}\\text{a}\\text{c}\\text{y}=\\frac{\\left(TP+TN\\right)}{\\left(TP+FP+TN+FN\\right)}\\ast\\:100\\%\\:\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eOverall correctness of the model in classifying both spam and non-spam messages.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003ePrecision\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\text{P}\\text{r}\\text{e}\\text{c}\\text{i}\\text{s}\\text{i}\\text{o}\\text{n}=\\frac{\\left(TP\\right)}{\\left(TP+FP\\right)}\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eThe proportion of messages predicted as spam (minimizes false positives).\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eRecall\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\text{R}\\text{e}\\text{c}\\text{a}\\text{l}\\text{l}=\\frac{\\left(TP\\right)}{\\left(TP+TN\\right)}\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eThe proportion of actual spam messages correctly identified by the model (minimizes false negatives).\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eF1-score\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\text{F}1\\:\\text{s}\\text{c}\\text{o}\\text{r}\\text{e}=\\frac{2\\ast\\:(Recall\\ast\\:Precision)}{\\left(Recall+Precision\\right)}\\:\\:\\:\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHarmonic means of precision and recall are balanced in the evaluation.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eThese performance metrics are critical in determining the efficacy of models that classify SMS or social media content, such as tweets, as spam or non-spam. By carefully analyzing these metrics, researchers and developers can fine-tune their models to optimize accuracy and reliability in real-world applications [\u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e40\u003c/span\u003e]. For instance, models that achieve high accuracy might underperform if precision or recall is low, indicating that a balance must be achieved between correctly detecting spam and avoiding the misclassification of legitimate messages.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003e5.4 Model Configuration\u003c/h2\u003e \u003cp\u003eThe proposed model is structured to maximize the effectiveness of feature extraction and fusion using a combination of CNN, Bi-LSTM, and BERT. The configuration is designed to handle the nuances of short text data, particularly SMS messages, while efficiently learning patterns that distinguish spam from legitimate communication.\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eInput Layer\u003c/b\u003e: This layer processes the tokenized and padded text sequences, ensuring uniform input for the following layers. Textual data is converted into word embeddings to capture semantic meaning.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eCNN Layer\u003c/b\u003e: A 1D Convolutional layer with 128 filters of size 3 is used to extract local patterns from the input text. CNN is responsible for identifying short-term dependencies and word n-grams commonly found in spam messages. The convolution operation helps capture spatial hierarchies and n-gram structures in the text.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eMax Pooling Layer\u003c/b\u003e: Following the CNN layer, a Global Max Pooling layer is applied to reduce the dimensionality of the feature maps. This layer highlights the most important local features while minimizing computational complexity.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eBi-LSTM Layer\u003c/b\u003e: A Bidirectional LSTM layer with 64 units is employed to capture sequential dependencies and the temporal flow of words. By processing the text forward and backward, the Bi-LSTM extracts long-term dependencies, which are critical in understanding the structure and intent behind spam messages.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eBERT Layer\u003c/b\u003e: BERT, pre-trained on a large corpus, is integrated to extract contextual features from the input text. Its transformer architecture, equipped with multi-head attention, enables the model to capture deep contextual relationships by considering the surrounding words in both directions. This helps in understanding the semantic meaning of the message and distinguishing deceptive language.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eFeature Fusion Layer\u003c/b\u003e: As described in the previous section, the outputs of the CNN, Bi-LSTM, and BERT layers are concatenated into a single feature vector. This fused vector aggregates the input message's local, temporal, and contextual features, providing a comprehensive representation.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eAttention Mechanism\u003c/b\u003e: To perform feature selection, an attention layer is applied to the fused feature vector. The attention mechanism assigns weights to each feature based on its importance for the classification task, allowing the model to focus on the most relevant information.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eFully Connected Layers\u003c/b\u003e: Two fully connected layers, with 256 and 128 units, respectively, follow the feature extraction process. Dropout is applied at a rate of 0.5 to prevent overfitting and ensure robustness during training.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eOutput Layer\u003c/b\u003e: The final layer uses softmax activation to classify the SMS as either spam or non-spam. The softmax function outputs the probabilities for each class, allowing the model to provide a confident prediction.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003eThis configuration ensures that the model captures different feature types (local, temporal, and contextual) and integrates them in a way that optimally supports the classification of spam and non-spam messages. Attention mechanisms further enhance the model\u0026rsquo;s ability to focus on the most critical features, improving its accuracy and reducing false classifications.\u003c/p\u003e \u003c/div\u003e"},{"header":"6. Results","content":"\u003cp\u003eUsing a 10-fold cross-validation approach, the model demonstrated robust performance across all folds, with accuracy results illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e. The average accuracy across folds was 99.68%, indicating consistent performance. In addition to accuracy, the model achieved the following average metrics across the folds: Precision: 99.65%, Recall: 99.70%, and F1-Score: 99.67%. These metrics reflect the model's effectiveness in correctly identifying spam messages while minimizing false positives and negatives. The consistently high values across all metrics suggest that the proposed model excels in overall accuracy and maintains a strong balance between precision and recall, making it highly reliable for practical spam detection applications.\u003c/p\u003e \u003cp\u003eThe comparative analysis of various methodologies for SMS spam detection highlights significant advancements in the field. Traditional approaches, such as T. Shahi et al.\u0026rsquo;s use of TF-IDF with Naive Bayes, achieved an accuracy of 92.67%. However, more sophisticated techniques, including SVM and hybrid models like CNN with LSTM, demonstrated improved performance, with accuracies reaching up to 99.28%. MRF Derakhshi et al.\u0026rsquo;s parallel CNN-LSTM model and the proposed dual-attention CNN-BiLSTM with the BERT approach outperformed previous methods, achieving an accuracy of 99.68%. This indicates the effectiveness of integrating multiple feature extraction techniques and leveraging deep learning architectures to enhance spam detection capabilities, ultimately leading to more robust and reliable models distinguishing between spam and legitimate messages. Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e compares SMS spam detection methods and their accuracy rates.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eComparison of SMS spam detection methods and their accuracy rates.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eReference\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMethodology\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAccuracy (%)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eShahi[\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTF-IDF\u0026thinsp;+\u0026thinsp;Naive Bayes\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e92.67\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eJehad [230]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eRNN\u0026thinsp;+\u0026thinsp;LSTM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e98.00\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e. Shaaban [234]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHybrid model (CNN\u0026thinsp;+\u0026thinsp;Pooling\u0026thinsp;+\u0026thinsp;Ensemble Learning)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e98.38\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGhourabi [227]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSequential CNN-LSTM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e98.37\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDerakhshi [228]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eParallel CNN-LSTM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e99.28\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLiu [206]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eModified Transformer Model\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e98.92\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHameed [65]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFuzzy Rule-Based\u0026thinsp;+\u0026thinsp;Binary PSO\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e98.50\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLarijani [204]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHybrid Classical-Quantum Transfer Learning\u0026thinsp;+\u0026thinsp;BERT\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e95.00\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGupta [221]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTF-IDF\u0026thinsp;+\u0026thinsp;CNN\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e99.10\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eProposed Method\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCNN\u0026thinsp;+\u0026thinsp;BiLSTM\u0026thinsp;+\u0026thinsp;BERT\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e99.68\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e compares the proposed model's performance with existing SMS spam detection methods. They highlight the superior accuracy and robustness of the proposed CNN-BiLSTM-BERT hybrid approach, demonstrating its effectiveness in capturing diverse feature types and outperforming traditional and hybrid models.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eWhile the proposed model demonstrates high accuracy and robustness, it has limitations. Relating to computationally intensive techniques like BERT and Bi-LSTM may make it less suitable for real-time applications or deployment in resource-constrained environments. Additionally, the model's performance has been validated on a single dataset, which may only partially capture the diversity of SMS spam in different languages or contexts.\u003c/p\u003e"},{"header":"7. Conclusion","content":"\u003cp\u003eThis paper introduced a parallel feature extraction method for SMS spam detection, combining CNN, Bi-LSTM, and BERT to capture diverse features: CNN for local patterns, Bi-LSTM for sequential dependencies, and BERT for contextual understanding. The model achieves a comprehensive representation by fusing these feature types into a unified vector, enhancing its accuracy and robustness against complex spam tactics. An attention-based feature selection step further refines this vector, retaining only the most relevant information, reducing computational complexity, and minimizing overfitting. Future research could focus on making the model adaptable to evolving spam tactics through real-time learning and adaptive updates. Expanding this approach to multimodal data, such as combining message content with metadata or user behavior, may also improve spam detection. Additionally, experimenting with other transformer models like RoBERTa or T5 and optimizing the model for deployment in resource-limited environments (e.g., mobile devices) would make it more versatile and practical for real-world applications.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eHussein Al-Kaabi conceptualized the study, developed the hybrid model, and conducted the primary analysis. Mohammed Al-Rekabi contributed to the methodology design, conducted experiments, and prepared the initial draft of the manuscript. Fuqdan Al-Ibraheemi reviewed the results, validated the findings, and assisted with the interpretation of the data. Ali Kadhim Jasim prepared figures and tables and provided editorial revisions to the manuscript. All authors contributed to the review and finalization of the manuscript and approved the submitted version.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eAprinia D (2022) Role Of Truecaller Application. Preventing Phone Call Text Message Scams Jurnal Mantik 6(2):1475\u0026ndash;1483\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu X et al (2020) Privacy and security issues in deep learning: A survey. IEEE Access 9:4566\u0026ndash;4593\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKamber AN (2020) A Variety of Electrocardiogram (ECG) Signal Processing. Solid State Technol 63(3):5370\u0026ndash;5377\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAl-kabi, Hussein, Sayyed Majid Mazinani (2021) DNCS: New UAV navigation with considering the no-fly zone and efficient selection of the charging station. Ain Shams Eng J 12(4):3669\u0026ndash;3676\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAl Saidat M, Rasol, Suleiman Y, Yerima (2024) Advancements of SMS Spam Detection: A Comprehensive Survey of NLP and ML Techniques. Procedia Comput Sci 244:248\u0026ndash;259\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAl-Kaabi H, Darroudi AD, Jasim AK (2024) Survey of SMS Spam Detection Techniques: A Taxonomy, \u003cem\u003eKJCS\u003c/em\u003e, vol. 2, no. 4, pp. 23\u0026ndash;34, Dec\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAlmeida TA, Jos\u0026eacute; Mar\u0026iacute;a G, Hidalgo, Yamakami A (2011) Contributions to the study of SMS spam filtering: new collection and results. \u003cem\u003eProceedings of the 11th ACM symposium on Document engineering\u003c/em\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAdewole K, Sakariyah et al (2019) SMSAD: a framework for spam message and spam account detection. Multimedia Tools Appl 78:3925\u0026ndash;3960\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAl Saidat M, Rasol, Suleiman Y, Yerima, Shaalan K (2024) A Novel Approach for Arabic SMS Spam Detection Using Hybrid Deep Learning Techniques. Procedia Comput Sci 244:260\u0026ndash;267\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTheodorus A et al (2021) Short message service (SMS) spam filtering using machine learning in Bahasa Indonesia. \u003cem\u003e2021 3rd East Indonesia Conference on Computer and Information\u003c/em\u003e Technology \u003cem\u003e(EIConCIT)\u003c/em\u003e. IEEE\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShahi TB, Yadav A (2013) Mobile SMS spam filtering for Nepali text using na\u0026iuml;ve bayesian and support vector machine. Int J Intell Sci 4(1):24\u0026ndash;28\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSjarif NN, Amir et al (2020) Support vector machine algorithm for SMS spam classification in the telecommunication industry. Int J Adv Sci Eng Inf Technol 10(2):635\u0026ndash;639\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSaeed VA (2023) A Method for SMS Spam Message Detection Using Machine Learning. Artif Intell Robot Dev J 3:214\u0026ndash;228\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJehad M, Nassif AB, AlShabi M (2023) Spam SMS detection based on long short-term memory and recurrent neural network. Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications V, vol 12538. SPIE\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShaaban MA, Yasser F, Hassan, Shawkat K (2022) Guirguis. Deep convolutional forest: a dynamic deep ensemble approach for spam detection in text. Complex Intell Syst 8(6):4897\u0026ndash;4909\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGhourabi A, Mahmood MA, Qusay M (2020) Alzubi. A hybrid CNN-LSTM model for SMS spam detection in arabic and english messages. Future Internet 12:156\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDerakhshi MR, Feizi et al (2024) Pclf: parallel cnn-lstm fusion model for sms spam filtering. \u003cem\u003eBIO Web of Conferences\u003c/em\u003e. Vol. 97. EDP Sciences\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu X, Lu H, Nayak A (2021) A spam transformer model for SMS spam detection. IEEE Access 9:80253\u0026ndash;80263\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eManasa P, Malik A, Batra I (2024) Detection of twitter spam using GLoVe vocabulary features, bidirectional LSTM and convolution neural network. SN Comput Sci 5(2):206\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHameed SM, Zuhair Hussein A (2021) SMS Spam Detection Based on Fuzzy Rules and Binary Particle Swarm Optimization. Int J Intell Eng Syst 14.2\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eArdeshir-Larijani (2024) Ebrahim, and Mohammad Mahdi Nasiri Fatmehsari. Hybrid classical-quantum transfer learning for text classification. Quantum Mach Intell 6(1):19\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGupta M et al (2018) A comparative study of spam SMS detection using machine learning classifiers. \u003cem\u003eeleventh international conference on contemporary computing (IC3)\u003c/em\u003e. IEEE, 2018\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAftan S, Shah H. A survey on bert and its applications. (2023) \u003cem\u003e20th Learning and Technology\u003c/em\u003e Conference \u003cem\u003e(L\u0026amp;T)\u003c/em\u003e. IEEE, 2023\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAl-Kabbi H, Alaa M-R Feizi-Derakhshi, and, Pashazadeh S (2024) A Hierarchical Two-Level Feature Fusion Approach for SMS Spam Filtering. Intell Autom Soft Comput 39:4\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKalyan K, Subramanyam (2024) A survey of GPT-3 family large language models including ChatGPT and GPT-4. Nat Lang Process J 6:100048\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAdam, Edriss Eisa Babikir (2020) Deep learning based NLP techniques in text to speech synthesis for communication recognition. J Soft Comput Paradigm (JSCP) 2(04):209\u0026ndash;215\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAl-Kabbi H, Alaa M-R (2023) Feizi-Derakhshi, and Saeid Pashazadeh. Multi-type feature extraction and early fusion framework for SMS spam detection. IEEE Access\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAlmarashy AH, Jawad (2023) Mohammad-Reza Feizi-Derakhshi, and Pedram Salehpour. Enhancing fake news detection by multi-feature classification. IEEE Access\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDerakhshi MR, Feizi et al (2024) Pclf: parallel cnn-lstm fusion model for sms spam filtering. \u003cem\u003eBIO Web of Conferences\u003c/em\u003e. Vol. 97. EDP Sciences\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang W et al (2023) Sentiment Text Spatio-Temporal Feature Analysis based on CharCNN and BiLSTM. \u003cem\u003e4th International Conference on Big Data \u0026amp; Artificial Intelligence \u0026amp; Software Engineering (ICBASE)\u003c/em\u003e. IEEE, 2023\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSalman Al-Tameemi IK et al (2023) An Efficient Sentiment Classification Method with the Help of Neighbors and a Hybrid of RNN Models. \u003cem\u003eComplexity\u003c/em\u003e 1 (2023): 1896556\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRamezani M, Mohammad-Reza Feizi-Derakhshi, and, Mohammad-Ali B (2022) Research Article Knowledge Graph-Enabled Text-Based Automatic Personality Prediction\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen X, Cong P, Lv S (2022) A long-text classification method of Chinese news based on BERT and CNN. IEEE Access 10:34046\u0026ndash;34057\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBusst MM, Azman et al (2024) Ensemble BiLSTM: A Novel Approach for Aspect Extraction From Online Text. IEEE Access\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChoi H et al (2020) Evaluation of bert and albert sentence embedding performance on downstream nlp tasks. \u003cem\u003e25th International conference on pattern recognition (ICPR)\u003c/em\u003e. IEEE, 2021\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePawłowski M (2023) Anna Wr\u0026oacute;blewska, and Sylwia Sysko-Romańczuk. Effective techniques for multimodal data fusion: A comparative analysis. Sensors 23:2381\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFarhan BI, Ammar D, Jasim (2023) Improving Detection for Intrusion Using Deep LSTM With Hybrid Feature Selection Method. Iraqi J Inform Communication Technol 6(1):40\u0026ndash;50\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMuhammed S, Mahdi (2023) Ghassan Abdul-Majeed, and Mahmoud Shuker Mahmoud. Prediction of heart diseases by using supervised machine learning algorithms. Wasit J Pure Sci 2(1):231\u0026ndash;243\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJabbooree A, Issa et al (2023) A novel facial expression recognition algorithm using geometry β\u0026ndash;skeleton in fusion based on deep CNN. Image Vis Comput 134:104677\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAlabid NN (2024) A Review of Sentiment Analysis in Social Media Perspectives. J Kufa Math Comput 112:1\u0026ndash;11\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Feature Fusion, Hybrid Deep Learning, SMS Spam Detection, Text Classification","lastPublishedDoi":"10.21203/rs.3.rs-5865706/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5865706/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eIn the era of pervasive digital communication, SMS spam poses significant threats, including financial fraud and phishing attacks, necessitating robust detection mechanisms. This paper introduces a novel hybrid model for SMS spam detection, leveraging advanced deep-learning techniques to capture diverse textual features comprehensively. The model integrates Convolutional Neural Networks (CNN) for local feature extraction, Bidirectional Long Short-Term Memory (Bi-LSTM) networks for sequential dependencies, and Bidirectional Encoder Representations from Transformers (BERT) for contextual embeddings. A parallel architecture combines these components to achieve a holistic representation of SMS content. Fused feature vectors undergo attention-based selection to enhance computational efficiency while preserving critical information. Evaluated on the UCI SMS Spam Collection dataset using a 10-fold cross-validation strategy, the proposed model achieves a remarkable accuracy of 99.68%, outperforming state-of-the-art techniques. This work addresses the limitations of traditional and hybrid methods, offering a highly reliable and adaptable solution to the evolving challenges of SMS spam detection. Future directions include real-time adaptability, multimodal integration, and resource-efficient deployment.\u003c/p\u003e","manuscriptTitle":"Fusion-Based Hybrid Model for SMS Spam Detection Integrating Local, Sequential, and Contextual Features","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-01-23 06:05:13","doi":"10.21203/rs.3.rs-5865706/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"6b353715-fd74-433e-b2c2-92852023812c","owner":[],"postedDate":"January 23rd, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-02-08T02:08:27+00:00","versionOfRecord":[],"versionCreatedAt":"2025-01-23 06:05:13","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5865706","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5865706","identity":"rs-5865706","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}