A novel reinforcement learning-based hybrid intrusion detection system on fog-to-cloud computing

A novel reinforcement learning-based hybrid intrusion detection system on fog-to-cloud computing | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A novel reinforcement learning-based hybrid intrusion detection system on fog-to-cloud computing Sepide Najafli, Abolfazl Toroghi Haghighat, Babak Karasfi This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3864298/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 17 You are reading this latest preprint version Abstract The increasing growth of the Internet of Things and its open and shared character has led to a rise in new attacks exponentially. As a consequence, the quick detection of attacks in IoT environments is essential. The intrusion detection system (IDS) is responsible for protecting and detecting the type of attacks. Creating an IDS that works in real-time and adapts to environmental changes is critical. In this paper, we propose a deep reinforcement learning-based (DRL) adaptive IDS that addresses the mentioned challenges. DRL-based IDS helps to create a decision agent, who controls the interaction with the indeterminate environment and performs binary detection (normal/intrusion) in fog. For multi-class classification of attacks, we use the ensemble method in the cloud. The proposed approach is evaluated on the CICIDS2018 dataset, in binary and multiclass classification. The results show that the proposed model achieves a comparable performance in detecting intrusions and identifying attacks and it reduces the prediction time significantly. Furthermore, it is a suitable solution for underrepresented attacks. Overall, combining multiple methods can be a great way for an intrusion detection system. IoT Intrusion detection system Deep learning Reinforcement learning Deep Q-networks Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Introduction The Internet of Things (IoT) provides a global network of devices, computers, and sensors able to interact with each other [1], [2]. Cloud computing is a centralized solution, and provides unlimited, remote computing and storage resources for the IoT applications. While cloud computing supplies a large amount of resources, it faces challenges like limited and unreliable bandwidth, security and high communication latency between IoT devices and cloud servers[3]. Therefore, latency-sensitive applications in IoT cannot be efficiently executed. To solve the mentioned problems Fog computing extends cloud computing and provides storage, processing, and other things. It is a heterogeneous paradigm and distributed in the proximity of IoT devices. The fog layer is an intermediate layer between the Cloud and the IoT and is able to deploy resources close to the edge of the network. Therefore, Fog is an optimal solution for latency-sensitive IoT applications. However, the fog has restricted resources in comparison to the cloud[4]. Fog-to-Cloud computing[5] is also a hybrid type of computing that is used in many aspects of IoT, such as healthcare, agriculture, driving, and industry. The characteristics of the IoT network have made it susceptible to various kinds of attacks. These attacks damage both devices and data. Therefore, the implementation of security systems is essential. An Intrusion Detection System (IDS) is a helpful security tool to detect malicious actions or attacks[6]. IDS can be implemented in IOT devices, cloud layer, or fog layer. It should be noted that due to limitations in processing and storage in most IOT devices, they are not able to run such applications. On the other hand, deploying IDS in the cloud layer will lead to delays. Consequence the most suitable option is to implement IDS in fog. Therefore, we need to design a new generation of intrusion detection systems that are suitable for the fog layer. The new generation (IDS) requires automatic and intelligent intrusion detection strategies to manage threats. The new IDS must be able to operate based on an independent agent and with minimal human intervention. At the same time, they should be able to evolve and improve themselves. Although deep learning methods had a significant impact on the development of intrusion detection systems, their efficiency is highly dependent on the selection of appropriate features and the setting of hyperparameters used for DL training[7]. In addition, DL-based techniques face limitations in adapting to changing attack patterns. In other words, they cannot adjust themselves to any possible change in the attack pattern of the environment. As a result, accuracy will decrease and FPR will increase. Reinforcement learning (RL) is the most effective solution to security IoT against new various attacks. It can integrate environmental behavior into the learning process simultaneously[8]. On the other hand, it can model an autonomous agent to take actions optimally without knowledge of the environment. Therefore, it is useful in real-time and adversarial environments. Reinforcement learning is modeled using a Markov decision process (MDP) consisting of five entities, state, action, reward, policy, and value. The foundation aim of reinforcement learning is to find an optimal policy. The policy function is used to compute the next best action, based on the current state and reward provided by the environment. Since the IoT environment is growing and becoming more complex, the RL agent will not be able to learn a larger number of parameters. On the other hand, the fog environment is stochastic in several aspects such as resource requirements of applications and the arrival rate of application requests. Thus, heuristic-based techniques are not able to efficiently adapt to changes in the fog. Deploying the deep reinforcement learning (DRL) model that combines deep learning (DL) and reinforcement learning (RL) can overcome this issue. DRL agents can learn the optimal policy and long-term rewards without prior knowledge[9]. In other words, deep learning significantly helps to optimize functions in reinforcement learning to create an optimal policy. DRL can deal with complex problems in dynamic and stochastic fog environments[10]. Moreover, DRL is also excellent for security applications where attacks are increasingly sophisticated, rapid, and ubiquitous[11]. The efficiency of IDS is highly dependent on the deep learning technique and their model training mechanisms. Therefore, feature selection methods and hyperparameter tuning greatly impact IDS performance, especially in complex environments where the number of parameters is increasing[12]. Using DRL for IDS causes DNNs as performance estimators to extend the scalability of RL as well as adapt to complex IoT. DLR increases the capability of continuous self-learning by interacting with the environment and improving its ability over time. To address the challenges raised and due to the advantages of DRL and DL, we have proposed a Deep Reinforcement Learning approach to separate normal events from intrusions in fog and we have used several machine learning methods to recognize types of attacks in the cloud. The main contributions of this paper are summarized as follows: · A two-level state-of-the-art suitable for online learning in a dynamic environment based on the Deep Reinforcement Learning approach for binary, and ensemble learning for multiclass detection. · Employing GRU to learn the internal relations between events based on a relatively long duration of the past time. Using Deep Q-learning to estimate the Q-function and choosing the appropriate action (permeation/normal) in fog. Utilizing multiple machine learning models in the cloud to detect the attack type of suspicious events · Sharing the information obtained in different fog nodes to the summary module in the cloud to update the parameters. · Extensive experiments were conducted to evaluate and compare the efficiency of the proposed DQL models. Our experimental results based on the CIC-IDS2018 dataset, demonstrate that it is effective in detecting intrusion of underrepresented classes and is excellent in achieving prediction time in comparison to another approach. The rest of the paper has been organized into six different sections. Section 2 is about the related work on IDS using deep learning and reinforcement learning techniques. Section 3 describes detailed of the model description. The experimental evaluation, results of the proposed model, and the comparison models are given in Section 4. Section 5 presents limitations and future scope. Finally, Section 6 concludes our paper. Related Work Cybersecurity concerns have become a significant obstacle to the faster adoption of IoT technology. Although numerous IDS have been proposed in the literature, they frequently have poor accuracy because of creating various attacks in IoT environments and the lack of adaptation of attack patterns. A lot of research has been conducted to enhance IDS performance. Researchers use a variety of techniques including machine learning and deep learning[ 13 ],[ 14 ],[ 15 ]. Recently Deep Reinforcement Learning has provided a promising solution that can improve the efficiency of IDS in dynamic environments[ 16 ],[ 17 ]. Considering the large number of works on IDS techniques, we study in this section, some related works of DRL-based and DL-based techniques in IoT environment. Sethi et al. [ 18 ] have proposed a deep reinforcement learning-based IDS. It is adaptive to cloud architecture. It consists of three components, namely the host network, agent network, and administrator network. This method can provide higher accuracy against new and complex attacks and is able to maintain a balance between high accuracy and low false positive rate (FPR) compared to state-of-the-art IDSs. Experiments have been done on the UNSW-NB15 test dataset. The result demonstrates 83.80% average accuracy and 2.6% FPR. Moreover, this model has been successful in detecting all types of attacks. Yang et al. [ 19 ] have conducted an intrusion detection model based on a two-layered bidirectional long short-term memory (Bi-LSTM) on fog. In the first stage fog nodes identify attacks based on traffic received from IoT devices and then send them to the cloud to summarize the global security condition. The first level is organized into three phases, including: preprocessing, categorizing and collecting data. If the predicted events are classified into normal groups, IoT devices are permitted to continue the process, otherwise activities are categorized as suspicious. The model has achieved the highest accuracy (99.05%), the highest detection rate (99.36%), and the highest f1-measure (99.154%) on the UNSW-NB15 dataset. Strickland et al. [ 20 ] have proposed a novel hybrid technique that can perform binary and multiclass classification. The model uses GANs to improve the learning capabilities of the proposed model. The GAN model is trained with the NSL-KDD dataset for four attack categories and normal traffic. DRL is used to implement this IDS. Although overall accuracy decreased, the results show generating synthetic data can improve the precision and recall for underrepresented classes in imbalanced datasets. Lazzarini et al. [ 21 ] suggest a novel intrusion detection approach named Deep Integrated Stacking for the IoT (DIS-IoT). This model works based on combining four different DL models (MLP, DNN, CNN and LSTM) into a fully connected DL layer, creating a standalone ensemble model. DIS-IoT implements binary and multiclass classification techniques. Experiments demonstrate the model on ToN-IoT and CICIDS2017 datasets has achieved great scores in accuracy, recall, precision and F1- Measure in both binary and multi-class classification. This method tries to show that the combination of several methods can be effective in improving the metrics. Authors in [ 22 ] present a distributed binary IDS method based on neural networks and genetic algorithms to efficiently detect various intrusions on Fog nodes. Genetic algorithm is used to optimize the weights of the network and the biases associated with each neuron. Experiments have been implemented on UNSW-NB15 and ToN_IoT datasets. The results show that optimized weights and biases can improve the performance model compared to the neural network without optimization. The proposed model compared to other state-of-the-art methods achieved a 16.35% and 37.07% reduction in execution time for both datasets. Therefore convergence process will increase and will save processing power. Stefanova et al. in [ 23 ] have proposed an intrusion response mechanism that is based on reinforcement learning techniques (RLT). The model uses a Q-learning approach. The agent can control the process of interacting with the indeterminate environment and finding an optimal policy to represent the intrusion response. Evaluation of the Model shows proposed IDS based on Q-learning establishes the balance between exploration and exploitation. Lopez-Martin et al. [ 24 ] present a novel application of several deep reinforcement learning (DRL) algorithms to IDS. It performs supervised learning based on a DRL framework. The model is suitable for dynamic environments where online learning is necessary. The authors supply a comparison study of four algorithms (DQN, DDQN, Policy gradient and actor-critic). They compare four algorithms with machine learning models on NSL-KDD and AWID datasets. Results illustrate DDQN algorithm has an excellent performance in accuracy, F1, precision and recall. Moreover, the performance of prediction scores, training and prediction times in DDQN was better than other approaches. In addition to the methods mentioned above, there are also other different proposals such as[ 25 ], [ 26 ], [ 27 ], [ 28 ], [ 29 ], [ 30 ] in intrusion detection. Stressing privacy preservation in the IoT is important. Therefore adopting a strategy to detect attacks is essential. Each of the existing suggested methods has examined specific attacks or specific aspects of the intrusion detection system and has advantages and disadvantages. Due to the effective features of deep learning and reinforcement learning, the study based on these methods in intrusion detection systems is still ongoing. Model description Intrusion detection and classifying the types of attacks are the two most important of our purpose. Our study uses deep Q-learning and machine learning approaches to binary and multiclass intrusion detection, respectively. Since we need fast response in binary diagnosis, binary detection is implemented in the fog layer. In other words, deep Q-learning identifies attacks in the fog layer. Events marked as intrusions are sent for more analysis and to identify types of attacks on the cloud. Cloud can perform complex analysis because there is no need for an urgent classifier and it has flexibility in response time. In the cloud layer, there is a robust method. It includes an ensemble machine learning approach of multi-class classification. In the first step, we model the environment using GRU architecture to learn the internal relations between events in fog. In general, we integrate the influence of the historical information of the dynamic environment on policy optimization. Because we aim to detect attacks with small mutants compared to previous attacks, and recognition zero-day attacks in an IoT environment. The important point is to send and share the information obtained in different fog nodes to the summary module in the cloud. That means the information created in each fog node is sent to the cloud so that it will be visible to all nodes. This module is responsible for updates based on received information. Generally, the proposed method is implemented in three phases, which are: 1) the preprocessing and environment modeling phase, 2) the binary detection and parameter updating phase, and 3) the multi-class detection phase. The overview of the proposed architecture is shown in Fig. 1 . Three proposed phases with detail are described in the following subsections. 3.1 preprocessing and environment modeling phase Since the IoT environment is completely variable and new attacks are being produced with very few differences compared to previous attacks, finding the relationship between events will significantly help in detecting intrusions and their types. GRU can learn values over short and long periods. It was proposed to solve the vanishing gradient problem in RNN. GRU is similar to LSTM but simpler than LSTM and uses fewer parameters. Therefore, it will be more computationally efficient. The use of GRU can lead to the benefit of the information on previous events in the detection of new attacks. In other words, prior information can be incorporated into an internal state that is a suitable representation of the interactive environment. These states are entered into Deep Q-Learning to perform binary recognition. We use Deep Q-Learning to deploy agents in discrete action space environments. According to Fig. 2 , B n (Mini Batch) records are sampled from the dataset in the first step. Choosing B n records means that each state (S) will be equal to B n records, and each record will contain S 1 to S m features. In the Next step (Fig. 3 ) (S i ), (h i -1) and (A i -1) which represent the current state, the hidden state of the previous step, and the previous action, respectively, enter the GRU module at any time (t). Eq. (1) calculations are performed in each GRU unit. Z i = δ (W z S i + U z h i−1 ) r i = δ (W r S i + U r h i−1 ) (1) h ' i = tanh (WS i ʘ Uh i−1 ) h i = (1-Z i ) ʘ h i−1 + z i ʘ h i ) Z i is the update gateway for state i. This gateway decides what information should be discarded and what new information should be kept in it. W z and U z are the weights of S i and h i−1 , respectively. Next, the sigmoid activator function is used to produce the output (0 or 1). In this stage, the model decides how much information to transfer to the future. This action also avoids the problem of gradient vanishing. r i is the reset gate. In this gate is decided what information from the past should be forgotten. W r shows the weight of S i and U r shows the weight of h i−1 . We use h ' i to specify how the two update and reset gates determine the output. The output of this function tells what to remove from the previous step. In the last step, the network decides to keep the information related to the current memory and transfer it to the network by calculating the h i−1 vector. This requires the update gate to determine what information to retrieve from the current memory contents of hi and what from the previous step h i−1 . This step is calculated by the equation h i . The output of this module includes the prediction of the next observation S i+1 and the next hidden state h i . In fact, the produced h i indicates what information from the previous step h i−1 and what information from the current observation S i should be stored. In addition to being used as an input at time t + 1 in the GRU, the generated output h i is also sent to the binary detection module to perform attack detection. In general, we perform binary detection on data that have useful information from previous events and current events. In other words, we have provided the ability to learn in earlier time steps. 3.2 Binary detection phase RL is based on the Markov Decision Process (MDP) theory. An MDP consists of tuples (S, A, T, R), where S is a set of states, A is a set of actions, T is a mapping function that presents the probability of transition from the pair (S, A) to a new state, and R is the reward function. The mapping function uses Markov probability. In such a way that the probability of transition to a new state depends only on the current state. MDP aims to learn the optimal policy so that it chooses the best action for each state. As mentioned deep Q-learning is used as a binary detection technique. Deep Q-learning is a model-free algorithm and it doesn’t create a model of the environment’s transition function. The purpose of Deep Q-learning is to estimate the Q-Value and solve the Q(S, A) function based on the experience samples in each time step according to the following formula: Q(S.A)←Q(S i ,A i )+α(R i + γQ(S i+1 ,A i+1 )-Q(S i ,A i )) (2) To obtain Q(S, A), it is necessary to calculate Q (S i ,A i ) which represents the function in state i, (S i+1 ,A i+1 ) the function in state i + 1, α is the learning rate and R i is the reward for the action is A i . Therefore, to update Q(S, A) for each action A in state S must calculate the estimated yield R i + γQ (S i+1 , A i+1 ). The estimated yield is also called TD-Target. Repeated execution of the update rule results in the correct Q(S, A). Before implementing binary detection, the initialization of the parameters and the DQL must be done. It means that the output from the modeling phase is entered into the DQL and it determines the QV, SV, AV and B n vectors. As shown in algorithm1, we use two categories of iteration (inner iteration and outer iteration) to detect intrusion in the binary detection phase. The outer iteration is used to train the DQL model, and the inner iteration is responsible for improving the Q-values. Each B n represents a state (S) which is determined at the beginning of the outer iteration. The specified current state (S i ) is entered into the outer loops to perform the DQL training process. At the end of the external iteration, the B n value will be reset and the DNN parameters will be updated if needed. As mentioned, inner iteration is used to improve Q-values. In the inner iteration, the Q-function is estimated by the DNN function. The values of B n records of the environment represent the current state of the environment (note that the current state in this step is the output of the modeling step. It means that previous events have also affected the current state). We use a deep neural network (Q-Network) to calculate Q(S, A). In each inner iteration, the characteristics of each record (values of variables) are entered into the DNN input layer. The DNN output layer also displays Q-values (intrusion/normal). All values obtained for each iteration are stored in the vector Q (Q-Value). The action that has the highest Q value (Q-Value) is predicted as the current action (A' i ) for the desired record according to the following equation. Action = argmax (Q-Value) (3) The noteworthy point is that the epsilon-greedy approach is used for exploration in DRL. Epsilon-greedy is a learning strategy based on the definition of reinforcement learning that helps the agent discover all possible actions by increasing the number of explorations and finding the optimal policy. An action selected by the epsilon-greedy approach is an action that is either randomly selected with probability ɛ or an action predicted with probability (1-ɛ). In the first inner iteration, the probability of choosing a random action is high. Over time, due to the use of the epsilon-greedy approach, this probability has decreased and actions are predicted with a probability of (1-ɛ). The DNN used is a three-layer deep neural network that uses the ReLU activation function for all layers. The loss function used is the Mean Square Error function. In the following, the values in (A ' i ) and the label of each state (A i ) are used to calculate the reward, Eq. (4). RV i =Reward (AV i ,label i ) (4) According to the Eq. (2) we need to calculate TD-Target. For this purpose, we must calculate QV i+1 . As you can see in Fig. 4 the next state (S i+1 ) is used to calculate QV i+1 and obtain TD-Target. Here we also use another deep neural network (Target-Network) with separate parameters from Q-Network). According to the Eq. (5): QT i =RV i + γ Q (S i+1 , Ai + 1 ) QT i =RV i + γ (QV i+1 ) (5) RV i is the earned reward for S i , [0, 1] € γ is the discount factor for future rewards and QV i+1 is the state vector of S i+1 . The value of γ is updated in each iteration. After each iteration, the loss function will be calculated to improve the performance of the neural network (Target-Network). According to Eq. (6) loss function is calculated at the end of each inner iteration. Loss= (QV i -QT i ) Loss = 1/n∑ (Q s -Q T ) r + γQ(S , ,a , )) 2 (6) At the end of the outer iteration, the updated parameters are sent to the Q-Network and Target-Network. Using two neural networks (Target-Network, Q-Network) will lead to better stability of the model. Note that outer iterations continue until the all dataset is covered. The event that is detected as an attack is provided to the cloud in order to detect the type of attack. 3.3 Multi-class detection phase Classifying attacks into different types requires a more complex model and more resources. The multi-class detection phase of the proposed method is less sensitive to delay. Therefore, are used several algorithms in the cloud. The main idea of ensemble methods and the combination of several conceptually different machine learning classifiers is to achieve higher improvement and robustness than individual classifiers. In other words, ensemble methods often provide better estimates than the best estimators. Ensemble models perform best when independent predictors are used as much as possible because the chance of making different mistakes increases and the accuracy of the model increases. The main challenge of Ensemble-based methods is how to combine base classifiers (base learner). Here we use majority voting to aggregate the results. All the information related to detected events (binary detection phase, multi-class detection phase) is sent to the summary module. This module is responsible for updating the model. The updated parameters are sent for the binary detection phase and the multi-class detection phase. Experimental Evaluation To evaluate the performance of the proposed method, experiments are carried out on CICIDS2018[ 31 ] datasets. First, we describe the dataset used in the experiment, then consider the metrics to evaluate the model, and elaborate on the setting parameters. Finally, we analyze the experimental results and compare our approach and existing techniques. 4.1 Dataset Assessing the dataset accuracy and collecting based on the real world is a challenging issue. In this study, CICIDS2018 is used as input and the model is evaluated using the CSE-CIC-IDS2018 dataset. As a result of cooperation between the Communications Security Establishment (CSE) and The Canadian Institute for Cybersecurity (CIC) the CIC-IDS2018 datasets were created. They used the concept of profiles to make a cybersecurity dataset that contains an explanation of intrusions and abstract distribution models for protocols and applications. Heartbleed, Botnet, Web attacks, Brute force attacks, DoS, DDoS and infiltration are seven kinds of attacks in the CIC-IDS2018 dataset. Table 1 shows the percentage distribution of data types in CIC-IDS2018 dataset. The dataset includes the captured network traffic and system logs of each machine, along with 80 features extracted. 4.2 Machine Learning Performance Evaluation We used different measurements to evaluate the performance of our DRL model and other ML algorithms, such as Accuracy, Precision, Recall, and F1- Measure (combines precision and recall). Accuracy is one of the most common metrics to evaluate a model. It measures the percentage of samples that are correctly classified and ignores samples that are misclassified. In general, the accuracy metric is insufficient to evaluate unbalanced datasets such as network traffic data, which usually include normal traffic. Therefore, using various metrics in IDS is necessary. All metrics are derived from the confusion matrix (True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN). We analyze our proposed method based on performance metrics as shown in Table 2 . Table 1 Value and percentage distribution of data types in CIC-IDS2018 dataset Traffic type Distribution (%) Benign 83.070 DDoS 7.786 DoS 4.031 Brute force 2.347 Botnet 1.763 Infiltration 0.997 Web attack 0.006 Table 2 Evaluation metrics Metric Formula Accuracy \(\frac{(TP+TN)}{(TP+FP+FN+TN)}\) Precision \(\frac{TP}{\left(TP+FP\right)}\) Recall \(\frac{TP}{\left(TP+FN\right)}\) F1- Measure \(\frac{2}{\frac{1}{Recal}+ \frac{1}{Precision}}\) 4.3 Implementation details and Parameter setting Experiments were operated on Intel Core i7-8700 LGA1151 @ 3.70 GHz, 16 GB DDR4 RAM 2400 MHz machine. The proposed model was implemented in Python language. Tensorflow, Keras and Scikit-learn are important machine learning libraries that were used. The hyperparameters involved in the experiment and the value of the hyperparameters of the algorithms are shown in Table 3 . Table 3 The hyperparameters value of the proposed model Parameters Values Inner itration 50 Outer itration 100 ɛ 0.9 Decoy rate 0.99 γ 0.001 α 0.001 Mini Batch 500 Activation function(Q-network) ReLU Activation function(target-network) ReLU Hiden layer (Q-network) 3 Hiden layer (Target-network) 2 4.4 Result At first, we experimented the proposed approach based on three different values of the discount factor (γ) (γ = 0.001, γ = 0.01, and γ = 0.1). The results obtained for Accuracy, Precision, Recall and F-Measure metrics are displayed in the table. According the Table 4 , it can be concluded that with the increase of the discount factor (γ), the investigated metrics decrease, and with the decrease of the discount factor, these metrics increase. In general, the amounts of loss and reward obtained during the DQN training process depend on the value of the discount factor. Table 4 Comparison of performance for different discount factors 0.001 = γ 0.01 = γ 0.1 = γ Accuracy 0.9597 0.8629 0.8234 Precision 0.9034 0.8276 0.8128 Recall 0.9171 0.7790 0.7640 F-Measure 0.9689 0.8803 0.8269 Since we get better results with γ = 0.001 in the proposed method, we will do the next comparisons with this value. Figure 5 shows the binary detection results obtained from the comparison of the proposed method with NB, KNN, DNN, [ 32 ] and [ 33 ] in terms of accuracy, Precision, Recall and F- Measure. As can be seen, the proposed approach has obtained the highest value compared to other methods in all metrics. For example, the proposed method achieved a Precision of 0.9969. Our proposed model can detect all kinds of attacks. In the continuation of the experiments, we determined the power of the proposed model in detecting types of attacks with accuracy and F1- Measure metrics. The results obtained are shown in Fig. 6 and Fig. 7 respectively. As it is clear the proposed method has achieved the highest accuracy and F1- Measure compared to other methods. According to Fig. 6 and Fig. 7 , our findings demonstrate that training the DRL can result in better performance in multi-class classifying minority classes (web attacks and infiltration) over training on the true imbalanced dataset. DRL models are suitable to be used in an industrial and online production environment. One of the most important advantages of it is that once trained, it provides the correct action for a specific state (intrusion features). Table 5 shows the training time and prediction time for the proposed DRL-based model and other models (KNN, NB and DNN). We have implemented a Bernoulli NB using only the discrete features in Naive Bayes (NB). The DNN was applied with three hidden layers with 1024, 512 and 128 nodes. Our proposed method requires more training time than methods KNN and NB, but it has much less prediction time. In general, DRL-based methods significantly reduce the prediction time, which makes them suitable for online detection and IoT network services. Table 5 Training time and prediction time for the proposed DRL-based model and other models (KNN, NB and DNN). Method Training Time (sec) Prediction Time (sec) KNN 92.73 84.39 NB 6.82 0.98 DNN 308.69 0.82 Our approach 276 0.52 4.5 Summary of results Our proposed DRL-based IDS provides binary detection and attack type detection in two separate steps. In the evaluation section, both components were examined and compared. The results showed that our proposed method was able to obtain better values in different metrics and provided a more robust solution. Reducing the number of false negatives is one of the main goals of IDS. Therefore Recall is important in evaluation. According to the results, the model has achieved a high Recall in comparison to the other models. In addition, F1- Measure has also obtained excellent results and it can act as a solution for imbalanced datasets Considering the outcomes presented in Table 5 , the proposed model provides much smaller prediction times than the experimented other model. Our solution makes the convergence slower but more stable. Overall, we can conclude that the performance of DRL models depends to a large extent on the choice of the value of the discount factor. Limitation and Future Scope The results of the experiment support the claim that our approach can provide an effective and practical solution to automatically IDS in IoT. However, like other intrusion detection systems, our proposed IDS has some defects. We consider these weaknesses. By examining the limitations and solving them, more effective approaches can be done in the future. One of the most fundamental limitations in the evaluation of intrusion detection systems is the lack of access to real datasets. We evaluated our proposed method on the CIC-IDS2018 dataset. This dataset has a limited number of attack types. Many other types of attacks that exist in IoT networks have not been considered. In addition, we have trained our method only on the CIC-IDS2018 dataset, thus, it needs to be retrained to be implemented in other environments. Therefore, investigating this method in other data sets with different types of attacks or creating a large dataset with various types of attacks can be future works. Examining the generative algorithms in the proposed solution to overcome the challenges of unbalanced data and deal with new attacks is another future work. Conclusion In this work, we have proposed a Deep Q-learning-based (DQL) IDS in IoT networks. We identify traffic behavior hierarchically. Binary classification is used to allow fast detection in fog, especially for benign traffic. In the next step attack identification can be trained for different classes in the cloud. We model the environment based on GRU. This process aims to find internal relationships between events over a relatively long time. Deep Q-learning received the internal state from GRU as the input to the Q-function approximator, which is used for the action (normal/intrusion). In each iteration, parameters are updated to achieve better results. As mentioned, all events detected as intrusions are sent to the cloud for intrusion detection. In the cloud, several machine learning methods are used to increase robustness and improve performance. Using the ensemble method can be an excellent method for IDS. We showed that the discount factor parameter is important in regulating the speed of convergence of the algorithm. The low value of this parameter converges the DQL algorithm quickly. Our experiment demonstrated that the suggested DQL approach can learn effectively from the environment in an autonomous manner, be successful in detecting unrepresented classes and can significantly reduce prediction times, which makes it suitable for online detection in IoT networks. Declarations Ethical Approval Not applicable. Competing interests Not applicable. Author Contributions All authors contributed to the writing of the manuscript text and reviewed it. Funding Not applicable. Availability of data and materials Not applicable. References Wójcicki K (2022) Biega nska, M.; Paliwoda, B.; Górna, J. Internet of Things in Industry: Research Profiling, Application, Challenges and Opportunities—A Review. Energies 2022, 15, 1806 Lee I, Lee K (2015) The Internet of Things (IoT): Applications, investments, and challenges for enterprises. Business horizons 58:431–440 Sabireen H, Neelanarayanan V (2021) A review on fog computing: Architecture, fog with IoT, algorithms and research challenges. Ict Express 7:162–176 Goudarzi M, Palaniswami M, Buyya R (2019) A fog-driven dynamic resource allocation technique in ultra dense femtocell networks. Journal of Network and Computer Applications 145:102407 De Souza CA, Westphall CB, Machado RB (2022) Two-step ensemble approach for intrusion detection and identification in IoT and fog computing environments. Computers & Electrical Engineering 98:107694 Labiod Y, Amara Korba A, Ghoualmi N (2022) Fog Computing-Based Intrusion Detection Architecture to Protect IoT Networks. Wireless Pers Commun 125:231–259. https://doi.org/10.1007/s11277-022-09548-7 Halim Z, Sulaiman M, Waqas M, Aydın D (2023) Deep neural network-based identification of driving risk utilizing driver dependent vehicle driving features: a scheme for critical infrastructure protection. J Ambient Intell Human Comput 14:11747–11765. https://doi.org/10.1007/s12652-022-03734-y Uprety A, Rawat DB (2020) Reinforcement learning for iot security: A comprehensive survey. IEEE Internet of Things Journal 8:8693–8706 Wang J, Hu J, Min G, et al (2020) Fast adaptive task offloading in edge computing based on meta reinforcement learning. IEEE Transactions on Parallel and Distributed Systems 32:242–253 Huang L, Bi S, Zhang Y-JA (2019) Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks. IEEE Transactions on Mobile Computing 19:2581–2593 Lu X, Xiao L, Xu T, et al (2020) Reinforcement learning based PHY authentication for VANETs. IEEE transactions on vehicular technology 69:3068–3079 Wang T, Liu Z, Zhang T, et al (2022) Adaptive feature fusion for time series classification. Knowledge-Based Systems 243:108459 Abou El Houda Z, Brik B, Khoukhi L (2022) “why should i trust your ids?”: An explainable deep learning framework for intrusion detection systems in internet of things networks. IEEE Open Journal of the Communications Society 3:1164–1176 Sadaf K, Sultana J (2020) Intrusion detection based on autoencoder and isolation forest in fog computing. IEEE Access 8:167059–167068 Illy P, Kaddoum G, Moreira CM, et al (2019) Securing fog-to-things environment using intrusion detection system based on ensemble learning. In: 2019 IEEE wireless communications and networking conference (WCNC). IEEE, pp 1–7 Deng Q, Goudarzi M, Buyya R (2021) FogBus2: a lightweight and distributed container-based framework for integration of IoT-enabled systems with edge and cloud computing. In: Proceedings of the International Workshop on Big Data in Emergent Distributed Environments. ACM, Virtual Event China, pp 1–8 Goudarzi M, Wu H, Palaniswami M, Buyya R (2020) An application placement technique for concurrent IoT applications in edge and fog computing environments. IEEE Transactions on Mobile Computing 20:1298–1311 Sethi K, Kumar R, Prajapati N, Bera P (2020) Deep reinforcement learning based intrusion detection system for cloud infrastructure. In: 2020 International Conference on COMmunication Systems & NETworkS (COMSNETS). IEEE, pp 1–6 Yang Y, Tu S, Ali RH, et al (2023) Intrusion detection based on bidirectional long short-term memory with attention mechanism Strickland C, Saha C, Zakar M, et al (2023) DRL-GAN: A Hybrid Approach for Binary and Multiclass Network Intrusion Detection. arXiv preprint arXiv:230103368 Lazzarini R, Tianfield H, Charissis P (2023) A Stacking Ensemble of Deep Learning Models for IoT Network Intrusion Detection. A Stacking Ensemble of Deep Learning Models for IoT Network Intrusion Detection Mohamed D, Ismael O (2023) Enhancement of an IoT hybrid intrusion detection system based on fog-to-cloud computing. J Cloud Comp 12:41. https://doi.org/10.1186/s13677-023-00420-y Stefanova ZS, Ramachandran KM (2018) Off-policy q-learning technique for intrusion response in network security. World Academy of Science, Engineering and Technology, International Science Index 136:262–268 Lopez-Martin M, Carro B, Sanchez-Esguevillas A (2020) Application of deep reinforcement learning to intrusion detection for supervised problems. Expert Systems with Applications 141:112963 Sharma P, Jain S, Gupta S, Chamola V (2021) Role of machine learning and deep learning in securing 5G-driven industrial IoT applications. Ad Hoc Networks 123:102685 Ashenafi A (2022) A Model to Detect MiTM Attack in IoT Networks: A Machine Learning Approach. PhD Thesis, St. Mary’s University Daoud WB, Mahfoudhi S (2022) SIMAD: Secure Intelligent Method for IoT-Fog Environments Attacks Detection. Computers, Materials & Continua 70: Sarwar A, Alnajim AM, Marwat SNK, et al (2022) Enhanced anomaly detection system for iot based on improved dynamic SBPSO. Sensors 22:4926 Sewak M, Sahay SK, Rathore H (2022) Deep Reinforcement Learning in the Advanced Cybersecurity Threat Detection and Protection. Inf Syst Front. https://doi.org/10.1007/s10796-022-10333-x Abou Ghaly M, Hannan SA (2024) Protecting Software Defined Networks with IoT and Deep Reinforcement Learning. International Journal of Intelligent Systems and Applications in Engineering 12:138–147 A Realistic Cyber Defense Dataset (CSE-CIC-IDS2018) - Registry of Open Data on AWS. https://registry.opendata.aws/cse-cic-ids2018/. Accessed 26 Dec 2023 Labiod Y, Amara Korba A, Ghoualmi N (2022) Fog Computing-Based Intrusion Detection Architecture to Protect IoT Networks. Wireless Personal Communications 1–29 de Souza CA, Westphall CB, Machado RB (2022) Two-step ensemble approach for intrusion detection and identification in IoT and fog computing environments. Computers & Electrical Engineering 98:107694 Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 24 Jun, 2024 Reviews received at journal 23 Jun, 2024 Reviews received at journal 23 Jun, 2024 Reviews received at journal 20 Jun, 2024 Reviews received at journal 19 Jun, 2024 Reviewers agreed at journal 10 Jun, 2024 Reviewers agreed at journal 05 Jun, 2024 Reviewers agreed at journal 03 Jun, 2024 Reviewers agreed at journal 02 Jun, 2024 Reviewers agreed at journal 02 Jun, 2024 Reviewers agreed at journal 02 Jun, 2024 Reviewers agreed at journal 02 Jun, 2024 Reviewers agreed at journal 02 Jun, 2024 Reviewers invited by journal 02 Jun, 2024 Editor assigned by journal 16 Jan, 2024 Submission checks completed at journal 16 Jan, 2024 First submitted to journal 14 Jan, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3864298","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":267486214,"identity":"7cd26fd8-cfc1-4f31-a797-6d362b12c183","order_by":0,"name":"Sepide Najafli","email":"","orcid":"","institution":"Faculty of Computer and Information Technology Engineering, Qazvin Branch, Islamic Azad University, Qazvin, Iran","correspondingAuthor":false,"prefix":"","firstName":"Sepide","middleName":"","lastName":"Najafli","suffix":""},{"id":267486215,"identity":"23a303a3-e71a-4c11-8b8b-5238b49e6fd6","order_by":1,"name":"Abolfazl Toroghi Haghighat","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA70lEQVRIiWNgGAWjYBACgwMMbAwJDAcYGNgbQCQIMDYQp4WH5wApWhhAWiQSiHXY8ePXHjz4dUfeXvLxxgM/2xjk+RuY2z7g02J/JqfcILHvmWGPdFrBwd42BsMZBxibZ+B3WE6aRGLPYcYe6RyDA7xtDIwbGBib8Tvs/BuwFvseyTMGB/+2MdgT1nIj/ZhEwo/DiT0SPAaHgbYkEqHlDZtEYsPh5J4zaQWHZc5JJM84TNBh6c8kf/w5bNvefnjzxzdlNrb97e2P8WphYOAxYGBsg2gHYgkGBmYCGoAJ5QEDwx+4llEwCkbBKBgFmAAAWi5VL2zoS1AAAAAASUVORK5CYII=","orcid":"","institution":"Faculty of Computer and Information Technology Engineering, Qazvin Branch, Islamic Azad University, Qazvin, Iran","correspondingAuthor":true,"prefix":"","firstName":"Abolfazl","middleName":"Toroghi","lastName":"Haghighat","suffix":""},{"id":267486216,"identity":"8468319a-4785-46fc-a21f-86aa22409fd3","order_by":2,"name":"Babak Karasfi","email":"","orcid":"","institution":"Faculty of Computer and Information Technology Engineering, Qazvin Branch, Islamic Azad University, Qazvin, Iran","correspondingAuthor":false,"prefix":"","firstName":"Babak","middleName":"","lastName":"Karasfi","suffix":""}],"badges":[],"createdAt":"2024-01-14 20:29:41","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3864298/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3864298/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":49743466,"identity":"56cbed89-bd91-4e84-b31b-a234cef57961","added_by":"auto","created_at":"2024-01-17 09:56:55","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":133277,"visible":true,"origin":"","legend":"\u003cp\u003eThree-phase architecture of the proposed model\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-3864298/v1/262dca4ad302f8e087748ae5.png"},{"id":49744195,"identity":"2fcb4d63-31fc-4c3f-8a71-805f2f30a43d","added_by":"auto","created_at":"2024-01-17 10:04:55","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":12919,"visible":true,"origin":"","legend":"\u003cp\u003eProcess of sampling record from the dataset\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-3864298/v1/11df3d5db3f085b27f549701.png"},{"id":49743463,"identity":"7a5a53c9-415d-4403-8c4b-5faf65e313c7","added_by":"auto","created_at":"2024-01-17 09:56:55","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":16223,"visible":true,"origin":"","legend":"\u003cp\u003eGRU architecture\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-3864298/v1/5b51df78aee896fd62246b0f.png"},{"id":49744586,"identity":"18a06c92-0c95-4aa0-a6ae-b7781c478107","added_by":"auto","created_at":"2024-01-17 10:12:55","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":103439,"visible":true,"origin":"","legend":"\u003cp\u003eTraining of DQL agent binary phase\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-3864298/v1/26cd0e69686c40a492cdb095.png"},{"id":49744192,"identity":"9b87a407-2eb8-4840-a896-5efab9facd95","added_by":"auto","created_at":"2024-01-17 10:04:55","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":56480,"visible":true,"origin":"","legend":"\u003cp\u003eComparison of the proposed method with NB, KNN, DNN, [32] and [33] in binary detection\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-3864298/v1/884cddb3635eb677df7842e4.png"},{"id":49743468,"identity":"5ed6f559-520e-4a66-b4a8-cebb8eafbc24","added_by":"auto","created_at":"2024-01-17 09:56:55","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":47074,"visible":true,"origin":"","legend":"\u003cp\u003eComparing our model’s Accuracy in multi-class classification with other studies in the literature\u003c/p\u003e","description":"","filename":"6.png","url":"https://assets-eu.researchsquare.com/files/rs-3864298/v1/b72ce3573b0c3bf8fa55064b.png"},{"id":49744194,"identity":"0d0070c9-6de7-4024-a5de-b23063ba81be","added_by":"auto","created_at":"2024-01-17 10:04:55","extension":"png","order_by":7,"title":"Figure 7","display":"","copyAsset":false,"role":"figure","size":53250,"visible":true,"origin":"","legend":"\u003cp\u003eComparing our model’s F1- Measure in multi-class classification with other studies in the literature\u003c/p\u003e","description":"","filename":"7.png","url":"https://assets-eu.researchsquare.com/files/rs-3864298/v1/3e80a19b88468e8128add076.png"},{"id":49744904,"identity":"1f10dc21-98ab-4306-ade1-9f231f7d88e4","added_by":"auto","created_at":"2024-01-17 10:20:57","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":738726,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3864298/v1/459f1e6d-4185-4cf5-9450-be80f8d86a03.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"A novel reinforcement learning-based hybrid intrusion detection system on fog-to-cloud computing","fulltext":[{"header":"Introduction","content":"\u003cp\u003eThe Internet of Things (IoT) provides a global network of devices, computers, and sensors able to interact with each other\u0026nbsp;[1],\u0026nbsp;[2]. Cloud computing is a centralized solution, and provides unlimited, remote computing and storage resources for the IoT applications. While cloud computing supplies a large amount of resources, it faces challenges like limited and unreliable bandwidth, security and high communication latency between IoT devices and cloud servers[3]. Therefore, latency-sensitive applications in IoT cannot be efficiently executed. To solve the mentioned problems Fog computing extends cloud computing and provides storage, processing, and other things. It is a heterogeneous paradigm and distributed in the proximity of IoT devices. The fog layer is an intermediate layer between the Cloud and the IoT and is able to deploy resources close to the edge of the network. Therefore, Fog is an optimal solution for latency-sensitive IoT applications. However, the fog has restricted resources in comparison to the cloud[4]. Fog-to-Cloud computing[5]\u0026nbsp;is also a hybrid type of computing that is used in many aspects of IoT, such as healthcare, agriculture, driving, and industry.\u003c/p\u003e\n\u003cp\u003eThe characteristics of the IoT network have made it susceptible to various kinds of attacks. These attacks damage both devices and data. Therefore, the implementation of security systems is essential. An Intrusion Detection System (IDS) is a helpful security tool to detect malicious actions or attacks[6]. IDS can be implemented in IOT devices, cloud layer, or fog layer. It should be noted that due to limitations in processing and storage in most IOT devices, they are not able to run such applications. On the other hand, deploying IDS in the cloud layer will lead to delays. Consequence the most suitable option is to implement IDS in fog. Therefore, we need to design a new generation of intrusion detection systems that are suitable for the fog layer.\u003c/p\u003e\n\u003cp\u003eThe new generation (IDS) requires automatic and intelligent intrusion detection strategies to manage threats. The new IDS must be able to operate based on an independent agent and with minimal human intervention. At the same time, they should be able to evolve and improve themselves.\u003c/p\u003e\n\u003cp\u003eAlthough deep learning methods had a significant impact on the development of intrusion detection systems, their efficiency is highly dependent on the selection of appropriate features and the setting of hyperparameters used for DL training[7]. In addition, DL-based techniques face limitations in adapting to changing attack patterns. In other words, they cannot adjust themselves to any possible change in the attack pattern of the environment. As a result, accuracy will decrease and FPR will increase. Reinforcement learning (RL) is the most effective solution to security IoT against new various attacks. It can integrate environmental behavior into the\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003elearning process simultaneously[8]. On the other hand, it can model an autonomous agent to take actions optimally without knowledge of the environment. Therefore, it is useful in real-time and adversarial environments. Reinforcement learning is modeled using a Markov decision process (MDP) consisting of five entities, state, action, reward, policy, and value. The foundation aim of reinforcement learning is to find an optimal policy. The policy function is used to compute the next best action, based on the current state and reward provided by the environment.\u003c/p\u003e\n\u003cp\u003eSince the IoT environment is growing and becoming more complex, the RL agent will not be able to learn a larger number of parameters. On the other hand, the fog environment is stochastic in several aspects such as resource requirements of applications and the arrival rate of application requests. Thus, heuristic-based techniques are not able to efficiently adapt to changes in the fog. Deploying the deep reinforcement learning (DRL) model that combines deep learning (DL) and reinforcement learning (RL) can overcome this issue. DRL agents can learn the optimal policy and long-term rewards without prior knowledge[9]. In other words, deep learning significantly helps to optimize functions in reinforcement learning to create an optimal policy. DRL can deal with complex problems in dynamic and stochastic fog environments[10]. Moreover, DRL is also excellent for security applications where attacks are increasingly sophisticated, rapid, and ubiquitous[11].\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe efficiency of IDS is highly dependent on the deep learning technique and their model training mechanisms. Therefore, feature selection methods and hyperparameter tuning greatly impact IDS performance, especially in complex environments where the number of parameters is increasing[12]. Using DRL for IDS causes DNNs as performance estimators to extend the scalability of RL as well as adapt to complex IoT. DLR increases the capability of continuous self-learning by interacting with the environment and improving its ability over time.\u003c/p\u003e\n\u003cp\u003eTo address the challenges raised and due to the advantages of DRL and DL, we have proposed a Deep Reinforcement Learning approach to separate normal events from intrusions in fog and we have used several machine learning methods to recognize types of attacks in the cloud.\u003c/p\u003e\n\u003cp\u003eThe main contributions of this paper are summarized as follows:\u003c/p\u003e\n\u003cp\u003e\u0026middot;\u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;A two-level state-of-the-art suitable for online learning in a dynamic environment based on the Deep Reinforcement Learning approach for binary, and ensemble learning for multiclass detection.\u003c/p\u003e\n\u003cp\u003e\u0026middot;\u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;Employing GRU to learn the internal relations between events based on a relatively long duration of the past time. Using Deep Q-learning to estimate the Q-function and choosing the appropriate action (permeation/normal) in fog. Utilizing multiple machine learning models in the cloud to detect the attack type of suspicious events\u003c/p\u003e\n\u003cp\u003e\u0026middot;\u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;Sharing the information obtained in different fog nodes to the summary module in the cloud to update the parameters.\u003c/p\u003e\n\u003cp\u003e\u0026middot; \u0026nbsp; \u0026nbsp; \u0026nbsp;Extensive experiments were conducted to evaluate and compare the efficiency of the proposed DQL models. Our experimental results based on the CIC-IDS2018 dataset, demonstrate that it is effective in detecting intrusion of underrepresented classes and is excellent in achieving prediction time in comparison to another approach.\u003c/p\u003e\n\u003cp\u003eThe rest of the paper has been organized into six different sections. Section 2 is about the related work on IDS using deep learning and reinforcement learning techniques. Section 3 describes detailed of the model description. The experimental evaluation, results of the proposed model, and the comparison models are given in Section 4. Section 5 presents limitations and future scope. Finally, Section 6 concludes our paper.\u003c/p\u003e"},{"header":"Related Work","content":"\u003cp\u003eCybersecurity concerns have become a significant obstacle to the faster adoption of IoT technology. Although numerous IDS have been proposed in the literature, they frequently have poor accuracy because of creating various attacks in IoT environments and the lack of adaptation of attack patterns.\u003c/p\u003e \u003cp\u003eA lot of research has been conducted to enhance IDS performance. Researchers use a variety of techniques including machine learning and deep learning[\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e],[\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e],[\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. Recently Deep Reinforcement Learning has provided a promising solution that can improve the efficiency of IDS in dynamic environments[\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e],[\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. Considering the large number of works on IDS techniques, we study in this section, some related works of DRL-based and DL-based techniques in IoT environment.\u003c/p\u003e \u003cp\u003eSethi et al. [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e] have proposed a deep reinforcement learning-based IDS. It is adaptive to cloud architecture. It consists of three components, namely the host network, agent network, and administrator network. This method can provide higher accuracy against new and complex attacks and is able to maintain a balance between high accuracy and low false positive rate (FPR) compared to state-of-the-art IDSs. Experiments have been done on the UNSW-NB15 test dataset. The result demonstrates 83.80% average accuracy and 2.6% FPR. Moreover, this model has been successful in detecting all types of attacks.\u003c/p\u003e \u003cp\u003eYang et al. [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e] have conducted an intrusion detection model based on a two-layered bidirectional long short-term memory (Bi-LSTM) on fog. In the first stage fog nodes identify attacks based on traffic received from IoT devices and then send them to the cloud to summarize the global security condition. The first level is organized into three phases, including: preprocessing, categorizing and collecting data. If the predicted events are classified into normal groups, IoT devices are permitted to continue the process, otherwise activities are categorized as suspicious. The model has achieved the highest accuracy (99.05%), the highest detection rate (99.36%), and the highest f1-measure (99.154%) on the UNSW-NB15 dataset.\u003c/p\u003e \u003cp\u003eStrickland et al. [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e] have proposed a novel hybrid technique that can perform binary and multiclass classification. The model uses GANs to improve the learning capabilities of the proposed model. The GAN model is trained with the NSL-KDD dataset for four attack categories and normal traffic. DRL is used to implement this IDS. Although overall accuracy decreased, the results show generating synthetic data can improve the precision and recall for underrepresented classes in imbalanced datasets.\u003c/p\u003e \u003cp\u003eLazzarini et al. [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e] suggest a novel intrusion detection approach named Deep Integrated Stacking for the IoT (DIS-IoT). This model works based on combining four different DL models (MLP, DNN, CNN and LSTM) into a fully connected DL layer, creating a standalone ensemble model. DIS-IoT implements binary and multiclass classification techniques. Experiments demonstrate the model on ToN-IoT and CICIDS2017 datasets has achieved great scores in accuracy, recall, precision and F1- Measure in both binary and multi-class classification. This method tries to show that the combination of several methods can be effective in improving the metrics.\u003c/p\u003e \u003cp\u003eAuthors in [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e] present a distributed binary IDS method based on neural networks and genetic algorithms to efficiently detect various intrusions on Fog nodes. Genetic algorithm is used to optimize the weights of the network and the biases associated with each neuron. Experiments have been implemented on UNSW-NB15 and ToN_IoT datasets. The results show that optimized weights and biases can improve the performance model compared to the neural network without optimization. The proposed model compared to other state-of-the-art methods achieved a 16.35% and 37.07% reduction in execution time for both datasets. Therefore convergence process will increase and will save processing power.\u003c/p\u003e \u003cp\u003eStefanova et al. in [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e] have proposed an intrusion response mechanism that is based on reinforcement learning techniques (RLT). The model uses a Q-learning approach. The agent can control the process of interacting with the indeterminate environment and finding an optimal policy to represent the intrusion response. Evaluation of the Model shows proposed IDS based on Q-learning establishes the balance between exploration and exploitation.\u003c/p\u003e \u003cp\u003eLopez-Martin et al. [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e] present a novel application of several deep reinforcement learning (DRL) algorithms to IDS. It performs supervised learning based on a DRL framework. The model is suitable for dynamic environments where online learning is necessary. The authors supply a comparison study of four algorithms (DQN, DDQN, Policy gradient and actor-critic). They compare four algorithms with machine learning models on NSL-KDD and AWID datasets. Results illustrate DDQN algorithm has an excellent performance in accuracy, F1, precision and recall. Moreover, the performance of prediction scores, training and prediction times in DDQN was better than other approaches.\u003c/p\u003e \u003cp\u003eIn addition to the methods mentioned above, there are also other different proposals such as[\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e], [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e], [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e], [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e], [\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e], [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e] in intrusion detection. Stressing privacy preservation in the IoT is important. Therefore adopting a strategy to detect attacks is essential. Each of the existing suggested methods has examined specific attacks or specific aspects of the intrusion detection system and has advantages and disadvantages. Due to the effective features of deep learning and reinforcement learning, the study based on these methods in intrusion detection systems is still ongoing.\u003c/p\u003e"},{"header":"Model description","content":"\u003cp\u003eIntrusion detection and classifying the types of attacks are the two most important of our purpose. Our study uses deep Q-learning and machine learning approaches to binary and multiclass intrusion detection, respectively. Since we need fast response in binary diagnosis, binary detection is implemented in the fog layer. In other words, deep Q-learning identifies attacks in the fog layer. Events marked as intrusions are sent for more analysis and to identify types of attacks on the cloud. Cloud can perform complex analysis because there is no need for an urgent classifier and it has flexibility in response time. In the cloud layer, there is a robust method. It includes an ensemble machine learning approach of multi-class classification.\u003c/p\u003e\n\u003cp\u003eIn the first step, we model the environment using GRU architecture to learn the internal relations between events in fog. In general, we integrate the influence of the historical information of the dynamic environment on policy optimization. Because we aim to detect attacks with small mutants compared to previous attacks, and recognition zero-day attacks in an IoT environment.\u003c/p\u003e\n\u003cp\u003eThe important point is to send and share the information obtained in different fog nodes to the summary module in the cloud. That means the information created in each fog node is sent to the cloud so that it will be visible to all nodes. This module is responsible for updates based on received information. Generally, the proposed method is implemented in three phases, which are: 1) the preprocessing and environment modeling phase, 2) the binary detection and parameter updating phase, and 3) the multi-class detection phase. The overview of the proposed architecture is shown in Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e. Three proposed phases with detail are described in the following subsections.\u003c/p\u003e\n\u003cdiv id=\"Sec5\" class=\"Section2\"\u003e\n \u003ch2\u003e3.1 preprocessing and environment modeling phase\u003c/h2\u003e\n \u003cp\u003eSince the IoT environment is completely variable and new attacks are being produced with very few differences compared to previous attacks, finding the relationship between events will significantly help in detecting intrusions and their types. GRU can learn values over short and long periods. It was proposed to solve the vanishing gradient problem in RNN. GRU is similar to LSTM but simpler than LSTM and uses fewer parameters. Therefore, it will be more computationally efficient. The use of GRU can lead to the benefit of the information on previous events in the detection of new attacks. In other words, prior information can be incorporated into an internal state that is a suitable representation of the interactive environment. These states are entered into Deep Q-Learning to perform binary recognition. We use Deep Q-Learning to deploy agents in discrete action space environments.\u003c/p\u003e\n \u003cp\u003eAccording to Fig. \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e, B\u003csub\u003en\u003c/sub\u003e (Mini Batch) records are sampled from the dataset in the first step. Choosing B\u003csub\u003en\u003c/sub\u003e records means that each state (S) will be equal to B\u003csub\u003en\u003c/sub\u003e records, and each record will contain S\u003csub\u003e1\u003c/sub\u003e to S\u003csub\u003em\u003c/sub\u003e features.\u003c/p\u003e\n \u003cp\u003eIn the Next step (Fig. \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e) (S\u003csub\u003ei\u003c/sub\u003e), (h\u003csub\u003ei\u003c/sub\u003e-1) and (A\u003csub\u003ei\u003c/sub\u003e-1) which represent the current state, the hidden state of the previous step, and the previous action, respectively, enter the GRU module at any time (t). Eq.\u0026nbsp;(1) calculations are performed in each GRU unit.\u003c/p\u003e\n \u003cp\u003eZ\u003csub\u003ei\u003c/sub\u003e\u0026thinsp;=\u0026thinsp;\u0026delta; (W\u003csub\u003ez\u003c/sub\u003eS\u003csub\u003ei\u003c/sub\u003e + U\u003csub\u003ez\u003c/sub\u003eh\u003csub\u003ei\u0026minus;1\u003c/sub\u003e)\u003c/p\u003e\n \u003cp\u003er\u003csub\u003ei\u003c/sub\u003e\u0026thinsp;=\u0026thinsp;\u0026delta; (W\u003csub\u003er\u003c/sub\u003eS\u003csub\u003ei\u003c/sub\u003e + U\u003csub\u003er\u003c/sub\u003eh\u003csub\u003ei\u0026minus;1\u003c/sub\u003e)\u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; (1)\u003c/p\u003e\n \u003cp\u003eh\u003csup\u003e\u0026apos;\u003c/sup\u003e\u003csub\u003ei\u003c/sub\u003e = tanh (WS\u003csub\u003ei\u003c/sub\u003e ʘ Uh\u003csub\u003ei\u0026minus;1\u003c/sub\u003e)\u003c/p\u003e\n \u003cp\u003eh\u003csub\u003ei\u003c/sub\u003e = (1-Z\u003csub\u003ei\u003c/sub\u003e) ʘ h\u003csub\u003ei\u0026minus;1\u003c/sub\u003e + z\u003csub\u003ei\u003c/sub\u003e ʘ h\u003csub\u003ei\u003c/sub\u003e)\u003c/p\u003e\n \u003cp\u003eZ\u003csub\u003ei\u003c/sub\u003e is the update gateway for state i. This gateway decides what information should be discarded and what new information should be kept in it. W\u003csub\u003ez\u003c/sub\u003e and U\u003csub\u003ez\u003c/sub\u003e are the weights of S\u003csub\u003ei\u003c/sub\u003e and h\u003csub\u003ei\u0026minus;1\u003c/sub\u003e, respectively. Next, the sigmoid activator function is used to produce the output (0 or 1). In this stage, the model decides how much information to transfer to the future. This action also avoids the problem of gradient vanishing. r\u003csub\u003ei\u003c/sub\u003e is the reset gate. In this gate is decided what information from the past should be forgotten. W\u003csub\u003er\u003c/sub\u003e shows the weight of S\u003csub\u003ei\u003c/sub\u003e and U\u003csub\u003er\u003c/sub\u003e shows the weight of h\u003csub\u003ei\u0026minus;1\u003c/sub\u003e. We use h\u003csup\u003e\u0026apos;\u003c/sup\u003e\u003csub\u003ei\u003c/sub\u003e to specify how the two update and reset gates determine the output. The output of this function tells what to remove from the previous step. In the last step, the network decides to keep the information related to the current memory and transfer it to the network by calculating the h\u003csub\u003ei\u0026minus;1\u003c/sub\u003e vector. This requires the update gate to determine what information to retrieve from the current memory contents of hi and what from the previous step h\u003csub\u003ei\u0026minus;1\u003c/sub\u003e. This step is calculated by the equation h\u003csub\u003ei\u003c/sub\u003e.\u003c/p\u003e\n \u003cp\u003eThe output of this module includes the prediction of the next observation S\u003csub\u003ei+1\u003c/sub\u003e and the next hidden state h\u003csub\u003ei\u003c/sub\u003e. In fact, the produced h\u003csub\u003ei\u003c/sub\u003e indicates what information from the previous step h\u003csub\u003ei\u0026minus;1\u003c/sub\u003e and what information from the current observation S\u003csub\u003ei\u003c/sub\u003e should be stored. In addition to being used as an input at time t\u0026thinsp;+\u0026thinsp;1 in the GRU, the generated output h\u003csub\u003ei\u003c/sub\u003e is also sent to the binary detection module to perform attack detection. In general, we perform binary detection on data that have useful information from previous events and current events. In other words, we have provided the ability to learn in earlier time steps.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec6\" class=\"Section2\"\u003e\n \u003ch2\u003e3.2 Binary detection phase\u003c/h2\u003e\n \u003cp\u003eRL is based on the Markov Decision Process (MDP) theory. An MDP consists of tuples (S, A, T, R), where S is a set of states, A is a set of actions, T is a mapping function that presents the probability of transition from the pair (S, A) to a new state, and R is the reward function. The mapping function uses Markov probability. In such a way that the probability of transition to a new state depends only on the current state. MDP aims to learn the optimal policy so that it chooses the best action for each state. As mentioned deep Q-learning is used as a binary detection technique. Deep Q-learning is a model-free algorithm and it doesn\u0026rsquo;t create a model of the environment\u0026rsquo;s transition function. The purpose of Deep Q-learning is to estimate the Q-Value and solve the Q(S, A) function based on the experience samples in each time step according to the following formula:\u003c/p\u003e\n \u003cp\u003eQ(S.A)\u0026larr;Q(S\u003csub\u003ei\u003c/sub\u003e,A\u003csub\u003ei\u003c/sub\u003e)+\u0026alpha;(R\u003csub\u003ei\u003c/sub\u003e\u0026thinsp;+\u0026thinsp;\u0026gamma;Q(S\u003csub\u003ei+1\u003c/sub\u003e,A\u003csub\u003ei+1\u003c/sub\u003e)-Q(S\u003csub\u003ei\u003c/sub\u003e,A\u003csub\u003ei\u003c/sub\u003e)) (2)\u003c/p\u003e\n \u003cp\u003eTo obtain Q(S, A), it is necessary to calculate Q (S\u003csub\u003ei\u003c/sub\u003e,A\u003csub\u003ei\u003c/sub\u003e) which represents the function in state i, (S\u003csub\u003ei+1\u003c/sub\u003e,A\u003csub\u003ei+1\u003c/sub\u003e) the function in state i\u0026thinsp;+\u0026thinsp;1, \u0026alpha; is the learning rate and R\u003csub\u003ei\u003c/sub\u003e is the reward for the action is A\u003csub\u003ei\u003c/sub\u003e. Therefore, to update Q(S, A) for each action A in state S must calculate the estimated yield R\u003csub\u003ei\u003c/sub\u003e\u0026thinsp;+\u0026thinsp;\u0026gamma;Q (S\u003csub\u003ei+1\u003c/sub\u003e, A\u003csub\u003ei+1\u003c/sub\u003e). The estimated yield is also called TD-Target. Repeated execution of the update rule results in the correct Q(S, A).\u003c/p\u003e\n \u003cp\u003eBefore implementing binary detection, the initialization of the parameters and the DQL must be done. It means that the output from the modeling phase is entered into the DQL and it determines the QV, SV, AV and B\u003csub\u003en\u003c/sub\u003e vectors.\u003c/p\u003e\n \u003cp\u003eAs shown in algorithm1, we use two categories of iteration (inner iteration and outer iteration) to detect intrusion in the binary detection phase. The outer iteration is used to train the DQL model, and the inner iteration is responsible for improving the Q-values. Each B\u003csub\u003en\u003c/sub\u003e represents a state (S) which is determined at the beginning of the outer iteration. The specified current state (S\u003csub\u003ei\u003c/sub\u003e) is entered into the outer loops to perform the DQL training process. At the end of the external iteration, the B\u003csub\u003en\u003c/sub\u003e value will be reset and the DNN parameters will be updated if needed.\u003c/p\u003e\n \u003cp\u003e\u003cimg src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAhAAAAFhCAYAAAAhlpNwAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAABhaVRYdFNuaXBNZXRhZGF0YQAAAAAAeyJjbGlwUG9pbnRzIjpbeyJ4IjowLCJ5IjowfSx7IngiOjUyOCwieSI6MH0seyJ4Ijo1MjgsInkiOjM1M30seyJ4IjowLCJ5IjozNTN9XX3/WMkBAABVz0lEQVR4Xu29v47dRpr+3/7tBezGglZoTPsaFAhjBQrccwEKbE3iSIAG2MywNhCwHsCJDCUOxgYcKVLPALqAsQLhC/VAga5hNCtoBcW7VzA/fig+7adLRZ7D0+cPefr5ANWHLNaft4ps1ltvFas++WfDQQghhBDCCP6/7jeEEEIIYWmiQIQQQghhNFEgQgghhDCaKBAhhBBCGE0UiBBCCCGMJgpECCGEEEYTBSKEEEIIo4kCEUIIIYTRRIEIIYQQwmiiQIQQQghhNFEgQgghhDCaKBAhhBBCGE0UiBBCCCGMJgpECCGEEEYTBSKEEEIIo4kCEUIIIYTRRIEIIYQQwmiiQIQQQghhNFEgQgghhDCaT/7Z0B1Pnj/+8Y+tCyGEEMbyX//1X60L62FWCkQIIYQQpkGGMEIIIYQwmigQIYQQQhhNFIgQQgghjGZWCsQnn3xy8Oc//7k7Ow/X/vGPf3Rn64G8fve733Vn5/n+++8P/vCHP3Rn6+Vvf/tbW56a6yv/JijlWFRe6kphP/300853eyDvLvKtgRzIs4hNPkchhLBRmEQ5B05OTv55dHT0z+Pj487nV/CnKK9fv+58NsO9e/faX2QhP51virJMDx8+bP343TTKy/On7qnrIYi36XrZJvtUlhBCWCezsUC8ePHi4PHjxwe//PLLR5aGv//9793R5qA3+ezZs/b4iy++OGgayvZ4m3zzzTdtvvfv31+7tcUhbfI4PT09+M1vftP5Hhz89a9/PWgUirbXfBnA2rPJeg4hhDkzCwWCl/jh4eHBb3/724OmB3zw6tWr7kodwrvpHadGj0ZBfm7u1nAFflwDmZdRHj777LO28eSam6a5jp8Pdch8rbRIm7TKcJJlTCOFEgGqAy+rp03+8ncTOedeB14WobSp75KmR37w/Pnz7mw8ffKqfkp/6lB1TBjdk7761Lny8fy8nnVv5DwdIK0vv/yyVVgVlzCSRXXqaXhdKg5I3jIu6BromfF7J7wccooXQgg7obNETBqGDGRKx0ReG8agKArDdeKAm9S57nHx57zpabfxPY1ymIIwbr4nXa5LNo6B9DwtheNX4ZTHIvrCKj0dKwx5ez4CuZFTQz2qA8LWhiQkcw3qoxZHEFd1VqMmL7i/jlWXSk/3BFeWU/dQZVM4yYp/7Zko76tDeKVHeE8fPB0/Vj0jH/6ShfQ8P13jV2XBKQ/CER765A8hhF0xCwsEQxfNy7TtdWFarw1jOJjd375925196L3B06dPD27dutUew48//timRU+7eTkfNC/pM5P9MsMUzUu8DSfIBzM/sjYv/rO0SBfLgQ8HXJRr166d9XhVN5TlzZs3ZxYE9VSbxqmtDw31UG5AJq6VdUnaQ9BLXoU+eaF5Fg/ev3/f+gvV5Z07d9pz3RPqXZYYQH7dQ0G5wIe39ExQPi+zwg5BnXEfv/rqq87nQ/3hj2WC51J4noRBXuTWs6L8dA14NnhmKC/lFu/evWt/+57pEELYFZNXIHhR8tKmgZHjJYsy0AcvdV7oNEb8+gtZDdZckdn6+vXr7S914XUj5YDGzv29wV2E0saMX8I8EClhPgywrDm9T17S+u6771q/TaOGHLkZmqLhXgUNBT148GDjc2KGnukQQtgFk1cgHj16dK6XD3fv3j34+eefu7OPYVyZXp4aKXHjxo2Dn3766az3Ro9YPcA5QCNN40FjRY+UXjfl9MabsqMA0LuXAkB5PYwsFFx3q4vgnHphDoD3dDVPQMoIPW3V8ZCCQj64PnllmdhWo4gsrpTW5nosA5YxrB6rxh9D3zMdQgg7o3kZTZamt9qOCfMrmpdo6yenMWWdQ9P4nQuD01gy48jyaxrP1o9r8iMueLimwT6X73/8x3+cu6ZjHGnWrpUyeR6k7bg8pSvDlvWh62UagmOXcYiybH4fani6pRN98vo91LGnR7jynugYV95DHZOWn5OGpyNHeiWepsuHP3g6uk46Htbl9GPC+PkPP/xwdky5vezkN/RMhxDCLhhuQWZKrTGo+V1GaHjUaC+LGrN9qUMa/rIOpl62PNMhhKkxm3UglgVzeDm8gcma4YuwGgwtNM9K+/kmY/DlJ49zg6EZZ+oTEvNMhxCmyN4pEIxHf/75521DJ8ciVNsYp546+nri6Oio/R2LFIm5T+A7OTlp60DPB8/LmEmm2ybPdAhhinyCGaI7DiGEEEJYir2zQIQQQghh80SBCCGEEMJoMoQRwiXmf//3f7uj5RkbZ5U8YBv5TDGP//u//+uOlmdfyg6bluvf/u3fDv77v/+7OwsXYW8ViCk+uLCNPGCKZcmLcXpy8TIdy9g4q+QB28hnG3n867/+a3e0HNsoB2yj7PskV/iYWSgQ33777cEf//jH7mw59uXBXfVB30Y+eTGOY6pyhRDCKmQII4QQQgijySTKEEIIIYwmCkQIIYQQRhMFIoQQQgijiQIRQgghhNFEgQghhBDCaKJAhBBCCGE0USBCCCGEMJooECGEEEIYTRSIEEIIIYwmCkQIIYQQRhMFIoQQQgijiQIRQgghhNHMQoH485//fPDJJ590Z+P49NNP27j81vjHP/7RXsd9//33ne9iFI/fIUjzD3/4Q3d20B6PyWcMkmkqcN9+97vfdWfzZptlIa++53Vd6H9KjmdHz2V5Tc6fW4792t/+9rfW3+Puy70PIfTAbpxT5uTkhN1CW7cq9+7da+Ofnp52Pr8ydG2Io6OjNt7r1687n4+R7OSxKZCbfIC8cGF1aveKOqZej4+PO5/NwfNEXjxfm+Lhw4dtHv7skp/nKTmEnmU9a6B6KSEueYQQ9pvJWyC++OKLg+ZF1Z2tTtMwHHz33Xfd2cV59uxZd9QPsjcv0u5sM3iZmhd3dxRWoc8y9Nvf/vagaTi7s83ym9/8ZqN5YWm4f/9++z9FXuLvf//7weeff96dfQzPcqNgHLx9+7bzCSFcdvZmDgQmXx8qKPn6668PfvnllzNTK2BuvXPnTnf2K26eLc2wnOP/6NGjzucDGj5YZLrlGvkC+WhIo4yHnEpPTvEE4SnTl19+2RvXh1gIU/N3vOzKj/SoX08X3FzNNdWBZFH5QGlIBsJ6nckP5O/XdY08SUN5y9SvcOQjXD5kgb60uU7D+tNPP7V+fVAervsQg9eZ3weVFZlUl6CwngbI/8WLF53Px7jcONVLX7lKnj592v6iFJX8+OOP3dHHUEYU1Nu3b3c+IYRLT2eJmDR9ptJlkVmaXzdR65i0yQP49TCYrT2+TLOlGdiPiePhPD7hMAPLJIwjDHE5FpiTJZPnW0JeMisrDZna/ZqngV/NRI6/4iqt//f//t+ZnLrmspX1JX+Vj2tKC6fr4DIoPCisrqscxNU15alzwE/+hPX0CePxy7TB71WJ5FNY4qs+8de91zF5+XWPpzpwed0fmVx2x/0lE/CL03Uvl0N+ZdpKRw75/Z7havWi+iwhbt/zGkLYHy7VVxhYG+hh0jOjV3jz5s3uyq88efLknD+9Mg1XEFc9MO+JqdfbvJjbnh9WgTdv3rR+Dmk1L/b2WMMbzYv54Jtvvmn9QL1Geqfv3r1rj6GWXh9//etfu6ODM5MzZaCHjXxYLJqXfOvv0PNFdsJQFviXf/mX1tzNuacr2ejJ+nCO/H34BlO50lDPVxYJQXjqBn/JhlldUA4NJRDOe8ukDYeHh+0vvHz5sk1HvXFAtr60l4F8kRMYOnr+/Hl73PwfHbx///4sH0BW6hsrBNeJx70l/88++6wN68+i4sBXX33V/pasUmcl1JHCCtKRH78+tIHs3MeaNePKlSvdUQjhMnKpFAhe0LxwGX6ggVFjULLKOC+NIy9buSFz8DIQn4aehoYG+qLpAQ2ty1iDxsLDqFEbgrFzGjHc9evXO9/FeGO3CVDOvCx993tV1NDSqKNQkIfD+YMHD9p7qCEUcJlcKVuGi9bZjRs32l+XZxFScJeNgzJ17dq17iyEsK9cKgUCeKHT8+sD6wM9R/Hq1auDu3fvtscoCZr7gD/QeNLI0nPzF+zQfIxlIL4amXU0tMjpPduafCq7epuM26t3PASWHRpQevjeex1CdeZzAyjnMgrLMtBQcp8lP7+e10XhOaLcSr9UBPCnjikPihvWCuqGZ8jrnmN68l4XWII4L+/ROuqMsChW3Ocx9UH5yji18gDP2RhFMoQwU5oGatI0L9+zcdghcZsXWXWcFn/FJS35NS/ij9LWuC2/8vM0iSP/4+Pj9he/8pr8fWyZNElL599+++25azrGgdJ3p7wcpenlVF3oXGPhnqbKWuKyEL4ss6eh+gTy9DS97E3Dc3ZMfFHWmdB5WQ6vM/y9zB5OcrgMhAU/L+vIww/VtcILl0PH1E1fXcnP/T1v4knekrF15nI6np8c8WrXajL2lae8FkLYXz7hT/NPHyYG1oxybgRWj3Wb4UMIIYRVuHRDGHMAM7Em6Ak+v4tZOIQQwlSIAjFBZGVg8p0cLDu/IIQQQtg0GcIIIYQQwmhigQghhBDCaKJAhBBCCGE0k1cg+MZc8wDGLH4jtB+BzyUYggmMvp9BCWnUVuVzkLP81n8V2UMIIYSpMuk5EFq0hkmFLMzDEsCnp6ejFxui8earhrGr/gGN/5hVIJGZFSTv3bu3ltUj5wT1zBLfmewZQgj7z6QtECgO+iIBpeH4+Phsr4VtgKVhaNXKGsj7cMNbeE8VX8EzhBDCfjO7ORBXr17tjs7DfgTlkrp9aIiBX4YkfMhC11Aejo4+bChFGCwL+PkwiOKXaZRwTdYU0lYcOS2HTBnkV4NwhPF8BenLzzdcIj/lKRkVDqe8lTa/ugaSSfKDhyFtUHjqDPnAy6p8SAc5lG4IIYSZwhDGXPBlkMfA0saK68vx4q+lgf2alvT1a6B4gmMteezHpKs0tKwwaYOWWQaWHdY58nmYsqy+hLHHUT6kpSWEdUw4wnMu8FMcHXvaypc4nHPN5VFYQTjJrfDg+SAL1/Tr4UIIIcyT2SgQaoxWwRtA8MZNDWKt4Ssby/Icao2ipwHkrUZWeOMPSkPOG31BXu5fnks+HNeAPJCnhPwJJxnKtFwx4Ff1x7HykFP6HKsOlL47rnlaIYQQ5ssshjAwibNT5NTADF/bxnkRlIddFMtJlk0D26aFG7sDJ8MC7LhJ3Kbx73w/RkMd7CbZNPyd7zgaBeBMTpzv2eE0ysK5cJlcGUII+8PkFQiNvWsypcbcd43G9Md+2UF5mGwoBYFz5lbQKHvZ+sqJkqHPSJ88edJuNc75L7/8spTS8fjx47ZhH/sli2A/DvLSfSFvzXlwbt261X6NIggjuUMIIewBTc9wsmBeR0R3NXM8YHL3IQFRmtLdBE9aQ9fAz3UsUz+/7sdvmYaX4S9/+cvZsTuZ/T090inRMIPC+VCAl1PXfRttpefy1cKRjqfl8vtwh/xwQukpXBnX4ylMCCGEeZK9MGYEVo+vvvpq9PBGCCGEsG5m9xlnCCGEEHZPFIiZwPwBVuJkDsTQmhMhhBDCNsgQRgghhBBGEwtECCGEEEYTBSKEEEIIo4kCEUIIIYTRTF6B0MqJOC1eNAZt2lS6qSxItSra+Gof0GJaolyYimu1e4ijHsbCva8tflVC/a6Sfg3SqsmP63uuPUwph19b9CxTVoXte2aYmMv1seUl3qK6XGc9hhCmw6QVCBqOFy9etMsgn56enlvZcFlYM+HevXvtss2kI7creJmuogiVUC6tJkk9LdMgLsu6ZFyWt2/fni1zTVmuXbvWHov379+f3T+eg6Ojo7P7uErDxBLifctvO9TvutbcIC09d75kOeXiufb7J2XjxJYCZ8VR/KRo4Xd8fNyGKZdEL6Gs1BuwemkJ+bG6KP8nY8qLzMQbAnn5eiiEsH9M3gKhlyMvYF6Y62LRS3dTsHfGunn06FF3tB42IWMfpcLw9OnTs2XLBfe+r8Efu5T41KBcKBEsby7lgMXC8PN6IBzP/0UsZygIP/3001k+AqWCa2ORTEOgGEp5CSHsF5NWIHzzJV6cQ40FPdFlXq613jVxZeL1a0oT/1oPHz/F09oMxOdcsug6L23C0GOjx6nwnobyJqzS4BeZS5QmYWgUaIDUG5cMOJfb0ySMh1Pcmox96a2DmsKwCsilsqksffeVculc8XQfVGYgjM51T/SrY+F54TydRUg5evXqVZsmFoobN260fg6KxaIe/xBsSEeDv4zCKSuInKPnQfXseD2EEPacf84AxMSx18IqsO+C0sD5PhNcY48GYE8KrnPevGjbY67XIIxfI7zOm97j2bHS5BcI5/tScA4ejl+cy+lozwmlSV7kCcjl9UQ4+XGscICfyu7HLmNfeuvC65B0+8osSnmAOMiFE163fkx+hMPP41EvqnsgH44VT+GUt9eR3++afA5p6L45+JOO8uW3RPIqvsuwCNIjrNIXKjfy+73wMhBG514+1RfXoawT1Z3HCSHsD7P4CqORszWDNi+scz3JMTQvtLN0HHrvmkuAxYNwL1++bK0dzUvv4M6dO+21Esy+vsU4QyLPnj3rzpaD+R30KOmtkRcw3k85oa9nPjROjezE917gu3fvzuLcvn27/QX8rly50oZTniV96a0Detteh+S1ijWCONy3pgHsfD74cQ+xLvjcGe5T07C1xx7Ph0iQi2eiaQw7nw/zFsDrnrkbwBCMWyP66nIRpMP9WMRFtkWnXDxrsiS9efPmo/T4H2NreKG6oYzaARaIp7oEWa54TnzH2RDCfjILBQJ48XkDsSqkUzZS/vIfgxqQiyDFRk7KzEWgnjzNvkYZMzsNBWGkwNRYNr2xrGv4ooZM8CgorghsApWB/JgwWCqpi1BjzlbpNMrci9pkRxROb7BXhTkuNPDUUZ+CvOr/BMqTnpMoECHsN7NRIABrwbobHF7IPib8/Pnzc730PmiYeAkLxq/VMwO9gPEH79EJpaGw9Pxq8x3GwNg59aR0+K1ZbciTHuOil/yy6V0U0q2N+68KDTDK2aYUFIf6YH6CGs4xSiDKA88AssoSQANPnXs9Ew6/RZN/sbgsavypE5QUZK7JiiLDs6F7TnrMbUC+w8PDc8894aSM8L9E/sKPQwh7SPPCmyzNS7UdY5UbonkhnhvDFfgrfvOC63zP42EYrwXCyq/pVbV+JS6f5014+SsdpUE4zskTPA3Jp3OFKXF5wdMAxqF1rjTKOOBl1HXkLGWspXdRqGeNl0Pt3pWUz4Pq1OUjDJC+/FQ28lDZcN9+++3ZcZm2x/c4pOXn5O35y0kO4emVrvZ81cJ7uPKaXO0Z97KpnvGTjKofnO5vWX+OPzcce1k9LeoFmT1sCGF/yGZaIVwQLAUafhBYDHxexarIQtE0vrP/ZDWEsF/MaggjhCnikzRh0RDCGFBC0PFv3bp1biJrCCHsmlggQrggWCBciTg6OsoEwhDC3hMFIoQQQgijyRBGCCGEEEYTBSKEEEIIo5mNAsF35mP2FxDlHgVy6/pGnbSWmTSH/MqbMfN8Ix9CCGHOzEaBYNGbVWAy271747fzRinQCoF91DYT6oMVCl93q/SxoiCLAoUQQghzZRYKBA25r/K4Dhat6LfMjoXLzrSXhULrBJA3Sk0IIYQwVyavQKjxZaOhIbAGLDMswFCCLxEM5RbEpIOFgAV8ZGXwIYhyKIUNsHSNcCXa38CvuwJDfutcOyCEEELYNJNXILAELLOiH9aAIasCygANOEMJDsoAew8wtMAwB+eyEHBOujTuxNPwh+8TANoHgTjsKFmDdFhNkHSQwxUG8vNVDEMIIYSpM2kFAktB326BY9EciNNip8RltiBmQywfciAd34TIt/Fme+Q+WIqYuFgjcCghsmzEAhFCCGFOTFqBePz48VmPnUaexp7ji1DbzluTG3F98xou0sAT14dXyEPWCuTBMhFCCCHMiUkrEOqx405OTtqGluN1QpreuNfmUdS2Ny7nUSzCt8QGrBaL5nWEEEIIU2XycyCWpW8SJf6aEFlbRwIlBYsAlg3czZs3W//Dw8OzeRPMT0CBkTWE4QesGJpgyTlfipAPrvb5J0MozLVQPnxVUlpCQgghhLmQvTB2jDZiynbNIYQQ5kQUiBBCCCGMZm+GMEIIIYSwPaJAhBBCCGE0USBCCCGEMJooECGEEEIYzSwUCD6/1OePi3bILPG4uEXwVUTtc09BGosWlULGcm2JsXJfVrZZV+QzdK93zabk4xnXJ8ghhLAyfIUxZU5PT1t3ER4+fPjP4+Pj7mwc9+7d646W4+TkhK9aRsfbB6jn169fd2fjIT51x++m4Zkir1Wfi00zRr4xzxr3h3SPjo46n3lxGf+vQpgqk7dAsFEVCzjVFonaNFgaWBhqDCwOxaJRlxEW3roIbJrGEt/bgCXEp3yflpUPa8KYZda1KNociRUvhGkxeQWCJZ8bRaf9HTLn9q1EWYMXkUzlDEl4urrGS5kVJoEwelH7MIjil2mUcI34QNqKI6clrimD/GoQjjCeryB9+bl5mvyUp2RUOJzyVtr86hpIJskPHoa0QeG1Kid4WZUP6SCH0u1D15U+EE/pKQ8gLI60vTwK62noHuKGNj7z+DiBP3n59RLlQTiQ3P4MIH+tjEPyyR9H/qTne8QQ18MpfyH/Fy9edD4fo2dVYb2ea/UveXWvkYtrCkccobSVDuGUVymr50UehEVBRaHHD7yuPB/SkjwufwhhzXwwRMwDzK6rDGdgEpcpWEMMOPxl0vVrMpP6NVA8wbFM9n5MukqDX66RNnBNUB6dI5+HKU3XkgXncZSP142OCUd4zgV+iqNjT1v5Eodzrrk8CisIJ7kVHjwfmeP16+FKVF9cV17Ew6kcfkzeOA+rY9EnI/5lPQulD5SFc6WL83oizxKXEThGBtJSGqTncoP8gTjKx+vTj0nTy+DyEEbh3J/wLptQ3eOA8JLHy+PHCq/6BcXxY0+b+ITnWPLJH/CjjEA45eXlBqUNlInr/Hq6IYTN8etbdgbwMvEX1bLoxSL8RcQLyF9EtWuiPAe9ZPvSAPIu5cavfBm6q73g/cUN5bnkw9Vexk75oi3T4lgy86v641h5yCl9jlUHSt8d1zytGqW85TnHpCVZVWavyz4Zy7zxq8lCOE8PyI+8hurJGQrXV8Zl5OPcy+tx/P7Lca2UhTh+7hDey1OeIxPpKr7yLCFPyUAY8LRKGTgmjo4VVw7IW+X29OV0zdMKIWyO2X3GefXq1e5o92AqZYOsph47n+XArMoGXj/++GPn84HmRdumhevbVrwPTLiff/55G7d5gXa+HyOT8YMHD1aeA9A0BGdy4pi7UKNpJM6FY/x9Fdi1VOZqaBqH9hdIk7TZBI3rlA+WlbEPDQdMhTH3zcu9rv1V+uq/xqr/Fw55eDlq8Jx7mPL/KYSwWWalQPDyZHLZFGCsF8a+oGkIGMuVgsA5L2caPB8L92MHJUON25MnT9pdPTlnHHwZpePx48dtw75qPWprczXU5F0bZ75161Y7Pi8II7mXhfCMeTMx9enTp+0Ey1IRQA7SJgzlYny/T0aUT9+W/fnz5+15KX9t+3YaxbEKEPcKiM8x9aE6IG9BGW/fvj0o3zL3DfloVMvn6MqVK23+qg/qiPO+Z0xzJJCV/Ifqv2TV/wsHRZjJ06ImJ/VAGfze9ZUnhLAhGs19spRmyiEwW8qE6WA29TQwnepY5ti+a+DnOiYv4Nf9+C3TQCad/+Uvfzk7dte8CD9Kj3RKqA/CKBxlE15OXW8UlTM/pefy1cKRjqfl8qt+++6L0lO4Mq7HU5gaSgenuuFXfl5+yuPyKnyfjH4fVdYaHp/8wGVQXJ0TvkRy8ovTPaDsHtfvdZ98tftGWJdTMujc/Tw+aapMJcqzjN9X/+4ndN3D+rNQltHD6/552oQHLwPhXCb5eTylFULYDNmNc0bQu6NnNnZ4I0wLesqHh4ejh1W2AUNhPGNYHUIIYYjZzYEIIYQQwu6JAjETGI9mQa3Xr1+f++Y9zAvG7JnzwDwYzUmYClhGmPPAXA3NZQghhD4yhBFCCCGE0cQCEUIIIYTRRIEIIYQQwmiiQIQQQghhNLNRIJh8xkp4m5pAyEJBpF+6dSxOQxrlYkXbgnJNdUIckwiRb92QrhZsgrLuue73eGqTGWET9430vNybWniJtL3+LwIyj31G+v6X+94d1IOH83r3tByPs6v/7RB2DpMopw4LzfhCNZuCxW60aI0YWvBobrCwTlm+i8AiQ74I0hi0CNAm7quXkXxqMip/fi8DWrzJy6tFnNaJFoXadb2W/8u632V5OacehMJ5XC3W5eGA50oLbYVwGZm8BQJNn6Vtd7V40j6tr//o0aPuaD2w38GqsOxy8wLuztYHPV/2zhAswXzZF0WiTvhstGnszi3HzXLTTYO51h70VBc5o9yU1curX192W88l9eVWFPy0rHgI4QOTViD4B+abeaiZER1Mjes0yWLGLE3bbs70a8q7z5yJ6VTh+dU54Ykr5M9vmRbHZd7UD+fKu2byxp9whNH6A8pTMuA8L0+TMB5OcZFRawZwDH3plSiM9lwQXkalOZZVFQaZyvmVDIAfx6pzwM/Lqmu1+yE/Oc5B6aqc5T30sIQhD37x73su5BRPvHr1qv2t7aPR9NTP7cvhKD/dc+Wl8nqetWdP4fn1ehBeh14mR+UWhPW0avn2gXLg5eWXzkmJnh/Vm0AB4/9nTJ4h7DWdJWKSYCLEbMgvcFyaEdcJZk+qRE75AtdkrpSZk3Pk4ZjrNZQmaRFeaSs8x6TneRNOZQc/Vt76xbmcTmlOJg+ZZsnDzbnKV3HchIsf18pjZFLefemVlPEVBz+vQ675+bJ4HNLsqxuvR69L1TNy6Tp1IX9+8ZfcpM85KA3P0+vEwwJp6d6A6sXzHnouJLcgntJwkN/DOaTvMpZw3dPUMWmqrv0YJDvgr2eJuJKDY89X5XMUvrwnSsPTLum7hr/y5dfldsjD5dY9xU/541fKHMJlYtIWiLdv37a9BvUI2I1z0ybS5gXB2+GjLYvpvasHh0yEe/nyZWv+bF5EB3fu3GmvlTAE0rwA22PiNy+d9tyHRt6/f38WjuvKR2Wlp05vnx4XeQFxmpd0e9zX4x6qK2Qnvnpy8O7du7M47A4p8GNHR8Ipz5K+9Bz13FQ+33GRnUXZkltQH8+ePevOloOeqaeBTMtYI7if3G/q1s3Z3Auu+ZCI7je7YwLpc07ZyvuBn3rvgD/3WPXAjqXq5dLTLu87DD0XyAZucSjrHFz+Gi5jCfVJPQJy8xwA+3ggG9YBeuVjWeZ50f+LULn9GXzz5k13tDwq71C5oVZvlJv74VaREC4rk1Yg+AcuzbHbgpdX2fjsShaQYiOnxuQiNL2vc2n2Nba8LDH1EkYKTI1l0+sDhfEiTHG+gysDJShpUkRWhQaWe0IjzD2qlZ/tyUFDDw5KGooM0KCqQdeQAun9/PPP7TENvJQWDT+g1PNsrsJFn5ex8P9LR0Dl5VdDpI4UPNVbCUokCj1lD+EyM2kFghcK/6j6h6aXevfu3fZ429Dr8EmIjJ96L32T0Av0SV28vFUnq3Ljxo325al0+K01MOTJPRhqCGGZ9Oi90nuUP5YVzhn/VxkFPfOL3GvyR6ZN4b1yqCl0+Hl5gXpUWG+Mr1692h6PBQVEDbBbtRzyobFmvoorwepFa1dQZFNavlMoikl5L8nXrSJDyEqgOiPfZZ+/dUG5UbRwKpt+3ZpAOPacQSnS/amBxcotISFcSpqXxaRhjBExcX3jldC8GAavL4L4yqdRFjrf83gYjX0SVn7NC6X1c5BJ1//0pz+dHePv6f3+978/O25eXmfHkqXmp3PSqeHpg6cBTQNwdq40yjjgZdR1yqqyKW4tvRIPQ7oezuUbey+5H6QthuK7DLgffvjh7BiZvLx+/5QmMnsY3Xede5m4Jn9ciWRxiK/wnv/QMyDXV+/g8ReFdfQ/6M93ea/5JX2XHfz/V3UmyjRKyv99T9vLwrHj4dyV4UQZ3sN5PuUzxTW9B0K4jGQzrRBGgqmfHvg6hpEuCkMNbi2gF4/pfaj3TI8bq1LTAJ6Le5mRhQIaRWmw/kIIH5j8OhAhhDoM/ZQwLLSo8WMMn34Dw3DMY3AT/mWFOqNOUB5QJKiXTQ6phLAPxAIRwgjUe4cp9FRp6JzT09NJWEZCCPtPFIgQQgghjCZDGCGEEEIYTRSIEEIIIYxm8gqEL24z5Qlf5SI8TMDSN+7IPLUJWciGrL4uACC7ylFel19t8t5YSJe0aiDbolUCtwVy6D4uw5DsUypXCCFclEkrELxwWSmPaRo4JohpFbmpoIaQRY8kJ7BoD9DYatLdlGBRLmabs3qjw2d91DOUkwQ5v3fvXu+CRWPQrHeB4kJd4ljIZwqg+FHmZRmSfUrlCiGEdTBpBYKVC70BYyW7ba3+uCysTnl8fHzue3qOT05O2qV/tZfBFEHp0TLFDrP4kbncjZD9N77++uvubL1oFUrutxSYXaN9L5ZlSPYplSuEENbBpBUIVx6A79blh3WCnv8Y8/ImYDle3xRKsAx3uba/hjm0zwD4kEE5PMO5rnkc3/rZ/ZU+bgiGU9j8C2WMHja94xLKVK71jwLn96SUl3SUP7JJfpntJTfhuC5/yUtj7eVBTqVXQ2l4HQryUn6Ss6+u8Ses5C3LhSKoeD4U5fVdDlHVZHIUt5StNtSlcnj4EELYOf+cCU1D17sU7a5AJqqQJXmH0BK+hCcs59D0SM8tj4u/zimrp6t8SENhmgb3rE6Iq/D4KY8aXo+E66tX8kRGkOyLIBxyCWRVPNLiOo60PZzqBwjHuZdTcgilgfM6IA7nZfp9dY1sSkd5cCyZSUPpKKyOVcd+LNlLmYBrLhPXlCflUZ6Oyik4Vj2FEMIumY0CwQtZL9sp4Y1NHzQUCsOvGhE1YIKGQdf49YaCcGqIuOYNESCHu/K6s2xY8lOe1L/Lg78azRLS070ijMKpUYWyMUUOpV9e47hWx2U4P0c+z4/zvromnMoJfk4YlQXKc8Ihu8o4JFPtWq1uauhe+T0IIYRdMpvPONnRb4or7DUNQHVL5nJ4oY+hLayZc1CDXROZXIpJG/O2aBqXdmIirm/3TEzkHo5jHPKWMN+BIRqGHKh/H74YmgvB3AqGO0iTYRAmkdaGSbbNRbcLF8zNoWzUP7uINopJd6WOhmpKeJ65T0N1o3x0v8phvRBC2BWzUCCm0Pj0wSRJGkhvyJGXOQSLNioa2sKaLZS/++679hjY44B5C6TNODgNCQ0/igSgyLgMfuyUeyVwTFy+yijRNdJC1mVhq2YmZ6JE0Eg2vfT2a49r1651IdYHdaDnY2i790XbhaseAdmp6xKUL5QB6oW8Hj58+NE8lxLuYW2OjOA697qvbph/4SDDlP8fQgiXiKZXM3kw77rZGDhH/NJ/VyCLnJupZeLGUQ4du+lafm5GB8LomszvTYNZ9QfyrfkD8XTN8yGc/HElul4iOfpAFpnlda+Ey6I0JLtva841L2vtGSCe4iotr1Ovh766xt/zkdzgspKPUJnkz6/S9HyUVq3MAr8hXLbyGQkhhF2RvTDCSmAFwfLSNNALe+GbAvM+vfu+4ZplYaiJIZp1rG+xClh4dpV3CCGsymzmQIRpoS2hd6U87AsMSdSGS0IIYepEgQizhHkArOz4+vXrC62NQAPO3Agmiy4z6XVdYHVgciRzUrL9dghhjmQII4QQQgijiQUihBBCCKOJAhFCCCGE0UxegdBCOri+BXnCrzCmP2ZOAGG3OfYfQghhP5i8AsFEudPT03bGPwpEGrs6TCpEybp69Wq75fkyigHXp7jV+BB9C2SFEELYLrMYwmDpYPAVFMN5qBuULNZF4KsClpBetBIm14+Pp7nVeI0ojyGEMB0mr0Dcu3fvrNfJcsN9jSLWiV30Thky0BALrg8tQb0qxFUetXLKAsG+HA8fPmyXSO5rcCVzbUjI8/Elkwlb85cfjuEmUFiVFzk4J98auneEIazKIsc5/vrcEj/wcBep2xBCCCvAZ5xTR0v5ThGWMdYSy8hZLiENLGPsyyCPRUs2g5ZE7oP8y6WSnVpaWm6ZZZJ1TDoK5+UijJZTxp80QEs765xrqhf8avUChCOeL9Hscvgxvx6uzE/hQgghbJ7JKxA0PDRENA7eYEwJNYK1RlIN67KOtGpQboW5CN4ggze8KAwui/Kq5Yk8pVJE2qoDyq2yLGrYSUfKhiN5FJ9fKRC1enXlIoQQwmaZ9BAGJuovv/yyXamPoYumAZncJDrM58w3aOqyuqwzsjeN7UHTGLZhFjmWiC7B9M+OjQqzSZqG+Zw8woctBOXqo2+r6kVDMaAhD4asGqWg8/2Ysk6zn0QIIWyPWUyi1Ng6TGkiZdk4cl4b50dmGsNVx+nfvn3bHX2gb27DMhweHp7b1pqvMDinjlFSfOtpNfDHx8cHjx49ao+B/CkTDbjLQhldiaptVa09NIYafLbTRpEZutdSzDz/qSmXIYSw1zQv8kmDSRwxcZi0++DaLkzYmNWXke+ikLbyWTQksAgNueB8CAP8mvvLD+dDNe7PsEYJ/kN4fopf1im/5OnPAmFxOpdfCCGE7ZC9MMJGwSqQoYUQQtg/okCEjcFwDgtbZbfJEELYP2YxByLMC6wOTILMVtUhhLC/xAIRQgghhNHEAhFCCCGE0USBCCGEEMJookCEEEIIYTSTVyBY4GjRyoW7Qismlq62auMYSNcXneK4byOqMfgqkOuQs6QvTfy4NgV4hnzxqUUskn0T9VhCHmPvP+FdrrLMfc/uqoudLQty1TZxWyf+zsANvTd0f2tl9w3kfDG7XYAMm37OQhjLpBUI/mE+++yzs5ULWelwzMt/07C8NismHtmSyqzayPmq8KLylSKB1Rtry2Q71NVQ3fAC0pbfOMm9DsVEkK5WjyRdvXQvUh/rhPphN88xDMm+6YYQuG+rwOqlQyt51p5dHMuPb0pR5xllafpNwj3mnfH69euzMpFv373S/8TDhw/bVVn9f4i6YCl1ri/6mmiTnZttPGchrMKkFYhXr16d2wuBpZFZ5njKsC8GrNpj4UXFy2wsvtR0CS83FBtf0En5bOqFrnoAXuZTgEZzaG+NGkOy08BsGhqvsQoYDaYvH44yd/v27e5sGP7HUNQ3AY31yclJd7Z+KDfKd7kMOgo493FR5wPZtKz7GEprz7rZxnMWwipMWoGgF+X/mCxK1Ada+iZ7Acvy5MmT9pcGmhcRcmnogLLgOMa5ydT937x50/l+gDT8pca5whKPctOz5uVX663QINy6das7+xU1Kkq7Zr5WPrwkyUfnwAuZY+JLfkAGXtj0BP2eEE7xCV+iNMirDEdeyl9l9HBebvwpSxnewY9rLp/uE65sbGoyOYpbylZrtFRvHn6dPH369JzFapE1wkHxY/8S4bL686G65Z56GfvKJX/WBulD91/O61nnuub/D4IOB9SsBSiOz58/787qXL9+vVWqeW778GcEebjPKOFYL/DTdX79eSec5Ncz11e3en71jDqeZq0OQtgqTQ9nsjQ9iXN7HHDe9Mba46kgGd3hh8x+LjhXeXwfCvenjFwD7RWhNDjXXhTs/YHTsdIqIX7tmmRUen0QRpCO0iK+5CKMh6MMuqZ8vEy1PJWGl0lhy/RJU+mBwpKnwioPZJHMhOMa8SUXcXB6tvxYYUqZhNICrnm5hu6H4ng99UGYRffIkaxAPn1yeF3JlWFdVh3jlIfXrZeF6x5G/tQd5zXcX/cc8Jd8QLq1MuGnMCXE6csXiKtyIqPuscoAni/yKT2O/ZkgnO4XaZblAurD0yY+535PJA/o3OUMYdfU/9smBP9U+ofC+T/dFOAfvu/FVF7zl4OXp3wB8ZLwc9IgLhCnBuno5VZC/No1XkSkR9pc7ysHsujFRz4K52kqLeEyl9dIb0ge4eeE93vPeZmGwpb16edlPZXnHJOOyjgkE3CMH/CreOQp/xqEI67qaAjCqv4XQZ4edkgO8pa8gvPSj3DI6mUtw+GvMHLUeZkH8pTpA+H8noHig+fNPfNnQZA24WoQvkzf4b57PZEOfp4PcqtsckC+nrafq7yUg/Q5B9LVMXBN9VKmB+RFHH9WQ9g1k/8Kg3H7Rs7Wwddff93+zpXmJXFWHtwqG025aXcZMEnX5o64yZf5AX2wxTemZ/K9efNm6zcF82k51LMqzBeQeRmal3372wf3sAbDBJidqZu+YQPlw7AS91/mdg0J4NycPZaLDF/A48eP2+En3V/Kw7wI/f8JxuUpA/L6MJA/28w9GMNFx/oZggCGAEp8GE9DA7jaEBPwDDAkWM4Hwd/LWIP6Z0gDqP+7d++2Q5v8v0lG4NoYkGXRMEwI22TyCgTopcs/75iX4dSgseDl7C8tXr7M7eCFo5c2L4lyRrhoeibnJkz2vQAdlC7y9Rc9eTF22/R2Op9+eOkxx0KNEy/Ely9fdlfXj+qB/JpeV3tccuPGjVYmKVPE8bB6gQONIkpQCXFJgzIpryFFCqh7yt8HcwhqeYn37993Rx9AbpwrymMb3j4on0+mXAaUBdAcHijlIV2UHP4Xea5o2DhGsfJnjOMrV660YdSoo4iWzyLof8MbfxQKKVjLgAzcQ55rPRcghUz3lnRV1333m3yb3n4rk0AR93tblsHh/5T/TeqfuUbUkStzKOL+tRXKxdBzBVKClvmfD2ErNP9EkwVTHiIuIybmP0x820TmbrnmZdNdOW/SdXOk+3scT4vwisNvGVbnOJlBPX4fHg/nJlSQCbUGcqh+VQbJA0pTaUhul1/3SOdl/kpXcZSWPwduwnV/4gj8Pd/yuSBdXVMZlDdO15Wm5+NpeToO/l43JV4HLncNz6OsrxLM5B6GuuqTo3x23Xkcz1/HXPf69Tzlh0Me8PojHunU8HuAEy6Dy+3PglOWrS8/0VcWKO+Ph1X+lFN+ZZkF8Up5XU49V56W/Lz8fr2v/CFsi8Utc9gr/AXkqFHb5UtJDchF4eVdvvi3iSsZYXeosU9DG8JmmMUQRlgfmGab+94OB2kc2E3oi0z4YRjqUvNEwm5h6IVnmiFBnvOLzC0JIXxMtvMOk4GXPBwdHa08oY4GXN/x37t3b6VJqqvAuDRj2k2vd21zGEIIYcpEgQghhBDCaDKEEUIIIYTRRIEIIYQQwmhmoUDwbbjGxx354/y7713Bd+G1iVq+cI1fd/n1zf06Ia+hb9WniuoER905fs3LxhwE+U/hWQghhL2HORBTxr8hd/gcUd938/nfom+9t4G+1/bv6AWf9tU+LeQTM8qyblQnM7jFZ+gTU/8WX9/Ke51Sj7VPJfGr1X0IIYT1M3kLBKsE8slhCUvDauU2Vnejp+qr2G0b8mbFQ74gYFXDElaD9NURBSvVjVltb1mQQcvw7rJexsAqf43CcG4pZj4r5csGtzaw2qPK5vD5pFb6CyGEsFlmOweCBsSX6aXhqK0tz9AAZu1NDBE4LNGLIoBSU1uvHvloCH0Z2tLUTkNfNvacyzRfuqElbdkngjz5lLHcQpl8iU+jzC91o3pyJ1k4VhwcYRXXG3aXVUMPCqd4fuyQ/uvXr9slqktQLFz5ksLldUV6vt07dVPmEUIIYY10lohJI9O2g3neTf+Yr3e54pzyxoSOrDVTOqZ5H2rZlLzkrWGAWt1xjvOhAq9Pr0sNgygNrilumban4cekpXC1YRxQWorjkBfXvE5J09PaVF2GEEKok68w1gA9YRYRomfdNJytX20YA9M8vWz1jMvdJInvvWrwXn3p+iwQ5M2GQoTRokqeLjKADxVgMXj37l139qtsWtBJcQCrBnHZKMkhLH7k6+EZhiAO/n0LO5Vp1fDhCTYowiohK47XpawlsUCEEMLmmK0Cwc54viMkjUbN/L0NGDpplLEz1/SGq9tnA9fY8ZDGrVzymGslNNSetruhZac9HI038zOGoGGX0sHw0CorOPLVB/eFPKVIOeVcBgflgDjMbSlhCIa4joZn2B0T5ejOnTvdlV+vhRBC2ByzVSBoMNRI0xiP3fp3XZB3uWUyvWO3NDjqOaNEuAVgXdCYlooUdeW99Ro07FI4VllGmrTJoxaXtJFBSzz37UlAnbC9tltLsLLgV1NoSJNrmn8SQghhizQNxqTR2Licw7i3/JsGu/M9z9DY+kXx/D19xubl3/SEO99f4XrNn/R8XsJYSFP5en0wH0H+LjP+wmWWI42+uJ6X0vE0FO/+/fvtL3H9XhK2Rnm/cX33FsiHtEuQbxP3PIQQwgeyF8ZEoAffNIbt8S5uCT19HxJBnlevXm3ESrIMyLPq5lRYfpj7QX2uYk0JIYSwmEyinAiM26M47EJ5YMig/PSUiZjXr1/vzrYPygx1cevWrXZeBm5ZGM4gbpSHEELYHFEgwpmVQQ21Gmv/6mFXSJHYhWIVQgihnwxhhBBCCGE0sUCEUIEvR4a+WgkhhMtOFIgQepjCEE4IIUyVWSgQTPLrm0S3yy2r+VLA5w3I+ToGyObXBL1b+fWtKLlvUNba2hiCa7V7ySqZfXXE/ff6Hgv51dalODw87I4+QJhYJEIIwWAOxJTRPgg1UbXuQG1NhW3BWgNaBwG0VoKvQaAylBCmtobBPqJ71bc2A3Xk9ShUv7Vr64B0kWtorQkxJH8IIVw2Jm+B4AuB5qXdnZ2H9QEa5aE7mwZaS8H3laAMTUP1Ue+bMEPLUe8Ti9ZyYBnt2meXLG3N0tpNAz9ovVgFLBdDW7CXIAM7g4YQQrgEcyBodBaZzteJTPDlAkxs881SzY5vP455fGgo5iImdA3zUA/KQ+eqG9UTjnwYNuBY5n1+OQcfutF1GMpH5zVIb0gRZC4CC0r53icCOVe9t4u2YAeUDA2RIAd7fVxkyCSEEPaGzhIxaTAb94nK8MWuhzCQTa5vSAITOddlKifesuZwzP/Lhi2hbsiXNAQyqs68bjn2cJ6vL7Ht5dDxonw0jFMrB3E8fUFYz78cxiBeX5rLoHtV3pshkGOXz1sIIUyFfIWxBjCBN3XZ9qLLLbqFetEylWOa9w2g1HsvoYfNJlUszaxe/5ArrRRsQkW+bnqnt80mVITXdt/EQx7fdIshBe2O6dYSyvr+/fs2vqjlw2ZnbHgFpUXGIZ+rV692Z79C3io3QxxNA3/O2sCwCHW/ClgRltmCneulxWFVS1AIIewTUSDWCI0o4/V9Ju4HDx60jVYNrtWgcaVhbnrZbcO9yC376WHTk67GQwliDwzKwPwAFI0SlBqGY4i3SfgSwmWkHmrbfa/Csluw4x9CCOFjokCsGRQIesu1cXl6+PR2sTaoZ74M9LRpsNfV82V/CWQUzEFQ2jdv3mwVBxpY5KXR5rq2CFe5ltngSooGKJ4sHg7hfNIpkCdbnztYN2oKzViQZcwW7CXLKmkhhLDXNL2vScP4NmLKORp3x/WNSyv+quPkQzCGrvxxGkN3/3JcnWvlWD4g3ybG1r2OvA6G6g4/zUmozT3gXHF13JeP/BpF5KNropxXoLA4R3444ni4sp4lT63+Fcdl8bRcFsKrLoBrfh5CCJeV7IUxEeiF0wNuGrVzcyMuC8w1WPejKGvCqvWJVQaLEUg27lN2+QwhhGymFSYCjT1DFOtqnBkCYbLoMkMty4LywPDOZVTwQgihJApEmAwoEUySZDLq1GDeCnJl/kMIIXwgCkQIIYQQRpOvMEIIIYQwmigQIYQQQhhNFIgQQgghjGYWCgSrIvqyyQI/uV0sL8xMf5dhXbKQbrlJVd/qlmMgnXXKGUII4fIyeQWChtNXTRQ0hsz/xN27d6/dJXHbsBU36zawVoBkYeVGrR2wCnyJUC53zaeIQ3tJAMoAikcfKAx8QSA5Jfc6FJMQQgiXj8krEDScNHYl/n3/119/3S7CNAW0p8UySyLXYI2Bhyvsv/Do0aPu6GPYpAvFxj+PVD415SyEEEJYxN7Mgejr9dOQ0/tetUEfizZ7ooEmTxYf0tABVgKchhB8mML9yx09ScPl51xhiYeCwB4RWC64VsL+HOx/UaK9JpS2y1Oi4RryCyGEEPZCgWD3SG3aVEJDjsl+k6sHYv1Qg05DjsWEhpaNo7iGVQIZGEJA0cGPc9Cwg/vT4AsadbeucK6dMBm6wfKAZYFjLAq1lRw9fg1tZOVWHeT34Q2Ga7BihBBCCLAXCsSLFy8WzhHYJDT+NOhyKCsoC5pnIOVFPX38UDZ++eWX1tpAQ03jrFUO79692/4CjTrhBXFUVhSHZVZt9Pg1rl692ioyNetFCCGEUGP2CgSN3jKN6FQolY1VZB87jMAE059//rk7+xUsN4CCg4UBUGZQbpCT+REcM0QSQgghOLNWIDDny9xPozr0FcIUoKFmOMHlpHHGAoBlQRYKNoHivFYeLBU+YXKZMmuSqSsC5IWCcHJy0vl8AOsGig3hubaqkhNCCGHPaRqISXN6espkgTMn3E+uafS6q7+i+Pyum4cPH/bmz7H8m0a/8z3v73E8LcIrDr9lWJ3jmka+9fP4fXg8nOKKo6Oj7uiDnOV1ZJEMIYQQLjfZTOuSgeWByZ3gt15fcjARU8MZDpYOvvJgaGNdW26HEEKYL1EgLimuSDDZc5NfqYQQQtg/okCEEEIIYTR7s5BUCCGEELZHFIgQQgghjCYKRAghhBBGMwsFQosblbAOBP61a7uALxlq+0n43hV+XeXCaQ2IdUGavhT1HGAtD9UHzuuqvFaupSH/EEIIW4JJlFOGtQgQsxSVdQ+0tsO9e/dat2tYRwE5a2slIJ+vByG8HOuCOkOWWn5TRffZ64I6K+875aLOSuZU1hBC2Acmb4Ho286bnST16eHNmzdbt0vo7T9+/LhdJ+Hp06ed76+wGiSrS5Zcu3Zt7Z9QsjcIspDf2GWvd4VWxfS6YAVM6tNX0GQjMVbqLPnqq6+6oxBCCNtgtnMgtPEUDG2mxdAApu11DxGUIAONHxth1Ro45G16yedM72XjjhLiww6ElWm+dH3DE6R5eHjYykLjq/0uBPEYGtCwivzK9KkvHOFUhzhQXJfBhxHU4PsQk6dRllv35vr16+2vQ3367qTcZ1/2G5DD45J/n+IkOfrqL4QQwpJ0lohJg1m7JqoPb9TM2ttE+TN8gTy1YQwNLYhNyEweypv03bSveizl83PCKw2FVRrIrrCetsKWx8AwhMrssji6j5LBIR9PD8ohq0X1WF7nnDxDCCGszqy/wtDGT01j0i6zvCvozZI/PVt6/VAbxkDeppE86z2zlbdT9oxXsUBoGIUwyOTDGFglmobz3NbhQPj37993ZwcHb9++PbcdOVuKC+JzjaEXwTn3AZlUfsEwhCwWno7DZmJDlGneuXOnXXYbatYM8uqzQIQQQlgPe/EZ5653i6TBpQGVa3q41e2zgWuM49PQlfM2uOawJ4Wn6642ZEOjyVwAD0fjW1NmHJQOlrWm4YXaXhiLIC7DOChIJSgYyOFzGRzNe6jJyXAQ25E7hEcJQsEiDvNhhK79z//8TysTTsodikwIIYT1sBcKBD1frBC7AEXAe+NAg+aWBodrWAVQImpKwEVgm+8yTeYQ9CkzAlmkcPRZCYag/mm0a4oc8yCYQMoGXMxl6FMiUJ5o6L3OCEtd1dJFUaJcWHHcmiL+/d///axMpM1vNgELIYQ10rxYJ42P27u4TY/2zM/Hw0sUn9910zRMZzJ4+oz1D8nG9Zo/6a06Nq/64FdoPoLcn/70p7Njz9/rUs7jIm9ZJj8GnbscCkeZcApTKzt4GLkhuF6rL2RFfkG9Ohep5xBCCB/IZloTgeGHptFtj7d9SxgK8GELWQE0tLBtsDwwx6FRNEYPT2EN4ZPQRomoWlMom3YhxUpUs16EEEJYzF4MYewDmoi4C+WhnMz55MmTnSkPgNKgehg7d4EhHOL2DcVQLtVzlIcQQlidWCBC20DTGxcnJydrn58RQghhv4gCEUIIIYTRZAgjhBBCCKOJAhFCCCGE0cxCgWBmPZPp+mDWPmF2iVZblHOGVpTk2kXhC45a2r4dNl8f+DWvL+1ZsauFlsjfV470PTRwfddcXi/f2GeB8Be9D8jo+ZIez6Vkcte3Fsa6IG+/9+ukr0y4Xf8PSo7yOfZnZgoy+vMcwqxhDsSU8bUBamidh11916/1EnytAcmstSFY90DrEnCssFwv1yi4CKy9oDwlV7nmQl9dsXbCsqxbZpenlLl2f8mfeCWEqZVtGcizrKsxlHWic+Qp65byrLMOHdXXmPs5Bk+XY9X3Rep+DIvuETKUzxBQ3/ofHOIiz8AieGaRbRk5QpgDk7dA8DVA81Lszj6GTw6bF1l3tn3okTUvnXPrKCAzftpims8Sa58M8knhKstGLwP5IUPZ22lepO2S0yW72A6bnjIrZeqLD+qSe+lrP1BHyMzaDoI6a17C1Z7cql+PkCfp+UqYq0IaN27c6M4+hvLUdmxdB6qvTdH3eSz1vmrdLwv3R3ugDNEoEW24VSxRm7QOZCXUsG/Meg4EDRDLJA8h0/Y6GoYaLLVc7mkBbPjU18jVqJmcy2ERd4ugvLxES8WApbTxd7moR3/5X9T8LZOx4Lj2Mmfpat/HgqWub9261Z39isL4PUQ5Yuluhz1JBGEpVx/IWJq6Hzx40C7rPRbycoXh5cuXg+toUG4vpw8L+PCG6pH0VX/cN4X1++T+5boeDmkpHE4ovqfjz8gYJDdO90DpqqzI4XnJKTzldT/CaqE1/IaUAzZnQ4lA6ewrg9e56pfw/D/j95//+Z/tL8+Iyyn5OFb9u6z+TOGv54xrjt8HjkOYJZ0lYtLILOtgBsQsCW5K3SbIgFy1vCUzvw4mUsm9bmQixdVM/IL6chmWkYc4Srt0ju4L5aZe+K3JUvMnrZostXpW/QquEa6PWro1ucryLAP31PFz5CJNd14Ola08pnyS2eNwrHL6fXR/ysW1Gl5m4uqc+DidL/M/VQvj99WPlb6H9/jIonornw3i4ef10wfpERZIT+mQvuqnzEtpEtfrjXA44XlLbtL0OKTLudLFKV/QucsTwlyZrQWC3uemzP/LssxKhleuXOmO+qFXU/aGQT2XmuujeXG1rnk5dT4fo42ogN6Pbwam3lQJpuvmeWld8/I7O8Y51An3hR44FgF647Ve/bt37z4qc/Oy747q+LbfpE949UQZlvH7QdrqLY6lr9fax+HhYXf0oT6xPjlNg3KuvujpqveKzPhRDi8/5cNSQTiuYyFST5Vw3CN6y1gbiEseKj/DQjUI5zub6v+H8up5cTO7W3SWBblJg7rXkuGg9N3Shbyeh+qdZ4bw/qzzvIxFQ2Fu1QGGj7DCkbZkrN1zrEr+f+LPm+RmN1i3JpEn94V6aJSMc/dFyHJW+ocwN2apQPDPrheAXqS8lMsXxTbgBcF22CW8BHnhLPOS6FOEeBF7w+NuCF5eNPJqpEr0EudlyBwSf6lz7A3ZReFl7ekPQeNW2zn01atX7S/lclBMGHbgefBGHKS0UAc8I9rSG7dOkzFpjRm+ABoWnlk1WshU2wqd+0z5uC5liHvjz4HPF1mGWkO5TkgfeQFFdghk133h1+dXND35c+WsPUOuYPcpiwyL8a4onyvugadf+z/VfaRMUoT5Xy/v+dCQUQ1k2tQcmBC2ySwVCP7Z/Z+fRpwXwtiX6TrgBU9j4C8wGmZeiDXFYltIKelTquilbkO+2pwGwJpQTipjPguNqMvMyxrlkPtbQqOi8D6XwpHlxK0mixr4MT3DZRSGEp4ZKZeyHpTPLuWmXKRNQ0yDwzHl9WeNMNQlz6AUI8KWzyRcv379XDgaRhrhMeVdBD1yGv9lrIPITnl0XwSNM42+5ORXPX/HFey+/Cgbz44rZzyTPFOCeupTrFBqKRPwvFF/fs8lq+IjK+UfQnN9+pSeEGZD8883aXwssU/c5gV8bmzVUXx+N0XzcjonY19+TaNxdh2ZHa6tSpm/xm3dX35C1/rkXAd990SQD3KUSGa5oXQ05lxCHC9zWX4oy0ld1NIagvwFeXp9cq0si5wjPz0f/EoWXVO65b1W/VE++RGvrxyko3BefvdzufvqXtdxXgcun8rj5fA8a/WjcpKv/Gpy1u6np1de55o/ax5W8nvdSA75KS5hFV64rKp3T0vhVR84v14rSwhzYPIKxJzQi8RfeMugl9kUXiRqiMqX5Cr0NWKCvPrK7C/YVVBcbzQcNWpeTo7VcITNU7v3U/gfCCEsRzbT2gCYW2UibRqwtZqI5wRj04seL+YoMKmzb54EJmFNdGsa99HDBcuCOZmx7F0Mg11GuK/cdx/G4v+G4ZhN3eMQwnqZ9ToQU4XGkIYTd1mVBzhaYjImcxSYi9E3Bk1jorrcVMOi8fUoD9uDe8n8Ak2CxDGRNMpDCPMhFogQQgghjCYWiBBCCCGMJgpECCGEEEYTBSKEEEIIo5mFAsEkNyZZlTDxzidh9U3E2wblqnjIrIVw5gLyen1qcuGmYUEhz1euXOp6U5DP3O5VCCHsmskrEDRivmqcwwpxmqGP28UXD1JiWNlRckCfzNuGxnlZ9MUD8Plp36eV64avH05OPuwboDrE+b4N66SsEz4lzOz/EEIYx+QVCBoxvv8voceodfR32XtkYxwaPl9Kl2MaxFU2AFonKF+7tMpclE18Vkl9sPRwCCGEizHbORDsckkvFeWChYb6zO0yy29KyaAxYkGcEhQf9eD7hlrk79c51gZQvlY+fpRR4XRNQyfEw49jetiy3LB2v66D4l9keMCHHLw3r/ypa/mr/rlW9vyHKONrYzCVV/dT9VKrM68v0qMOtDYFfnpmOFb98Ks47g8617VNPVMhhDALmkZ48mhZ4z64Pnb56HXAMsnIxRLWQ7hsWu4a+MXpupZXpjyEk7+WulY81YeWafZjwmp5ZtLw5aRJj7jg4Uo8vRLS9OWGSVPnyotfpS0/wng9lKhe5Fy2shxcpxyEUXjVmcJxrvx0n/jVsSCMrulcSCZQOJ2Td5ZdDiFcZvbiKwzGr7c14c5ZZs4FvVSXDatE08i1/k2j1fr5cr5No9SWhyV9BaZ84jQNWnvOdc61zfUy0HMmP6w19J59B8ExsGqkb2XMuW9NTNqsLqnhB3YdxI845e6bJZSpeSZbtwxeL5rDoDzYMZF5KcB9Is3a/XKZhu6VwumewdhtnEMIYZ/YCwUCdrVkNA1MbVtsGh2Z0xc1nNtEDTSOhn4V+uZ2aOtsFAYNOTAfBL/vvvuu9QcNN+D6hjXWMf9hlQZ+SvcqhBCmzF4oEIxlHx4edmfbhYaOeQbeENKzZ14EjSc9Y3qtGm8HGin1mMfAXgFA+uSpORbw/v379vfZs2etdcHzAxSso6Ojc3L2Nd41pBBgUXjw4EF7DPT0NQdEvXcUBuQD4iEvyoV681I0cMsqCmrYiQ/aYKsPLB5uZSGe4vaxznsVQgh7T/MSnzQa75cTPv7t4+Ulis/vJpEsOB9Hh6ZROnddePiyPDpuGt02LL84+Xt5yvCqD687hde5+wkPXzof7/f83N/lkz9hNX9AZSnx9HA1dE15IGsph4cB9/N7Ij+uSzYc9N0rD+fpqpwhhHDZyGZaM4Ge/NC21yGEEMI22Zs5ECGEEELYHlEgZgBzFZhTwLoOi8bxQwghhG2QIYwQQgghjCYWiBBCCCGMJgpECCGEEEYzCwWC7/K1CFEN7Y9Qrn2waZiboAWRcFpzQHtC4MastQDEyTyHEEIIU2fyCgRKQd/W2DS0NLjaSnvbnziyCBLLKB8dHbX5azVMFpA6PT1tr41ZUVGLNS2Ccm9bWdo0YxWtEEIIu2XyCgRKAY1xDVYj5JpvpT1nll1ammWh9wnfQTOEEMI8mO0cCBqd4+PjdnGloaECWSl2PSyADBqKKeXRUEhZBg+vZaKxUuiTTlksPFxfY0zYvnAaAsLJsoF8+BMPf4ZnfGjGrSWEU3ilobAeDjwvINz9+/fbZaflp3uG8zrROb+SM4QQwo7gM86poyWWHZYrZiljlh7W8sNsv7xtyNOXSRbILHm0DLIvsaw4HFMOIDzhiAuEqR2TjtLGz/P3+MKXfAauc0y9kY7k0rEv5+xpKY4fE15hOdcyz/wqHcVxubmufL0OFEdQNuLwq3RDCCHsntlaINjk6M6dO+28A1zTAB28ffu2u7o9fNvtPrQRlOZDMOTSNJRtr/7nn39uywHlHA7iXblype1xE74Gm1lxTT12KHfLJN+msa5uB06eDJ3Qs9dcE+qzURzauR2+kVTzvLSbdikfIC7hCK85IKRNGXUuZDkhPlYH1YujLcpVHsrGfVXY27dvt78hhBB2y2wVCEzhfdtKTwFkW0a5GALz/+eff9423DTSfaA8EUZuzGRSDRfcvHnzTMHogzpn/gV5rAoKgeSsKRCAAqIwuH2Z4xJCCPvEbBUI5j74ttKMoe+id6otoMu5B/S0y22g1btm/J5Gkh66GmXQvAgmh2KdoMfe18gKbVutuPz2zQ+obQf+5MmTg4cPHy5UOpT+shM9a1Bmn9NQzvmA69evt7KpDJp7EUIIYWI0PbxJo/F6OcfH9n2s3lH8vuvroFEgzuSQw8/Br2lAz6478tN1yerhNQeAdFVu/IA5Ah6uBml5espD9aO4/P7+978/8yOO0HUP62kyP0HHfm9wwtPQfAiXn/K5TDjweCGEEHZP9sLYEhrPL+cFbAuGQ7IdeAghhHUx2yGMEEIIIeyOKBBbgHkOcDQwEXKTMNdAX0BoLkMIIYRwETKEEUIIIYTRxAIRQgghhNFEgQghhBDCaGahQLAmAF8xOIzla7VCuU3sj8A6BGU+crV1DIZQWvyGEEIIc2byCgRKgZZYdljpkekbcqzGyCJE64bPLkmfCZCnp6fn8hwLq0qGEEII+8DkFQjWLaDhLqkpC9teY0F7WyzLs2fPuqMQQghh3sx2DoQrCwxnHB4edmfn0VDHuj9f9OELH5rQ8IYPU2hL7EePHnU+IYQQwrzZi0mU7EjZtw8G+1Ew3FDuS7EK7FEhBcHR+g7a+Oq42+kSUDRu3brV+vcpOSGEEMLc2AsF4vnz51sZvvA5EA5LVINvfKWtxX2Tr2xFHUIIYV+YvQLBUAE9/G0zdv5DCCGEsE/MXoF4+vRpu6X1LmB4YtHcCoY3NPdBwxr5GiOEEMLcmfxS1jTQzD0QpbhMUPzrX//anX2M4jP8sMo8CCwcfXtYMNeBvDUngnAoBwxbwMnJSfu1iOITnj0pdrkrZwghhLAOshdGCCGEEEazF5MoQwghhLBdokCEEEIIYTRRIEIIIYQwmigQIYQQQhhNFIgQQgghjCYKRAghhBBGMwsFgi29y/0ngIWctDfF999/3/mOg3hKgzUffJOsqYGsY+XTZmIlXu7S+UZgQunIuRzlNe7XtqE8rAmyiE8//bSVN4QQwgVhHYgpc3JywjoVrXPwPzo6ao9fv37dXud3DKenp2dpAMfHx8fd2eZ4+PDhaFlXpVZ3oiw/UH7Cc00gb1m/hCvjrnIP1gGykveie1crWwghhNWYvAXiiy++aFeRrEFvElZd1fHdu3dnaQCbYW1jhcj79+93R5unr+76YGXNpqE9+Oqrr9pzrBHISzpeN4RrlIWVLT/rhBVGGyWnO+sHmRulpzsLIYRwEWY7BwLFgmWhMZfjaEBqjb/M6zWzNctMkwbXhW+S5UMkMtljJudcioeGAmS29zi1PPEHGjLiEo80SU/XavkCx2qwSVvmeIV1lN6TJ086n+V58OBBqxygPGj/jtoy4Pfu3Wt3Ql0FyS45qQfVZTkUoTAKJ5BP/m/evOl8P6C0yjghhBDWRGeJmDQyUZdgLse/acg6n9UgDZyb5JV2eQzk52ZwHWPqlyx9MgP+pKkwOoe+fDWMwK/8cTLbI7vkwI8hHkAepVFC+HIYAiQXv8q3Bml7fMKpHENoKEHha2VTOl7XuqZzD4ccqgvKruMyPa+nEEIIqzPrrzDY5bIpw8GzZ8+WmkDXB2k0DUzr6LEC1gz86b02jU7rJ27evHnw8uXL9phe9JUrV9pjeuNspEUa2gCsNiFR0Ktnw62msTuznvTl+80337Q9fiBM0wi2130jMYZkyA+rChYauHPnTvu7CpTr2rVr3VkdrANj0VACZVC5qQPKqHNBfcr6wbVGyWjrnvrxert79277Cy9evDizLKkO379/3/6GEEJYD7NVIGTKB+Yu4MaaqgmvODRENNw0OBp6oAGiMUKxcGicf/755/aYRtsbPRQC0pErG8Rl6Mt3W3z33XdtPSA7wzxQq1sUt1u3bnVnm2NICesDRcPvwyo7sYYQQuhn1hYIGjDn6tWr3dHyfPnll93Rh4aKRpuet3q4PifCYdvuslGlMfX0UHLGNn6L8l2EFBYpWLKULGuhIRy998ePH7fnpIflg3J5WZQeVoMhLmIZAuoCS5PAynP79u32XiOnlD38OafcWIiY+Cl5qVOFCyGEsCaa3tmkYbwaMeWcpnE582f8vIbi18a9GSsv0/dw8mt642e/QvEahaPz+YDmHOA4ruHp1cLKz8NpngDuhx9+ODumDrwekMvLpGslnp47L6NThvdwZR26q92X8r7p2OsOJ7ye/P54XNWDKK+B51vetxBCCOP4hD/NCzWEpZCFommgF1ofQggh7C+zHsII24cJkOicDBkwV+OiQxQhhBDmSSwQIYQQQhhNLBAhhBBCGE0UiBBCCCGMJgpECCGEEEYzCwWC7/iZsFeyaN+JIVgjQHFL5/tPLIPSGrvmQwghhDBXJq9AoDz44kxCCyUxB/T09LRdOnpMA84CScQ96pZU5lhuLCwqFUIIIVwmJq9A9G3nzWeE2ueBZYqPj48Pnj592p5flLGrQJYrYoYQQgj7zmznQLD3BftQiL49JxjaYHhh2SEOH77woQkNb7iVgzUQ8POllkMIIYTLwGwVCIYNtF/DEFgnGJYY2kyJ4Q8pCA7DG0BepIGV49WrV60figZ7X+B/eHjY+oUQQgiXhdkqEAwzaMtmHNs+37hxo7s6Dp8D4Wg3TKwd4u3bt+0v+bGpE+g3hBBCuCzMVoEANfpsoY11YB1bNq+6C2YIIYRwmZi1AgF8jfHgwYN2j4Z1wfDEojkTDG9o7oOGNfI1RgghhMvC5BUIGnLmKIDPUdAExjdv3pwbYijpm0SpiZEMU/gcCBzXsGZoDsSnn37aKhUMmdy/f7/9tJQvLxjGILzmYuRrjBBCCJeFbKYVQgghhNHMfggjhBBCCNsnCkQIIYQQRhMFIoQQQgijiQIRQgghhNFEgQghhBDCaKJAhBBCCGE0USBCCCGEMJooECGEEEIYTRSIEEIIIYwmCkQIIYQQRhMFIoQQQggjOTj4/wFDW6mNBZ/cmgAAAABJRU5ErkJggg==\"\u003e\u003c/p\u003e\n \u003cp\u003eAs mentioned, inner iteration is used to improve Q-values. In the inner iteration, the Q-function is estimated by the DNN function. The values of B\u003csub\u003en\u003c/sub\u003e records of the environment represent the current state of the environment (note that the current state in this step is the output of the modeling step. It means that previous events have also affected the current state).\u003c/p\u003e\n \u003cp\u003eWe use a deep neural network (Q-Network) to calculate Q(S, A). In each inner iteration, the characteristics of each record (values of variables) are entered into the DNN input layer. The DNN output layer also displays Q-values (intrusion/normal). All values obtained for each iteration are stored in the vector Q (Q-Value). The action that has the highest Q value (Q-Value) is predicted as the current action (A\u0026apos;\u003csub\u003ei\u003c/sub\u003e) for the desired record according to the following equation.\u003c/p\u003e\n \u003cp\u003eAction\u0026thinsp;=\u0026thinsp;argmax (Q-Value) (3)\u003c/p\u003e\n \u003cp\u003eThe noteworthy point is that the epsilon-greedy approach is used for exploration in DRL. Epsilon-greedy is a learning strategy based on the definition of reinforcement learning that helps the agent discover all possible actions by increasing the number of explorations and finding the optimal policy. An action selected by the epsilon-greedy approach is an action that is either randomly selected with probability ɛ or an action predicted with probability (1-ɛ). In the first inner iteration, the probability of choosing a random action is high. Over time, due to the use of the epsilon-greedy approach, this probability has decreased and actions are predicted with a probability of (1-ɛ). The DNN used is a three-layer deep neural network that uses the ReLU activation function for all layers. The loss function used is the Mean Square Error function.\u003c/p\u003e\n \u003cp\u003eIn the following, the values in (A\u003csup\u003e\u0026apos;\u003c/sup\u003e\u003csub\u003ei\u003c/sub\u003e) and the label of each state (A\u003csub\u003ei\u003c/sub\u003e) are used to calculate the reward, Eq.\u0026nbsp;(4).\u003c/p\u003e\n \u003cp\u003eRV\u003csub\u003ei\u003c/sub\u003e =Reward (AV\u003csub\u003ei\u003c/sub\u003e,label\u003csub\u003ei\u003c/sub\u003e) (4)\u003c/p\u003e\n \u003cp\u003eAccording to the Eq.\u0026nbsp;(2) we need to calculate TD-Target. For this purpose, we must calculate QV\u003csub\u003ei+1\u003c/sub\u003e. As you can see in Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e4\u003c/span\u003e the next state (S\u003csub\u003ei+1\u003c/sub\u003e) is used to calculate QV\u003csub\u003ei+1\u003c/sub\u003e and obtain TD-Target. Here we also use another deep neural network (Target-Network) with separate parameters from Q-Network). According to the Eq.\u0026nbsp;(5):\u003c/p\u003e\n \u003cp\u003eQT\u003csub\u003ei\u003c/sub\u003e=RV\u003csub\u003ei\u003c/sub\u003e\u0026thinsp;+\u0026thinsp;\u0026gamma; Q (S\u003csub\u003ei+1\u003c/sub\u003e, Ai\u003csub\u003e+\u0026thinsp;1\u003c/sub\u003e)\u003c/p\u003e\n \u003cp\u003eQT\u003csub\u003ei\u003c/sub\u003e=RV\u003csub\u003ei\u003c/sub\u003e\u0026thinsp;+\u0026thinsp;\u0026gamma; (QV\u003csub\u003ei+1\u003c/sub\u003e) (5)\u003c/p\u003e\n \u003cp\u003eRV\u003csub\u003ei\u003c/sub\u003e is the earned reward for S\u003csub\u003ei\u003c/sub\u003e, [0, 1] \u0026euro; \u0026gamma; is the discount factor for future rewards and QV\u003csub\u003ei+1\u003c/sub\u003e is the state vector of S\u003csub\u003ei+1\u003c/sub\u003e. The value of \u0026gamma; is updated in each iteration. After each iteration, the loss function will be calculated to improve the performance of the neural network (Target-Network). According to Eq.\u0026nbsp;(6) loss function is calculated at the end of each inner iteration.\u003c/p\u003e\n \u003cp\u003eLoss= (QV\u003csub\u003ei\u003c/sub\u003e-QT\u003csub\u003ei\u003c/sub\u003e)\u003c/p\u003e\n \u003cp\u003eLoss\u0026thinsp;=\u0026thinsp;1/n\u0026sum; (Q\u003csub\u003es\u003c/sub\u003e-Q\u003csub\u003eT\u003c/sub\u003e) r\u0026thinsp;+\u0026thinsp;\u0026gamma;Q(S\u003csup\u003e,\u003c/sup\u003e,a\u003csup\u003e,\u003c/sup\u003e))\u003csup\u003e2\u003c/sup\u003e (6)\u003c/p\u003e\n \u003cp\u003eAt the end of the outer iteration, the updated parameters are sent to the Q-Network and Target-Network.\u003c/p\u003e\n \u003cp\u003eUsing two neural networks (Target-Network, Q-Network) will lead to better stability of the model. Note that outer iterations continue until the all dataset is covered.\u003c/p\u003e\n \u003cp\u003eThe event that is detected as an attack is provided to the cloud in order to detect the type of attack.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec7\" class=\"Section2\"\u003e\n \u003ch2\u003e3.3 Multi-class detection phase\u003c/h2\u003e\n \u003cp\u003eClassifying attacks into different types requires a more complex model and more resources. The multi-class detection phase of the proposed method is less sensitive to delay. Therefore, are used several algorithms in the cloud. The main idea of ensemble methods and the combination of several conceptually different machine learning classifiers is to achieve higher improvement and robustness than individual classifiers. In other words, ensemble methods often provide better estimates than the best estimators. Ensemble models perform best when independent predictors are used as much as possible because the chance of making different mistakes increases and the accuracy of the model increases. The main challenge of Ensemble-based methods is how to combine base classifiers (base learner). Here we use majority voting to aggregate the results.\u003c/p\u003e\n \u003cp\u003eAll the information related to detected events (binary detection phase, multi-class detection phase) is sent to the summary module. This module is responsible for updating the model. The updated parameters are sent for the binary detection phase and the multi-class detection phase.\u003c/p\u003e\n\u003c/div\u003e"},{"header":"Experimental Evaluation","content":"\u003cp\u003eTo evaluate the performance of the proposed method, experiments are carried out on CICIDS2018[\u003cspan class=\"CitationRef\"\u003e31\u003c/span\u003e] datasets. First, we describe the dataset used in the experiment, then consider the metrics to evaluate the model, and elaborate on the setting parameters. Finally, we analyze the experimental results and compare our approach and existing techniques.\u003c/p\u003e\n\u003cdiv id=\"Sec9\" class=\"Section2\"\u003e\n \u003ch2\u003e4.1 Dataset\u003c/h2\u003e\n \u003cp\u003eAssessing the dataset accuracy and collecting based on the real world is a challenging issue. In this study, CICIDS2018 is used as input and the model is evaluated using the CSE-CIC-IDS2018 dataset. As a result of cooperation between the Communications Security Establishment (CSE) and The Canadian Institute for Cybersecurity (CIC) the CIC-IDS2018 datasets were created. They used the concept of profiles to make a cybersecurity dataset that contains an explanation of intrusions and abstract distribution models for protocols and applications. Heartbleed, Botnet, Web attacks, Brute force attacks, DoS, DDoS and infiltration are seven kinds of attacks in the CIC-IDS2018 dataset. Table \u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e shows the percentage distribution of data types in CIC-IDS2018 dataset. The dataset includes the captured network traffic and system logs of each machine, along with 80 features extracted.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec10\" class=\"Section2\"\u003e\n \u003ch2\u003e4.2 Machine Learning Performance Evaluation\u003c/h2\u003e\n \u003cp\u003eWe used different measurements to evaluate the performance of our DRL model and other ML algorithms, such as Accuracy, Precision, Recall, and F1- Measure (combines precision and recall). Accuracy is one of the most common metrics to evaluate a model. It measures the percentage of samples that are correctly classified and ignores samples that are misclassified. In general, the accuracy metric is insufficient to evaluate unbalanced datasets such as network traffic data, which usually include normal traffic. Therefore, using various metrics in IDS is necessary. All metrics are derived from the confusion matrix (True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN). We analyze our proposed method based on performance metrics as shown in Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e.\u003c/p\u003e\n \u003cdiv class=\"gridtable\"\u003e\u0026nbsp;\u003cbr\u003e\n \u003ctable id=\"Tab1\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eValue and percentage distribution of data types in CIC-IDS2018 dataset\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003ccolgroup cols=\"2\"\u003e\u003c/colgroup\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eTraffic type\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eDistribution (%)\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eBenign\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e83.070\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eDDoS\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e7.786\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eDoS\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e4.031\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eBrute force\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e2.347\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eBotnet\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e1.763\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eInfiltration\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.997\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eWeb attack\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.006\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003c/div\u003e\n \u003cdiv class=\"gridtable\"\u003e\n \u003cdiv align=\"left\" class=\"colspec\"\u003e\u003cbr\u003e\u003c/div\u003e\u0026nbsp;\u003ctable id=\"Tab2\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eEvaluation metrics\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003ccolgroup cols=\"2\"\u003e\u003c/colgroup\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eMetric\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eFormula\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eAccuracy\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\frac{(TP+TN)}{(TP+FP+FN+TN)}\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003ePrecision\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\frac{TP}{\\left(TP+FP\\right)}\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eRecall\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\frac{TP}{\\left(TP+FN\\right)}\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eF1- Measure\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\frac{2}{\\frac{1}{Recal}+ \\frac{1}{Precision}}\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec11\" class=\"Section2\"\u003e\n \u003ch2\u003e4.3 Implementation details and Parameter setting\u003c/h2\u003e\n \u003cp\u003eExperiments were operated on Intel Core i7-8700 LGA1151 @ 3.70 GHz, 16 GB DDR4 RAM 2400 MHz machine. The proposed model was implemented in Python language. Tensorflow, Keras and Scikit-learn are important machine learning libraries that were used.\u003c/p\u003e\n \u003cp\u003eThe hyperparameters involved in the experiment and the value of the hyperparameters of the algorithms are shown in Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e.\u003c/p\u003e\n \u003cdiv class=\"gridtable\"\u003e\u0026nbsp;\u003ctable id=\"Tab3\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eThe hyperparameters value of the proposed model\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003ccolgroup cols=\"2\"\u003e\u003c/colgroup\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eParameters\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eValues\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eInner itration\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e50\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eOuter itration\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e100\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eɛ\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.9\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eDecoy rate\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.99\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u0026gamma;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.001\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u0026alpha;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.001\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eMini Batch\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e500\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eActivation function(Q-network)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eReLU\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eActivation function(target-network)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eReLU\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eHiden layer (Q-network)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eHiden layer (Target-network)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec12\" class=\"Section2\"\u003e\n \u003ch2\u003e4.4 Result\u003c/h2\u003e\n \u003cp\u003eAt first, we experimented the proposed approach based on three different values of the discount factor (\u0026gamma;) (\u0026gamma;\u0026thinsp;=\u0026thinsp;0.001, \u0026gamma;\u0026thinsp;=\u0026thinsp;0.01, and \u0026gamma;\u0026thinsp;=\u0026thinsp;0.1). The results obtained for Accuracy, Precision, Recall and F-Measure metrics are displayed in the table. According the Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e4\u003c/span\u003e, it can be concluded that with the increase of the discount factor (\u0026gamma;), the investigated metrics decrease, and with the decrease of the discount factor, these metrics increase. In general, the amounts of loss and reward obtained during the DQN training process depend on the value of the discount factor.\u003c/p\u003e\n \u003cdiv class=\"gridtable\"\u003e\u0026nbsp;\u003ctable id=\"Tab4\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eComparison of performance for different discount factors\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003ccolgroup cols=\"4\"\u003e\u003c/colgroup\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\u0026nbsp;\u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003e0.001\u0026thinsp;=\u0026thinsp;\u0026gamma;\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003e0.01\u0026thinsp;=\u0026thinsp;\u0026gamma;\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003e0.1\u0026thinsp;=\u0026thinsp;\u0026gamma;\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eAccuracy\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.9597\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.8629\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.8234\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003ePrecision\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.9034\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.8276\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.8128\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eRecall\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.9171\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.7790\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.7640\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eF-Measure\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.9689\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.8803\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.8269\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003c/div\u003e\n \u003cp\u003eSince we get better results with \u0026gamma;\u0026thinsp;=\u0026thinsp;0.001 in the proposed method, we will do the next comparisons with this value.\u003c/p\u003e\n \u003cp\u003eFigure \u003cspan class=\"InternalRef\"\u003e5\u003c/span\u003e shows the binary detection results obtained from the comparison of the proposed method with NB, KNN, DNN, [\u003cspan class=\"CitationRef\"\u003e32\u003c/span\u003e] and [\u003cspan class=\"CitationRef\"\u003e33\u003c/span\u003e] in terms of accuracy, Precision, Recall and F- Measure. As can be seen, the proposed approach has obtained the highest value compared to other methods in all metrics. For example, the proposed method achieved a Precision of 0.9969.\u003c/p\u003e\n \u003cp\u003eOur proposed model can detect all kinds of attacks. In the continuation of the experiments, we determined the power of the proposed model in detecting types of attacks with accuracy and F1- Measure metrics. The results obtained are shown in Fig. \u003cspan class=\"InternalRef\"\u003e6\u003c/span\u003e and Fig. \u003cspan class=\"InternalRef\"\u003e7\u003c/span\u003e respectively. As it is clear the proposed method has achieved the highest accuracy and F1- Measure compared to other methods. According to Fig. \u003cspan class=\"InternalRef\"\u003e6\u003c/span\u003e and Fig. \u003cspan class=\"InternalRef\"\u003e7\u003c/span\u003e, our findings demonstrate that training the DRL can result in better performance in multi-class classifying minority classes (web attacks and infiltration) over training on the true imbalanced dataset.\u003c/p\u003e\n \u003cp\u003eDRL models are suitable to be used in an industrial and online production environment. One of the most important advantages of it is that once trained, it provides the correct action for a specific state (intrusion features). Table \u003cspan class=\"InternalRef\"\u003e5\u003c/span\u003e shows the training time and prediction time for the proposed DRL-based model and other models (KNN, NB and DNN). We have implemented a Bernoulli NB using only the discrete features in Naive Bayes (NB). The DNN was applied with three hidden layers with 1024, 512 and 128 nodes. Our proposed method requires more training time than methods KNN and NB, but it has much less prediction time. In general, DRL-based methods significantly reduce the prediction time, which makes them suitable for online detection and IoT network services.\u003c/p\u003e\n \u003cdiv class=\"gridtable\"\u003e\u0026nbsp;\u003ctable id=\"Tab5\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 5\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eTraining time and prediction time for the proposed DRL-based model and other models (KNN, NB and DNN).\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003ccolgroup cols=\"3\"\u003e\u003c/colgroup\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eMethod\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eTraining Time (sec)\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003ePrediction Time (sec)\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eKNN\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e92.73\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e84.39\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eNB\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6.82\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.98\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eDNN\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e308.69\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.82\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eOur approach\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e276\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.52\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec13\" class=\"Section2\"\u003e\n \u003ch2\u003e4.5 Summary of results\u003c/h2\u003e\n \u003cp\u003eOur proposed DRL-based IDS provides binary detection and attack type detection in two separate steps. In the evaluation section, both components were examined and compared. The results showed that our proposed method was able to obtain better values in different metrics and provided a more robust solution.\u003c/p\u003e\n \u003cp\u003eReducing the number of false negatives is one of the main goals of IDS. Therefore Recall is important in evaluation. According to the results, the model has achieved a high Recall in comparison to the other models. In addition, F1- Measure has also obtained excellent results and it can act as a solution for imbalanced datasets Considering the outcomes presented in Table \u003cspan class=\"InternalRef\"\u003e5\u003c/span\u003e, the proposed model provides much smaller prediction times than the experimented other model. Our solution makes the convergence slower but more stable. Overall, we can conclude that the performance of DRL models depends to a large extent on the choice of the value of the discount factor.\u003c/p\u003e\n\u003c/div\u003e"},{"header":"Limitation and Future Scope","content":"\u003cp\u003eThe results of the experiment support the claim that our approach can provide an effective and practical solution to automatically IDS in IoT. However, like other intrusion detection systems, our proposed IDS has some defects. We consider these weaknesses. By examining the limitations and solving them, more effective approaches can be done in the future.\u003c/p\u003e \u003cp\u003eOne of the most fundamental limitations in the evaluation of intrusion detection systems is the lack of access to real datasets. We evaluated our proposed method on the CIC-IDS2018 dataset. This dataset has a limited number of attack types. Many other types of attacks that exist in IoT networks have not been considered. In addition, we have trained our method only on the CIC-IDS2018 dataset, thus, it needs to be retrained to be implemented in other environments. Therefore, investigating this method in other data sets with different types of attacks or creating a large dataset with various types of attacks can be future works.\u003c/p\u003e \u003cp\u003eExamining the generative algorithms in the proposed solution to overcome the challenges of unbalanced data and deal with new attacks is another future work.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eIn this work, we have proposed a Deep Q-learning-based (DQL) IDS in IoT networks. We identify traffic behavior hierarchically. Binary classification is used to allow fast detection in fog, especially for benign traffic. In the next step attack identification can be trained for different classes in the cloud. We model the environment based on GRU. This process aims to find internal relationships between events over a relatively long time. Deep Q-learning received the internal state from GRU as the input to the Q-function approximator, which is used for the action (normal/intrusion). In each iteration, parameters are updated to achieve better results. As mentioned, all events detected as intrusions are sent to the cloud for intrusion detection. In the cloud, several machine learning methods are used to increase robustness and improve performance. Using the ensemble method can be an excellent method for IDS.\u003c/p\u003e \u003cp\u003eWe showed that the discount factor parameter is important in regulating the speed of convergence of the algorithm. The low value of this parameter converges the DQL algorithm quickly. Our experiment demonstrated that the suggested DQL approach can learn effectively from the environment in an autonomous manner, be successful in detecting unrepresented classes and can significantly reduce prediction times, which makes it suitable for online detection in IoT networks.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eEthical Approval\u0026nbsp;\u003c/strong\u003eNot applicable.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting\u0026nbsp;interests\u0026nbsp;\u003c/strong\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor Contributions\u0026nbsp;\u003c/strong\u003eAll authors contributed to the writing of the manuscript text and reviewed it.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding\u0026nbsp;\u003c/strong\u003eNot applicable.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAvailability of data and materials\u0026nbsp;\u003c/strong\u003eNot applicable.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n \u003cli\u003eW\u0026oacute;jcicki K (2022) Biega nska, M.; Paliwoda, B.; G\u0026oacute;rna, J. Internet of Things in Industry: Research Profiling, Application, Challenges and Opportunities\u0026mdash;A Review. Energies 2022, 15, 1806\u003c/li\u003e\n \u003cli\u003eLee I, Lee K (2015) The Internet of Things (IoT): Applications, investments, and challenges for enterprises. Business horizons 58:431\u0026ndash;440\u003c/li\u003e\n \u003cli\u003eSabireen H, Neelanarayanan V (2021) A review on fog computing: Architecture, fog with IoT, algorithms and research challenges. Ict Express 7:162\u0026ndash;176\u003c/li\u003e\n \u003cli\u003eGoudarzi M, Palaniswami M, Buyya R (2019) A fog-driven dynamic resource allocation technique in ultra dense femtocell networks. Journal of Network and Computer Applications 145:102407\u003c/li\u003e\n \u003cli\u003eDe Souza CA, Westphall CB, Machado RB (2022) Two-step ensemble approach for intrusion detection and identification in IoT and fog computing environments. Computers \u0026amp; Electrical Engineering 98:107694\u003c/li\u003e\n \u003cli\u003eLabiod Y, Amara Korba A, Ghoualmi N (2022) Fog Computing-Based Intrusion Detection Architecture to Protect IoT Networks. Wireless Pers Commun 125:231\u0026ndash;259. https://doi.org/10.1007/s11277-022-09548-7\u003c/li\u003e\n \u003cli\u003eHalim Z, Sulaiman M, Waqas M, Aydın D (2023) Deep neural network-based identification of driving risk utilizing driver dependent vehicle driving features: a scheme for critical infrastructure protection. J Ambient Intell Human Comput 14:11747\u0026ndash;11765. https://doi.org/10.1007/s12652-022-03734-y\u003c/li\u003e\n \u003cli\u003eUprety A, Rawat DB (2020) Reinforcement learning for iot security: A comprehensive survey. IEEE Internet of Things Journal 8:8693\u0026ndash;8706\u003c/li\u003e\n \u003cli\u003eWang J, Hu J, Min G, et al (2020) Fast adaptive task offloading in edge computing based on meta reinforcement learning. IEEE Transactions on Parallel and Distributed Systems 32:242\u0026ndash;253\u003c/li\u003e\n \u003cli\u003eHuang L, Bi S, Zhang Y-JA (2019) Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks. IEEE Transactions on Mobile Computing 19:2581\u0026ndash;2593\u003c/li\u003e\n \u003cli\u003eLu X, Xiao L, Xu T, et al (2020) Reinforcement learning based PHY authentication for VANETs. IEEE transactions on vehicular technology 69:3068\u0026ndash;3079\u003c/li\u003e\n \u003cli\u003eWang T, Liu Z, Zhang T, et al (2022) Adaptive feature fusion for time series classification. Knowledge-Based Systems 243:108459\u003c/li\u003e\n \u003cli\u003eAbou El Houda Z, Brik B, Khoukhi L (2022) \u0026ldquo;why should i trust your ids?\u0026rdquo;: An explainable deep learning framework for intrusion detection systems in internet of things networks. IEEE Open Journal of the Communications Society 3:1164\u0026ndash;1176\u003c/li\u003e\n \u003cli\u003eSadaf K, Sultana J (2020) Intrusion detection based on autoencoder and isolation forest in fog computing. IEEE Access 8:167059\u0026ndash;167068\u003c/li\u003e\n \u003cli\u003eIlly P, Kaddoum G, Moreira CM, et al (2019) Securing fog-to-things environment using intrusion detection system based on ensemble learning. In: 2019 IEEE wireless communications and networking conference (WCNC). IEEE, pp 1\u0026ndash;7\u003c/li\u003e\n \u003cli\u003eDeng Q, Goudarzi M, Buyya R (2021) FogBus2: a lightweight and distributed container-based framework for integration of IoT-enabled systems with edge and cloud computing. In: Proceedings of the International Workshop on Big Data in Emergent Distributed Environments. ACM, Virtual Event China, pp 1\u0026ndash;8\u003c/li\u003e\n \u003cli\u003eGoudarzi M, Wu H, Palaniswami M, Buyya R (2020) An application placement technique for concurrent IoT applications in edge and fog computing environments. IEEE Transactions on Mobile Computing 20:1298\u0026ndash;1311\u003c/li\u003e\n \u003cli\u003eSethi K, Kumar R, Prajapati N, Bera P (2020) Deep reinforcement learning based intrusion detection system for cloud infrastructure. In: 2020 International Conference on COMmunication Systems \u0026amp; NETworkS (COMSNETS). IEEE, pp 1\u0026ndash;6\u003c/li\u003e\n \u003cli\u003eYang Y, Tu S, Ali RH, et al (2023) Intrusion detection based on bidirectional long short-term memory with attention mechanism\u003c/li\u003e\n \u003cli\u003eStrickland C, Saha C, Zakar M, et al (2023) DRL-GAN: A Hybrid Approach for Binary and Multiclass Network Intrusion Detection. arXiv preprint arXiv:230103368\u003c/li\u003e\n \u003cli\u003eLazzarini R, Tianfield H, Charissis P (2023) A Stacking Ensemble of Deep Learning Models for IoT Network Intrusion Detection. A Stacking Ensemble of Deep Learning Models for IoT Network Intrusion Detection\u003c/li\u003e\n \u003cli\u003eMohamed D, Ismael O (2023) Enhancement of an IoT hybrid intrusion detection system based on fog-to-cloud computing. J Cloud Comp 12:41. https://doi.org/10.1186/s13677-023-00420-y\u003c/li\u003e\n \u003cli\u003eStefanova ZS, Ramachandran KM (2018) Off-policy q-learning technique for intrusion response in network security. World Academy of Science, Engineering and Technology, International Science Index 136:262\u0026ndash;268\u003c/li\u003e\n \u003cli\u003eLopez-Martin M, Carro B, Sanchez-Esguevillas A (2020) Application of deep reinforcement learning to intrusion detection for supervised problems. Expert Systems with Applications 141:112963\u003c/li\u003e\n \u003cli\u003eSharma P, Jain S, Gupta S, Chamola V (2021) Role of machine learning and deep learning in securing 5G-driven industrial IoT applications. Ad Hoc Networks 123:102685\u003c/li\u003e\n \u003cli\u003eAshenafi A (2022) A Model to Detect MiTM Attack in IoT Networks: A Machine Learning Approach. PhD Thesis, St. Mary\u0026rsquo;s University\u003c/li\u003e\n \u003cli\u003eDaoud WB, Mahfoudhi S (2022) SIMAD: Secure Intelligent Method for IoT-Fog Environments Attacks Detection. Computers, Materials \u0026amp; Continua 70:\u003c/li\u003e\n \u003cli\u003eSarwar A, Alnajim AM, Marwat SNK, et al (2022) Enhanced anomaly detection system for iot based on improved dynamic SBPSO. Sensors 22:4926\u003c/li\u003e\n \u003cli\u003eSewak M, Sahay SK, Rathore H (2022) Deep Reinforcement Learning in the Advanced Cybersecurity Threat Detection and Protection. Inf Syst Front. https://doi.org/10.1007/s10796-022-10333-x\u003c/li\u003e\n \u003cli\u003eAbou Ghaly M, Hannan SA (2024) Protecting Software Defined Networks with IoT and Deep Reinforcement Learning. International Journal of Intelligent Systems and Applications in Engineering 12:138\u0026ndash;147\u003c/li\u003e\n \u003cli\u003eA Realistic Cyber Defense Dataset (CSE-CIC-IDS2018) - Registry of Open Data on AWS. https://registry.opendata.aws/cse-cic-ids2018/. Accessed 26 Dec 2023\u003c/li\u003e\n \u003cli\u003eLabiod Y, Amara Korba A, Ghoualmi N (2022) Fog Computing-Based Intrusion Detection Architecture to Protect IoT Networks. Wireless Personal Communications 1\u0026ndash;29\u003c/li\u003e\n \u003cli\u003ede Souza CA, Westphall CB, Machado RB (2022) Two-step ensemble approach for intrusion detection and identification in IoT and fog computing environments. Computers \u0026amp; Electrical Engineering 98:107694\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"the-journal-of-supercomputing","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [The Journal of Supercomputing](https://www.springer.com/journal/11227)","snPcode":"11227","submissionUrl":"https://submission.nature.com/new-submission/11227/3","title":"The Journal of Supercomputing","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"IoT, Intrusion detection system, Deep learning, Reinforcement learning, Deep Q-networks","lastPublishedDoi":"10.21203/rs.3.rs-3864298/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3864298/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe increasing growth of the Internet of Things and its open and shared character has led to a rise in new attacks exponentially. As a consequence, the quick detection of attacks in IoT environments is essential. The intrusion detection system (IDS) is responsible for protecting and detecting the type of attacks. Creating an IDS that works in real-time and adapts to environmental changes is critical. In this paper, we propose a deep reinforcement learning-based (DRL) adaptive IDS that addresses the mentioned challenges. DRL-based IDS helps to create a decision agent, who controls the interaction with the indeterminate environment and performs binary detection (normal/intrusion) in fog. For multi-class classification of attacks, we use the ensemble method in the cloud. The proposed approach is evaluated on the CICIDS2018 dataset, in binary and multiclass classification. The results show that the proposed model achieves a comparable performance in detecting intrusions and identifying attacks and it reduces the prediction time significantly. Furthermore, it is a suitable solution for underrepresented attacks. Overall, combining multiple methods can be a great way for an intrusion detection system.\u003c/p\u003e","manuscriptTitle":"A novel reinforcement learning-based hybrid intrusion detection system on fog-to-cloud computing","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-01-17 09:56:50","doi":"10.21203/rs.3.rs-3864298/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-06-24T13:37:25+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-06-23T15:08:25+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-06-23T09:36:36+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-06-20T13:02:38+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-06-19T21:46:21+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"53548963470293172068972580568386089014","date":"2024-06-10T23:28:23+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"58700146384281430380819636710599348579","date":"2024-06-05T08:52:06+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"89288933737788290539486401073979774761","date":"2024-06-03T07:46:58+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"176132202722306813010442872860071667861","date":"2024-06-03T03:38:35+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"323762484269280497789426880733900608836","date":"2024-06-03T00:55:50+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"269169283406570830733766692181816297725","date":"2024-06-02T20:40:02+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"110007741285604553115716697442816774281","date":"2024-06-02T19:41:09+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"44766995121441806675596245028260865914","date":"2024-06-02T19:39:53+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-06-02T18:15:46+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-01-16T20:08:41+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-01-16T07:29:57+00:00","index":"","fulltext":""},{"type":"submitted","content":"The Journal of Supercomputing","date":"2024-01-14T20:25:45+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"the-journal-of-supercomputing","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [The Journal of Supercomputing](https://www.springer.com/journal/11227)","snPcode":"11227","submissionUrl":"https://submission.nature.com/new-submission/11227/3","title":"The Journal of Supercomputing","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"33012efd-978b-4516-8892-457ee67c8829","owner":[],"postedDate":"January 17th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2024-07-30T18:21:26+00:00","versionOfRecord":[],"versionCreatedAt":"2024-01-17 09:56:50","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-3864298","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3864298","identity":"rs-3864298","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}