{"paper_id":"2bb1f920-0cce-4e17-a1ea-1ce23a087fbf","body_text":"Intelligent Agents in Educational Institutions: AEdBOT– A Chatbot for Administrative Assistance using Deep Learning Hybrid Model Approach | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Intelligent Agents in Educational Institutions: AEdBOT– A Chatbot for Administrative Assistance using Deep Learning Hybrid Model Approach Muhammad Shahroze Ali, Muhammad Waseem Anwar, Farooque Azam, Muhammad Hashir Ashraf This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4257811/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Chatbots substantially improve administrative support for educational institu- tions facing immense pressure during admissions. Chatbots not only automate repetitive tasks, handle large volumes of inquiries, collecting data from inter- actions but also provide an additional way for students to access information. Existing chatbots are built on traditional Artificial Intelligence (AI) approaches where the accuracy required for seamless real-time interactions is usually compro- mised. This article presents novel AEdBOT - an AI-based Educational ChatBOT system where a novel Deep Learning (DL)-based Hybrid model approach is proposed grounded on integrating informational retrieval and generative neural networks. Moreover, a novel Natural Language Processing (NLP) pipeline is developed on top of the open-source Rasa platform to aid with BERT (Bidi- rectional Encoder Representation Transformer) for dense feature extraction and DIET (Dual Intent and Entity Transformer) Classifier for intent classification and entity extraction from the natural language text. Furthermore, the customized dual fallback classifier algorithm is developed to provide the self-learning ability to a chatbot on out-of-scope inputs and acts as a recommendation system. The effec- tiveness of the proposed chatbot is established through two real-life datasets from educational institutes. For the first dataset, AEdBOT achieved 94.7%, 96.0%, 96.0%, and 95.1% precision, accuracy, recall, and F1-Score, respectively at an average mean response time of 216.43ms per query and a user-friendliness score of 77.5 on the System Usability Scale (SUS). The second dataset is used from the literature for comparative analysis, and AEdBOT attained 76.2%, 83.7%, 77.7%, and 79.1% accuracy, precision, F1-Score, and recall, respectively. Experiment results reveal that AEdBOT significantly improves response accuracy and outperforms state-of-the-art educational chatbots. Artificial Intelligence (AI) Chatbot Educational Institutions Deep Learning (DL) Natural Language Processing (NLP) Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 Figure 9 Figure 10 Figure 11 Figure 12 Figure 13 Figure 14 Figure 15 Figure 16 Figure 17 Figure 18 Figure 19 Figure 20 Figure 21 Figure 22 Figure 23 1 Introduction Deep Learning-based chatbot systems have seen increased adaption in the educational domain in recent years owing to increased sophistication in the Artificial Intelligence (AI) domain [1]. However, most communication between students and educational institutions is still performed physically. Several studies have revealed that chatbots are a great way to make the educational process more interactive and engaging. Chatbots are useful in education because they provide students with real-time information and quickly answer questions [2]. In the educational domain, chatbots are increasingly being utilized to answer frequently asked questions, manage student forums, promote information sharing and student guidance without human interference [3], and provide administrative support to reduce the administrative staff’s burden during admissions times [4],[5]. Chatbots are intelligent agents communicating with users through natural language conversations. Most educational institutions use chatbots for different purposes depending on their requirements. Many studies have been conducted to make a chat- bot for educational purposes. Some chatbots were made for inquiries purposes related to students [4], [6], some were for assessment purposes [7], [8], and some were built for administrative purposes [9]. During the admission time at educational institutions, many local and international students want to know about the organization they are interested in. Many educational institutions need to employ more staff to reply to all questions and requests of local and international students and provide the necessary information for the hour, which is sometimes impossible. So, there is a need for an additional way in the form of a chatbot for the informational service providers through that the local and international students have quicker access to the information and can find the relevant information about the organization of the educational institution in an efficient way. From the design perspective, chatbot approaches and platforms are extremely important. Developers and researchers need a solid understanding of the chatbot approaches and platforms to build an effective smart chatbot. There are three primary approaches in Artificial Intelligence (AI) to building chatbots [10]: a rule-based approach, a machine learning approach, and a deep learning approach. Rule-based chatbots are programmed with an explicit set of rules and instructions. They tend to underperform and create wrong responses when they run over a sentence with no known pattern. Conversely, machine-learning chatbots seek to overcome the limitations of hard- coded rule-based chatbots. The machine learning approach can manually extract patterns from data instead of relying on its rule through feature engineering tech- niques. For this, more expertise and domain knowledge are required. The need for a deep learning approach stems from the limitations of the knowledge base and the rigidity of machine learning approaches, which can automatically learn the patterns from the data representation. The deep learning approach allows models to automatically learn the patterns from the data representation by transforming inputs. It skips this step-in feature engineering that allows building more general models to analyze data on a large scale. Deep learning chatbots are categorized into informational retrieval-based and generative-based neural networks chatbots. Informational Retrieval-based Neural Networks chatbots pick the appropriate message from a knowledge base with the pre- diction based on the highest confidence level to respond to user queries. Still, these chatbots cannot generate a response on runtime based on the conversation history and recommend something to the users. Unlike informational retrieval-based neural net- work chatbots, generative-based neural network chatbots can generate responses based on the knowledge of the current and previous history of the user messages. However, generative chatbots are considered more difficult to develop and train. Notable among these generative-based neural network chatbots models are the Large Language Mod- els (LLMs) like Generative Pre-trained Transformers (GPT), with GPT-4 [11], GPT-3 [12], GPT-2 [13], and XLNet [14], has significantly impacted the landscape of natural language processing, particularly in the context of chatbot applications. However, It needs many training data examples, close source access due to paid API restrictions and high computational requirements to achieve a good conversation. Moreover, the previously built chatbots [15],[16] on generative-based neural network approach generally generate less accurate and inconsistent responses with significant grammatical errors due to traditional Sequence-to-Sequence model and Transformer-XL model approaches like Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) models, and DialoGPT [17], which do not perform well when they must respond to longer sentences and tend to give vague answers [5]. There is a need to propose a new execution platform and new model techniques to make a more productive chatbot to support the development of the chatbot advances in the administrative area of the educational domain. The previously built chatbots [4],[18],[19] in the educational domains are domain-specific due to fixed rule- based design, generated grammatical errors during response due to obsoleted deep approaches resulting in accuracy issues, required costly subscription plans for web and social sites integration due to close source models and platforms, and no action as a recommender system due to their model implementation techniques. This paper pro- poses a novel chatbot system named AEdBOT to provide administrative support to educational institutions. The key contributions of this study are as follows: A novel Deep Learning (DL)-based Hybrid Model approach is developed that com- bines Information Retrieval and Generative Neural Networks to build an advanced conversational agent. It uses customized NLP pipeline featurizers to extract sparse features and a pre-trained BERT model to extract dense features simultaneously from Natural Language (NL)-based text. The model uses the DIET (Dual Intent Entity Transformer) Classifier for intent and entity extraction and a customized dual fallback classifier algorithm for self-learning ability to act as a recommendation system on out-of-scope inquiries. From a platform’s perspective, an open-source Rasa platform is utilized for deploy- ment on two distribution channels; a custom web-based UI and a Facebook Messenger with the integration of the MYSQL database through webhook after analyzing the entities from the Natural language (NL)-text entered by the user by making the automated SQL query generation for informational retrieval-based response generation. Evaluation of the AEdBOT methodology is done by developing the customized dataset relevant to the administrative support inquiries at NUST referred to the National University of Sciences and Technology and got an intent confusion matrix and multi-classification performance metrics like Precision, Accuracy, Recall, and F1-Score, which come out to be 94.7%, 96.0%, 96.0%, and 95.1%, respectively, at an average mean response time of 216.43ms per query and a user-friendliness score of 77.5 on the System Usability Scale (SUS). The experimental and theoretical analysis is performed through proposed methodology for comparative purpose on the state- of-the-art chatbot ’JAQ’ dataset [20] and outperformed as well with an accuracy of 76.2%. Hence, AEdBOT fulfilled its goals for accuracy issues during response generation, acted as a recommendation system, and could easily handle wider scopes and vague inputs. This paper adopts a research flow regarding chatbot development with stages in several parts. The structural overview of the article is shown in Fig. 1. These stages are organized as follows: Section 2 presents the Literature Review. Section 3 describes the Proposed Framework. The implementation of the AEdBOT is done in Section 4. The experimentation of AEdBOT is done in Section 5. Section 6 compares the AEdBOT proposed methodology with the state-of-the-art chatbot from the literature. Sections 7 and 8 discuss the main results of this work, future perspectives, and a conclusion, respectively. 2 Literature Review We conducted a detailed literature review to ascertain the current state-of-the-art approaches and platforms employed in chatbot development, as given in Sections 2.1 and 2.2, respectively. It allows us to identify a realistic research gap, as given in Section 2.3. 2.1 Chatbot in Administrative Area of Educational Domain From the literature, we identified that in Artificial Intelligence-based chatbots [10], three approaches are currently used to make chatbots: a rule-based approach, a machine learning approach, and a deep learning approach, as given in Section 2.1.1, 2.1.2, and 2.1.3, respectively. 2.1.1 Rule-based Chatbots Based on the literature review, we identified some related work to the rule-based approach that many authors utilized to develop chatbots in the administrative area of the educational domain. Nouman et al.[21] developed an innovative e-mentor-based learning model by the utilization of a rule-based approach that aims to customize courses for individuals, adapting to their unique learning styles, preferences, abilities, and prior knowledge. The outcomes reveal that e-mentoring enhances the learning process and increases learner satisfaction, establishing it as a preferred choice. Despite these positive findings, there exists a notable research gap in understanding the specific challenges and nuances associated with implementing e-mentoring in the context of personalized learning. In addition, the authors didn’t express the model quality using its BLEU (Bilingual Evaluation Understudy) score or multi-classification performance metrics. This gap highlights the need for ongoing research to optimize and enhance personalized learning in the evolving landscape of online education.B. R. Ranoliya et al. [4] adjusted a chatbot-preparing program to the FAQ in the School of Computing at the University of Leeds, England. It recovers data by pattern-matching design without utilizing any linguistic device. The chatbot answers inquiries utilizing FAQ data about the university. K. Bala et al. [22] carried out an intelligent chatbot for university-related FAQs to meet the scholastic requirements of the users. The application answers inquiries from students or guardians about university admissions and supports inquiries con- nected with Manipal College in India. The chatbot depended on Artificial Intelligence Markup Language (AIML) language, an XML-based markup language intended to make artificial intelligence-based applications using the pattern matching method. A. Vichare et al. [23] fostered a chatbot for instructing users about daily activities. The chatbot’s knowledge base is cared for with sports-related information coded utiliz- ing AIML. Their paper depicted the examination of three existing chatbots named ELIZA, ALICE, and Siri. The authors assessed these three frameworks, took specific imperative highlights, and carried them out in their proposed technique. Besides, the writers have developed an approach utilized in their chatbot. The proposed model utilized a pattern-matching approach that comprised depth-first search (DFS) as an implementation technique. 2.1.2 Machine Learning Chatbots We identified some related work to the machine learning approach. G. S. Sai Vikas et al. [18] introduced the informational Chatbot for College Management System Using Multinomial Naive Bayes. Chatbot aimed to help students in pandemic and socially distant situations. The system is built using the supervised learning Multinomial Naive Bayes algorithm, which is used to classify text messages into different categories. One downside of this model is unable to handle the unsupervised sentence. The chat- bot is designed to answer questions about course schedules, grades, attendance, and other important information related to college management. The proposed chatbot system was evaluated using a dataset of frequently asked questions related to college management and achieved an accuracy of 87.14% in classifying questions into their respective categories. The authors concluded that the system could provide an efficient and effective way for students and staff to access information related to college management. W. Raees et al. [24] forested a chatbot for admission into the NED University of Engineering and Technology. The purpose of this chatbot is to handle inquiries related the admission. The chatbot is trained on the 20000 plus previous inquiries by using the machine learning model of SVM for classification purposes. The GUI of the chatbot consists of Ajax, jQuery, and JavaScript, and MS SQL is used as a database for backend purposes. The authors concluded that the chatbot could provide an efficient and effective way for students to access information related to admissions and reduce the workload of the admissions office. W. Mahanan et al. [25] developed a chatbot system that uses a machine learning approach to answer questions about Digital Industry Integration (DII) curriculum administrative tasks at Chiang Mai University in Thailand. The authors employed a heuristic algorithm combining informational retrieval-based and generative-based approaches to determine the best response. The results show that our proposed chatbot can answer basic questions with 78.8% accuracy and advanced questions with 57.9% accuracy. However, the primary concern of this proposed framework is the sequential conversation based on the conversation history between the chatbot and humans. 2.1.3 Deep Learning Chatbots We also identified some related work to the deep learning approach. M. Rana et al. [19] carried out a methodology for semantics search to address students’ requests about a college and provided the data based on the college’s site by utilizing the BERT model technique. It is an adaptable chatbot framework using three-choice techniques with the fundamental platform utilizing DialogFlow. Given the solicitation constraints in the DialogFlow chatbot formation, this chatbot has a few impediments, and the bot accomplished 56% accuracy only. The authors used a generative-based approach for the development of the chatbot. J. Thakkar et al. [26] forested is an Artificial Intelligence chatbot that addresses inquiries on academic world data. The creators proposed Erasmus by utilizing cloud administrations, Mlab (MongoDB cloud), (DialogFlow), and IBM Bluemix (webhook API). This chatbot took over a seriously extended redundancy in reacting to the clients since the thing applied to too many cloud administrations. The authors used an informational retrieval-based approach for the development of the chatbot. Y. Windiatmoko et al. [15] made a chatbot incorporated with MySQL data set and an API for University requests. This straightforward chatbot is just equipped for replying to customers with few purposes. Also, Indonesian is very different from different dialects like English, and the creators should have specified their tokenizer and the pipeline. They used Long Short-Term Memory Networks (LSTMs), a type of Simple RNN that is used for how far back one wants to go in the information of a conversation. The authors used a generative-based approach for the development of the chatbot. M. T. Nguyen et al. [16] introduced an intelligent system (a chatbot) that could support the admission process by automatically answering questions. The chatbot is developed using the rasa platform with the help of the BERT model, and for entity extraction purposes, the DIET classifier is used. The approach for designing the chatbot is a generative base and deployed the chatbot on Facebook Messenger. S. Meshram et al. [27] forested a web-based chatbot application that analyses and under- stands users’ queries and provides an instant and accurate response. The platform used to develop the chatbot is Rasa, the authors used an informational retrieval-based approach for entity extraction purposes. H. T. Hien et al. [9] forested a chatbot named FIT-EBot, which aimed to provide administrative and learning support at the Faculty of Information Technology of the Ho Chi Minh City University of Science, Vietnam (FIT-HCMUS). The chatbot was developed with the close source platform DialogFlow and deployed on Facebook Messenger. The average F1-Score of this chatbot is 82.33%. K. Lee et al. [28] forested a chatbot for administrative-related tasks in their college. The authors developed the chatbot with the closed-source DialogFlow Platform, and the informational retrieval base model technique was used to develop it. To check the effect of the chatbot on the administrative staff, they experimented by hiring two office workers and measuring their workload using NASA-TLX. X. Gonsalves and S. Deshmukh [29] proposed an Interactive Chatbot for Educational Assistance using Rasa Framework that allows students to submit questions about their academics, relevant courses, and programs. To handle the inquiries effectively, the authors used deep learning policies in the domain file. These policies checked the intent response and performed actions against the inquiries entered by the user on the web-based user interface (UI). However, the chatbot is not evaluated using performance evaluation metrics, and the bot cannot handle contextually relevant and interactive queries, which is unaware of the context in which a conversation with a student is taking place. Channabasamma et al. [30] developed a chatbot based on a deep-learning approach integrated with the website via the Flask framework. The chatbot model is linked to HTML, CSS, and Java for the front end, Python for the back end, and JSON for the database format, which provides answers to FAQs (Frequently Asked Questions), general queries, and all relevant information about the GRIET (Gokaraju Rangaraju Institute of Engineering and Technology) organization. The GRIET website provides available information such as administration, admission, departments, placements, social media contact information, and a location navigator. However, the proposed chatbot needed to have a fallback policy to respond to the user and not act as a recommendation system. 2.2 Chatbot’s Platforms A platform is a software infrastructure that aims to facilitate and speed up a devel- oper’s task. It gives them a ready-to-use architecture and components to solve common obstacles encountered in the applications it relates to. There are lots of different plat- forms intended to create different types of software. Conceivably, a certain number exist specifically to build chatbots, each having its characteristics, tackling problems differently, and offering different useful features. From the literature review, we identified that the most used platforms in the edu- cational domain are DialogFlow and Rasa. After reviewing different ones, the most conspicuous characteristic to categorize them is whether they are closed or open source. Usually, closed-source platforms are meant to be used online and run on a server owned by the company to which the platforms belong. On the contrary, open-source plat- forms can be used on a local machine and are typically developed to be lightweight enough to execute on slow hardware. Of course, it is also possible to put the pro- duced chatbot on a privately owned server and communicate with the user through an internet connection. The closed-source platforms cannot be used without access to an internet connection, while open-source ones usually can. This approach is true when designing and interacting with the produced chatbot. Open-source platforms are thus a better solution if the chatbot is meant to be used on a local machine or when data privacy is critical [31],[32]. Determining the chatbot’s platform is profoundly important to improve the accu- racy of the chatbot. Multiple different closed-source solutions are available to the public. Those solutions usually use a ”pay per request” system. In opposition to this, open-source solutions are available for free services providing. For this, we analyzed five chatbot platforms [33] and evaluated them on seven dimensions based on the usability criteria described in Table 1. 2.3 Research Gap From studies, we analyzed that there is a need for some improvement regarding the technical advancement of the chatbot models to generate better results. To the extent that education and research go, chatbots in this domain are built using a rule-based approach that consists of a defined set of fixed rules. They may create wrong responses when they run over a sentence with no known pattern. Moreover, pattern-matching standards could be stronger, exceptionally domain-specific, and must move better from one domain to the next [34]. The machine learning approach is emerged to overcome the rule-based approach issues. By machine learning algorithms, there is no longer the need to define rules manually and code new pattern matching, which permits chatbots to be more adaptable and, as of now, not subject to domain-specific knowledge. Traditional machine learning methods include feature engineering as a data preprocessing step. For this purpose, the developer needs to analyze the output parameter manually, and too much expertise and domain knowledge are required. A deep learning approach is introduced to overcome these issues. It is an approach known as feature engineering that could automatically identify the features from data to classify them greatly. The first approach in deep learning is an informational retrieval- based neural network approach that offers more prominent flexibility and response to the users in a well-defined manner based on pre-defined responses. The disadvantage of this approach is that the informational retrieval-based neural networks approach cannot recommend anything to the user because it cannot generate a response as per the conservation flow on the run time [35]. The second approach in deep learning is the generative-based neural networks approach that can generate responses based on the knowledge of the current and pre- vious history of the user messages and has some memory-saving abilities. The main feature of this approach is that the generative-based model is not domain-specific and can be trained on different data sets and applied to different domains. This approach uses recurrent neural networks (RNNs) models like LSTM in their design approach. The disadvantage of the generative approach is not answering the longer sentence appropriately but generating vague output. Their replies often need to be more con- sistent and have a lot of grammatical errors during response generation [5]. In deep learning, Transformers are the greatest innovation based on the attention mechanism. It is also a generative-based neural network technique with some advancement. Trans- formers replaced the traditional recurrent neural networks (RNNs) models like long short-term memory (LSTM) with the pre-trained model like BERT (Bidirectional Encoder Representation Transformer) and GPT (Generative Grained Pre-Trained Transformer) that permits the model to train on the larger data sets. Transformers also replace the context’s fixed-length vectorization support in the generative base model by weighing the input data in an equivalence sum of hidden vectors [36]. In recent years, the development of large language models (LLMs) has significantly impacted the landscape of natural language processing, particularly in the context of chatbot applications. Notable among these models is the Generative Pre-trained Transformers (GPT), with GPT-4 [11] standing out as the epitome of scale, boast- ing an impressive 1 trillion parameters. This cutting-edge model boasts multimodal capabilities, demonstrating its proficiency in processing both image and text inputs to generate insightful text outputs. However, accessibility to GPT-4 is restricted due to paid API. One of its predecessors is GPT-3 [12], boasting an impressive 175 billion parameters but limited access due to API restrictions and high computational requirements. Necessitating consideration of alternative models like its predecessor GPT-2 [13], which, with 1.5 billion parameters, strikes a balance between performance and availability. Moving beyond the GPT series, Bidirectional Encoder Representations from Transformers (BERT) [37] introduces bidirectional context understanding, prov- ing effective in various NLP tasks, while the Text-to-Text Transfer Transformer (T5) [38] takes a versatile approach with a text-to-text framework, excelling in tasks ranging from language generation to understanding. Additionally, XLNet [14] combines autore- gressive and autoencoding approaches, capturing bidirectional context with intricate architecture. For conversational contexts, DialoGPT [17] has been fine-tuned, offering context-aware responses. Each model comes with its strengths, including considerations for parameters, accessibility, resource requirements, versatility, and whether the model is open source. Understanding these aspects is crucial for choosing the most suitable model for specific chatbot applications. The choice of the ”best” model depends on specific requirements, available resources, and the intended application. For open-source availability and versatility, BERT, T5, XLNet, and DialoGPT may be more accessible options, but T5 and XLNet have high computational demands. On the other hand, GPT-4, GPT- 3, and GPT-2 could provide unparalleled performance if access is available free. For model selection, we analyzed seven large language models (LLMs) and evaluated them on five dimensions based on the usability criteria described in Table 2. In this study, we proposed a novel Deep Learning (DL)-based Hybrid model approach that used an advanced form of deep learning attention-based mechanism of Transformers to overcome the issues in the traditional rule-based, machine learning, and deep learning approaches that had been obsoleted by developing a customized NLP pipeline for creating a smart chatbot called AEdBOT by utilizing the open-source Rasa platform for educational institutions. The developed customized NLP pipeline used the featurizers and pre-trained open source large language BERT model to extract sparse and dense features from the Natural Language (NL)-based text simultaneously along with the aid of DIET Classifier for dual intent and entity extraction and process sentences as a whole by using the attention base mechanism of the transformer rather than requiring 8 time-steps of traditional deep learning DialoGPT [17] models to pro- cess the sentences word by word. The developed customized dual fallback classifier is integrated into the NLP pipeline, providing the self-learning ability to AEdBOT to act as a recommendation system on vague inquiries to efficiently enhance the prediction of response generation without grammatical errors to overcome the accuracy issue. The developed chatbot has third-party integration like a web-based UI and Facebook messenger connectivity to make it more social. 3 Proposed Framework This section describes the development of the proposed framework based on the pro- posed methodology adopted. The architecture of AEdBOT is discussed in Section 3.1. It allows us to develop the customized natural language processing (NLP) pipeline for processing Natural Language (NL)-based text as given in Section 3.2. The working principles of the NLP pipeline for response generation after processing of Natural Language (NL)-based text is described in Section 3.3. 3.1 AEdBOT’s Architecture The architecture of the AEdBOT is shown in Fig. 2. The chatbot is built utilizing the open-source Rasa platform, which comprises two primary parts: RASA Core and RASA NLU. RASA Core is the dialogue manager; the domain file in RASA Core characterizes the settings for the chatbot, for example, what it ought to comprehend, and what it could use to answer. RASA NLU took in NLU training data and conversation examples to prepare the chatbot. The predicated-based word embedding Bidirectional Encoder Representation Transformer (BERT) model is used for the extraction of dense features from the Natural Language (NL)-based text after parsing the text from WhitespaceSpaceTokenizer and ProfanityAnalyzer for the parting of text into tokens and performing sentimental analysis on the text for the removal of negative words from the text, respectively. RegexFeaturizer, LexicalSyntacticFeaturizer, CRFEntityExtractor , and CountVec- torFeaturize r extract sparse features from the text to provide more information to the model. It could then deal with intent classification and entity extraction through the DIET classifier from the Natural Language (NL)-based text. The conversation flow design, different functionalities, for example, storing and bringing data from a data frame, and AEdBOT run-time recommendations based on confidence level score are carried out through Transformer Embedding Dialogue (TED) policy , Memoization Policy , and custom Dual Fallback Classifier respectively. In addition, the AEdBOT interacts with the MySQL database system to generate an informational retrieval- based response by automatically creating a SQL query from the Natural Language (NL)-based text. The developed chatbot is deployed on two distribution channels with the help of Flask on the web page and Facebook messenger through the ngrok python library to allow communication between the rasa server and the Facebook messenger app to make it more social. The proposed architecture provides an effective solution for building an advanced conversational agent to handle various user inquiries. The combi- nation of two different deep learning-based approaches, i.e., information retrieval and generative neural networks, along with the customized NLP pipeline and self-learning ability, makes the approach more robust and accurate 3.2 Natural Language Processing (NLP) Pipeline There are several components in developing a customized AEdBOT Natural Language Processing (NLP) Pipeline. RASA NLU is responsible for the sequential processing of Natural Language (NL)-based text in the NLP pipeline as described in Section 3.2.1 to perform pre-processing and processing on the Natural Language (NL)-based text entered by the user as discussed in Section 3.2.2 and 3.2.3, respectively. 3.2.1 RASA Natural Language Understanding (NLU) RASA NLU performs the NLU task by sequentially applying the components specified in a pipeline configuration file on the sample examples in a labeled dataset. The pipeline configuration file contains a specification that defines the sequential processing steps required to classify the initially unstructured user utterances and extract the relevant entities. It is also specified in yalm format (config.yml). It assigns one or more components to each of the three stages depicted in Fig. 3. These are commonly used in deep learning approaches for Natural Language (NL)-based text analysis. The first step is tokenizing the utterance, which involves breaking down the textual data into words, symbols, or other meaningful elements known as tokens. The algorithm divides the input sentence into words, but there are more complex alternatives that help with specialized tasks. Each token is converted into numeric features in the second stage. Several featurizers can be used at the same time. The features produced by all components are concatenated into a single vector in this case. The number of features can be sparse or dense. Dense features are typically floating-point values obtained from pre-trained embedding such as BERT [36], GloVe [39], ConveRT [40], or other Hugging Face models. On the other hand, sparse features include vectors with many zero-values, such as Bag of Words (BoW) and n-gram representations or categorical data counts. Sparse features for this CLS [36] token are calculated as the sum of each token’s sparse features. The computation of dense features is determined by the capabilities provided by the concrete featurizer. Some models can compute the sequence’s contextualized aggregate representation. When this is impossible, they are computed as the sum or mean of the token representations. The original sentence is converted into numeric features, and then passed to the intent classification model. 3.2.2 Pre-Processing on NL-based text Several components are involved in the customized NLP pipeline to perform the pre- processing on the NL-based text, as shown in Fig. 4, which are: • Whitespace Tokenizer: A whitespace tokenizer is a simple text tokenization method that tokenizes a piece of text by breaking it up at every instance of a whitespace character. For example, the string ”Information about getting a nust accommodation” would be tokenized as [‘Information’, ‘about’, ‘getting’, ‘a’, ‘nust’, ‘accommodation’] using a whitespace tokenizer. • Profanity Analyzer: A customized Profanity Analyzer python algorithm is devel- oped for AEdBOT to perform sentimental analysis to filter out the negative words from the Natural Language (NL)-based text entered by the user. It will help to refine the input after filtering out the notorious or criminal words from the Natural Language (NL)-based text. • BERT Model: BERT (Bidirectional Encoder Representations from Transformers) [36] is a state-of-the-art natural language processing (NLP) model developed by Google as shown in Fig. 5. The internal functionality of BERT is based on the transformer architecture, which was introduced by [41] in their paper ”Attention is All You Need” (2017). The transformer architecture consists of self-attention and feed-forward layers stacked on top of each other to form a deep neural network. The BERT model uses 12 layers of transformers block with a hidden size of 768 and several self-attention heads as 12 and has around 110M trainable parameters. The feed-forward layers process the weighted input from the self-attention layers and transform it into dense output features. Since BERT’s goal is to generate a language representation model, it only needs the encoder part. The input to the encoder for BERT is a sequence of tokens, which are first converted into vectors and then processed in the neural network. But before processing can start, BERT needs the input to be massaged and decorated with some extra metadata: Token embeddings: A [CLS] token is added to the input word tokens at the beginning of the first sentence, and a [SEP] token is inserted at the end of each sentence. Segment embeddings: A marker indicating Sentence A or Sentence B is added to each token. This allows the encoder to distinguish between sentences. Positional embeddings: A positional embedding is added to each token to indicate its position in the sentence. One unique aspect of BERT is that it is bidirectional, meaning it considers the context of a word both to the left and the right of the word. In contrast, most previous NLP models could only consider the context to the left of a word. It is achieved using a “masked language model” training task, in which a portion of the words in the input is randomly masked, and the model is trained to predict the masked words based on the context provided by the unmasked words. For example, consider the following sentence in Fig. 5: “Information about getting a nust accommodation.” If the word “accommodation” is masked, the model must use the context provided by the words “Information about getting a nust “to predict the correct word. It allows BERT to learn rich representations of language that can capture the meaning and context of words in a sentence rather than just their meanings. • Featurizers: In the Rasa natural language processing platform, featurizers take raw input data, such as a message or user input, and convert it into a numerical representation that can be used as input to a deep learning model. Each of the three featurizers in this configuration Natural Language pre-processing pipeline generates a set of sparse features for each token produced by the WhiteSpaceTokenizer . The two components of CountVectorFeaturizer produce sparse features associated with the LexicalSyntacticFeaturizer, RegexFeaturizer , and CRFEntityExtractor for each token, including the CLS, based on the appearance of words and n-grams in the sentence. These features are computed for all labeled samples during training and passed to the DIET classifier to build the model. They are computed from the user utterance at inference time and passed to the classifier to predict the intent and extract the entities. • DIET Classifier: DIET Classifier [42] is a simple transformer architecture that can be fully parameterized using the Rasa toolkit. Fig. 6 depicts a schematic represen- tation of this architecture with entity recognition and masking disabled. The user’s utterance is split into tokens in the first stage, using the tokenizer algorithm. At the end of the sentence, the special CLS token is added. It is possible to generate sparse and dense features through the DIET Classifier simultaneously. Sparse features are then routed through a feed-forward network with shared output concatenated with the dense features from the pre-trained word embedding layers before passing through another feed-forward network. The last feed-forward network’s outputs are fed into the transformer. The transformer output for the CLS token and the intent label are separately embedded into a single semantic vector space using embedding layers. The dot-product of Total Loss is then applied to maximize similarity to the intent target label while minimizing similarity to all other intent labels. The dot- product similarity is used at inference time to rank all possible intent labels, and the scores for all intents are combined to yield a confidence value. The Intent Loss module calculates the loss of the transformer’s output and the sentence’s intent clas- sification (given by the similarity module). Then goes as an input to the Total Loss module, which will provide a measure of learning. The Conditional Random Field (CRF) module is used during training to calculate an Entity Loss for the entities recognized by the NLU pipeline and the transformer’s output; this loss measures the ability of the transformer to recognize entities related to the input sentence. The pipeline specification in Rasa allows the designer to configure multiple param- eters. The number of transformer layers (2 by default), the output dimension of the embedding layers (20 by default), the fraction of weights set to non-zero values for all feed-forward layers in the model (0.2 by default), the size of the vector com- ing out of the transformer (256), and the number and size of hidden layer sizes are chosen for the training of AEdBOT’s dataset in feed-forward networks. 3.2.3 Processing on NL-based text When executing a dialogue management solution, the fundamental task is to conclude what occurs next concerning the conversation. The class rasa.core.policies conclude what move to be made during every conversation flow with the bot [43]. There are two sorts of approaches: Deep Learning and Rule-based policies. These policies will help the bot conclude which moves should be made toward a conversation. Depending upon AEdBOT’s requirements, we picked deep learning and customized dual fallback classifier policies to make AEdBOT work more effectively. The deep learning policies included the Transformer Embedding Dialogue (TED) Policy and Memoization Policy. These policies are activated to predict the response on the confidence level score returned by the DIET Classifier during pre-processing of the Natural Language (NL)-based text entered by the user. If the confidence level score returned by the DIET Classifier during intent and entity extraction from the Natural Language (NL)-based text is equal to 1, then Memoization Policy is activated. Memoization is a technique for optimizing the performance of a chatbot by storing the results of expensive function calls and returning the stored result when the same inputs occur again. Fig. 7 shows the policies utilized in the NLP pipeline. In Rasa, memoization can be used to optimize the performance of the Rasa.core.featurizers.Max.History. Tracker Featurizer class, which converts training data and conversations into machine-readable features. It will cause the featurizer to store the results of its featurization process in a cache and return the stored result whenever the same input is encountered again. It can significantly improve the per- formance of the featurizer, especially when dealing with large amounts of data. If the confidence level score is less than 1 but greater than 0.3, then the Transformer Embed- ding Dialogue (TED) Policy is activated. The TED policy in Rasa is a variant of the Transformer model specifically designed for dialogue management tasks. It takes in a sequence of messages in a conversation and produces a probability distribution over the next action to take in the conversation. The TED policy is trained on data from past conversations to learn patterns and improve its performance over time. AEdBOT can also handle out-of-scope queries efficiently entered by the user. We developed a customized Dual Fallback Classifier algorithm to handle the fallback sitduations in a conversation; if the confidence level score returned by the DIET Classifier is less than or equal to 0.3. A fallback situation is when the chatbot cannot under- stand or respond to a user’s message and needs to provide a default response or take other action to keep the conversation going. The customized Dual Fallback Classifier algorithm consists of two separate clas- sifiers: a primary fallback classifier and a secondary fallback classifier. The primary fallback classifier identifies fallback situations and triggers a fallback action. The sec- ondary fallback classifier is used to confirm the fallback situation, determine the appropriate fallback action based on the form and mapping policies, and provide a self- learning ability to AEdBOT as a recommendation system based on the conversation history between the AEdBOT and the user. The primary fallback classifier compares the confidence of the intent prediction made by the chatbot with a threshold value. If the confidence is below the threshold of 0.3, the primary fallback classifier will trigger a fallback action. The secondary fallback classifier then determines the appropriate recommendation based on the conversation’s context. 3.3 Natural Language Generation (NLG) NLG systems use natural language processing (NLP) techniques to analyze structured data and generate natural language text that is easy for humans to understand. The proposed framework could retrieve and generate responses according to the Natural Language (NL)-based text entered by the users. The NLG process of AEdBOT is demonstrated in Fig. 8. The Deep Learning (DL)-based Hybrid model approach is used to build an effective model for classifying Natural Language (NL)-based text that the user will enter. It is a combination of informational retrieval and generative neu- ral networks. An informational retrieval-based neural networks approach is developed by providing a training dataset of text-based information or by interacting with the MYSQL Db after extracting an entity from the Natural Language (NL)-based text entered by the user. Then the approach will retrieve the information using a DIET Classifier in correspondence to Natural Language (NL) - based text entered by the user. On the other hand, the generative-based neural networks approach is developed on the pre-trained data set of the BERT model. The pre-trained model is then used for training the data set through a transfer learning approach as a word embedding model. The BERT model could easily predict response and activate the dual fall- back classifier as a recommendation system by efficiently extracting the dense features based on the conversation history between the user and the chatbot. For example, the user entered the Natural Language (NL)-based text ”Eligibility criteria of NUST” on the AEdBOT UI. The developed Deep Learning (DL)-based Hybrid model parses the query in the customized Natural Language Processing (NLP) pipeline, which first performs the pre-processing on the text to extract intent and entity. The Natural Language (NL)-based text is passed from WhitespaceTokenizer for parting text into tokens and then passed the text into the customized ProfanityAnalyzer algorithm to perform the sentimental analysis like removing negative words from the sentence. After performing the data cleaning on the Natural Language (NL)-based text, the sec- ond step is feature extraction to convert textual data into categorical form through a one-hot encoding technique by representing Natural Language (NL)-based text into vector space. For this, a generative-based BERT model is utilized to extract dense features. For the extraction of sparse features, three featurizers are used. The first one is RegexFeaturizer which works on the pre-specified regular expression and is respon- sible for extracting numbers, dates, and times in specified formats. The second one is the LexicalSyntacticFeaturizer which acts as character n-grams, representing the doc- ument as a sequence of characters. The third one is CountVectorFeaturizer which is responsible for the bag-of-words representation of Natural Language (NL)-based text. These concatenate features (dense and sparse) are passed the DIET Classifier, which is responsible for the extraction of intent and entity from the Natural Language (NL)- based text by predicting the [intent: admission, confidence: 0.7877], entity: [”nust”: Organization, confidence: 0.9123].After the extraction of intent and entity confidence level score returned by the DIET classifier, we made the customized processing pipeline responsible for the retrieval of response by interacting with the MYSQL Db or gener- ation of response based on the deep learning policies specified in the pipeline. If the Confidence level score is 1, then the Memoization policy is activated, which generates the fixed response on which the Hybrid model is trained. Suppose the confidence level score is less than or equal to 0.3. In that case, the customized dual fallback algorithm is activated to provide the self-learning ability to AEdBOT to act as a recommendation system. If the confidence level score is less than 1 but greater than 0.3, then the Trans- formerEmbeddingDialogue (TED) Policy is activated, a unidirectional transformer. It will check the previous history of the bot responses based on the slots, which is the bot’s memory, and passes the text through Embedding, Similarity layers to perform the next system action and generate the response. 3.3.1 Informational Retrieval-based Response Generation The informational retrieval-based models usually used the deep learning approach in most cases, but they also used the pattern matching approach, a rule-based model technique. Informational retrieval-based neural networks usually contain a set of pre- defined pairs of question answers in their knowledge base and pick the one answer from the knowledge base based on the user’s Natural Language (NL)-based text. The knowledge base is a corpus of the pre-defined set of question-answer pairs. The chat index is provided to the question-answer pair in the knowledge base. When the user enters the Natural Language (NL)-based text from the web interface or any other ter- minal, the model treats the input as a user query, matches the pattern of this question in its knowledge base concerning the chat index, and retrieves the information accord- ingly. For retrieval-based response generation purposes in AEdBOT, we categorized it into two blocks for the intent and entity extraction from the Natural Language (NL)- based text entered by the user. The first is a non-informational retrieval-based intent response. The second is an informational retrieval-based intent response, as shown in Fig. 9. For each intent category, corresponding blocks are categorized, and solutions are identified to generate the responses; for Non-Informational Retrieval based intent, such as Welcome and Good Bye, fixed and unchangeable responses are generated. For example, when an AEdBOT receives a user’s Natural Language (NL)-based text that is identified to be a Welcome intent, the AEdBOT automatically generates the answer that is ”Welcome. I’m AEdBOT. I can help you with finding a Hostel or accom- modation and providing latest news for them”. When an AEdBOT receives a user’s Natural Language (NL)-based text identified as a Good Bye intent, the AEdBOT automatically generates the ”GoodBye” answer to the user.The AEdBOT interacts with MYSQL for other intents to generate the Informational-Retrieval based intent response. For example, for asking about the PG programs at NUST, the proposed system executes a SQL statement, ”SELECT program from NUST programs where program=’PG’”, to return the PG programs to the user. The AEdBOT applies a retrieval-based neural network model approach that can perform queries from the MYSQL database system APIs. Suppose the AEdBOT cannot identify the intent and entity extraction from the user’s Natural Language (NL)-based text. In that case, the system will respond to the user with an intent fallback message: ”Sorry, I didn’t under- stand it, please rephrase again”. For example, a user asks for the programs at NUST but doesn’t specify the information about which program it is. At the same time, the system is trained on the program’s asking at NUST with the entity @program. In this scenario, the AEdBOT will remind the user:” Please, tell me about which program you are asking?” after calling an intent fallback. After the user provides the program information, the AEdBOT generates the answer. 3.3.2 Generative-based Response Generation As the name suggests, the Generative-based Neural Networks models generate responses to the user input. These models could generate a new sentence based on the user’s Natural Language (NL)-based text. However, the model should be trained on such types of Natural Language (NL)-based text. Generative-based models are usu- ally trained on a large dataset and learn the pattern, syntax, and vocabulary from the data that has been fed. AEdBOT has a feature to give a conditional block a goal-based approach for registering accommodation in NUST. Nonetheless, we quickly realized there were two main problems if we used this: First, the user cannot change their mind once the chatbot tries to fulfill a goal. For example, If the user wants to register the hostel and asks the system: ”I want to register a hostel”, the system will respond with an intent message: ”ok, Please provide your First Name”. The user enters the name, for example: ”Shahroze Ali”. After that, the system requests information about the date of birth: ” Please provide your Date-of-Birth (DOB)”, and waits for the entity that the user would enter in return for it, but the user asks the counter-question on it: ”where do you live”, and the Bot generates the run-time response in return of it: ”The Virtual World is my Playground, I always live here.” and at the same time Bot again ask the previous inquiry to the user, and the user doesn’t want to provide the information about the DOB to the Bot. We want to avoid that behaviour since it annoys the user when the NLU component detects a question associated with a complex interaction. To overcome this issue, we employed Rasa’s Slot filling feature to check whether the required entity is already contained in the corresponding context. Suppose the context supposed to contain this entity value is not defined. In that case, the answer related to the intent will be recommended, asking the user for the required entity value. A context wait Ei will be activated to indicate which entity is expected. On the other hand, if the entity value is known, the webhook will be called and generate a relevant answer. We made a ’custom fallback action’ script by using Python language which is our customized Dual Fallback Classifier algorithm to handle the fallback situations in a conversation. Its code contains a data structure that stores each complex question according to the conditions (i.e., which entity is required) and the associated answers. Knowing the current intent and known entity values, the webhook can generate an appropriate answer. When the NLU component detects that the user is providing information and there is a pending question (the context pend Qi is active), the webhook is called to generate the correct answer in the same way as before. If no question is pending, it will activate the related context ctx Ei so that it remembers it for later turns and returns an answer acknowledging an entity was provided and asking the user what they want. Lastly, if the NLU component didn’t understand the latest user message and an entity was expected, it will tell the user it didn’t understand and which entity it was expecting. If it were not expecting any entity, the returned message would tell the user the chatbot didn’t understand and suggest the user rephrase the text again. The implemented logic using the contexts is shown in Fig. 10. Secondly, suppose the user wants information about the type of test conducted at the educational institution, for example, NUST, and asks the system: ”What type of test is required for admission at NUST”, if the chatbot does not understand the message the user is providing, it will keep asking for it. It can stall the chatbot conver- sation, as the user might need to learn why they get asked the same question again and how to escape the loop. We avoided those drawbacks once again by using a webhook. We used to manage complex interactions as much more intricate as simple slot-filling (in the goal-based approach to dialog management). When an entity is required to answer a question, we activate a context so that the chatbot remembers it expects an entity. Then, the user can ask something else and get an answer before providing the previously asked entity and having an answer for the first question. Since the context has a certain lifespan (measured in the number of conversation turns), the chatbot can ”forget” a question is pending if the user asks for other information for a long time (and hence refuses to give the value of the entity), as it would happen in a human- to-human conversation. Having dual fallback intents depending on the active contexts prevents the problem: the chatbot can recommend the user, as shown in Fig. 11, based on the previous history of the conversation if it does not understand the entity value it provided. 4 Implementation An important feature of the Rasa platform is easily integrating the developed chatbot into third-party services, such as web platforms and social media services like Facebook Messenger, Slack, and Twitter. [44]. These services represent the UI component of the chatbot. We only integrated it into two different interfaces: a Custom User Interface (UI) and Facebook Messenger, as described in Section 4.1 and Section 4.2, respectively. The project code [45] is publically available for the online communities to implement the state-of-the-art AEdBOT proposed methodology in their developing chatbots to get better accuracy. 4.1 Custom Web-based User Interface (UI) The custom webpage using HTML/CSS and JavaScript is built for AEdBOT deploy- ment, where AEdBOT would like to run. The credentials.yml file is configured to allow communication between the Rasa server and the custom web page. In this file, the socketio channel is activated for the API calling of the bot on the UI. Then this webhook API is wrapped into Flask to route the HTML home page.This interface is meant to be executed on a web server. Internally, the Custom User Interface (UI) works as follows: • Web Front-End: The HTML, CSS, and JavaScript-based front-end of the AEdBOT permit the user to send a message request to Rasa. The message made by the user is processed and delivered as an API call generally anticipated by the Python frame- work that runs on the WSGI server. A request is made to the Python framework utilizing a POST retrieval API request which assists in transmitting the information to the WSGI server. • Web Back-End: The chatbot’s back-end is created using Flask. Every time the user sends a message to the UI, the WSGI web server makes an API call, including the user message to the Rasa server through Socket.io, which hosts AEdBOT. This server handles the requests and returns the chatbot’s answer, displayed to the user. • Exchange Flow APIs: Whenever a user types in a Natural Language (NL)-based text, the front-end calls to the Python framework, and the WSGI server do the processing. In Rasa, the TransformerEmbedding- Dialogue (TED) policy is a variant of the Transformer model designed explicitly for dialogue management tasks. It takes in a sequence of messages in a conversation and produces a probability distribution over the following action to take in the conversation. The TED policy is trained on data from past conversations to learn patterns and improve its performance over time. Rasa server gets the request from Socket.io and generates a response utilizing a Deep Learning (DL)-based Hybrid Model (informational retrieval and generative) approach. The deep learning policies included the Transformer Embedding Dialogue (TED) Policy is activated at that moment to predict the response on the confidence level score returned by the DIET Classifier during pre-processing of the Natural Language (NL)-based text entered by the user and transmits the output to the user. An MYSQL database is also connected to Rasa via webhook for retrieval-based response generation. Building it as a web page allows us to easily integrate it into another web page, such as the official website of the NUST. The custom web user interface (UI) of AEdBOT is shown in Fig. 12. 4.2 Facebook User Interface (UI) The user might be familiar with Facebook Messenger, the messaging application included in the Facebook social network. It allows Facebook users to send each other messages in different formats, such as text, images, audio, recordings, or files. The interface of AEdBOT on Facebook Messenger is implemented as follows: • Facebook’s Page Creation: A Facebook Page is created because Facebook currently allows only a Facebook page to be connected to a Facebook application. • Facebook’s App Development: The application uses the API to communicate with a Facebook platform through a webhook call-back. The credentials.yml file is config- ured to allow communication between the Rasa server and the Facebook application by passing the generated secret key and page-access-token generated during the Facebook application’s development. • Facebook’s App Connection with RASA Server through Ngrok: For running a Rasa server on localhost, most external channels won’t find the server URL since localhost is not open to the internet. To make a port on the local machine that will be publicly available on the internet, the ngrok is used. An event will be sent to the appropriate webhook whenever a user sends a message to the Facebook application’s page. AEdBOT can acquire user-submitted data for subsequent processing as a result. Integrating a chatbot into Facebook Messenger allows users of Facebook to interact with the chatbot as if they were normal users of this social media platform. The user interface (UI) of Facebook Messenger is shown in Fig. 13. 5 Experimentation This section describes the testing of the proposed framework on the case study of an educational institution named NUST, referring to the National University of Sci- ence and Technology. For experimentation purposes, we made a customized dataset of inquiries relevant to administrative tasks in NUST, as described in Section 5.1. The simulation of the dataset is done in Section 5.2. The performance of AEdBOT is val- idated using widely accepted metrics such as 1) Model Evaluation, 2) Model Testing, 3) Response Time, and 4) User Experience, as described in Sections 5.3, 5.4, 5.5, and 5.6, respectively. 5.1 Dataset Preparation Once we have decided which topics the chatbot should support, we can gather data that covers them. In our case, there are a certain number of ways we could find this data: our knowledge as an NUST student, Frequently Asked Questions (FAQs) on the official website of the university, other websites related to it, and employees of NUST. The main source of information used is the FAQs found on NUST’s website; obviously, all this data is not formatted well-defined, nor is the information contained there directly exploitable. We thus had to read through all this documentation, understand the information there and make it exploitable, which cannot be done automatically. To choose which questions AEdBOT should support, we had to imagine what being an international and local student would be like. We took inspiration primarily from the FAQs and divided all the information we could find into 2 (correlated) topics: • Accommodation • Admission We listed all the questions about the topics that seemed relevant to us. To create the answers to those questions, we organized the gathered data to let us see the big picture and easily retrieve some information. To do that, we made simple schema diagrams of the relevant topics. We also tried to automate this data gathering but scraping the website of NUST is quite difficult since its HTML code could be clearer. Moreover, we would have required an extremely powerful algorithm to find useful information automatically within the data we could gather. Manually gathering data was the most effective but extremely time-consuming. To speed up this process, chatbot designers commonly use a dataset generator. Such tools allow them only to provide templates of examples that the tools used to generate the datasets. As Natural Language Understanding (NLU) algorithms usually expect a dataset in a formatted way, such tools can also handle the formatting of the generated data. Many tools exist for data set generation purposes. Each has capabilities and is intended to be used with a certain NLU algorithm. We identified five data set generator platforms, i.e., Chatito [46], Chatl [47], Expando [48], Tracery [49], and Chatette [50]. All these tools are open source. We also identified that the first four use similar syntax for writing the templates and allow certain interoperability between them. To make the training dataset for the chatbot, we used Chatette for two reasons: it is intended to create large datasets as needed and uses a simple syntax to create the scripts. The generated dataset from the Chatette data set generator consisted of 108 intents with almost 4263 examples. The dataset is publicly available at [51] for further exploration. The flow of dataset preparation is shown in Fig. 14. 5.2 Dataset Simulation Rasa’s documentation gives a short tutorial [52] on building a basic chatbot. The main files are presented here: • An nlu.md file, which incorporates all NLU-preparing information. • A domain.yml file characterizes the chatbot’s domain or knowledge base. • A Stories.md file to create a conversation flow. 5.2.1 RASA NLU Rasa NLU (Natural Language Understanding) [52] is a tool for building natural lan- guage processing systems for chatbots and other applications. It allows building a custom language model to understand and respond to user inputs naturally. With Rasa NLU, we can: • Define the intents and entities the model should recognize in user inputs. • Train a model to recognize and classify user inputs based on defined intents and entities. • Use the trained model to parse user inputs and extract relevant information. To use Rasa NLU, training data is needed in the form of examples of user inputs and the corresponding intents and entities that the model should recognize. Then this data is used to train a model using the Rasa NLU training pipeline. Once the model is trained, Rasa NLU processes and understands user inputs in real-time. The generated training examples for AEdBOT (Section 5.1) are placed in the nlu.md files as shown in Fig. 15. 5.2.2 Domain The domain characterizes the universe in which chatbots live. It should know all the responses, intents, and actions for response generation. The model will use the information in the domain to understand the user’s input and generate appropriate responses. To define the domain in Rasa NLU, we created a domain.yml file. This file should contain a list of intents, entities, and actions that AEdBOT should be able to recognize and respond to as shown in Fig. 16. 5.2.3 RASA Training Data to Stories In Rasa, a ”story” is a sequence of conversations between a user and an AI assistant. These conversations train the AI assistant to understand and respond to user inputs naturally and engagingly. Rasa uses a deep learning-based approach to understand user inputs and generate appropriate responses. By training the AI assistant on a large dataset of example conversations (called ”stories”), it can learn to recognize patterns in user input and generate appropriate responses. In Rasa, stories are defined in a simple mark-up language, consisting of lines of text representing user inputs and AI responses. The training data consisted of intents and examples in the nlu.md file, and their actions and responses in domain.yml were then placed in stories.md file for the creation of conversation flow as shown in Fig. 17. For example, the story consists of two conversations: a greeting and a mood great. The AEdBOT will respond with the utter how can I help action when it receives the greet input from the user, and it will respond with ”Look’s good! Have a great day.” when it receives the mood great input from the user. 5.3 Model Evaluation When designing a software product, it is important to assess its quality. Quality can refer to many concepts: correctness, performance, reliability, development costs, and ease of use. In our situation, we will mainly be interested in measuring the following characteristics: NLU Performance. When we talk about the quality of a chatbot, its NLU performance, i.e., its ability to understand messages, directly crosses into minds. We usually use metrics such as precision and recall in machine learning tasks to char- acterize their performance. However, those metrics are meant for binary classification problems. In AEdBOT case, the NLU component is concerned with a multi-class clas- sification problem. We could have measured the precision and recall for each intent, as we would have ended up with many measurements (number of intents 108 with almost 4263 examples). Note that it appears the Loebner Prize Competition [53], an application of the Turing test, is often employed as a criterion to assess a chatbot. However, it has been argued this did not represent a good measurement of quality [54]. 5.3.1 Performance of Intent Classification Intent classification is the task of identifying the purpose or intention of a user’s input. In Rasa, this is done using a deep learning model trained on a dataset of examples of the user input command. The trained model will be able to predict the intent of new, unseen input by matching it against the corresponding intent labels. A train- ing dataset must important in Rasa because it allows the model to understand the purpose or intention behind a user’s input. In machine learning tasks, performance metrics for multi-class classification are essential tools for evaluating machine learn- ing models’ effectiveness in scenarios with more than two classes or categories. These metrics provide valuable insights into a model’s ability to classify instances across multiple classes correctly. One commonly used metric is accuracy, which measures the proportion of correctly classified instances out of the total. However, in situations with imbalanced class distributions, accuracy can be misleading. Precision, Recall, and F1-score are valuable alternatives. Precision measures the ratio of true positive predictions to the total positive predictions for each class, providing information about the model’s ability to minimize false positives. On the other hand, Recall measures the ratio of true positive predictions to the total actual positives, indicating the model’s ability to capture all instances of a class. The F1-score combines precision and recall to provide a balanced measure of a model’s performance. Here are some common per- formance metrics that we used for the performance evaluation of the Hybrid model on the test dataset as defined in Table 3; where True Positive is abbreviated as TPi , FPi denotes False Positive, TNi denotes True Negative, and FNi denotes False Nega- tive. These metrics collectively help data scientists and machine learning practitioners assess the strengths and weaknesses of their models and make informed decisions about model selection and optimization for multi-class classification tasks.The intent confusion matrix is shown in Fig. 18. Utilizing the ”rasa test” [52] command inside the CLI gives the trained model to run the tests on an 80/20 ratio of training and testing datasets, respectively, against 20 epochs. It assists us in envisioning our trained Deep Learning (DL)-based Hybrid Model in the form of a confusion matrix and histograms. Assuming each intent is mapped or predicted accurately. The matrix would have brought about all the slots being topped off diagonally, meaning the model has a true prediction for that specific intent. Comparative is the situation for our trained model, which shows that our model has accurately predicted each intent with no clutter. Also, the diagonal values accu- rately address the absolute number of tests it has predicted for that specific intent. The intent histogram in Fig. 19 allows us to visualize the training sample’s confidence level against each intent. The true and false predictions are shown separately by blue and red bars, meaning that the original intents classified the predicted intents suc- cessfully based on the training data split into an 80/20 ratio of training and testing data sets, respectively. For our trained Hybrid model, we got mostly hits (blue bars) and a few misses (red bars) by considering the 108 intents from around 4263 training samples. Each example was predicted with a confidence score of roughly 0.98-0.99. Rasa utilizes the strategy of classifying the intent behind the message considering the confidence level rank, i.e., the intent that gets the most remarkable confidence level (on a scale of 0 to 1) is ranked first for that specific message. It is picked to be the predicted intent, where 0 is for the least certainty value, and 1 is for the most con- fidence value. The confidence score completely relies upon the kind of training data that we have given in the NLU dataset, i.e., the bigger the training dataset given to each intent, the more confident the model would have the ability to distinguish or classify the predicted intent from the client’s message with more prominent accuracy. For the Hybrid model, we are getting a confidence score for a weighted and macro average between 0.98-0.99, which clearly expresses that the model could correctly identify or predict the intent of the user message. Macro is the arithmetic mean of the individual scores, while weighted includes the individual sample sizes. We have an imbalanced dataset but want to assign greater contributions to classes with more examples in the dataset, then the weighted average is preferred. This is because, in weighted averaging, the contribution of each class to the F1 average is weighted by its size. The weighted average results for multi-class classification are shown in Table 4. 5.4 Model Testing We selected 25 evaluators from varying backgrounds for Model testing and asked them to use AEdBOT for the admission and accommodation FAQ use case. The test examples results concerning the predicted intents are shown in Table 5. Each evaluator ran 2 test inquiries on AEdBOT. We made a test data set of 50 unseen inquiries based on the inquiries entered by the evaluators that did not belong to the training samples. Rasa gives us the provision to make a test.md file in the project file where we can place the test data set. Through this, we can see how a well-trained Deep Learning (DL)-based Hybrid model predicts the intents from our developed test set. The Deep Learning (DL)-based Hybrid model predicted the 47 inquiries correctly out of 50 test examples. We performed two types of testing approaches at that moment. The first one is the end-to-end level testing for the test data set. In Rasa, ”end-to- end level testing” typically refers to testing the entire conversation flow of a chatbot from start to finish. This might involve testing that the chatbot can correctly handle multiple turns of conversation and that it can correctly handle a variety of input from the user. The second one is the Action level testing. In Rasa, Action-level testing refers to testing a chatbot’s actions as part of its conversation flow. These actions might include making an API call, sending a message to the user, or storing data in a database. In Rasa, it is possible to write unit tests to test the behavior of specific actions to ensure that they are working correctly. We tested the whole data set against the 20 epochs, and the Hybrid model correctly identified 96 stories (including 50 stories from the test set) out of 100. The action confusion matrix shown in Fig. 20, which is based on a Deep Learning (DL)-based Hybrid trained core model (comprising of stories and utterances), shows that 3 intents are mixed up with other intents were topped off in the non-diagonal areas, while the rest, 47 intents, were in the diagonal areas. It is analyzed from the action confusion matrix that AEdBOT can classify the inquiries with 96% accuracy on the test data set. From the results, we can see that AEdBOT misunderstands about 0.04 of the user messages that are out of the scope of AEdBOT; it seems to misunderstand less often than that thanks to the fact that when the classifier misclassifies an utterance, it is usually not that far off. Indeed, upon misclassification, the chatbot often answers a question about the same topic. It is rare for AEdBOT not to understand an utterance. It is because, by default, Rasa considers an intent not to be matched when the confidence in the intent classification is smaller than 0.3. Considering that the classification problem is a multi-class problem, we get an average accuracy and error rate. In Rasa, action-level testing is a way to evaluate the performance of a trained model by testing it on a set of user input examples and comparing the predicted action sequence with the expected action sequence. This testing is often used to ensure a model can correctly predict the appropriate action to respond to a given input. The results of action-level testing can be considered the model’s results because they provide a clear and concise evaluation of the model’s ability to take the correct actions in response to user input. The average per-class Accuracy, Precision, Recall, and F1-Score are near what we got in model testing, as shown in Table 6. 5.5 Response Time Answering speed is another quality of AEdBOT that should be assessed. It is quite easy to ask the chatbot questions and record the time it takes to get a response. We can then use statistical tools to characterize those durations. The response times we measure here are the sum of several servers’ response times and data handling. The information flows as follows: the user transmits a message to the web server that hosts the UI; this web server forwards it to a WSGI server which oversees Rasa; this server runs AEBOT’s settings on the user message and selects the chatbot answer; the answer is sent back to the UI server which displays it for the user to see. We do not know what happens in the WSGI server; more than one server may be contacted to handle Rasa tasks. Nonetheless, we consider that part of the user message’s handling time by characterizing those durations. We are interested in measuring the response time of Rasa rather than the cumulated response time of Rasa and our server since we could make specific experiments to test our server with much larger precision. To avoid recording the response time of our server, we can either take the relevant measurements directly from the server or create a new program that uses the API of Rasa. We settled for the second option for the sake of simplicity. Response time can be seen as a continuous-time random variable. In our analysis, we will consider the response time between subsequent questions to be independent. It eases the analysis and should be close to reality since Rasa servers answer many differ- ent requests between the ones we send them. We could model the distribution of such a random variable in a few different ways: a log-normal distribution, a Pareto distribu- tion, and a Gamma distribution. All of them correspond to right-skewed probability distributions. Rather than choosing one of them a priori, we plotted the histogram of our measurements and fitted the different distributions to see which corresponds best, as illustrated in Fig. 21. We used the sum of the distribution’s squared errors (SSE) as a criterion to compare them. The distribution that fits our measurements best is the Gamma distribution. It has two parameters: its shape k and rate , both positive real numbers. Its probability density function is: 5.6 User Experience User experience (UX) plays a pivotal role in the testing phase of chatbot development for a variety of reasons. At its core, a positive user experience is synonymous with user satisfaction. When users find the interaction with a chatbot enjoyable and seamless, they are more likely to continue engaging with it and extracting value from its func- tionalities. The intuitive nature of a chatbot’s design is a key factor in user adoption. An easily navigable and user-friendly interface encourages users to actively participate, making the chatbot a valuable tool for accomplishing its intended objectives. The effectiveness of a chatbot is closely intertwined with user experience. A well- designed chatbot allows users to effortlessly navigate, access information, or receive assistance, contributing to the overall success of the chatbot in meeting user needs. Success is also measured by the users’ ability to complete tasks successfully and achieve their goals within the chatbot interface. A positive user experience fosters user reten- tion, as users are more likely to return for future interactions if their initial experience was positive. Importantly, a well-designed chatbot helps minimize user frustration. By identify- ing potential pain points and areas of confusion during testing, developers can make improvements that enhance the user experience, reducing frustration and ensuring a smoother interaction. Gathering feedback on the user experience during testing is invaluable for iterative improvements. Understanding user preferences, expectations, and challenges allows developers to refine the chatbot for better performance over time. The user experience also contributes to the overall brand image associated with the chatbot. A positive experience enhances the brand perception and fosters a favorable impression among users. Additionally, considering the diversity of users – in terms of background, technical expertise, and communication preferences – is vital. A well- crafted user experience ensures that the chatbot is accessible and usable by a broad range of individuals. In a competitive landscape where multiple chatbots may offer similar function- alities, a superior user experience becomes a significant differentiator. It can give a chatbot a distinct advantage over others in the same domain. Ultimately, prioritiz- ing user experience in chatbot testing is essential for creating a chatbot that not only functions correctly but is also well-received and embraced by its users. This user- centric approach contributes to the overall success and effectiveness of the chatbot in delivering a positive and satisfying user experience. Throughout this research, we have emphasized one of the key objectives of this study, which is to develop an additional system for international students to access general information about the university. A crucial aspect of this objective is to ensure the system is user-friendly and provides a positive user experience, irrespective of the user’s background. The quality of user experience is influenced by three main factors: • The ease-of-use of the User Interface (UI). • The abilities of the chatbot in understanding and responding to questions. • The behavior of the chatbot towards the user, aiming to avoid any awkwardness or difficulty in understanding. Each of these factors is the responsibility of a different component of AEdBOT: its UI, Natural Language Understanding (NLU) component, and Natural Language Generation (NLG) unit. Quantifying these aspects can be challenging as they are not easily measurable in a quantitative manner. A common approach to assess these factors is to gather user feedback by asking individuals to grade their experience on a given scale. To achieve this, we conducted tests where a selected group of external individuals, representing the intended users (present and future international students), interacted with AEdBOT. We employed the System Usability Scale (SUS) [ 55 ], a widely used industry standard, to evaluate usability. In this test, users are asked to rate 10 statements on a scale from ”strongly disagree” to ”strongly agree.” The scores obtained are typically divided into five steps and assigned numerical values (1 for ”strongly disagree” and 5 for ”strongly agree”). The results are then used to calculate a score out of 100. It’s important to note that this score is not a percentage; rather, it indicates how convenient and user-friendly the system appears to first-time users, allowing for comparisons between different systems. We conducted the test with several participants and aggregated their results by computing the mean and median scores for each question, ultimately deriving the final score described above. The 10 questions asked in the standard SUS [55] test, as well as in our conducted test, are as follows: • I think that I would like to use this system frequently. • I found the system unnecessarily complex. • I thought the system was easy to use. • I think that I would need the support of a technical person to be able to use this system. • I found the various functions in this system were well integrated. • I thought there was too much inconsistency in this system. • I would imagine that most people would learn to use this system very quickly. • I found the system very cumbersome to use. • I felt very confident using the system. • I needed to learn a lot of things before I could get going with this system. After interviewing 11 people, the average grades for each question, in the same order as above, are as follows in Table 7 : This yields a total mean score of 77.5 out of 100 and a total median score of 85. Of course, interpreting those results heavily depends on the type of system at hand. Nevertheless, knowing that the mean score for this test is 68, this result is very good, and the system seems easy to use and understandable even to non-technical users. 6 Comparative Analysis We compared the proposed chatbot methodology with the developed approaches in the literature to identify areas where we can improve performance. By comparing the performance of our developed chatbot to other approaches, we can identify areas where we need to improve in the proposed approach and take steps to improve. We per- formed the theoretical and experimental comparison with the state-of-the-art chatbot concerning the performance as given in Section 6.1 and Section 6.2, respectively. 6.1 Theoretical Comparison We identified the various chatbots in the literature review (Section 2) that are built for the administrative area of the educational domain [22],[23],[18],[24],[19],[26],[15],[16], [29],[25],[30]. But unfortunately, their data set is private to perform comparative analysis. We got only one data set [20] that was available publicly. So, to perform a comparative analysis, we decided to compare our proposed methodology with the chatbot [56]. University students from Belgium developed chatbot [56] for their administrative support at the university level [56]. The produced chatbot, named ’JAQ’, is aimed to be a Proof of Concept of an additional way for those students to obtain general information about the university. The scope of topics relevant to chatbot being very extensive, the authors implemented a complete system on top of DialogFlow platform to aid with data management and maintenance. The theoretical comparison between our proposed methodology chatbot and the chatbot [56] is performed in Table 8. 6.2 Experimental Comparison To test the chatbot as mentioned above [56] with our proposed methodology, we settled on an environment to test the validity of our proposed approach. A Core i5 6th generation – 2.56 GHz Processor with 12 GB RAM having an Amd Radeon r5 430 -2GB graphic card is used on the hardware side. For testing purposes, the same NLP pipeline is used that we proposed to develop the AEdBOT. The action confusion matrix is shown in Fig. 22. We ran the 121 examples of test data as mentioned in the dataset [20] on our proposed methodology that we adopted to develop AEdBOT and performed end-to- end level and action-level testing. To run the test examples on the Hybrid model to evaluate the performance of the proposed methodology, we used the ‘rasa evaluate’ command to generate a confusion matrix for the Hybrid model. This command will run the test data set on the Hybrid model and summarize the model’s performance, including the confusion matrix. Then we used the ‘rasa test’ command to get a detailed breakdown of the predictions made by the Hybrid model on a specific test set [20]. Out of 121 test examples, the Hybrid model predicts 78 test examples correctly at end-to-end level testing. At the action level testing, the Hybrid model predicted 182 stories correctly out of 239. The experimental result comparison graph and proposed framework’s performance evaluation results between the chatbot [20] and our proposed Deep Learning (DL)-based Hybrid model methodology chatbot are shown in Fig. 23 and Table 9, respectively 7 Discussion and Future Work To discuss whether AEdBOT is a good Proof of Concept for an assistant chatbot, we can look at the results of our different experiments; in terms of performance, we could argue that AEdBOT works well enough to be usable with 96% accuracy. More- over, upon misclassification, AEdBOT recommends something related to the topic that the user would like to know about based on conversation history, making that misclassification less annoying than if it had given information utterly unrelated to the question. During the conception of AEdBOT, we roughly tracked the evolution of the NLU performance and saw that it varied. The first challenge that stood out during those experiments is that the proposed NLP pipeline needs deterministic model train- ing. Indeed, running the same experiments twice on the default Rasa NLP pipeline yields similar results. However, training two models, such as the BERT and DIET Classifier, with the same training data and executing the same experiments produces different results. We performed a fine-tuning by comparing different configurations in the customized NLP pipeline and concluded that: 1) A small confidence threshold (0.3 compared to the default 0.6) improves the overall performances; 2) Augmenting the number of examples in the training set by only adding a question mark at the end of some examples does not improve performance; 3) Increasing the number of examples in the training set never decreases performance. Moreover, from a user perspective, capitalizing the first letter of a message or adding a question mark at the end does not influence its classification. Logically, defining many intents gives the model more classes to classify user messages as and, thus, more potential incorrect classifications. Therefore, we anticipated that activating the Smalltalk module would decrease the performance, as it would add many new intents. On the contrary, removing some of the intents we defined and whose examples were very close to examples of other intents improved the classification performances. Verifying the correctness of generated responses from a chatbot is a crucial step in ensuring the quality and reliability of the proposed system. A knowledge-based integration in AEdBOT relies on structured data and cross-reference responses with the data in the knowledge base to ensure that information is accurate and up to date. Another challenge we identified during experiments was how well AEdBOT worked when users spoke poor English. It is extremely relevant since AEdBOT is meant to be used by people from abroad whose native language might not be English. Therefore, having a chatbot that behaves well in this situation is essential. When a user message contains a typo or slightly off grammar, the NLU component could still classify the intents with the same performance. However, when the question was phrased utterly incorrectly, the chatbot would either need help understanding it or completely misclassifying it. In other words, the poorer the quality of the English in the user messages, the worse the user experience was. From a user-friendliness perspective, AEdBOT is thus able to cope with users who speak bad English but not with users whose first language is not English. To make AEdBOT multilingual, AEdBOT’s NLP pipeline can use multiple pre-trained language models, such as spaCy or Hugging Face’s transformers, and machine translation services, such as Google Cloud Translation, and Microsoft Azure Translator, which support various languages. These models and services can process the user’s input in their languages, and then the output can be passed to the AEdBOT. In the future, we will add the principles of continuous learning and adaptation of the Never-ending learning (NEL) [57] approach in AEdBOT that enable it to proac- tively discover new user intents through active engagement, monitoring interactions, and topic modeling, ensuring its knowledge base remains current and relevant. To improve over time, the AEdBOT leverages feedback loops, version control, and adap- tive learning, allowing it to adjust responses to user expectations and enhance overall user satisfaction. Knowledge enrichment is fundamental through data enrichment, knowledge graph expansion, and integration of external data sources. It ensures the AEdBOT can respond accurately to a wide range of queries, making it a dynamic and valuable conversational partner. In addition, developing an advanced automated SQL query generation algorithm through NLP are other potential areas for improvement. However, these improvements would require further investigation to avoid overfitting and classification issues. Conclusion In this paper, we were interested in studying the modern chatbot Artificial Intelligence (AI) approaches and platforms for building an effective chatbot, AEdBOT, to assist local and international students of Educational Institutions. The difference with previ- ous works was developing a novel Deep Learning (DL)-based Hybrid Model approach on top of the RASA platform with a customized NLP pipeline. The developed NLP pipeline can generate a response by efficient prediction of intent and entity extraction from Natural Language (NL)-based text and process text as a whole by using the attention base mechanism of the transformer and provide the chatbot with greater predictive accuracy to act as a recommendation system. We performed several experi- ments and a comparative analysis to assess its performance and usability. During those experiments the Deep Learning (DL)-based Hybrid Model approach performed well on the generated dataset and achieved encouraging results on performance metrics like Precision, Accuracy, Recall, and F1-Score, which come out to be 94.7%, 96.0%, 96.0%, and 95.1%, respectively, with a response time from the mean and standard deviation is 216.43 ms and 50.09 ms, respectively on an average basis. We also performed the User experience (UX) testing in the testing phase of chatbot development and AEdBOT showed its user-friendliness with a total mean score of 77.5 out of 100 and a total median score of 85 on the System Usability Scale (SUS), this result is very good, even though there is room for improvement. The proposed methodology is compared with the state-of-the-art chatbot and outperformed as well on it with an accuracy of 76.2%. We concluded that the challenges faced in developing a chatbot with a large scope make it difficult to create a training set of good quality and can decrease the performance and user experience. The modeler design and structured implementation methodology enable further improvements to be incorporated, i.e., adding more questions, conversational flows, and examples of utterances and entity values easily. Despite these challenges, AEdBOT fulfills its intended purpose as a practical expert chatbot system and is already operational in helping students query information. The results proved that the chatbot outperformed well, and users perceived it as enjoyable and user-friendly. Declarations Funding This work is partly supported by the Higher Education Commission (HEC), Pakistan, the National University of Science and Technology (NUST), Pakistan, and Malardalen University, Sweden. Conflict of interest The authors have no relevant financial or non-financial interests to disclose. Author contribution Material Preparation, Data Collection, Methodology Design, Experimentation and Analysis were performed by Muhammad Shahroze Ali. The first draft of the manuscript was written by Muhammad Shahroze Ali and Muhammad Waseem Anwar commented on previous versions of the manuscript. All authors read and approved the final manuscript. Data availability The Project Data is publicly available for the online communi- ties on [51]. Code availability The Project Code is publicly available for the online communities on [45]. References Roos, S.: CHATBOTS IN EDUCATION A passing trend or a valuable pedagog- ical tool ? In: Chatbots in Education A, p. 58 (2018). Wu, E.H.K., Lin, C.H., Ou, Y.Y., Liu, C.Z., Wang, W.K., Chao, C.Y.: Advantages and constraints of a hybrid model K-12 E-Learning assistant chatbot. IEEE Access 8 , 77788–77801 (2020) https://doi.org/10.1109/ACCESS.2020.2988252 Fakhri, S.A., Lutfi, H.U., Wardana, W.K., ...: Aplikasi Chatbot Informasi Kam- pus Polban Menggunakan Aplikasi LINE Messenger. https://jurnal.polban.ac.id/ proceeding/article/view/1403 (2019) Ranoliya, B.R., Raghuwanshi, N., Singh, S.: Chatbot for university related FAQs. In: 2017 International Conference on Advances in Computing, Communications and Informatics, ICACCI 2017, vol. 2017-January, pp. 1525–1530 (2017). https: //doi.org/10.1109/ICACCI.2017.8126057 Nguyen, T.T., Le, A.D., Hoang, H.T., Nguyen, T.: NEU-chatbot: Chatbot for admission of National Economics University. Elsevier Ltd (2021). https://doi. org/10.1016/j.caeai.2021.100036 Clarizia, F., Colace, F., Lombardi, M., Pascale, F., Santaniello, D.: Chatbot: An education support system for student. In: 2018 International Symposium on Cyberspace Safety and Security, Amalfi, pp. 291–302 (2018). https://doi.org/10. 1007/978-3-030-01689-0 23 Okonkwo, C.W., Ade-Ibijola, A.: Python-bot: A chatbot for teaching python programming, vol. 29, pp. 25–34 (2021) Pham, X.L., Pham, T., Nguyen, Q.M., Nguyen, T.H., Cao, T.T.H.: Chatbot as an intelligent personal assistant for mobile language learning. In: Proceedings of the 2018 2nd International Conference on Education and E-Learning, pp. 16–21 (2018) Hien, H.T., Cuong, P.-N., Nam, L.N.H., Nhung, H.L.T.K., Thang, L.D.: Intel- ligent assistants in higher-education environments: The fit-ebot, a chatbot for administrative and learning support. In: Proceedings of the Ninth International Symposium on Information and Communication Technology (2018) Goodfellow, I., Bengio, Y., Courville, A.: Deep learning. In: The MIT Press, Cambridge, MA (2016) OpenAI: GPT-4 Technical Report. http://arxiv.org/abs/2303.08774 (2023) Chan, A.: GPT-3 and InstructGPT: technological dystopianism, utopianism, and “Contextual” perspectives in AI ethics and industry. AI and Ethics 3 (1), 53–64 (2023) https://doi.org/10.1007/s43681-022-00148-6 Yenduri, G., M, R., G, C.S., Y, S., Srivastava, G., Maddikunta, P.K.R., G, D.R., Jhaveri, R.H., B, P., Wang, W., Vasilakos, A.V., Gadekallu, T.R.: Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions, 1–40 (2023) arXiv:2305.10435 Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., Le, Q.V.: XLNet: Generalized autoregressive pretraining for language understanding. In: Advances in Neural Information Processing Systems, vol. 32, pp. 1–18 (2019) Windiatmoko, Y., Hidayatullah, A.F.: Developing FB chatbot based on deep learning using RASA framework for university enquiries. https://arxiv.org/pdf/ 2009.12341. ArXiv Preprint ArXiv:2009.12341 (2020) Nguyen, M.T., Tran-Tien, M., Viet, A.P., Vu, H.T., Nguyen, V.H.: Building a Chatbot for Supporting the Admission of Universities, vol. 2021-November (2021). https://doi.org/10.1109/KSE53942.2021.9648677 Zhang, Y., Sun, S., Galley, M., Chen, Y.C., Brockett, C., Gao, X., Gao, J., Liu, J., Dolan, B.: DIALOGPT: Large-scale generative pre-training for conversational response generation. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 270–278 (2020). https://doi.org/10.18653/v1/ 2020.acl-demos.30 Sai Vikas, G.S., Kumar, I.D., Shareef, S.A., Roy, B.R., Geetha, G.: Informa- tion chatbot for college management system using multinomial naive bayes. In: 2021 2nd International Conference on Smart Electronics and Communica- tion (ICOSEC), Trichy, India, pp. 1149–1153 (2021). https://doi.org/10.1109/ ICOSEC51865.2021.9591757 Rana, M.: EagleBot: A Chatbot-Based Multi-Tier Question Answering Sys- tem for Retrieving Answers from Heterogeneous Sources Using BERT. https: //digitalcommons.georgiasouthern.edu/etd (2019) gellens: Master thesis JAQ code. https://github.com/gellens/Master thesis JAQ code (2024) Nouman, N., Shaikh, Z.A., Wasi, S.: A novel personalized learning framework with interactive e-mentoring. IEEE Access 12 (November 2023), 10428–10458 (2024) https://doi.org/10.1109/ACCESS.2024.3354167 Bala, K., Kumar, M., Hulawale, S., Pandita, S.: Chat-Bot for College Management System Using A.I. www.irjet.net. (September) (2017) Vichare, A., Shrikhande, Y., Gyani, A., Rathod, N.: Derek: A chatbot that shows intelligence in behavior using nlp, pp. 232–235 (2016). (March) Raees, W., Ismail, M.A., Aziz, R., Afshan, A.: NED Chatbot for Admission Related Queries Using Prescriptive Analysis, vol. 19, pp. 133–138 (2019) Mahanan, W., Thanyaphongphat, J., Sawadsitang, S., Sangamuang, S.: College agent: The machine learning chatbot for college tasks. In: 7th International Con- ference on Digital Arts, Media and Technology, DAMT 2022 and 5th ECTI Northern Section Conference on Electrical, Electronics, Computer and Telecom- munications Engineering, NCON 2022, pp. 329–332 (2022). https://doi.org/10. 1109//ECTI-DAMTNCON53731.2022.9720420 Thakkar, J., Raut, P., Doshi, Y., Parekh, K.: Erasmus ai chatbot, vol. 6, pp. 498–502 (2018). https://doi.org/10.26438/ijcse/v6i10.498502 Meshram, S., Naik, N., Megha, V.R., More, T., Kharche, S.: College enquiry chatbot using rasa framework. In: 2021 Asian Conference on Innovation in Tech- nology, ASIANCON, pp. 1–8 (2021). https://doi.org/10.1109/ASIANCON51346. Lee, K., Jo, J., Kim, J., Kang, Y.: Can chatbots help reduce the workload of administrative officers? - implementing and deploying faq chatbot service in a univerity. International Conference on Human-Computer Interaction (2019) Gonsalves, X., Deshmukh, S.: Designing an interactive chatbot for educational assistance using rasa framework. In: IDCIoT 2023 - International Conference on Intelligent Data Communication Technologies and Internet of Things, Proceed- ings, pp. 68–74 (2023). https://doi.org/10.1109/IDCIoT56793.2023.10053457 Channabasamma, Lakshmi Soumya, P., Indu, M., Swetha, N., Saritha, C.: Smart chatbot for college information enquiry using deep neural network. In: 2023 9th International Conference on Advanced Computing and Communica- tion Systems, ICACCS 2023, vol. 1, pp. 991–994 (2023). https://doi.org/10.1109/ ICACCS57279.2023.10112919 Braun, D., Mendez, A.H., Matthes, F., Langen, M.: Evaluating natural lan- guage understanding services for conversational question answering systems. In: SIGDIAL 2017 - 18th Annual Meeting of the Special Interest Group on Dis- course and Dialogue, Proceedings of the Conference, pp. 174–185 (2017). https: //doi.org/10.18653/v1/w17-5522 Kang, A.: Understanding the Differences between Alexa, API.ai, Wit.ai, and LUIS/Cortana. https://medium.com/@abraham.kang/ understanding-the-differences-between-alexa-api-ai-wit-ai-and-luis cortana-2404ece0977c. Accessed: 2018-8-11 (2017) Bobriakov, I.: A Comparative Analysis of Chatbots APIs. https://medium.com/activewizards-machinelearning-company/ a-comparative-analysis-of-Chatbots-APIs-f9d240263e1d. Accessed: 2018-8-13 (2018) Liu, G., Wang, S., Yu, J., Yin, J.: A survey on multimodal dialogue systems: Recent advances and new frontiers. In: Proceedings – 2022 5th International Con- ference on Advanced Electronic Materials, Computers and Software Engineering, AEMCSE 2022, pp. 845–853 (2022). https://doi.org/10.1109/AEMCSE55572. Yan, R., Song, Y., Wu, H.: Learning to respond with deep neural networks for retrieval-based human-computer conversation system. In: Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’16), Pisa, Italy, pp. 55–64 (2016). ACM Press Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidi- rectional transformers for language understanding. In: NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, vol. 1(Mlm), pp. 4171–4186 (2019) Devlin, J., Chang, M.-W., Lee, K., Google, K.T., Language, A.I.: BERT: Pre- training of Deep Bidirectional Transformers for Language Understanding. Naacl- Hlt 2019 (Mlm), 4171–4186 (2018) Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text- to-text transformer. In: Journal of Machine Learning Research, vol. 21, pp. 1–67 (2020) Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word rep- resentation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014) Henderson, M., Casanueva, I., Mrkˇsi´c, N., Su, P.-H., Wen, T.-H., Vulic, I.: Con- vert: Efficient and accurate conversational representations from transformers. In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2161–2174 (2020). https://doi.org/10.18653/v1/2020.findings-emnlp.196 Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017) Bunk, T., Varshneya, D., Vlasov, V., Nichol, A.: DIET: Lightweight Language Understanding for Dialogue Systems. http://arxiv.org/abs/2004.09936 (2020) Sharma, R.K.: An analytical study and review of open source chatbot framework, rasa. In: International Journal of Engineering Research And, vol. V9, pp. 1011– 1014 (2020). https://doi.org/10.17577/ijertv9is060723 Rasa: Connecting to Messaging and Voice Channels. https://rasa.com/docs/rasa/ messagingand-voice-channels. Accessed 5 June 2021 (2021) Ali, M.S.: AEdBOT. https://github.com/MuhammadShahrozeAli/AEdBOT (2024) Pimentel, R.: Chatito. https://github.com/rodrigopivi/Chatito. Consulted in April 2019 (2019) Leicher, J.: Chatl. https://github.com/atlassistant/chatl. Consulted in April 2019 (2019) Buck, M.: Expando. https://github.com/voxable-labs/expando. Consulted in April 2019 (2019) Galaxy, K.: Tracery. http://tracery.io/. Consulted in April 2019 (2019) Gustin, S.: Chatette. https://github.com/SimGus/Chatette. Consulted in April 2019 (2019) Ali, M.S.: AEdBOT - Data Repository. https://github.com/ MuhammadShahrozeAli/AEdBOT/tree/main/data (2024) Rasa: Rasa Tutorial. https://rasa.com/docs/rasa/user-guide/rasa-tutorial/ (2024) AISB: Loebner Prize. http://aisb.org.uk/events/loebner-prize. Consulted in June 2019 (2019) Shawar, B.A., Atwell, E.: Different measurements metrics to evaluate a chatbot system. In: Proceedings of the Workshop on Bridging the Gap: Academic and Industrial Research in Dialog Technologies, pp. 89–96 (2007). Association for Computational Linguistics Brooke, J., Jordan, P.W., Thomas, B., Weerdmeester, B.A., McClelland, I.L.: Sus: A quick and dirty usability scale. In: Usability Evaluation in Industry, vol. 189, pp. 4–7 (1996). https://doi.org/10.1016/B978-0-12-566251-8.50006-1 Gellens, A., Gustin, S.: Jaq: A chatbot for foreign students, Ecole polytechnique de Louvain, Universit´e catholique de Louvain (2019) Mitchell, T., Cohen, W., Hruschka, E., Talukdar, P., Yang, B., Betteridge, J., Carlson, A., Dalvi, B., Gardner, M., Kisiel, B., Krishnamurthy, J., Lao, N., Mazaitis, K., Mohamed, T., Nakashole, N., Platanios, E., Ritter, A., Samadi, M., Settles, B., Wang, R., Wijaya, D., Gupta, A., Chen, X., Saparov, A., Greaves, M., Welling, M.: Never-ending learning. In: Communications of the ACM, vol. 61, pp. 103–115 (2018). https://doi.org/10.1145/3191513 Tables Tables 1 to 3 are available in the Supplementary Files section Additional Declarations No competing interests reported. Supplementary Files Tables.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {\"props\":{\"pageProps\":{\"initialData\":{\"identity\":\"rs-4257811\",\"acceptedTermsAndConditions\":true,\"allowDirectSubmit\":true,\"archivedVersions\":[],\"articleType\":\"Research Article\",\"associatedPublications\":[],\"authors\":[{\"id\":298830700,\"identity\":\"137d8e26-8102-4591-a33a-2a060245e963\",\"order_by\":0,\"name\":\"Muhammad Shahroze Ali\",\"email\":\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABXElEQVRIie2RP2vCQBTA7wikS6RrRKufoHAhkCIW/SoJQm6JUOji0OFJ4SbFVSe/QlxCx5SDuAS6ZoxTlw5xq1Db3l0X/7QInUrJD+694x6/d487hEpK/iY4dmU6G4JMDRnMnbImln7ofCkGj2WyZV0oRHWDnxQVTV+ZHpxSLic8j3PU6Z6jwCrWwTWdz0dWtn54Q1cmfc7RoO1Bjbk7ipP5RAzW82aQ2tVZ5PfDxLBbs5Sg1jSwAKXUg3oS7ylIKrFLHkeOVol4P9QNp1ZhBJEswIAZ98CksKs8LQupdLvccLRt9EGbTChbpdAV4PdjJQ7ULTjUdV/DkdgnQsFKcS3AIBV/f7DgRkzV86apxqvjqGeFiX/bGjPbIOmLNXUTarN64u4Ptli9DsSLzVfDYhN1ms17vsg2rNEgS5oXxV37YlJj5PBj0PGJ+FoVZXvd/KZ8gl8oJSUlJf+JT2Ong7B1wyiJAAAAAElFTkSuQmCC\",\"orcid\":\"\",\"institution\":\"National University of Sciences and Technology\",\"correspondingAuthor\":true,\"prefix\":\"\",\"firstName\":\"Muhammad\",\"middleName\":\"Shahroze\",\"lastName\":\"Ali\",\"suffix\":\"\"},{\"id\":298830704,\"identity\":\"c044d0d0-88cd-4f20-ab8b-ea53b1576d8e\",\"order_by\":1,\"name\":\"Muhammad Waseem Anwar\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Mälardalen University\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Muhammad\",\"middleName\":\"Waseem\",\"lastName\":\"Anwar\",\"suffix\":\"\"},{\"id\":298830707,\"identity\":\"b29d2d7a-645f-4efd-a0ae-1b1f8e8680c5\",\"order_by\":2,\"name\":\"Farooque Azam\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"National University of Sciences and Technology\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Farooque\",\"middleName\":\"\",\"lastName\":\"Azam\",\"suffix\":\"\"},{\"id\":298830710,\"identity\":\"59f1a1ce-44fa-44b5-baef-a3b43691e3fb\",\"order_by\":3,\"name\":\"Muhammad Hashir Ashraf\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"National University of Sciences and Technology\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Muhammad\",\"middleName\":\"Hashir\",\"lastName\":\"Ashraf\",\"suffix\":\"\"}],\"badges\":[],\"createdAt\":\"2024-04-12 12:35:40\",\"currentVersionCode\":1,\"declarations\":\"\",\"doi\":\"10.21203/rs.3.rs-4257811/v1\",\"doiUrl\":\"https://doi.org/10.21203/rs.3.rs-4257811/v1\",\"draftVersion\":[],\"editorialEvents\":[],\"editorialNote\":\"\",\"failedWorkflow\":false,\"files\":[{\"id\":55992994,\"identity\":\"321c390b-2fcc-4e16-a373-bbc73acc032b\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 09:44:43\",\"extension\":\"png\",\"order_by\":1,\"title\":\"Figure 1\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":119951,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eArticle Structure Overview.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"1.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/d894956b898ade27dc7eb111.png\"},{\"id\":55993004,\"identity\":\"4cc1934a-b2d2-4eaa-bc17-8f66b8352238\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 09:44:43\",\"extension\":\"png\",\"order_by\":2,\"title\":\"Figure 2\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":101868,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eAEdBOT’s Architecture: A Detailed Breakdown of System Modules and Communication Channels.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"2.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/f42307c7d1fe976223354d5c.png\"},{\"id\":55994265,\"identity\":\"e6123a5c-fc9b-4c2d-90cb-37bdfa17e294\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 10:00:43\",\"extension\":\"png\",\"order_by\":3,\"title\":\"Figure 3\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":36720,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eNLP Pipeline Steps.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"3.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/a1612f92d67740a1bba755ea.png\"},{\"id\":55993485,\"identity\":\"85c3a508-ba1f-4289-8330-85adf5d46be6\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 09:52:43\",\"extension\":\"png\",\"order_by\":4,\"title\":\"Figure 4\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":85700,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003ePre-Processing on NL-Based Text Pipeline.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"4.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/d18e67d239b8892e92cb6e5c.png\"},{\"id\":55994266,\"identity\":\"146ab8be-8f42-4328-8254-2e3e661fc94c\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 10:00:43\",\"extension\":\"png\",\"order_by\":5,\"title\":\"Figure 5\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":76713,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eBERT’s model architecture in the Proposed Framework.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"5.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/e68fc9fa822e543155b9af5d.png\"},{\"id\":55993003,\"identity\":\"84d904c4-d0a4-40d0-b6d7-e245fd6e521c\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 09:44:43\",\"extension\":\"png\",\"order_by\":6,\"title\":\"Figure 6\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":89822,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eDIET Classifier architecture used in the Proposed Framework.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"6.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/aaa0d8a280ec5fc0e67dbdfb.png\"},{\"id\":55992997,\"identity\":\"4df965f8-307f-427d-aa06-6cd88bfe4fae\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 09:44:43\",\"extension\":\"png\",\"order_by\":7,\"title\":\"Figure 7\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":118152,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eProcessing on NL-Based Text Pipeline.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"7.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/b89792b0154bfb800e5e3c1b.png\"},{\"id\":55993007,\"identity\":\"477176e3-8d1e-4732-98ed-71d96e3e09b6\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 09:44:43\",\"extension\":\"png\",\"order_by\":8,\"title\":\"Figure 8\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":246376,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eNLG process of Proposed Framework.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"8.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/7fe2a836f8606178dd1c1397.png\"},{\"id\":55993014,\"identity\":\"5c7396ee-099b-45af-bff9-df731732e300\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 09:44:44\",\"extension\":\"png\",\"order_by\":9,\"title\":\"Figure 9\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":81595,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eInformational Retrieval-based Response Generation of Proposed Framework.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"9.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/524c4787dd8ce8d1ceefc0a0.png\"},{\"id\":55993481,\"identity\":\"dd9e264d-695b-451f-a740-44fd26343808\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 09:52:43\",\"extension\":\"png\",\"order_by\":10,\"title\":\"Figure 10\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":74393,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eCustomized Dual Fallback Classifier Algorithm’s Implementation of Proposed Framework\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"10.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/2ce81e99264d866c5b19009e.png\"},{\"id\":55993015,\"identity\":\"50ee19a7-07f7-40bb-83f5-81a4a7edbf90\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 09:44:44\",\"extension\":\"png\",\"order_by\":11,\"title\":\"Figure 11\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":92329,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eProposed Framework’s Recommendation.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"11.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/b344f7951a510732bce5a06f.png\"},{\"id\":55993008,\"identity\":\"3302ccdf-d0d0-4ae3-a8a4-695960e378e7\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 09:44:43\",\"extension\":\"png\",\"order_by\":12,\"title\":\"Figure 12\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":104688,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eAEdBOT Custom Web-based UI [45].\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"12.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/35052c0ea942dbb798d0a76d.png\"},{\"id\":55993487,\"identity\":\"1e542d6c-e452-4f7e-a0e8-fffed079c682\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 09:52:44\",\"extension\":\"png\",\"order_by\":13,\"title\":\"Figure 13\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":154343,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eAEdBOT Facebook Messenger UI [45].\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"13.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/0eca51f2c397a332406e2c2a.png\"},{\"id\":55993488,\"identity\":\"0468037e-52cb-4473-9d29-9cb5a91416a3\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 09:52:44\",\"extension\":\"png\",\"order_by\":14,\"title\":\"Figure 14\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":53531,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eCharting the Stages of Data Preparation Flow [51].\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"14.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/451335ce160b7835b43390ea.png\"},{\"id\":55993486,\"identity\":\"c90d8a19-0740-43d5-9f9d-850d9431a239\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 09:52:43\",\"extension\":\"png\",\"order_by\":15,\"title\":\"Figure 15\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":52472,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eTraining Examples for AEdBOT.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"15.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/0460f9b3135fca5883589eba.png\"},{\"id\":55993009,\"identity\":\"8a20143a-646b-40c4-b6b4-5f2c67c74f75\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 09:44:43\",\"extension\":\"png\",\"order_by\":16,\"title\":\"Figure 16\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":49968,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eGenerated Actions for AEdBOT.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"16.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/a9ed06515f29a1ce826903bf.png\"},{\"id\":55994639,\"identity\":\"4dd9f25a-5c8e-4605-b1aa-e6e6ca3ed648\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 10:08:43\",\"extension\":\"png\",\"order_by\":17,\"title\":\"Figure 17\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":42828,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eTraining Data to Stories for AEdBOT.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"17.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/5c6e96589aada098f5155dae.png\"},{\"id\":55993010,\"identity\":\"6797f4b9-4b5a-42c2-be86-a6433f47cd7f\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 09:44:44\",\"extension\":\"png\",\"order_by\":18,\"title\":\"Figure 18\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":321490,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003ePerformance Assessment of Intent Classification in a Confusion Matrix.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"18.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/b7d5f943425332f38a002fcd.png\"},{\"id\":55993013,\"identity\":\"ba7e884f-4c16-401a-9357-14ee44443b60\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 09:44:44\",\"extension\":\"png\",\"order_by\":19,\"title\":\"Figure 19\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":29365,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eIntent Prediction Confidence Distribution Histogram.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"19.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/d665cb40f0495895d69ae22c.png\"},{\"id\":55993012,\"identity\":\"c437983d-7bab-460c-a71a-66c866954748\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 09:44:44\",\"extension\":\"png\",\"order_by\":20,\"title\":\"Figure 20\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":246433,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003ePerformance Assessment of Model’s Predictions in a Confusion Matrix.\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"20.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/d4369c6cd9e631d034cb64a8.png\"},{\"id\":55995041,\"identity\":\"be35e186-df4a-4396-8d78-1a57f86682af\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 10:16:43\",\"extension\":\"png\",\"order_by\":21,\"title\":\"Figure 21\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":49812,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eDistribution of Response Time (in milliseconds).\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"21.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/5c0ebec8572e1648cfbac69e.png\"},{\"id\":55993016,\"identity\":\"075e509d-a598-4637-b9d5-825329f04df5\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 09:44:44\",\"extension\":\"png\",\"order_by\":22,\"title\":\"Figure 22\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":349511,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eAction Confusion Matrix of the state-of-art-chatbot dataset [20].\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"22.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/c2fd778475ea572375621c9c.png\"},{\"id\":55993017,\"identity\":\"0e080506-10f7-450c-80e2-60417ad61656\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 09:44:44\",\"extension\":\"png\",\"order_by\":23,\"title\":\"Figure 23\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":35998,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eExperimental Result Comparison w.r.t Performance\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"23.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/bacc6af6467d822b3238e866.png\"},{\"id\":56662090,\"identity\":\"9d3d389f-ed07-4bfd-a64b-d006965886a0\",\"added_by\":\"auto\",\"created_at\":\"2024-05-17 11:52:36\",\"extension\":\"pdf\",\"order_by\":0,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"manuscript-pdf\",\"size\":3305633,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"manuscript.pdf\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/136d76fb-64a4-47ba-a1c5-ce6d2b7b5660.pdf\"},{\"id\":55993479,\"identity\":\"bf1ed0fa-6854-47c9-9dc4-f770f55f6955\",\"added_by\":\"auto\",\"created_at\":\"2024-05-07 09:52:43\",\"extension\":\"docx\",\"order_by\":1,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"supplement\",\"size\":83904,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"Tables.docx\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-4257811/v1/7348b5c87ea69fcd852decf9.docx\"}],\"financialInterests\":\"No competing interests reported.\",\"formattedTitle\":\"Intelligent Agents in Educational Institutions: AEdBOT– A Chatbot for Administrative Assistance using Deep Learning Hybrid Model Approach\",\"fulltext\":[{\"header\":\"1 Introduction\",\"content\":\"\\u003cp\\u003eDeep Learning-based chatbot systems have seen increased adaption in the educational domain in recent years owing to increased sophistication in the Artificial Intelligence (AI) domain [1]. However, most communication between students and educational institutions is still performed physically. Several studies have revealed that chatbots are a great way to make the educational process more interactive and engaging. Chatbots are useful in education because they provide students with real-time information and quickly answer questions [2]. In the educational domain, chatbots are increasingly being utilized to answer frequently asked questions, manage student forums, promote information sharing and student guidance without human interference [3], and provide administrative support to reduce the administrative staff\\u0026rsquo;s burden during admissions times [4],[5].\\u003c/p\\u003e\\n\\u003cp\\u003eChatbots are intelligent agents communicating with users through natural language conversations. Most educational institutions use chatbots for different purposes depending on their requirements. Many studies have been conducted to make a chat- bot for educational purposes. Some chatbots were made for inquiries purposes related to students [4], [6], some were for assessment purposes [7], [8], and some were built for administrative purposes [9]. During the admission time at educational institutions, many local and international students want to know about the organization they are interested in. Many educational institutions need to employ more staff to reply to all questions and requests of local and international students and provide the necessary information for the hour, which is sometimes impossible. So, there is a need for an additional way in the form of a chatbot for the informational service providers through that the local and international students have quicker access to the information and can find the relevant information about the organization of the educational institution in an efficient way.\\u003c/p\\u003e\\n\\u003cp\\u003eFrom the design perspective, chatbot approaches and platforms are extremely important. Developers and researchers need a solid understanding of the chatbot approaches and platforms to build an effective smart chatbot. There are three primary approaches in Artificial Intelligence (AI) to building chatbots [10]: a rule-based approach, a machine learning approach, and a deep learning approach. Rule-based chatbots are programmed with an explicit set of rules and instructions. They tend to underperform and create wrong responses when they run over a sentence with no known pattern.\\u003c/p\\u003e\\n\\u003cp\\u003eConversely, machine-learning chatbots seek to overcome the limitations of hard- coded rule-based chatbots. The machine learning approach can manually extract patterns from data instead of relying on its rule through feature engineering tech- niques. For this, more expertise and domain knowledge are required. The need for a deep learning approach stems from the limitations of the knowledge base and the rigidity of machine learning approaches, which can automatically learn the patterns from the data representation. The deep learning approach allows models to automatically learn the patterns from the data representation by transforming inputs. It skips this step-in feature engineering that allows building more general models to analyze data on a large scale. Deep learning chatbots are categorized into informational retrieval-based and generative-based neural networks chatbots. Informational Retrieval-based Neural Networks chatbots pick the appropriate message from a knowledge base with the pre- diction based on the highest confidence level to respond to user queries. Still, these chatbots cannot generate a response on runtime based on the conversation history and recommend something to the users. Unlike informational retrieval-based neural net- work chatbots, generative-based neural network chatbots can generate responses based on the knowledge of the current and previous history of the user messages. However, generative chatbots are considered more difficult to develop and train. Notable among these generative-based neural network chatbots models are the Large Language Mod- els (LLMs) like Generative Pre-trained Transformers (GPT), with GPT-4 [11], GPT-3 [12], GPT-2 [13], and XLNet [14], has significantly impacted the landscape of natural language processing, particularly in the context of chatbot applications. However, It needs many training data examples, close source access due to paid API restrictions and high computational requirements to achieve a good conversation. Moreover, the previously built chatbots [15],[16] on generative-based neural network approach generally generate less accurate and inconsistent responses with significant grammatical errors due to traditional Sequence-to-Sequence model and Transformer-XL model approaches like Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) models, and DialoGPT [17], which do not perform well when they must respond to longer sentences and tend to give vague answers [5].\\u003c/p\\u003e\\n\\u003cp\\u003eThere is a need to propose a new execution platform and new model techniques to make a more productive chatbot to support the development of the chatbot advances in the administrative area of the educational domain. The previously built chatbots [4],[18],[19] in the educational domains are domain-specific due to fixed rule- based design, generated grammatical errors during response due to obsoleted deep approaches resulting in accuracy issues, required costly subscription plans for web and social sites integration due to close source models and platforms, and no action as a recommender system due to their model implementation techniques. This paper pro- poses a novel chatbot system named AEdBOT to provide administrative support to educational institutions. The key contributions of this study are as follows:\\u003c/p\\u003e\\n\\u003cul\\u003e\\n \\u003cli\\u003eA novel Deep Learning (DL)-based Hybrid Model approach is developed that com- bines Information Retrieval and Generative Neural Networks to build an advanced conversational agent. It uses customized NLP pipeline featurizers to extract sparse features and a pre-trained BERT model to extract dense features simultaneously from Natural Language (NL)-based text. The model uses the DIET (Dual Intent Entity Transformer) Classifier for intent and entity extraction and a customized dual fallback classifier algorithm for self-learning ability to act as a recommendation system on out-of-scope inquiries.\\u003c/li\\u003e\\n \\u003cli\\u003eFrom a platform\\u0026rsquo;s perspective, an open-source Rasa platform is utilized for deploy- ment on two distribution channels; a custom web-based UI and a Facebook Messenger with the integration of the MYSQL database through webhook after analyzing the entities from the Natural language (NL)-text entered by the user by making the automated SQL query generation for informational retrieval-based response generation.\\u003c/li\\u003e\\n \\u003cli\\u003eEvaluation of the AEdBOT methodology is done by developing the customized\\u003c/li\\u003e\\n\\u003c/ul\\u003e\\n\\u003cp\\u003edataset relevant to the administrative support inquiries at NUST referred to the National University of Sciences and Technology and got an intent confusion matrix and multi-classification performance metrics like Precision, Accuracy, Recall, and F1-Score, which come out to be 94.7%, 96.0%, 96.0%, and 95.1%, respectively, at an average mean response time of 216.43ms per query and a user-friendliness score of 77.5 on the System Usability Scale (SUS). The experimental and theoretical analysis is performed through proposed methodology for comparative purpose on the state- of-the-art chatbot \\u0026rsquo;JAQ\\u0026rsquo; dataset [20] and outperformed as well with an accuracy of 76.2%.\\u003c/p\\u003e\\n\\u003cp\\u003eHence, AEdBOT fulfilled its goals for accuracy issues during response generation, acted as a recommendation system, and could easily handle wider scopes and vague inputs. This paper adopts a research flow regarding chatbot development with stages in several parts. The structural overview of the article is shown in Fig. 1. These stages are organized as follows: Section 2 presents the Literature Review. Section 3 describes the Proposed Framework. The implementation of the AEdBOT is done in Section 4. The experimentation of AEdBOT is done in Section 5. Section 6 compares the AEdBOT proposed methodology with the state-of-the-art chatbot from the literature. Sections 7 and 8 discuss the main results of this work, future perspectives, and a conclusion, respectively.\\u003c/p\\u003e\"},{\"header\":\"2 Literature Review\",\"content\":\"\\u003cp\\u003eWe conducted a detailed literature review to ascertain the current state-of-the-art approaches and platforms employed in chatbot development, as given in Sections 2.1 and 2.2, respectively. It allows us to identify a realistic research gap, as given in Section 2.3.\\u003c/p\\u003e\\n\\u003ch2\\u003e2.1 Chatbot in Administrative Area of Educational Domain\\u003c/h2\\u003e\\n\\u003cp\\u003eFrom the literature, we identified that in Artificial Intelligence-based chatbots [10], three approaches are currently used to make chatbots: a rule-based approach, a machine learning approach, and a deep learning approach, as given in Section 2.1.1, 2.1.2, and 2.1.3, respectively.\\u003c/p\\u003e\\n\\u003ch3\\u003e2.1.1 Rule-based Chatbots\\u003c/h3\\u003e\\n\\u003cp\\u003eBased on the literature review, we identified some related work to the rule-based approach that many authors utilized to develop chatbots in the administrative area of the educational domain. Nouman et al.[21] developed an innovative e-mentor-based learning model by the utilization of a rule-based approach that aims to customize courses for individuals, adapting to their unique learning styles, preferences, abilities, and prior knowledge. The outcomes reveal that e-mentoring enhances the learning process and increases learner satisfaction, establishing it as a preferred choice. Despite these positive findings, there exists a notable research gap in understanding the specific challenges and nuances associated with implementing e-mentoring in the context of personalized learning. In addition, the authors didn\\u0026rsquo;t express the model quality using its BLEU (Bilingual Evaluation Understudy) score or multi-classification performance metrics. This gap highlights the need for ongoing research to optimize and enhance personalized learning in the evolving landscape of online education.B. R. Ranoliya et al. [4] adjusted a chatbot-preparing program to the FAQ in the School of Computing at the University of Leeds, England. It recovers data by pattern-matching design without utilizing any linguistic device. The chatbot answers inquiries utilizing FAQ data about the university.\\u003c/p\\u003e\\n\\u003cp\\u003eK. Bala et al. [22] carried out an intelligent chatbot for university-related FAQs to meet the scholastic requirements of the users. The application answers inquiries from students or guardians about university admissions and supports inquiries con- nected with Manipal College in India. The chatbot depended on Artificial Intelligence\\u003c/p\\u003e\\n\\u003cp\\u003eMarkup Language (AIML) language, an XML-based markup language intended to make artificial intelligence-based applications using the pattern matching method. A. Vichare et al. [23] fostered a chatbot for instructing users about daily activities. The chatbot\\u0026rsquo;s knowledge base is cared for with sports-related information coded utiliz- ing AIML. Their paper depicted the examination of three existing chatbots named ELIZA, ALICE, and Siri. The authors assessed these three frameworks, took specific imperative highlights, and carried them out in their proposed technique. Besides, the writers have developed an approach utilized in their chatbot. The proposed model utilized a pattern-matching approach that comprised depth-first search (DFS) as an implementation technique.\\u003c/p\\u003e\\n\\u003ch3\\u003e2.1.2 Machine Learning Chatbots\\u003c/h3\\u003e\\n\\u003cp\\u003eWe identified some related work to the machine learning approach. G. S. Sai Vikas et al. [18] introduced the informational Chatbot for College Management System Using Multinomial Naive Bayes. Chatbot aimed to help students in pandemic and socially distant situations. The system is built using the supervised learning Multinomial Naive Bayes algorithm, which is used to classify text messages into different categories. One downside of this model is unable to handle the unsupervised sentence. The chat- bot is designed to answer questions about course schedules, grades, attendance, and other important information related to college management. The proposed chatbot system was evaluated using a dataset of frequently asked questions related to college management and achieved an accuracy of 87.14% in classifying questions into their respective categories. The authors concluded that the system could provide an efficient and effective way for students and staff to access information related to college management.\\u003c/p\\u003e\\n\\u003cp\\u003eW. Raees et al. [24] forested a chatbot for admission into the NED University of Engineering and Technology. The purpose of this chatbot is to handle inquiries related the admission. The chatbot is trained on the 20000 plus previous inquiries by using the machine learning model of SVM for classification purposes. The GUI of the chatbot consists of Ajax, jQuery, and JavaScript, and MS SQL is used as a database for backend purposes. The authors concluded that the chatbot could provide an efficient and effective way for students to access information related to admissions and reduce the workload of the admissions office.\\u003c/p\\u003e\\n\\u003cp\\u003eW. Mahanan et al. [25] developed a chatbot system that uses a machine learning approach to answer questions about Digital Industry Integration (DII) curriculum administrative tasks at Chiang Mai University in Thailand. The authors employed a heuristic algorithm combining informational retrieval-based and generative-based approaches to determine the best response. The results show that our proposed chatbot can answer basic questions with 78.8% accuracy and advanced questions with 57.9% accuracy. However, the primary concern of this proposed framework is the sequential conversation based on the conversation history between the chatbot and humans.\\u003c/p\\u003e\\n\\u003ch3\\u003e2.1.3 Deep Learning Chatbots\\u003c/h3\\u003e\\n\\u003cp\\u003eWe also identified some related work to the deep learning approach. M. Rana et al. [19] carried out a methodology for semantics search to address students\\u0026rsquo; requests about a college and provided the data based on the college\\u0026rsquo;s site by utilizing the BERT model technique. It is an adaptable chatbot framework using three-choice techniques with the fundamental platform utilizing DialogFlow. Given the solicitation constraints in the DialogFlow chatbot formation, this chatbot has a few impediments, and the bot accomplished 56% accuracy only. The authors used a generative-based approach for the development of the chatbot. J. Thakkar et al. [26] forested is an Artificial Intelligence chatbot that addresses inquiries on academic world data. The creators proposed Erasmus by utilizing cloud administrations, Mlab (MongoDB cloud), (DialogFlow), and IBM Bluemix (webhook API). This chatbot took over a seriously extended redundancy in reacting to the clients since the thing applied to too many cloud administrations. The authors used an informational retrieval-based approach for the development of the chatbot. Y. Windiatmoko et al. [15] made a chatbot incorporated with MySQL data set and an API for University requests. This straightforward chatbot is just equipped for replying to customers with few purposes. Also, Indonesian is very different from different dialects like English, and the creators should have specified their tokenizer and the pipeline. They used Long Short-Term Memory Networks (LSTMs), a type of Simple RNN that is used for how far back one wants to go in the information of a conversation. The authors used a generative-based approach for the development of the chatbot.\\u003c/p\\u003e\\n\\u003cp\\u003eM. T. Nguyen et al. [16] introduced an intelligent system (a chatbot) that could support the admission process by automatically answering questions. The chatbot is developed using the rasa platform with the help of the BERT model, and for entity extraction purposes, the DIET classifier is used. The approach for designing the chatbot is a generative base and deployed the chatbot on Facebook Messenger. S. Meshram et al. [27] forested a web-based chatbot application that analyses and under- stands users\\u0026rsquo; queries and provides an instant and accurate response. The platform used to develop the chatbot is Rasa, the authors used an informational retrieval-based approach for entity extraction purposes. H. T. Hien et al. [9] forested a chatbot named FIT-EBot, which aimed to provide administrative and learning support at the Faculty of Information Technology of the Ho Chi Minh City University of Science, Vietnam (FIT-HCMUS). The chatbot was developed with the close source platform DialogFlow and deployed on Facebook Messenger. The average F1-Score of this chatbot is 82.33%.\\u003c/p\\u003e\\n\\u003cp\\u003eK. Lee et al. [28] forested a chatbot for administrative-related tasks in their college. The authors developed the chatbot with the closed-source DialogFlow Platform, and the informational retrieval base model technique was used to develop it. To check the effect of the chatbot on the administrative staff, they experimented by hiring two office workers and measuring their workload using NASA-TLX. X. Gonsalves and\\u003c/p\\u003e\\n\\u003cp\\u003eS. Deshmukh [29] proposed an Interactive Chatbot for Educational Assistance using Rasa Framework that allows students to submit questions about their academics, relevant courses, and programs. To handle the inquiries effectively, the authors used deep learning policies in the domain file. These policies checked the intent response and performed actions against the inquiries entered by the user on the web-based user interface (UI). However, the chatbot is not evaluated using performance evaluation metrics, and the bot cannot handle contextually relevant and interactive queries, which is unaware of the context in which a conversation with a student is taking place.\\u003c/p\\u003e\\n\\u003cp\\u003eChannabasamma et al. [30] developed a chatbot based on a deep-learning approach integrated with the website via the Flask framework. The chatbot model is linked to HTML, CSS, and Java for the front end, Python for the back end, and JSON for the database format, which provides answers to FAQs (Frequently Asked Questions), general queries, and all relevant information about the GRIET (Gokaraju Rangaraju Institute of Engineering and Technology) organization. The GRIET website provides available information such as administration, admission, departments, placements, social media contact information, and a location navigator. However, the proposed chatbot needed to have a fallback policy to respond to the user and not act as a recommendation system.\\u003c/p\\u003e\\n\\u003ch2\\u003e2.2 Chatbot\\u0026rsquo;s Platforms\\u003c/h2\\u003e\\n\\u003cp\\u003eA platform is a software infrastructure that aims to facilitate and speed up a devel- oper\\u0026rsquo;s task. It gives them a ready-to-use architecture and components to solve common obstacles encountered in the applications it relates to. There are lots of different plat- forms intended to create different types of software. Conceivably, a certain number exist specifically to build chatbots, each having its characteristics, tackling problems differently, and offering different useful features.\\u003c/p\\u003e\\n\\u003cp\\u003eFrom the literature review, we identified that the most used platforms in the edu- cational domain are DialogFlow and Rasa. After reviewing different ones, the most conspicuous characteristic to categorize them is whether they are closed or open source. Usually, closed-source platforms are meant to be used online and run on a server owned by the company to which the platforms belong. On the contrary, open-source plat- forms can be used on a local machine and are typically developed to be lightweight enough to execute on slow hardware. Of course, it is also possible to put the pro- duced chatbot on a privately owned server and communicate with the user through an internet connection. The closed-source platforms cannot be used without access to an internet connection, while open-source ones usually can. This approach is true when designing and interacting with the produced chatbot. Open-source platforms are thus a better solution if the chatbot is meant to be used on a local machine or when data privacy is critical [31],[32].\\u003c/p\\u003e\\n\\u003cp\\u003eDetermining the chatbot\\u0026rsquo;s platform is profoundly important to improve the accu- racy of the chatbot. Multiple different closed-source solutions are available to the public. Those solutions usually use a \\u0026rdquo;pay per request\\u0026rdquo; system. In opposition to this, open-source solutions are available for free services providing. For this, we analyzed five chatbot platforms [33] and evaluated them on seven dimensions based on the usability criteria described in Table 1.\\u003c/p\\u003e\\n\\u003ch2\\u003e2.3 Research Gap\\u003c/h2\\u003e\\n\\u003cp\\u003eFrom studies, we analyzed that there is a need for some improvement regarding the technical advancement of the chatbot models to generate better results. To the extent that education and research go, chatbots in this domain are built using a rule-based approach that consists of a defined set of fixed rules. They may create wrong responses when they run over a sentence with no known pattern. Moreover, pattern-matching standards could be stronger, exceptionally domain-specific, and must move better from one domain to the next [34]. The machine learning approach is emerged to overcome the rule-based approach issues. By machine learning algorithms, there is no longer the need to define rules manually and code new pattern matching, which permits chatbots to be more adaptable and, as of now, not subject to domain-specific knowledge. Traditional machine learning methods include feature engineering as a data preprocessing step. For this purpose, the developer needs to analyze the output parameter manually, and too much expertise and domain knowledge are required. A deep learning approach is introduced to overcome these issues. It is an approach known as feature engineering that could automatically identify the features from data to classify them greatly. The first approach in deep learning is an informational retrieval- based neural network approach that offers more prominent flexibility and response to the users in a well-defined manner based on pre-defined responses. The disadvantage of this approach is that the informational retrieval-based neural networks approach cannot recommend anything to the user because it cannot generate a response as per the conservation flow on the run time [35].\\u003c/p\\u003e\\n\\u003cp\\u003eThe second approach in deep learning is the generative-based neural networks approach that can generate responses based on the knowledge of the current and pre- vious history of the user messages and has some memory-saving abilities. The main feature of this approach is that the generative-based model is not domain-specific and can be trained on different data sets and applied to different domains. This approach uses recurrent neural networks (RNNs) models like LSTM in their design approach. The disadvantage of the generative approach is not answering the longer sentence appropriately but generating vague output. Their replies often need to be more con- sistent and have a lot of grammatical errors during response generation [5]. In deep learning, Transformers are the greatest innovation based on the attention mechanism. It is also a generative-based neural network technique with some advancement. Trans- formers replaced the traditional recurrent neural networks (RNNs) models like long short-term memory (LSTM) with the pre-trained model like BERT (Bidirectional Encoder Representation Transformer) and GPT (Generative Grained Pre-Trained Transformer) that permits the model to train on the larger data sets. Transformers also replace the context\\u0026rsquo;s fixed-length vectorization support in the generative base model by weighing the input data in an equivalence sum of hidden vectors [36].\\u003c/p\\u003e\\n\\u003cp\\u003eIn recent years, the development of large language models (LLMs) has significantly impacted the landscape of natural language processing, particularly in the context of chatbot applications. Notable among these models is the Generative Pre-trained Transformers (GPT), with GPT-4 [11] standing out as the epitome of scale, boast- ing an impressive 1 trillion parameters. This cutting-edge model boasts multimodal capabilities, demonstrating its proficiency in processing both image and text inputs to generate insightful text outputs. However, accessibility to GPT-4 is restricted due to paid API. One of its predecessors is GPT-3 [12], boasting an impressive 175 billion parameters but limited access due to API restrictions and high computational requirements. Necessitating consideration of alternative models like its predecessor GPT-2 [13], which, with 1.5 billion parameters, strikes a balance between performance and availability. Moving beyond the GPT series, Bidirectional Encoder Representations from Transformers (BERT) [37] introduces bidirectional context understanding, prov- ing effective in various NLP tasks, while the Text-to-Text Transfer Transformer (T5) [38] takes a versatile approach with a text-to-text framework, excelling in tasks ranging from language generation to understanding. Additionally, XLNet [14] combines autore- gressive and autoencoding approaches, capturing bidirectional context with intricate architecture. For conversational contexts, DialoGPT [17] has been fine-tuned, offering context-aware responses.\\u003c/p\\u003e\\n\\u003cp\\u003eEach model comes with its strengths, including considerations for parameters, accessibility, resource requirements, versatility, and whether the model is open source. Understanding these aspects is crucial for choosing the most suitable model for specific chatbot applications. The choice of the \\u0026rdquo;best\\u0026rdquo; model depends on specific requirements, available resources, and the intended application. For open-source availability and versatility, BERT, T5, XLNet, and DialoGPT may be more accessible options, but T5 and XLNet have high computational demands. On the other hand, GPT-4, GPT- 3, and GPT-2 could provide unparalleled performance if access is available free. For model selection, we analyzed seven large language models (LLMs) and evaluated them on five dimensions based on the usability criteria described in Table 2.\\u003c/p\\u003e\\n\\u003cp\\u003eIn this study, we proposed a novel Deep Learning (DL)-based Hybrid model approach that used an advanced form of deep learning attention-based mechanism of Transformers to overcome the issues in the traditional rule-based, machine learning, and deep learning approaches that had been obsoleted by developing a customized NLP pipeline for creating a smart chatbot called AEdBOT by utilizing the open-source Rasa platform for educational institutions. The developed customized NLP pipeline used the featurizers and pre-trained open source large language BERT model to extract sparse and dense features from the Natural Language (NL)-based text simultaneously along with the aid of DIET Classifier for dual intent and entity extraction and process sentences as a whole by using the attention base mechanism of the transformer rather than requiring 8 time-steps of traditional deep learning DialoGPT [17] models to pro- cess the sentences word by word. The developed customized dual fallback classifier is integrated into the NLP pipeline, providing the self-learning ability to AEdBOT to act as a recommendation system on vague inquiries to efficiently enhance the prediction of response generation without grammatical errors to overcome the accuracy issue. The developed chatbot has third-party integration like a web-based UI and Facebook messenger connectivity to make it more social.\\u003c/p\\u003e\"},{\"header\":\"3 Proposed Framework\",\"content\":\"\\u003cp\\u003eThis section describes the development of the proposed framework based on the pro- posed methodology adopted. The architecture of AEdBOT is discussed in Section 3.1. It allows us to develop the customized natural language processing (NLP) pipeline for processing Natural Language (NL)-based text as given in Section 3.2. The working principles of the NLP pipeline for response generation after processing of Natural Language (NL)-based text is described in Section 3.3.\\u003c/p\\u003e\\n\\u003ch2\\u003e3.1 AEdBOT\\u0026rsquo;s Architecture\\u003c/h2\\u003e\\n\\u003cp\\u003eThe architecture of the AEdBOT is shown in Fig. 2. The chatbot is built utilizing the open-source Rasa platform, which comprises two primary parts: RASA Core and RASA NLU. RASA Core is the dialogue manager; the domain file in RASA Core characterizes the settings for the chatbot, for example, what it ought to comprehend, and what it could use to answer. RASA NLU took in NLU training data and conversation examples to prepare the chatbot. The predicated-based word embedding Bidirectional Encoder Representation Transformer (BERT) model is used for the extraction of dense features from the Natural Language (NL)-based text after parsing the text from \\u003cem\\u003eWhitespaceSpaceTokenizer\\u0026nbsp;\\u003c/em\\u003eand \\u003cem\\u003eProfanityAnalyzer\\u0026nbsp;\\u003c/em\\u003efor the parting of text into tokens and performing sentimental analysis on the text for the removal of negative words from the text, respectively.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cem\\u003eRegexFeaturizer, LexicalSyntacticFeaturizer, CRFEntityExtractor\\u003c/em\\u003e, and \\u003cem\\u003eCountVec- torFeaturize\\u003c/em\\u003er extract sparse features from the text to provide more information to the model. It could then deal with intent classification and entity extraction through the DIET classifier from the Natural Language (NL)-based text. The conversation flow design, different functionalities, for example, storing and bringing data from a data frame, and AEdBOT run-time recommendations based on confidence level score are carried out through \\u003cem\\u003eTransformer Embedding Dialogue (TED) policy\\u003c/em\\u003e, \\u003cem\\u003eMemoization\\u0026nbsp;\\u003c/em\\u003e\\u003cem\\u003ePolicy\\u003c/em\\u003e, and custom \\u003cem\\u003eDual Fallback Classifier\\u0026nbsp;\\u003c/em\\u003erespectively. In addition, the AEdBOT interacts with the MySQL database system to generate an informational retrieval- based response by automatically creating a SQL query from the Natural Language (NL)-based text. The developed chatbot is deployed on two distribution channels with the help of Flask on the web page and Facebook messenger through the ngrok python library to allow communication between the rasa server and the Facebook messenger app to make it more social. The proposed architecture provides an effective solution for building an advanced conversational agent to handle various user inquiries. The combi- nation of two different deep learning-based approaches, i.e., information retrieval and generative neural networks, along with the customized NLP pipeline and self-learning ability, makes the approach more robust and accurate\\u003c/p\\u003e\\n\\u003ch2\\u003e3.2 Natural Language Processing (NLP) Pipeline\\u003c/h2\\u003e\\n\\u003cp\\u003eThere are several components in developing a customized AEdBOT Natural Language Processing (NLP) Pipeline. RASA NLU is responsible for the sequential processing of Natural Language (NL)-based text in the NLP pipeline as described in Section 3.2.1 to perform pre-processing and processing on the Natural Language (NL)-based text entered by the user as discussed in Section 3.2.2 and 3.2.3, respectively.\\u003c/p\\u003e\\n\\u003ch3\\u003e3.2.1 RASA Natural Language Understanding (NLU)\\u003c/h3\\u003e\\n\\u003cp\\u003eRASA NLU performs the NLU task by sequentially applying the components specified in a pipeline configuration file on the sample examples in a labeled dataset. The pipeline configuration file contains a specification that defines the sequential processing steps required to classify the initially unstructured user utterances and extract the relevant entities. It is also specified in yalm format (config.yml). It assigns one or more\\u003c/p\\u003e\\n\\u003cp\\u003ecomponents to each of the three stages depicted in Fig. 3. These are commonly used in deep learning approaches for Natural Language (NL)-based text analysis. The first step is tokenizing the utterance, which involves breaking down the textual data into words, symbols, or\\u003c/p\\u003e\\n\\u003cp\\u003eother meaningful elements known as tokens. The algorithm divides the input sentence into words, but there are more complex alternatives that help with specialized tasks. Each token is converted into numeric features in the second stage. Several featurizers can be used at the same time. The features produced by all components are concatenated into a single vector in this case. The number of features can be sparse or dense. Dense features are typically floating-point values obtained from pre-trained embedding such as BERT [36], GloVe [39], ConveRT [40], or other Hugging Face models.\\u003c/p\\u003e\\n\\u003cp\\u003eOn the other hand, sparse features include vectors with many zero-values, such as Bag of Words (BoW) and n-gram representations or categorical data counts. Sparse features for this CLS [36] token are calculated as the sum of each token\\u0026rsquo;s sparse features. The computation of dense features is determined by the capabilities provided by the concrete featurizer. Some models can compute the sequence\\u0026rsquo;s contextualized aggregate representation. When this is impossible, they are computed as the sum or mean of the token representations. The original sentence is converted into numeric features, and then passed to the intent classification model.\\u003c/p\\u003e\\n\\u003ch3\\u003e3.2.2 Pre-Processing on NL-based text\\u003c/h3\\u003e\\n\\u003cp\\u003eSeveral components are involved in the customized NLP pipeline to perform the pre- processing on the NL-based text, as shown in Fig. 4, which are:\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; \\u003cstrong\\u003eWhitespace Tokenizer:\\u0026nbsp;\\u003c/strong\\u003eA whitespace tokenizer is a simple text tokenization method that tokenizes a piece of text by breaking it up at every instance of a whitespace character. For example, the string \\u0026rdquo;Information about getting a nust accommodation\\u0026rdquo; would be tokenized as [\\u0026lsquo;Information\\u0026rsquo;, \\u0026lsquo;about\\u0026rsquo;, \\u0026lsquo;getting\\u0026rsquo;, \\u0026lsquo;a\\u0026rsquo;, \\u0026lsquo;nust\\u0026rsquo;, \\u0026lsquo;accommodation\\u0026rsquo;] using a whitespace tokenizer.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; \\u003cstrong\\u003eProfanity Analyzer:\\u0026nbsp;\\u003c/strong\\u003eA customized Profanity Analyzer python algorithm is devel-\\u003c/p\\u003e\\n\\u003cp\\u003eoped for AEdBOT to perform sentimental analysis to filter out the negative words from the Natural Language (NL)-based text entered by the user. It will help to refine the input after filtering out the notorious or criminal words from the Natural Language (NL)-based text.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; \\u003cstrong\\u003eBERT\\u003c/strong\\u003e\\u003cstrong\\u003eModel:\\u003c/strong\\u003eBERT (Bidirectional Encoder Representations from Transformers)\\u003c/p\\u003e\\n\\u003cp\\u003e[36] is a state-of-the-art natural language processing (NLP) model developed by Google as shown in Fig. 5. The internal functionality of BERT is based on the\\u003c/p\\u003e\\n\\u003cp\\u003etransformer architecture, which was introduced by [41] in their paper \\u0026rdquo;Attention is All You Need\\u0026rdquo; (2017). The transformer architecture consists of self-attention and feed-forward layers stacked on top of each other to form a deep neural network. The BERT model uses 12 layers of transformers block with a hidden size of 768 and several self-attention heads as 12 and has around 110M trainable parameters. The feed-forward layers process the weighted input from the self-attention layers and transform it into dense output features. Since BERT\\u0026rsquo;s goal is to generate a language representation model, it only needs the encoder part. The input to the encoder for BERT is a sequence of tokens, which are first converted into vectors and then processed in the neural network. But before processing can start, BERT needs the input to be massaged and decorated with some extra metadata:\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cem\\u003eToken embeddings:\\u0026nbsp;\\u003c/em\\u003eA [CLS] token is added to the input word tokens at the beginning of the first sentence, and a [SEP] token is inserted at the end of each sentence.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cem\\u003eSegment embeddings:\\u0026nbsp;\\u003c/em\\u003eA marker indicating Sentence A or Sentence B is added to each token. This allows the encoder to distinguish between sentences.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cem\\u003ePositional embeddings:\\u0026nbsp;\\u003c/em\\u003eA positional embedding is added to each token to indicate its position in the sentence. One unique aspect of BERT is that it is bidirectional, meaning it considers the context of a word both to the left and the right of the word. In contrast, most previous NLP models could only consider the context to the left of a word. It is achieved using a \\u0026ldquo;masked language model\\u0026rdquo; training task, in which a portion of the words in the input is randomly masked, and the model is trained to predict the masked words based on the context provided by the unmasked words. For example, consider the following sentence in Fig. 5: \\u0026ldquo;Information about getting a nust accommodation.\\u0026rdquo; If the word \\u0026ldquo;accommodation\\u0026rdquo; is masked, the model must use the context provided by the words \\u0026ldquo;Information about getting a nust \\u0026ldquo;to predict the correct word. It allows BERT to learn rich representations of language that can capture the meaning and context of words in a sentence rather than just their meanings.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; \\u003cstrong\\u003eFeaturizers:\\u0026nbsp;\\u003c/strong\\u003eIn the Rasa natural language processing platform, featurizers take raw input data, such as a message or user input, and convert it into a numerical representation that can be used as input to a deep learning model. Each of the three featurizers in this configuration Natural Language pre-processing pipeline generates a set of sparse features for each token produced by the \\u003cem\\u003eWhiteSpaceTokenizer\\u003c/em\\u003e. The two components of \\u003cem\\u003eCountVectorFeaturizer\\u0026nbsp;\\u003c/em\\u003eproduce sparse features associated with the \\u003cem\\u003eLexicalSyntacticFeaturizer, RegexFeaturizer\\u003c/em\\u003e, and \\u003cem\\u003eCRFEntityExtractor\\u0026nbsp;\\u003c/em\\u003efor each token, including the CLS, based on the appearance of words and n-grams in the sentence. These features are computed for all labeled samples during training and passed to the DIET classifier to build the model. They are computed from the user utterance at inference time and passed to the classifier to predict the intent and extract the entities.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; \\u003cstrong\\u003eDIET\\u0026nbsp;\\u003c/strong\\u003e\\u003cstrong\\u003eClassifier:\\u0026nbsp;\\u003c/strong\\u003eDIET Classifier [42] is a simple transformer architecture that can be fully parameterized using the Rasa toolkit. Fig. 6 depicts a schematic represen- tation of this architecture with entity recognition and masking disabled. The user\\u0026rsquo;s utterance is split into tokens in the first stage, using the tokenizer algorithm. At the end of the sentence, the special CLS token is added. It is possible to generate sparse and dense features through the DIET Classifier simultaneously. Sparse features are then routed through a feed-forward network with shared output concatenated with the dense features from the pre-trained word embedding layers before passing through another feed-forward network. The last feed-forward network\\u0026rsquo;s outputs are fed into the transformer. The transformer output for the CLS token and the intent label are separately embedded into a single semantic vector space using embedding layers. The dot-product of Total Loss is then applied to maximize similarity to the intent target label while minimizing similarity to all other intent labels. The dot- product similarity is used at inference time to rank all possible intent labels, and the scores for all intents are combined to yield a confidence value. The Intent Loss module calculates the loss of the transformer\\u0026rsquo;s output and the sentence\\u0026rsquo;s intent clas- sification (given by the similarity module). Then goes as an input to the Total Loss module, which will provide a measure of learning. The Conditional Random Field (CRF) module is used during training to calculate an Entity Loss for the entities recognized by the NLU pipeline and the transformer\\u0026rsquo;s output; this loss measures\\u003c/p\\u003e\\n\\u003cp\\u003ethe ability of the transformer to recognize entities related to the input sentence. The pipeline specification in Rasa allows the designer to configure multiple param- eters. The number of transformer layers (2 by default), the output dimension of the embedding layers (20 by default), the fraction of weights set to non-zero values for all feed-forward layers in the model (0.2 by default), the size of the vector com- ing out of the transformer (256), and the number and size of hidden layer sizes are chosen for the training of AEdBOT\\u0026rsquo;s dataset in feed-forward networks.\\u003c/p\\u003e\\n\\u003ch3\\u003e3.2.3 Processing on NL-based text\\u003c/h3\\u003e\\n\\u003cp\\u003eWhen executing a dialogue management solution, the fundamental task is to conclude what occurs next concerning the conversation. The class rasa.core.policies conclude what move to be made during every conversation flow with the bot [43]. There are two sorts of approaches: Deep Learning and Rule-based policies. These policies will help the bot conclude which moves should be made toward a conversation. Depending upon AEdBOT\\u0026rsquo;s requirements, we picked deep learning and customized dual fallback classifier policies to make AEdBOT work more effectively.\\u003c/p\\u003e\\n\\u003cp\\u003eThe deep learning policies included the \\u003cem\\u003eTransformer Embedding Dialogue\\u0026nbsp;\\u003c/em\\u003e\\u003cem\\u003e(TED)\\u0026nbsp;\\u003c/em\\u003ePolicy and \\u003cem\\u003eMemoization\\u0026nbsp;\\u003c/em\\u003ePolicy. These policies are activated to predict the response on the confidence level score returned by the DIET Classifier during pre-processing of the Natural Language (NL)-based text entered by the user. If the confidence level score returned by the DIET Classifier during intent and entity extraction from the Natural Language (NL)-based text is equal to 1, then \\u003cem\\u003eMemoization\\u0026nbsp;\\u003c/em\\u003ePolicy is activated. \\u003cem\\u003eMemoization\\u0026nbsp;\\u003c/em\\u003eis a technique for optimizing the performance of a chatbot by storing the results of expensive function calls and returning the stored result when the same inputs occur again. Fig. 7 shows the policies utilized in the NLP pipeline.\\u003c/p\\u003e\\n\\u003cp\\u003eIn Rasa, \\u003cem\\u003ememoization\\u0026nbsp;\\u003c/em\\u003ecan be used to optimize the performance of the\\u0026nbsp;\\u003cem\\u003eRasa.core.featurizers.Max.History. Tracker\\u0026nbsp;\\u003c/em\\u003eFeaturizer class, which converts training\\u003c/p\\u003e\\n\\u003cp\\u003edata and conversations into machine-readable features. It will cause the featurizer to store the results of its featurization process in a cache and return the stored result whenever the same input is encountered again. It can significantly improve the per- formance of the featurizer, especially when dealing with large amounts of data. If the confidence level score is less than 1 but greater than 0.3, then the \\u003cem\\u003eTransformer Embed- ding Dialogue (TED) Policy\\u0026nbsp;\\u003c/em\\u003eis activated. The TED policy in Rasa is a variant of the Transformer model specifically designed for dialogue management tasks. It takes in a sequence of messages in a conversation and produces a probability distribution over the next action to take in the conversation. The TED policy is trained on data from past conversations to learn patterns and improve its performance over time.\\u003c/p\\u003e\\n\\u003cp\\u003eAEdBOT can also handle out-of-scope queries efficiently entered by the user. We developed a customized \\u003cem\\u003eDual Fallback Classifier\\u0026nbsp;\\u003c/em\\u003ealgorithm to handle the fallback sitduations in a conversation; if the confidence level score returned by the DIET Classifier is less than or equal to 0.3. A fallback situation is when the chatbot cannot under- stand or respond to a user\\u0026rsquo;s message and needs to provide a default response or take other action to keep the conversation going.\\u003c/p\\u003e\\n\\u003cp\\u003eThe customized \\u003cem\\u003eDual Fallback Classifier\\u0026nbsp;\\u003c/em\\u003ealgorithm consists of two separate clas- sifiers: a primary fallback classifier and a secondary fallback classifier. The primary fallback classifier identifies fallback situations and triggers a fallback action. The sec- ondary fallback classifier is used to confirm the fallback situation, determine the appropriate fallback action based on the form and mapping policies, and provide a self- learning ability to AEdBOT as a recommendation system based on the conversation history between the AEdBOT and the user. The primary fallback classifier compares the confidence of the intent prediction made by the chatbot with a threshold value. If the confidence is below the threshold of 0.3, the primary fallback classifier will trigger a fallback action. The secondary fallback classifier then determines the appropriate recommendation based on the conversation\\u0026rsquo;s context.\\u003c/p\\u003e\\n\\u003ch2\\u003e3.3 Natural Language Generation (NLG)\\u003c/h2\\u003e\\n\\u003cp\\u003eNLG systems use natural language processing (NLP) techniques to analyze structured data and generate natural language text that is easy for humans to understand. The proposed framework could retrieve and generate responses according to the Natural Language (NL)-based text entered by the users. The NLG process of AEdBOT is demonstrated in Fig. 8. The Deep Learning (DL)-based Hybrid model approach is used to build an effective model for classifying Natural Language (NL)-based text that the user will enter. It is a combination of informational retrieval and generative neu- ral networks. An informational retrieval-based neural networks approach is developed by providing a training dataset of text-based information or by interacting with the MYSQL Db after extracting an entity from the Natural Language (NL)-based text entered by the user. Then the approach will retrieve the information using a DIET Classifier in correspondence to Natural Language (NL) - based text entered by the user. On the other hand, the generative-based neural networks approach is developed on the pre-trained data set of the BERT model. The pre-trained model is then used for training the data set through a transfer learning approach as a word embedding model. The BERT model could easily predict response and activate the dual fall- back classifier as a recommendation system by efficiently extracting the dense features based on the conversation history between the user and the chatbot. For example, the user entered the Natural Language (NL)-based text \\u0026rdquo;Eligibility criteria of NUST\\u0026rdquo; on the AEdBOT UI. The developed Deep Learning (DL)-based Hybrid model parses the query in the customized Natural Language Processing (NLP) pipeline, which first performs the pre-processing on the text to extract intent and entity. The Natural Language (NL)-based text is passed from \\u003cem\\u003eWhitespaceTokenizer\\u0026nbsp;\\u003c/em\\u003efor parting text into tokens and then passed the text into the customized \\u003cem\\u003eProfanityAnalyzer\\u0026nbsp;\\u003c/em\\u003ealgorithm to perform the sentimental analysis like removing negative words from the sentence. After performing the data cleaning on the Natural Language (NL)-based text, the sec- ond step is feature extraction to convert textual data into categorical form through a one-hot encoding technique by representing Natural Language (NL)-based text into vector space. For this, a generative-based BERT model is utilized to extract dense features. For the extraction of sparse features, three featurizers are used. The first one is \\u003cem\\u003eRegexFeaturizer\\u0026nbsp;\\u003c/em\\u003ewhich works on the pre-specified regular expression and is respon- sible for extracting numbers, dates, and times in specified formats. The second one is the \\u003cem\\u003eLexicalSyntacticFeaturizer\\u0026nbsp;\\u003c/em\\u003ewhich acts as character n-grams, representing the doc- ument as a sequence of characters. The third one is \\u003cem\\u003eCountVectorFeaturizer\\u0026nbsp;\\u003c/em\\u003ewhich is responsible for the bag-of-words representation of Natural Language (NL)-based text. These concatenate features (dense and sparse) are passed the DIET Classifier, which is responsible for the extraction of intent and entity from the Natural Language (NL)- based text by predicting the [intent: admission, confidence: 0.7877], entity: [\\u0026rdquo;nust\\u0026rdquo;: Organization, confidence: 0.9123].After the extraction of intent and entity confidence level score returned by the DIET classifier, we made the customized processing pipeline responsible for the retrieval of response by interacting with the MYSQL Db or gener- ation of response based on the deep learning policies specified in the pipeline. If the Confidence level score is 1, then the \\u003cem\\u003eMemoization\\u0026nbsp;\\u003c/em\\u003epolicy is activated, which generates the fixed response on which the Hybrid model is trained. Suppose the confidence level\\u003c/p\\u003e\\n\\u003cp\\u003escore is less than or equal to 0.3. In that case, the customized dual fallback algorithm is activated to provide the self-learning ability to AEdBOT to act as a recommendation system. If the confidence level score is less than 1 but greater than 0.3, then the \\u003cem\\u003eTrans- formerEmbeddingDialogue (TED)\\u0026nbsp;\\u003c/em\\u003ePolicy is activated, a unidirectional transformer. It will check the previous history of the bot responses based on the slots, which is the bot\\u0026rsquo;s memory, and passes the text through Embedding, Similarity layers to perform the next system action and generate the response.\\u003c/p\\u003e\\n\\u003ch3\\u003e3.3.1 Informational Retrieval-based Response Generation\\u003c/h3\\u003e\\n\\u003cp\\u003eThe informational retrieval-based models usually used the deep learning approach in most cases, but they also used the pattern matching approach, a rule-based model technique. Informational retrieval-based neural networks usually contain a set of pre- defined pairs of question answers in their knowledge base and pick the one answer from the knowledge base based on the user\\u0026rsquo;s Natural Language (NL)-based text. The knowledge base is a corpus of the pre-defined set of question-answer pairs. The chat index is provided to the question-answer pair in the knowledge base. When the user enters the Natural Language (NL)-based text from the web interface or any other ter- minal, the model treats the input as a user query, matches the pattern of this question in its knowledge base concerning the chat index, and retrieves the information accord- ingly. For retrieval-based response generation purposes in AEdBOT, we categorized it into two blocks for the intent and entity extraction from the Natural Language (NL)- based text entered by the user. The first is a non-informational retrieval-based intent response. The second is an informational retrieval-based intent response, as shown in Fig. 9.\\u003c/p\\u003e\\n\\u003cp\\u003eFor each intent category, corresponding blocks are categorized, and solutions are identified to generate the responses; for Non-Informational Retrieval based intent, such as Welcome and Good Bye, fixed and unchangeable responses are generated. For example, when an AEdBOT receives a user\\u0026rsquo;s Natural Language (NL)-based text that is identified to be a Welcome intent, the AEdBOT automatically generates the answer that is \\u0026rdquo;Welcome. I\\u0026rsquo;m AEdBOT. I can help you with finding a Hostel or accom- modation and providing latest news for them\\u0026rdquo;. When an AEdBOT receives a user\\u0026rsquo;s Natural Language (NL)-based text identified as a Good Bye intent, the AEdBOT automatically generates the \\u0026rdquo;GoodBye\\u0026rdquo; answer to the user.The AEdBOT interacts with MYSQL for other intents to generate the Informational-Retrieval based intent response. For example, for asking about the PG programs at NUST, the proposed system executes a SQL statement, \\u0026rdquo;SELECT program from NUST programs where program=\\u0026rsquo;PG\\u0026rsquo;\\u0026rdquo;, to return the PG programs to the user. The AEdBOT applies a retrieval-based neural network model approach that can perform queries from the MYSQL database system APIs. Suppose the AEdBOT cannot identify the intent and entity extraction from the user\\u0026rsquo;s Natural Language (NL)-based text. In that case, the system will respond to the user with an intent fallback message: \\u0026rdquo;Sorry, I didn\\u0026rsquo;t under- stand it, please rephrase again\\u0026rdquo;. For example, a user asks for the programs at NUST but doesn\\u0026rsquo;t specify the information about which program it is. At the same time, the system is trained on the program\\u0026rsquo;s asking at NUST with the entity @program. In this scenario, the AEdBOT will remind the user:\\u0026rdquo; Please, tell me about which program\\u003c/p\\u003e\\n\\u003cp\\u003eyou are asking?\\u0026rdquo; after calling an intent fallback. After the user provides the program information, the AEdBOT generates the answer.\\u003c/p\\u003e\\n\\u003ch3\\u003e3.3.2 Generative-based Response Generation\\u003c/h3\\u003e\\n\\u003cp\\u003eAs the name suggests, the Generative-based Neural Networks models generate responses to the user input. These models could generate a new sentence based on the user\\u0026rsquo;s Natural Language (NL)-based text. However, the model should be trained on such types of Natural Language (NL)-based text. Generative-based models are usu- ally trained on a large dataset and learn the pattern, syntax, and vocabulary from the data that has been fed. AEdBOT has a feature to give a conditional block a goal-based approach for registering accommodation in NUST. Nonetheless, we quickly realized there were two main problems if we used this: First, the user cannot change their mind once the chatbot tries to fulfill a goal. For example, If the user wants to register the hostel and asks the system: \\u0026rdquo;I want to register a hostel\\u0026rdquo;, the system will respond with an intent message: \\u0026rdquo;ok, Please provide your First Name\\u0026rdquo;. The user enters the name, for example: \\u0026rdquo;Shahroze Ali\\u0026rdquo;. After that, the system requests information about the date of birth: \\u0026rdquo; Please provide your Date-of-Birth (DOB)\\u0026rdquo;, and waits for the entity that the user would enter in return for it, but the user asks the counter-question on it: \\u0026rdquo;where do you live\\u0026rdquo;, and the Bot generates the run-time response in return of it: \\u0026rdquo;The Virtual World is my Playground, I always live here.\\u0026rdquo; and at the same time Bot again ask the previous inquiry to the user, and the user doesn\\u0026rsquo;t want to provide the information about the DOB to the Bot.\\u003c/p\\u003e\\n\\u003cp\\u003eWe want to avoid that behaviour since it annoys the user when the NLU component detects a question associated with a complex interaction. To overcome this issue, we employed Rasa\\u0026rsquo;s Slot filling feature to check whether the required entity is already contained in the corresponding context. Suppose the context supposed to contain this entity value is not defined. In that case, the answer related to the intent will be recommended, asking the user for the required entity value. A context wait Ei will be activated to indicate which entity is expected.\\u003c/p\\u003e\\n\\u003cp\\u003eOn the other hand, if the entity value is known, the webhook will be called and generate a relevant answer. We made a \\u0026rsquo;custom fallback action\\u0026rsquo; script by using Python language which is our customized \\u003cem\\u003eDual Fallback Classifier\\u0026nbsp;\\u003c/em\\u003ealgorithm to handle the fallback situations in a conversation. Its code contains a data structure that stores each complex question according to the conditions (i.e., which entity is required) and the associated answers. Knowing the current intent and known entity values, the webhook can generate an appropriate answer. When the NLU component detects that the user is providing information and there is a pending question (the context pend Qi is active), the webhook is called to generate the correct answer in the same way as before. If no question is pending, it will activate the related context ctx Ei so that it remembers it for later turns and returns an answer acknowledging an entity was provided and asking the user what they want. Lastly, if the NLU component didn\\u0026rsquo;t understand the latest user message and an entity was expected, it will tell the user it didn\\u0026rsquo;t understand and which entity it was expecting. If it were not expecting any entity, the returned message would tell the user the chatbot didn\\u0026rsquo;t understand and suggest the user rephrase the text again. The implemented logic using the contexts is shown in Fig. 10.\\u003c/p\\u003e\\n\\u003cp\\u003eSecondly, suppose the user wants information about the type of test conducted at the educational institution, for example, NUST, and asks the system: \\u0026rdquo;What type of test is required for admission at NUST\\u0026rdquo;, if the chatbot does not understand the message the user is providing, it will keep asking for it. It can stall the chatbot conver- sation, as the user might need to learn why they get asked the same question again and how to escape the loop. We avoided those drawbacks once again by using a webhook. We used to manage complex interactions as much more intricate as simple slot-filling (in the goal-based approach to dialog management). When an entity is required to answer a question, we activate a context so that the chatbot remembers it expects an entity. Then, the user can ask something else and get an answer before providing the previously asked entity and having an answer for the first question. Since the context has a certain lifespan (measured in the number of conversation turns), the chatbot can \\u0026rdquo;forget\\u0026rdquo; a question is pending if the user asks for other information for a long time (and hence refuses to give the value of the entity), as it would happen in a human- to-human conversation. Having dual fallback intents depending on the active contexts prevents the problem: the chatbot can recommend the user, as shown in Fig. 11, based on the previous history of the conversation if it does not understand the entity value it provided.\\u003c/p\\u003e\"},{\"header\":\"4\\tImplementation\",\"content\":\"\\u003cp\\u003eAn important feature of the Rasa platform is easily integrating the developed chatbot into third-party services, such as web platforms and social media services like Facebook Messenger, Slack, and Twitter. [44]. These services represent the UI component of the chatbot. We only integrated it into two different interfaces: a Custom User Interface (UI) and Facebook Messenger, as described in Section 4.1 and Section 4.2, respectively. The project code [45] is publically available for the online communities to implement the state-of-the-art AEdBOT proposed methodology in their developing chatbots to get better accuracy.\\u003c/p\\u003e\\n\\u003ch2\\u003e4.1 Custom Web-based User Interface (UI)\\u003c/h2\\u003e\\n\\u003cp\\u003eThe custom webpage using HTML/CSS and JavaScript is built for AEdBOT deploy- ment, where AEdBOT would like to run. The credentials.yml file is configured to allow communication between the Rasa server and the custom web page. In this file, the \\u003cem\\u003esocketio\\u0026nbsp;\\u003c/em\\u003echannel is activated for the API calling of the bot on the UI. Then this webhook API is wrapped into Flask to route the HTML home page.This interface is meant to be executed on a web server. Internally, the Custom User Interface (UI) works as follows:\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; \\u003cem\\u003eWeb Front-End:\\u0026nbsp;\\u003c/em\\u003eThe HTML, CSS, and JavaScript-based front-end of the AEdBOT permit the user to send a message request to Rasa. The message made by the user is processed and delivered as an API call generally anticipated by the Python frame- work that runs on the WSGI server. A request is made to the Python framework utilizing a POST retrieval API request which assists in transmitting the information to the WSGI server.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; \\u003cem\\u003eWeb Back-End:\\u0026nbsp;\\u003c/em\\u003eThe chatbot\\u0026rsquo;s back-end is created using Flask. Every time the user\\u003c/p\\u003e\\n\\u003cp\\u003esends a message to the UI, the WSGI web server makes an API call, including the user message to the Rasa server through Socket.io, which hosts AEdBOT. This server handles the requests and returns the chatbot\\u0026rsquo;s answer, displayed to the user.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; \\u003cem\\u003eExchange Flow APIs:\\u0026nbsp;\\u003c/em\\u003eWhenever a user types in a Natural Language (NL)-based text, the front-end calls to the Python framework, and the WSGI server do the processing. In Rasa, the \\u003cem\\u003eTransformerEmbedding- Dialogue (TED)\\u0026nbsp;\\u003c/em\\u003epolicy is a variant of the Transformer model designed explicitly for dialogue management tasks. It takes in a sequence of messages in a conversation and produces a probability distribution over the following action to take in the conversation. The TED policy is trained on data from past conversations to learn patterns and improve its performance over time. Rasa server gets the request from Socket.io and generates a response utilizing a Deep Learning (DL)-based Hybrid Model (informational retrieval and generative) approach. The deep learning policies included the Transformer Embedding Dialogue (TED) Policy is activated at that moment to predict the response on the confidence level score returned by the DIET Classifier during pre-processing of the Natural Language (NL)-based text entered by the user and transmits the output to the user. An MYSQL database is also connected to Rasa via webhook for retrieval-based response generation.\\u003c/p\\u003e\\n\\u003cp\\u003eBuilding it as a web page allows us to easily integrate it into another web page, such as the official website of the NUST. The custom web user interface (UI) of AEdBOT is shown in Fig. 12.\\u003c/p\\u003e\\n\\u003ch2\\u003e4.2 Facebook User Interface (UI)\\u003c/h2\\u003e\\n\\u003cp\\u003eThe user might be familiar with Facebook Messenger, the messaging application included in the Facebook social network. It allows Facebook users to send each other messages in different formats, such as text, images, audio, recordings, or files. The interface of AEdBOT on Facebook Messenger is implemented as follows:\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; \\u003cem\\u003eFacebook\\u0026rsquo;s Page Creation:\\u0026nbsp;\\u003c/em\\u003eA Facebook Page is created because Facebook currently allows only a Facebook page to be connected to a Facebook application.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; \\u003cem\\u003eFacebook\\u0026rsquo;s App Development:\\u0026nbsp;\\u003c/em\\u003eThe application uses the API to communicate with a\\u003c/p\\u003e\\n\\u003cp\\u003eFacebook platform through a webhook call-back. The credentials.yml file is config- ured to allow communication between the Rasa server and the Facebook application by passing the generated secret key and page-access-token generated during the Facebook application\\u0026rsquo;s development.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; \\u003cem\\u003eFacebook\\u0026rsquo;s App Connection with RASA Server through Ngrok:\\u0026nbsp;\\u003c/em\\u003eFor running a Rasa server on localhost, most external channels won\\u0026rsquo;t find the server URL since localhost is not open to the internet. To make a port on the local machine that will be publicly available on the internet, the ngrok is used. An event will be sent to the appropriate webhook whenever a user sends a message to the Facebook application\\u0026rsquo;s page. AEdBOT can acquire user-submitted data for subsequent processing as a result. Integrating a chatbot into Facebook Messenger allows users of Facebook to interact with the chatbot as if they were normal users of this social media platform. The user interface (UI) of Facebook Messenger is shown in Fig. 13.\\u003c/p\\u003e\"},{\"header\":\"5\\tExperimentation\",\"content\":\"\\u003cp\\u003eThis section describes the testing of the proposed framework on the case study of an educational institution named NUST, referring to the National University of Sci- ence and Technology. For experimentation purposes, we made a customized dataset of inquiries relevant to administrative tasks in NUST, as described in Section 5.1. The simulation of the dataset is done in Section 5.2. The performance of AEdBOT is val- idated using widely accepted metrics such as 1) Model Evaluation, 2) Model Testing, 3) Response Time, and 4) User Experience, as described in Sections 5.3, 5.4, 5.5, and 5.6, respectively.\\u003c/p\\u003e\\n\\u003ch2\\u003e5.1 Dataset Preparation\\u003c/h2\\u003e\\n\\u003cp\\u003eOnce we have decided which topics the chatbot should support, we can gather data that covers them. In our case, there are a certain number of ways we could find this data: our knowledge as an NUST student, Frequently Asked Questions (FAQs) on the official website of the university, other websites related to it, and employees of NUST. The main source of information used is the FAQs found on NUST\\u0026rsquo;s website; obviously, all this data is not formatted well-defined, nor is the information contained there directly exploitable. We thus had to read through all this documentation, understand the information there and make it exploitable, which cannot be done automatically.\\u003c/p\\u003e\\n\\u003cp\\u003eTo choose which questions AEdBOT should support, we had to imagine what being an international and local student would be like. We took inspiration primarily from the FAQs and divided all the information we could find into 2 (correlated) topics:\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; \\u003cem\\u003eAccommodation\\u003c/em\\u003e\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; \\u003cem\\u003eAdmission\\u003c/em\\u003e\\u003c/p\\u003e\\n\\u003cp\\u003eWe listed all the questions about the topics that seemed relevant to us. To create the answers to those questions, we organized the gathered data to let us see the big picture and easily retrieve some information. To do that, we made simple schema diagrams of the relevant topics. We also tried to automate this data gathering but scraping the website of NUST is quite difficult since its HTML code could be clearer. Moreover, we would have required an extremely powerful algorithm to find useful information automatically within the data we could gather. Manually gathering data was the most effective but extremely time-consuming.\\u003c/p\\u003e\\n\\u003cp\\u003eTo speed up this process, chatbot designers commonly use a dataset generator. Such tools allow them only to provide templates of examples that the tools used to generate the datasets. As Natural Language Understanding (NLU) algorithms usually expect a dataset in a formatted way, such tools can also handle the formatting of the generated data.\\u003c/p\\u003e\\n\\u003cp\\u003eMany tools exist for data set generation purposes. Each has capabilities and is intended to be used with a certain NLU algorithm. We identified five data set generator platforms, i.e., Chatito [46], Chatl [47], Expando [48], Tracery [49], and Chatette [50]. All these tools are open source. We also identified that the first four use similar syntax for writing the templates and allow certain interoperability between them. To make the training dataset for the chatbot, we used Chatette for two reasons: it is intended to create large datasets as needed and uses a simple syntax to create the scripts. The generated dataset from the Chatette data set generator consisted of 108 intents with almost 4263 examples. The dataset is publicly available at [51] for further exploration. The flow of dataset preparation is shown in Fig. 14.\\u003c/p\\u003e\\n\\u003ch2\\u003e5.2 Dataset Simulation\\u003c/h2\\u003e\\n\\u003cp\\u003eRasa\\u0026rsquo;s documentation gives a short tutorial [52] on building a basic chatbot. The main files are presented here:\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; An \\u003cem\\u003enlu.md\\u0026nbsp;\\u003c/em\\u003efile, which incorporates all NLU-preparing information.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; A \\u003cem\\u003edomain.yml\\u0026nbsp;\\u003c/em\\u003efile characterizes the chatbot\\u0026rsquo;s domain or knowledge base.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; A \\u003cem\\u003eStories.md\\u0026nbsp;\\u003c/em\\u003efile to create a conversation flow.\\u003c/p\\u003e\\n\\u003ch3\\u003e5.2.1 RASA NLU\\u003c/h3\\u003e\\n\\u003cp\\u003eRasa NLU (Natural Language Understanding) [52] is a tool for building natural lan- guage processing systems for chatbots and other applications. It allows building a custom language model to understand and respond to user inputs naturally. With Rasa NLU, we can:\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; Define the intents and entities the model should recognize in user inputs.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; Train a model to recognize and classify user inputs based on defined intents and entities.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; Use the trained model to parse user inputs and extract relevant information.\\u003c/p\\u003e\\n\\u003cp\\u003eTo use Rasa NLU, training data is needed in the form of examples of user inputs and the corresponding intents and entities that the model should recognize. Then this data is used to train a model using the Rasa NLU training pipeline. Once the model is trained, Rasa NLU processes and understands user inputs in real-time. The generated training examples for AEdBOT (Section 5.1) are placed in the nlu.md files as shown in Fig. 15.\\u003c/p\\u003e\\n\\u003ch3\\u003e5.2.2 Domain\\u003c/h3\\u003e\\n\\u003cp\\u003eThe domain characterizes the universe in which chatbots live. It should know all the responses, intents, and actions for response generation. The model will use the information in the domain to understand the user\\u0026rsquo;s input and generate appropriate responses. To define the domain in Rasa NLU, we created a domain.yml file. This file should contain a list of intents, entities, and actions that AEdBOT should be able to recognize and respond to as shown in Fig. 16.\\u003c/p\\u003e\\n\\u003ch3\\u003e5.2.3 RASA Training Data to Stories\\u003c/h3\\u003e\\n\\u003cp\\u003eIn Rasa, a \\u0026rdquo;story\\u0026rdquo; is a sequence of conversations between a user and an AI assistant. These conversations train the AI assistant to understand and respond to user inputs naturally and engagingly. Rasa uses a deep learning-based approach to understand user inputs and generate appropriate responses. By training the AI assistant on a large dataset of example conversations (called \\u0026rdquo;stories\\u0026rdquo;), it can learn to recognize patterns in user input and generate appropriate responses. In Rasa, stories are defined in a simple mark-up language, consisting of lines of text representing user inputs and AI responses. The training data consisted of intents and examples in the nlu.md file, and their actions and responses in domain.yml were then placed in stories.md file for the creation of conversation flow as shown in Fig. 17. For example, the story consists of two conversations: a greeting and a mood great. The AEdBOT will respond with the \\u003cem\\u003eutter how can\\u0026nbsp;\\u003c/em\\u003e\\u003cem\\u003eI\\u0026nbsp;\\u003c/em\\u003e\\u003cem\\u003ehelp\\u0026nbsp;\\u003c/em\\u003eaction when it receives the greet input from the user, and it will respond with \\u0026rdquo;Look\\u0026rsquo;s good! Have a great day.\\u0026rdquo; when it receives the \\u003cem\\u003emood great\\u0026nbsp;\\u003c/em\\u003einput from the user.\\u003c/p\\u003e\\n\\u003ch2\\u003e5.3 Model Evaluation\\u003c/h2\\u003e\\n\\u003cp\\u003eWhen designing a software product, it is important to assess its quality. Quality can refer to many concepts: correctness, performance, reliability, development costs, and ease of use. In our situation, we will mainly be interested in measuring the following characteristics: NLU Performance. When we talk about the quality of a chatbot, its NLU performance, i.e., its ability to understand messages, directly crosses into minds. We usually use metrics such as precision and recall in machine learning tasks to char- acterize their performance. However, those metrics are meant for binary classification problems. In AEdBOT case, the NLU component is concerned with a multi-class clas- sification problem. We could have measured the precision and recall for each intent, as we would have ended up with many measurements (number of intents 108 with almost 4263 examples).\\u003c/p\\u003e\\n\\u003cp\\u003eNote that it appears the \\u003cem\\u003eLoebner Prize Competition\\u0026nbsp;\\u003c/em\\u003e[53], an application of the Turing test, is often employed as a criterion to assess a chatbot. However, it has been argued this did not represent a good measurement of quality [54].\\u003c/p\\u003e\\n\\u003ch3\\u003e5.3.1 Performance of Intent Classification\\u003c/h3\\u003e\\n\\u003cp\\u003eIntent classification is the task of identifying the purpose or intention of a user\\u0026rsquo;s input. In Rasa, this is done using a deep learning model trained on a dataset of examples of the user input command. The trained model will be able to predict the intent of new, unseen input by matching it against the corresponding intent labels. A train- ing dataset must important in Rasa because it allows the model to understand the purpose or intention behind a user\\u0026rsquo;s input. In machine learning tasks, performance metrics for multi-class classification are essential tools for evaluating machine learn- ing models\\u0026rsquo; effectiveness in scenarios with more than two classes or categories. These metrics provide valuable insights into a model\\u0026rsquo;s ability to classify instances across multiple classes correctly. One commonly used metric is accuracy, which measures the proportion of correctly classified instances out of the total. However, in situations with imbalanced class distributions, accuracy can be misleading. Precision, Recall, and F1-score are valuable alternatives. Precision measures the ratio of true positive predictions to the total positive predictions for each class, providing information about the model\\u0026rsquo;s ability to minimize false positives. On the other hand, Recall measures the ratio of true positive predictions to the total actual positives, indicating the model\\u0026rsquo;s ability to capture all instances of a class. The F1-score combines precision and recall to provide a balanced measure of a model\\u0026rsquo;s performance. Here are some common per- formance metrics that we used for the performance evaluation of the Hybrid model on the test dataset as defined in Table 3; where True Positive is abbreviated as \\u003cem\\u003eTPi\\u003c/em\\u003e, \\u003cem\\u003eFPi\\u0026nbsp;\\u003c/em\\u003edenotes False Positive, \\u003cem\\u003eTNi\\u0026nbsp;\\u003c/em\\u003edenotes True Negative, and \\u003cem\\u003eFNi\\u0026nbsp;\\u003c/em\\u003edenotes False Nega- tive. These metrics collectively help data scientists and machine learning practitioners assess the strengths and weaknesses of their models and make informed decisions about model selection and optimization for multi-class classification tasks.The intent confusion matrix is shown in Fig. 18.\\u003c/p\\u003e\\n\\u003cp\\u003eUtilizing the \\u0026rdquo;rasa test\\u0026rdquo; [52] command inside the CLI gives the trained model to run the tests on an 80/20 ratio of training and testing datasets, respectively, against\\u003c/p\\u003e\\n\\u003cp\\u003e20 epochs. It assists us in envisioning our trained Deep Learning (DL)-based Hybrid Model in the form of a confusion matrix and histograms. Assuming each intent is mapped or predicted accurately. The matrix would have brought about all the slots being topped off diagonally, meaning the model has a true prediction for that specific intent. Comparative is the situation for our trained model, which shows that our model has accurately predicted each intent with no clutter. Also, the diagonal values accu- rately address the absolute number of tests it has predicted for that specific intent. The intent histogram in Fig. 19 allows us to visualize the training sample\\u0026rsquo;s confidence level against each intent. The true and false predictions are shown separately by blue\\u003c/p\\u003e\\n\\u003cp\\u003eand red bars, meaning that the original intents classified the predicted intents suc- cessfully based on the training data split into an 80/20 ratio of training and testing data sets, respectively. For our trained Hybrid model, we got mostly hits (blue bars) and a few misses (red bars) by considering the 108 intents from around 4263 training samples. Each example was predicted with a confidence score of roughly 0.98-0.99. Rasa utilizes the strategy of classifying the intent behind the message considering the confidence level rank, i.e., the intent that gets the most remarkable confidence level (on a scale of 0 to 1) is ranked first for that specific message. It is picked to be the predicted intent, where 0 is for the least certainty value, and 1 is for the most con- fidence value. The confidence score completely relies upon the kind of training data that we have given in the NLU dataset, i.e., the bigger the training dataset given to each intent, the more confident the model would have the ability to distinguish or classify the predicted intent from the client\\u0026rsquo;s message with more prominent accuracy. For the Hybrid model, we are getting a confidence score for a weighted and macro average between 0.98-0.99, which clearly expresses that the model could correctly identify or predict the intent of the user message. Macro is the arithmetic mean of the individual scores, while weighted includes the individual sample sizes. We have an imbalanced dataset but want to assign greater contributions to classes with more examples in the dataset, then the weighted average is preferred. This is because, in weighted averaging, the contribution of each class to the F1 average is weighted by its size. The weighted average results for multi-class classification are shown in Table 4.\\u003c/p\\u003e\\n\\u003ch2\\u003e5.4 Model Testing\\u003c/h2\\u003e\\n\\u003cp\\u003eWe selected 25 evaluators from varying backgrounds for Model testing and asked them to use AEdBOT for the admission and accommodation FAQ use case. The test examples results concerning the predicted intents are shown in Table 5.\\u003c/p\\u003e\\n\\u003cp\\u003eEach evaluator ran 2 test inquiries on AEdBOT. We made a test data set of 50 unseen inquiries based on the inquiries entered by the evaluators that did not belong to the training samples. Rasa gives us the provision to make a test.md file in the project file where we can place the test data set. Through this, we can see how a well-trained Deep Learning (DL)-based Hybrid model predicts the intents from our developed test set.\\u003c/p\\u003e\\n\\u003cp\\u003eThe Deep Learning (DL)-based Hybrid model predicted the 47 inquiries correctly out of 50 test examples. We performed two types of testing approaches at that moment. The first one is the end-to-end level testing for the test data set. In Rasa, \\u0026rdquo;end-to- end level testing\\u0026rdquo; typically refers to testing the entire conversation flow of a chatbot from start to finish. This might involve testing that the chatbot can correctly handle multiple turns of conversation and that it can correctly handle a variety of input from the user.\\u003c/p\\u003e\\n\\u003cp\\u003eThe second one is the Action level testing. In Rasa, Action-level testing refers to testing a chatbot\\u0026rsquo;s actions as part of its conversation flow. These actions might include making an API call, sending a message to the user, or storing data in a database. In Rasa, it is possible to write unit tests to test the behavior of specific actions to ensure that they are working correctly.\\u003c/p\\u003e\\n\\u003cp\\u003eWe tested the whole data set against the 20 epochs, and the Hybrid model correctly identified 96 stories (including 50 stories from the test set) out of 100. The action confusion matrix shown in Fig. 20, which is based on a Deep Learning (DL)-based Hybrid trained core model (comprising of stories and utterances), shows that 3 intents are mixed up with other intents were topped off in the non-diagonal areas, while the rest, 47 intents, were in the diagonal areas.\\u003c/p\\u003e\\n\\u003cp\\u003eIt is analyzed from the action confusion matrix that AEdBOT can classify the inquiries with 96% accuracy on the test data set. From the results, we can see that AEdBOT misunderstands about 0.04 of the user messages that are out of the scope of AEdBOT; it seems to misunderstand less often than that thanks to the fact that when the classifier misclassifies an utterance, it is usually not that far off. Indeed, upon misclassification, the chatbot often answers a question about the same topic. It is rare for AEdBOT not to understand an utterance. It is because, by default, Rasa considers an intent not to be matched when the confidence in the intent classification is smaller than 0.3. Considering that the classification problem is a multi-class problem, we get an average accuracy and error rate. In Rasa, action-level testing is a way to evaluate the performance of a trained model by testing it on a set of user input examples and comparing the predicted action sequence with the expected action sequence. This testing is often used to ensure a model can correctly predict the appropriate action to respond to a given input. The results of action-level testing can be considered the model\\u0026rsquo;s results because they provide a clear and concise evaluation of the model\\u0026rsquo;s ability to take the correct actions in response to user input. The average per-class Accuracy, Precision, Recall, and F1-Score are near what we got in model testing, as shown in Table 6.\\u003c/p\\u003e\\n\\u003ch2\\u003e5.5 Response Time\\u003c/h2\\u003e\\n\\u003cp\\u003eAnswering speed is another quality of AEdBOT that should be assessed. It is quite easy to ask the chatbot questions and record the time it takes to get a response. We can then use statistical tools to characterize those durations. The response times\\u003c/p\\u003e\\n\\u003cp\\u003ewe measure here are the sum of several servers\\u0026rsquo; response times and data handling. The information flows as follows: the user transmits a message to the web server that hosts the UI; this web server forwards it to a WSGI server which oversees Rasa; this server runs AEBOT\\u0026rsquo;s settings on the user message and selects the chatbot answer; the answer is sent back to the UI server which displays it for the user to see. We do not know what happens in the WSGI server; more than one server may be contacted to handle Rasa tasks. Nonetheless, we consider that part of the user message\\u0026rsquo;s handling time by characterizing those durations. We are interested in measuring the response time of Rasa rather than the cumulated response time of Rasa and our server since we could make specific experiments to test our server with much larger precision. To avoid recording the response time of our server, we can either take the relevant measurements directly from the server or create a new program that uses the API of Rasa. We settled for the second option for the sake of simplicity.\\u003c/p\\u003e\\n\\u003cp\\u003eResponse time can be seen as a continuous-time random variable. In our analysis, we will consider the response time between subsequent questions to be independent. It eases the analysis and should be close to reality since Rasa servers answer many differ- ent requests between the ones we send them. We could model the distribution of such a random variable in a few different ways: a log-normal distribution, a Pareto distribu- tion, and a Gamma distribution. All of them correspond to right-skewed probability distributions. Rather than choosing one of them a priori, we plotted the histogram of our measurements and fitted the different distributions to see which corresponds best, as illustrated in Fig. 21.\\u003c/p\\u003e\\n\\u003cp\\u003eWe used the sum of the distribution\\u0026rsquo;s squared errors (SSE) as a criterion to compare them. The distribution that fits our measurements best is the Gamma distribution. It has two parameters: its shape k and rate , both positive real numbers. Its probability density function is:\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cimg src=\\\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAtEAAADgCAYAAADFcDomAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAHY6SURBVHhe7b3fyx5JWv//zPcPcPHHUQghbPZEFPZE4xJmIp8BE9GTBYPuZEUGXGaNwoIYZhQDKmbBLDmSdcYNCoNIsqM58EBkZoQoyRo0eiIqHriREEOOFMF/YL796tzvZ97Pleru6r77/vXc1wvqvrur68dVV11VXV1d3f3KJw0HSZIkySR++Zd/+eDq1asHn/3sZxc+/XznO985ePjw4cHbb7+98BnH5z73uYPHjx8fXLx48eDDDz9c+K6Gb3/72wcnT54syvuf//mfB48ePTr40pe+tPBJkiTZL/6/xX+SJEkSYMD7yiuvHPzkT/7kwudlGEzWDqDh9u3bB5cuXVrsjee73/3uwY0bNxZ7q4UB8quvvnpw7ty5hc+n3Lx58+Ds2bOLvSRJkv0jB9FJkiQdMODlZt3rr7++8DkKM7UcY5DNYBvHwLuPjz/+uJ3BVXhmsrcd9CB0YfHee+8dnDlzZuGbJEmyf+QgOkmSpIN33323/T916lT7H7l//347q8yyCpZXMOBm5tb5xje+cTiw5v+tt95qZ3ivXLnSLssgD8JoUB0dyzfmhEF7KR9cacY9+lG+O3futLPhuRowSZJ9JgfRSZIkE2FW+e7du+0AtLQ+mcHxrVu3DgfWzOgyICf8+fPnD5eBsN6YAWnJsXxjThi0l/LBxTIgPzPtyOq8//77xSUeSZIk+0QOopMkSQZg0BhhKQezxAySmVV2NNv7zjvvtLPNms1lCQSz17DqB/I0u4ybMpvNWm/KxgCfhwsF/gzs44x7kiTJvpGD6CRJkh4YEJdmgxkMX7t2rZ2NxoHWNzPby5IHYBDNDC9LORhsc4ylH+yvYj20loY8ePCgnV3mHxkY/I7hwoULh+VyWM/NsSRJkr2n6WSTJEmSAs1AmEW/nzSD34XPC5pBaesvmkHxkX3A78yZM4u9F/ukB0pX+2NQXjhPH5Qu8glkj+GGILzLpjzxI71S3kmSJPtGvic6SZKkADO3zUCxfYCOdcxjll943Knvg54CM9DKk5nu1157rZVj7nXVSZIkSS7nSJIkKcJSCwagDEifPn268K2Dhw1hmfdBj4U12sA6bAbTb775ZrucIwfQSZIkqyEH0UmSJAHWFX/00UfFNcE13Lt3r133POYjLHPB+mcfPDMrrgF2kiRJMh85iE6SJDFYBsFs7p07dw4HwU+ePGn/ax4EZNDKAJyZYMKPfaBvKnqDBl8SFJSF2fT8NHeSJMn85CA6SZJkgdYR8xYNH3iePn26XSJRw/Pnz9v/N95448i7oFeNPoLCa/SQFXf9+vV2VjpJkiSZn3ywMEmSpWHw+c1vfvPggw8+OPje7/3egz/90z89+Kmf+qn22P/8z/8c/PEf/3G7vOHzn/9861fiW9/61sFnPvOZnDVNkiRZI9wt4zmOX/zFXzz4/u///oVvN/T3//Zv/3bw1a9+deGzv+RMdJIkS/Gbv/mb7ezt//t//6+d9WT5wH/913+1x/7qr/7q4Fd+5VfazrlvAA3qkPkwCQPvJEmSZLXwvASTHDxAHQfQ9MNMbtAnc2dLH43irtcXvvCF7KsbciY6SZLJ0AGzbOEP//APX5qV4BgfI/mHf/iHztkNOmAG2f7gG532H/3RH7UfKKmZFUmSJEnG88///M8HP/MzP1N8g4+OAcvCzp49+9LSNCZJfuu3fuvgH//xHxc++0fORCdJMglu6fHg3I/8yI+8NIDm9iDH6Hy7BsJ0wD/2Yz/WzmA7pPW///u/7eA6SZIkWQ0MkumjIwyg6Zf5MimTICyxKz3bwZI9+mreZrSv5CA6SZJRMGvMrT2WcNCB/tM//dNLD93xhgiO/cRP/MTC51MYfH/uc587+Omf/un2dWy/9Eu/dHibULz11lvt+mrCJkmSJPNCP07/ywyzw91BBtD4v/vuu4N3A3/u537u4Pd+7/cWe/tHDqKTJBkFsxKsAuMNFqD3Egs6Yd4QwdroUgfMejpuH/IAIh0wcVm64fzwD/9w+3/79u32vwQDbL2FosblgDxJkuQFf/EXf9H+xxlm1kczAYL/j/7oj7Z95/d93/d1zjbzNVfCc2dxH8lBdJIkk2Amg6UcsRP+93//9/af2eYuiEvH+wu/8AsLn6N8z/d8T/vft9aOwTgD8FpH+CRJkuSg/ZAUb0yK/Pmf/3n7z+s5mdz47//+73bihHfn8xB55Id+6Ifa/3/9139t//eNHEQnSTIaZh0YBH/lK19Z+IyDWRBmovUavC5YKtLF3DPRv/3bv12Ml266+/3f//2FdpMk2Sbov0vQ5+oDTdxJxLGsAz/uMCZHyUF0kiSj+cu//Mv2v7TmeQgeOmS9s5aD9MFMdxdzz0QziC7FSzfdfe1rX1toN0mSbYJJjNLr6Rgsl+DOYtfAe5/JQXSSJKPpWsoBP/iDP9j+l16bBH/913/d/v/sz/5s+98Ha/KSJEmSeeHNG6U7ffjznAuTHZHS8o9nz561/3qOZd/IQXSSJKPQeuZf+7VfW/gchdt/zDJ3dcS8A5oBOB9f6XpY5eHDh+3/5cuX2/8kSZJkPr74xS+2/7GP/t3f/d12NpqHvnWMd/c/evTo4MaNG+2+8/Tp06qleceVHEQnSTIKrWfuW8px9erVNgwdb4QBOI6OmS8ZluDhFjrxfBgwSZJkfljzzGBZdwYFkyC8G5pjOJ5t+Ju/+ZvWlb46S1/967/+64u9/SO/WJgkSTWsofuBH/iBdkaCz8T2wYz10BcLS+grWPnFwiRJktXR98XCGnhY+1d/9Vfzi4VJkiQ18A5RlmIMDaCBmY7333//4Mtf/nLxAZYSdOp/8id/kgNog1uqLHtxHbKvN2DED9VwDD0mSZL0wcwyb9ApvbpuCPqjb37zmy+943/fyJnoJEk68dnku3fvHvzGb/xG+zBK6YHCLuhsuWXIA4el24GCGej/+7//awffyQvQ/7/8y78cfP3rX1/4vIAZIJa6MIi+c+fOSzrjk+u85zV1mSTJEFyo00dfunSpavKCi/S///u/P/jqV7+68NlfchCdJEknDGx//ud/vt1mQMZDJzlDvB6GbrUywH7jjTfaV8mV4M0mv/M7v7O3D/wkSZKsmhxEJ0mSbCG8l/X69euds8ks4+COAB9CKMEgmxnp//iP/8gLnyRJkhWQa6KTJEnWCINjrWeWY7DrMADmFYFnz55d+ByF268fffTRwenTp19KS/D2FN6CwjKcJEmSZH5yEJ0kSbImGCx//PHHBw8ePGg/XMCNQFycTeY1gtC19lwDYx7wJC1QWoLZZ14zyHu5kyRJkvnJ5RxJkiRrghlkBsb6yEzXW06+7/u+r52F7nryndlsvizGTPS9e/c6w7Hkgxnr7OaTJEnmJ2eikyRJ1oRmlhn4njt3rt0uwTKMLngzB0s93nvvvYN33nln718xlSRJsilyEJ0kSbJmeONG39cYWYbR9W7t27dvt/+aXWZQnSRJkqyfHEQnSZKsEZZ0DMFSDd7HXYIZaL4YCXyWV4PqEgzW+ThOkiRJMj85iE6SJFkjz58/b9c0Q3wrh/jiF7/Y/scBN2/tAD6KAAy2eVARlKbDso+vfOUri70kSZJkTnIQnSRJskZOnDjRPuzH6+j4qmAJ3g3NLDNfEXP4jDpv9dDa6suXL7cDZdLSYFrwoRyWhWjAnSRJksxLvp0jSZJkCxn6YuEQvJnja1/7Wn6xMEmSZEXkIDpJkmRLYTb57/7u7w6+/vWvL3zq+Na3vnXwmc98pvNrh0mSJMny5CA6SZJki2FdNMs6WJZR8/luBtBf+MIXDj7/+c8vfJIkSZJVkIPoJEmSJEmSJBlJPliYJEmSJEmSJCPJQXSSJEmSJEmSjCQH0UmSJEmSJEkykr0bRPPaJ1yy3/CpZN6tu8ufTObDG5Sh5gt4yWbYxf6GD8CUPtyyTr7xjW+0tr3PzKkD6rPrwz7HnXW2wU30ydQteeLIf50y5DmogQcLdwFEdffgwYNPrly5csTv4sWLi9CfHDkmf/5juG0E2ZPVge3INtiek1XWnad9586dwzI8fvx44Ztskl1tty4329jUmTNnFj7r58aNG4e2va8sqwOvU+qSdLbdPnex/VBP6n830SdTtzqHqe2WIAzyOciI/FPJc9ALdmYmupH1oKnwdrupsINXX3314N133z1ojKP149iHH37YbgPHmko+aAzr0J9/vva17bz33nuLrWQVYDuymznhajx+NW5O3C54/y/2nWwHq677VcGdGJ9Fot+kz9wkb7/99mFfv68sowNmBx0+1sPXL7eZXW0/t27dWmytv0+m7TIW4guoQNtlnFTi9u3bi61PuXv37mJrGnkOesFOLeegY4FHjx61/8CAiIGxG7O4du3awdWrVxd7u8G+3nI7Dqyy7tIutptdrZ8333xzsZUcF954443F1u6wi+2H5TYMYjfFs2fPFlv9MNiOE3NctLzzzjuLvWQZdm5NNAPm999/f7H3gs9+9rOtMfv6VoyEtUIcK8FxrSMqNWD8uo6zT9pKw9dbaY0Qbuy6QtKRscd0a/C8cZF4PM5Y6Lh0pzTQK9v8I5PCgOvR/YXC0+Hg/Dh5yF/xo0ygNOQiHMe5LKU69XxKHZDKiYNSGl1Qlo8++qi1w5j/kN6HoGxDduE6igzV0RDbXv+kg769/kjTUR0QFhnicZehpN8++uo+yqJ8gHhs8w9uJ1E+GNL1WEgDmZG9K0/lVdKJy6syjMHLw3ZXe4t6cqJtuE6UvtttDCN0TOHH4vaDi3aMH2FcZzEMSM5l9An0F6U8+toIUAc63lUffSA/TvnEtiDn5WO7pu+cohMgrpdVdQWkOSVtZNQglFn+Uny3uYjqSq5kk32Qny6WyN/TQDbJg/5ee+21dpvwhPuzP/uzwzsTlAE/t3npBFeyUdLX8aShXdSxQ2gdTtPgFj6frnX2NVWs9SFshLCsI8JBV3rEB/w5rn2tOyK+wvEPvg0KMwbSIP2xEE9lgpi3yim0r3JpH0c87bOWquSPXnRMxDVZpK30VS7pmW25GIZ0BXl6vbLv6Wif8qr8LqMg7aifmBf7gm3PtwbCex4gWYT2VeZapBtHaZGnyhHTHqqjIRQft431r3rkn2OgdARhVS9se55A+p4Hx32/BsLHuicdz0ty4QiLLJLHyxnlh2XrsQvyld6EyqL0lbfXCzJ6PNd/LdKBtj2+dBD1hP2JqHOOS4eSWU6ykwf7QulKDuWr/Rq8TQB5RLlw+Em+KDsQT8clF24sno/ADycdl2yMY/iD8td+DdIt/yqL5HCZVDd9dQnkLXnB5a+FfLwcKjeO9Cinyury1BDrHeRH2rI59l2PKr+g7L5fSyl/peW67CoffrF+8ZPcSl/7QPpeBxzHuQz7xvia2wK88qlgtqlYN5yuxhbDycBkKOz7cYhxZKjRcAjjfmqwYwxMccZCHG8kpOMysy2dCcrleanRxMamRh/jowcPq3DyIzxhBNuui1Ka+CmO0vM4bHsYqKlT3wel7X6eF/+eRw2EdzmgRu81kEaMo/ryMpC2y812Xx3VoDjbWv/omHKLGE9xBMc9Xz8GpB39hiBOrHsgHc+LbZcNiOflUb16GI4vW48l0JvrDkrlj3khs8tXKtcQpEE5hMvRpSfXJXJ6nFgW6cjziHojPU+T/DjufkOovoRkd9gv1bGI+1BKp4ZoT4BfSTfSL/+EcQgf/YYgTknmWHeE8X3kjXmx7/UvfbhfDTGvUjpRvhpUZ56O/KJde32wTTgRbbKWUv4QdcnxUvr4eZk5Hu2GMPKLNgNdMuwTO/mKu8YoD9dAP3z48ODSpUvt2r6mItvbEtzW6FrGAaVbL7q9z3pr0tHtCpxuNTmNkR7JgzwJg7/i6XbP8+fP2/9Vods4J0+ebP+B9eM8UAKS7dSpU+2+0HpIxRdnz55dbB3l3Llzi60X8CCIbhHhdNvo6dOn7T/hdVuRW0A8+NBXL0DdSh7qFjwO2x5GDNUpsH6+D9JV/QHyLsNYvU9FD5YIT3eojsawzfVfSlftjmPeLrEDPV+h25g6htPSmbnqp8SQHiJz1mMN6CuivNDLHH3dhQsX2nIQn3rwB8NF1NOTJ08WWy/aJ3GQhzTop4dQH6Cy3Lt37/CWNk7l9nyG4AGr5lzabtMPSReR06dPL7Y+RTZ2//791q5XSV8bqT3v1VAqB+ci2pyWZwwxl4310Wdby7KuPnlOsEH113Ig2emPqY+xfddxZycH0T5gphOkUjXwo6J56vTy5cvt/hQwFDrF6GpArhhvaPC2yzRXoi+VVwMUys0++lTj9M5k2+CETHkAmRn4HQf66mhZdqX+OYnfWLztADnietBYBty2nSxWWY9TWLavYxD8YPGWHAYSrCEdA7ZEXTIYJ++pg1DsIpZj7AW01pEyQJKd7RrLnPeG0OCZgdqYNI/r+XTb2rK4cuXKS3KVLm6TT9nJQTRX/nD9+vWD119/vd3mhEcnygw1A+upDY3ZXBpuPNkPDah0wvU3hwCdhy/aXwXKWzN3grwph47HBzK58qXjnDpY4MRBp+iQnwYo/LPPAIbGSF43b95sj3VBWNWpZnDjww0epgbN0A/VA3WsWSU6OQZ+0Q7GsCq9j2GojpZhV+pfMnGSQg4GOZrV0oxRtA3Jvi2ssh7HIrtdtq+jvelCi8E0M59j4jN4pi6xh1pUp7qrQlk4X0TGXEATlvqhHFPbNLPUNTPpq2Lqea8G0mTmlTquvTiZy8a2kW1qyw42yEVgRDZAf1yykX1nJwfRwICZTsdvMWuGepnBCZ06adNBOzWGQ4dOZ+FhGUCNGdBr4EAaYzowDQy8g6GhShcMCtGXN1TCcyEyFV4hyEDTBzncBfA6cT3SecRbmn7rE9moP12RM6ClLlyn5OVhakD/DOB02wz03kz8JL+XRQPvMbZE2ZANVHdz6X2qXdTU0VR2pf7B2wa6xB6A+mX2xW0DdIerllLdz8mq6pEyahA6tr9Ztq/jhK12oYuZeBu8D/Ttt+C7BqHe1igjNiU5eQVqbJ/o+Pz584u9YdCBD+RLg/IhWJIIPhuvJYtj7+DQzhS+tk6XOe8NoeUXWl5XGgSX2s8cNrZKdI6gfNhPra5W2SeXUD/G5A26V770gbQf5EZ++lTqwG2G8OqzNXnpNqIJItIq1ete0Fw97yTN4OSlBxEaA2gXubMAvgThOY5jWwvl5ZpGuwj56QMScqLLX5CGH++SpQ/FpYxj4AGAvrxJr+t4l9wxTpSpT4ds+3F05yi8juNKuM5jnY+pU/eXrlwH+LkO/VgNsr8Yt0/vY1B80ivVV9SF4JiHdZ0Mse3173EJV5KXeiEfTwc/J7adeHyIWPe+j4u2hYthkC/qg7TEkK7xI48xeJrI4zqSn+97+n2y1kBanoZsq0ZPEPP3eKCyeTjFdfrKWEOsF5fjgw8+eOlYl9660hmLtwHQNo7yl9qIiPU/hq7+BzxdthVW9eF14Ha0jI2V6tXrBtdlW7WoHMhZkrVLJ7GuiTsGTxentLvqT+X28klelws8ftRHSaf847+vvMJPo4QkWTvMsDQNefSsYnI8yPqfH2aRxq7nPa4wM8YdhmbAsjWzl0mSHC92djlHkiRJcpQ5br8nSZIkdeQgOkmSZMdhTSMz+/kkfZIkyfrI5RzJ2mG27MziwS64cuVK3oLeI7L+k1XDw1M8lCbu3Llz+GBUkiTJXOQgOkmSJEmSJElGkss5kiRJkiRJkmQkOYhOkiRJkiRJkpHkIDpJkiRJkiRJRrKSQTQPdZS+soSffxlqTvjSk3/tqUuGuVh1+sl6yfrcHagj6oo62ySxz+mCL8jVfjkuSaawLW1iH0DPqxrHjEFv5Ek2y+yDaBqxPxXt8AzjmA8rjDnx8Gonvd6pT4YpkF78pCVPelOeMZ8G3lUYKIxprOrQ5YY6dupZYTcxiJ1iL8SRzGPsVLrEDX0mlTwYgPVBGqQ1Bde7HPqP9Sf/bYE2R9tb99sWvJ6px67PTEf4JHS+fSShrQ71hbV4WrRNf9tNsjroC7cFxlL5XojNM/sgmhMbrxNaFjqGjz/+eLE3jrlkEPo+/L6hwVTtYAHUofOVMBr448eP2wFq6eSh9N9777027KYuSsbaC2WhTJKZctQMpBl4aQCIXviaWtdAmjRrBvakMRUGdsgCfDlQ+schX/TfZ6gn6kRwwX7x4sXFXpIMc/v27cXW8nha3l6T1aL+MknE1q6JHjO7t0oYMI0ZRB4nNOBjIFXLzZs328GFPrNLGsS/du1auy802OYdwbvWMVEWH3Szz4WAD7IiDMKwI81IohfKfv369XY/cuHChcFBGoPyVQzkVDecmPOT3C948803F1tJMh7aP33EHMyZVpIkyzHbINpvDZeg4XOMf8EAFT9uWXPi1joj9hlwcBLnuAbUvgZI8QT7XeuUOCYnlDdOMulWu9IlPc0GMuPn8ZGpdKtdaeJIL4I/6Xr+bNfgcZSOgzzIpXB+XGUrxZsT6jHOWp47d66tSx9kapA45TY38qssuKg/lXFIx0M2WwJboSxnz55d+By0FwwMOh89erTweRlmjuKA9/Lly8ULNGRncB316FCe119/vXVzQt7UTd/ss/RKfbpdRTiuYzi1Y1B/gFM78rQU1utQeP5Ctu/pluzc67x0vARhqXPqqhTPy+llBB1z+/PwbMc4Q3h8HHi51O+4LogjpvYFpfoRnlfpOHFxLvuYcpfK51AOHcchj6PjgK2wLbtzG4v6ULmQW/FwwuMqT+nJ09cdI84nHPP6UHxPo4uhtMB1UUrPy+F22QdhSdfLq7glvThuN1G/bg+4WLfUu/TYFaYPT1vpiNq0ZQO4Wn25veKEdOW27/rBeX0qb/4VTsfZj/J63eNcXukavzE2EusMXObS8b2iOVkuzZUrVz5pBgiLvXZasXXNiafdf/DgwaEf28CxZuBxuM2xGzdutPtAmjoOHFMa5HXnzp3D4/L3+ByXv2Db05Rckglivl1hYlqAH/mKGIZ9+ZEGxPy6iHJIBkEa7KMbjvEvfXBMcikdl3MI6b4G8o3libKrbryOcLKXPhRHYWN+Sgu/Ph1LR6JWBuUf8fxKqF6cqBdgu09uQEbJPqZuuiC+0vE2VMLrDNkke4yrNq3j2nedA36kKWL9guvN81cY5GCf/z69RH0qHU+rC9KNsrNPekozyq4y47yMhHfZY7q1ENd1XrKnaHPEkSwK77J14XnFeCqn9qV7lZHysU8aOCCsh+mD+CqH8vJys610QflHeSQDaSgdT1vhhGSUk6xeDujSu4dRfpIJ5Kd0lb/2u+hLizxJByhbtC3CSE6Vz+UuQRilLV2pfDjJSxjlLTguOaPdgMeJ8pCH8uUflMaQzODyxrxr05afyqh4sZwluupTMgHbyCAIH2XGUY/SD+mxL3/hx4FjShs/pYXfkI1IT7FOgPg6Lhm1v48sd/ZtiEYGsTJB4VQZqlSBvxtmNC6QUSoNB3+PXyND3IeYbykMxHDkHeWVDG5g7KuRgMIMITlUnlL5yD/KQLzYSAgT/fqQ3muQXF4Xii9ZKT/7LgMy1eQR9VWSjf0+HUddQkmfJWJaAvk9zwhljcdLtuU6IXysT/AwY+qmC+K7o4x9SAdRbi8f2y4nSNa+eNJJtB+nVFfRppVODBPTLem3BGnH8rDv8ckrlk9+rlPiRB1MIepYebk+vbzkGfOKeitRsnnXZSy3dB/LOKSrEkqL8IK0vIwcjzZLXp4fssZ0OO7plOyqz6+vvKRbKq/LiUxePwrjfiVKaZXisu0yEN7LC8SJfiViuJJeoj7Rx5C9eTlK5SI9/JwYpgvycn0Qz/dr0o5pQEynj5ge264zZHB9sO37sq1SfjEsaXt5iOP7NTYS98HLUFOn+8bSyzkePnzY3sruu/VcgvDE0y0BbonXrr/UetuxnDhxov1/9uxZ+z8n9+7de+l2kd4e8PTp0/ZfnD59erH1KfGWXIQyN/XV6o1bKV0PnXEr3qF+dBtarmlM7RsDVgFlbhriwTvvvHOYH9vRRppGd/g2FdDDm0O3y0gfPQD6Ju0SfTqearOrhlt8Q8tbuHUW15fPAXWGXtELtlVzi07tSbgN81Bw1O+lS5faf29/rDX29Z0se2lOLAe3bt1q97EHlgPVUKrP58+fL7Ze8OTJk8XWwcGpU6fatrAMsc3DUP9CG9XyMG6lejsYg5YDSe93795tdSd9kjZlFFP7Avov7MIhjvpr9I7t0Edhw10Pu07RlY573aIvtRPdij558mT7L956661i3c7R5rWUa9nzCOcM7yelY7fRsXh9g+vg/v37rW14/cPQuUeU+tQ+auwNu6FPp7+JNiZK/vGcWkI2Sh+iskb60kYvyFvb/5SgPfqLCUjbbRA7xp7JCxnRV4kaGWrPjX02gu3Fdqo6gnWPJ3aBjT5YiOI5eQOVsfdrawbQOqT4YNsQDFhpCO5WafR69Y4cdD1AJ+KArA8aOXpgoCb7WRc6WccTDx3J+fPnF3svQ8cZ4+gkrItCTnB06uqc2CddtnVxweBSAzCcOkq252g/2AX2Qrq+bm9VqHOmfOiHEzV6pNzsc+KfetEcwQZ9wM7JrevEvUo4cT548KDdpi7jusZa0AvyM3gGToAMrIHBJSc8X7sPq+oLtAaUulPZkmF08epuynMitTCoi/lNvYirYcje6LOwG/DB3Bxo8EwfQr6bwC90cXEAix8y6jkU9LUM6zg3rns8se0sPYjWbA7GMAbC04A04KLCu66c5kKzUjpxjxm4DcEgSY1FaHuZK1mBrjBUdDVmUEH9lK5u1zFAAho1jU46B060UVcinvQjyE2a6GHqrNJUmwUNXPwhQs2G9cleeoiQzt07TXVIcpzwyItt6U82IKeOku3aOzlDcFLVjOZUO6GOfMAKan+lgR26YDDIbLXKqsHhXJCuX6RQH5vo/NEpdkSdMeBEDtnQWDj5MqPJgAEbk30yo8+g2tvI1L6gq73ooo28qWvK4+18DnTRGvWjvNWHx9fHUfZlByRdRDueeh6hbqi7yKr6Zi5QS6+NXVV+Q/ZGnXLOx7bm6rsEtsodNdrX1IsStR0uRqei9shbq+jPYvug/dKHz9EPoddlz43YSKnO1N42PZ7YRpYeRMsofBmBbl9gPH0nBxqQjlM5hBdUpq5Ml6kgPxEz61O6OlMjQRbN/mGMoA6SWUNOFl3lUUN1PcTXvS0DJwXpAxh01KCBg892TRk8jgU9MVBBj3GmQzJ5vbLNwG2o8SO7dzilk9AQy9gsMKPpy2nYx676ZMcGsAWVmXJga6ucdVoG5EIXyKi2MAaVy+0OPZXqWEs6fNBHOPqHvtn9sSAL9cxJRq4W5JLdLXvCYCATB4FTB2K6OKNc6mdYyoA+4+33qX2B4nl78Xh++xuWGXRE1G7iEhFsBagX2h7lpX8Gta05lz3RlwvstWTHfecRhUVXHEfWq1evtnUnWwD8h2y+lFYNDFSRye2X+NFO5mLI3nQnThclfW83GovSVB5DfXoX6ocUX7rGr7YfUHssQX3IlqE0QK0F3S57btTFjNcZZWd8BlP7kGNNcyJZmsYQOBsdusbw2n/8QQve5bS4vun82gXp8ld48DRZzK40Y1gPhyOcIJ4fYz+CDDqOLMRvjGRx9AWSUWm7zDjH/V2WkpyeN64knyjF13bUL/JFKJOOx/L14eni9ICBULqqD5errzzgMrmu+oh16vl98MEHLx3r0nGXPlWOITxdtiM6FvEyD+WFTEN1JTkiXfk7rjs5yeRyylH3JX16WJc36rikJ1A4txfVc9RRKX/fx/ZL/Q2UyuvH+/B8kCmWOcqBnCU/iHbp+atvqZFJxPAlfTpd9TWE4pTi+THXM+Ws0dUQngYu4vrEud3Eepd+5NB5jC/doVf2/TjpRfw46REm6khyeP8cZSmlXcLTKulTx+VcH+7vspQoyVfSle/HNGP9O37MZWabdLSPG6srj8+28mJ7TNoeVjKig1qUdqk9Rl26DmR7vi9KOo314Gn9wR/8wZFjfTYSdUH5I6X895VX+GkUkSRJcqxhxq9025hZlVWuC50Cs1zbepdin2DmkbtOzcDipZnnJEmSjT5YmCRJsg64Jem3TZ25P1izLKu8xZ4kSZLMRw6ikyQ59rBelXWJcW0ks9Na77cNMCvO2vG5H7RKkiRJ5ieXcyRJkiRJkiTJSHImOkmSJEmSJElGkoPoJEmSJEmSJBlJDqKTJEmSJEmSZCRrH0TzyiB9MWzKRxySZN3w8Bn2OhVeV3ZcbH2Z9qu4e/9y/i2hZNfUDX76qESyGXjA1D9oAatuP6TtH34pyTAnq04/SdYCDxauC14Wrhd364XeY15anhxf/OX224S/DL8WL4teaH8cXkg/tv1yTC/w9w8HyC/ZHFPseh3ED0Co3Wxr/7AK9HGP0kcuxjJVb3PKANtYf+rD3Lmc9FmlD6QkibPWmWg+TasX1vPf5J+vckra145t6+wk9tkMOBZ7w1AOPuss+GBG0zEv9nabse331q1bi60Xn4ttTkqLvWTTjLXrVaMZcF5DiF3J0Zbw9zZ13OHDP83gdbE3nWXuJswlg+j67PUmUR+mdsC2f+CI/i5JhljrINq/654k4s0331xs7T4s3TiujGm/3BZ+/PjxYi9J+rlw4cLBmTNn2oGMw0AHO0pbGg9fWtwGdrFP5ALko48+WuwlSTezDaJZ28SMgZyjtVx0hFyRsl3TsJih9DR9vZbSxOkDCpLB12r6Gj/y9PBdEJ+wnr/y9nKWZk+1zhBXWu9F2jqOcyQrjm3pyNNUnkrH85C8/EtOl1FpKIxQvujI81IY7XetgXWdeB1Bny6BfeyCDsuPeXiosRdQHcuNKaejMnWVuQRhKQflIW5JZuVZsg23aRz7ER1D5pi+yifndT+E1yHOkVy17Zdj77zzTrvNwKikQ88vMlc5SjLi15c2x5FXMqieuvzB661UVh2LdSYbJ03iKVxEecrFcun4HHatspCmIA7O5YgyAH46HvuBPsgT2+IjMyUYSJfu5nhZcaTjSA6vH4Xp0zfHKIvqB6fySH8415HQMVyXjrtQvJJuAf+YpnTOP2VDZtUTqL16uRUWf5VLZS3ZjOvPZYu25Pbh+tIsNP7sCw8nXOd9x/lX2XGlupgKeeoC5LXXXmvTFyozYXA1+RIH5/qRHr0MJd2jWx1XHQqPG+MjG36gNKLtJDPRXPkvDeumfC2R1lM1HePC5wWsb6tdG8VaJE9D+x5ffr5uieNaR0dcjuOQSesy+9Y5EZcw/BMHtH6QfcUtlYV99yOO7xNfaUo20hakqfJ6/iDZXaeennQhOT288lJclceP4chT8iht/EHhSNchjvwkg/Y5pnQlp/J2lJfjYdiOui4R0/Z8h8rpsK/8PF4NxCN9R35KQ3pyO1R9Ce1LTkAuxWHb81Gagjx9vw8vL7BPXMrukJ+H60PyexryIx2Vg30v4zLlIF2lpXRki0BalE0obcmifdIhHGlJNyV/8G1QGMG20meb4yBdyElPHFcYkP0pDe0rD+3jiKfyc9zlAPYpC3g84TJJHuIo7Si7wgDputxKJ4YroTyGwjlRBtIo5Y+fyiwZcYpHGOkMOKZ40h/H8WNf9eDpCvcr2V8fHlZlU/5AupJLEE75KY7kgygj+0oXf/4pm2SN8TmOX8zD9cUxlwliGOnPYT+GkxyqG+0rf5eTPKWvWM6xlORTXq4Pwklexam1bWSUnrx+VQa2cY7nH+s3ykzaiq9jOPyREce+8kvmo+7s1IOMzY1JFRYNe4yxYxAxLPueV8nQCSNjhS5Z+nCDhFI+XUbvSF4RyxT1wX7MQ6gRuZ6jDJJTDV2w737SifziPrDtskMMQ35RB1F3XbrsKwd4GP5r6g/ZyE8Qx/e7yulhpGenpIsuYp6AX4zPvndoxHG5AJ14PPY9juuMPPyY9Ox+JUr1IT1FnSNj9OtCevR05UeegjJ4mlPLobQd16nK5HkDflGP+Lnc0OVPHu4nW5FfX51JZo8vP8lJvh4HlIfCEJ99tx+23Q5L+lE6Tkkm8ve0lJ/r0nUNMf8+CBfzFDompzxieUplYd9tq1Q20vcwgJ/rXDbo5eV4rBfiuQ6I4/tdlOq4lD7hyENQHg9DXrFOSmXzNERtGaMMcR9iuUt1AzEceUV52Sec6kxyUnZBmCjnGErylfRBOJeP7ZLNRpDNdaT2E8vgYTjmeQFx5IcsffWgMkVbd30n87D0co6HDx+2/3rgCNhuDGepWyzcFj99+vRi7wWXL19u/58/f97+j+H8+fOLrTq8PDXoVorfXtFtLOmBhzV4cEG3pBoDb/0F6wJ1+4gwhJ/CuXPnFlsvuHfvXnt7XXI1jan1f/LkSfsvTp06tdiqg7rXMgw5yhTXzpZ0OVSH2A9ykib4Ax9d8LCU8iae9B+J5fR6uH///qwP1Ajp3Hn69Gn7j30gQ5RLa8VlP6+//np7m5GycWvP7YMHr3QMhx2B8uhiVe23jxMnTiy2XuD5TC0Hx6OOsQU9+Pjo0aP2P+bdnHxeslfSKdls9Fe9yU5xWsYi++6rsxJnz55t/589e9b+o48oy6VLl9p/hRGrtOvSreCYv/cnyOL599F3m5m6aQYy7Tb/qk8eVG3OX+028aX3SDyH1FKq/yFkb1r+UAt1jJ2MBVtR/8tyAfJ+9dVXF0e74TwzFfRZW69jGXPOP3ny5GLrBbENrwLOq76cjXNSrZ2MXUpBe1VecqC+svZcF+WL5/xkedb+nujjDh17dDJkDZ5Z+4d/POnTKHXCYPDg68eWpbkCfUmumoHpEJyYY7pzdGgMNpqr8XYbPWkN2RDqcJCDAdJxgo5TJzAGZrFjRl+xLjTo2CV2rRzUSZRXg5mhOjsu0Kf5iZw3G8T+rQsNIO/evdv+14IuaesMQunfNo0GzwyAsIFVw3mFfOiDNZj2dbHJvNCm0Td2rQHuqiYagPOX+hM5vwgnfxz+x+1ct0ssPYjW7Edc9M5AasrVtcBQ/RVZoJkPnaDirNImkSyxE/OHDxgYM0Domo1ioKiGymCajlHpxSvvMdDZMhsdqR2YdkHdI2Nk2XSBNDTbhM7otIY6LC461PFMhZmQUplWiS6y4iuVNLuq4+hEJ04GZji1OwYUnLwd9IX99bGq9juVZcqBPqKNKJ7aTxyoEX7qzJzqRbPcAl2q3fbVWQnNuGlGGn3EWaYYpoZV2zVtFVvViZ28ai+mdYHEbPJQGxfoFd2gV9XDJkFuLpLot8dOTlCOUv88hOyM8wl6YDDd9XDmXDCT6QO2qTP9JWrO+ZtE53LsGn0j782bNxdH5wW9cnEY0bl1jnNdMg9LD6LpPGm8dCDqAGncnCyWmT1iQEEaMhq4du1accZBt6TpUDjhEI+OaZ3QkWPUuv0s6Bw5Jt3EW/gOjUYnfQ3K44WCTtjomBMVbmjG+urVq204H4gQf+wSl4hOnJ6/yjkG9KMTruqbetRgQwOgoZMlaXj+pU5oCN0q9zKpY6+ZeaDzU7267Q7BhUKsIwYVflIkb6UpXUg3tA3XGTBgjEt7Iqtqv5KLAZ9OPjUsUw5s0QfEnicnYdonOtUAl390TvuYCv2R6w7ou3TS76sz4SdilvAgp8JqQOb2iE14mBqWteshSJdyc1KXG4PaDHXodd8F8vogfcogdE50YaNBn2ysBurc2z5lYx83dB7z8w324DZBXNXrmL7IQQalQb3QNmN7oe5iPrQzlUcX6oTpk2PMOX/V6LxLfVJu1af3L+h3zosIR3ewXBfIoPzmONclM9F0drPQnIjpNVvXdIQL3xc0A4TDY3L4DdFc1Q/GaRrY4XFkaE4uh/nH+IQdwsOTXpSdfdLXfiwr+Xv4piEsjhw9RjzpTGlw3MsTy+vx2SY+DkpyOsjhx4kPJR15PrgYRnFFlz48TpcuwdOXvsjD5SDMEDF9j//BBx8cOVYqp/IulZf/GlzPpKM6luuqB4jyxzIT1sNEe45y19i7cDmjTUe5cKq7PmQXyIHz+MjaZTfLlMPjxXJArHMn1pXo8helsom+OpO/x3d7ENFmPI2SrmIZa+26VI5YR111E/OUq7ETpysd1yl0lQUX2zrHSmXzfeoYol+0ffa77NbthG2FU9p9uHzEI47H87RxgCxeDpcFXHbQNs7TLpVRuF5j+kLlVFz+ZRfCj/f1gbFeXJaSnCW9KA0vY4koB85lUdryo0wu31D64LphO5aPNEtlEO7v+UVdeD19+ctfPnIslrNG7qSeV/hpFJskSZKsEWa4mMVuTnKjZpW3EWYdS3cumKEeepgySZJkV8kHC5MkSZLJcJu566n/TayrT5IkWRc5iE6SJEkmwzMprJWN64CZndZ62CRJkuNIDqKTJEnWDANMlnLAmTNnRj2Itm2wjOPOnTvtQ256OwcOeOAzSZLkuLK1g2jWC6oz9qdQk2TV+NPYqwbblp2Tb1KGQecy/cC26Zk3Y/A4ilzXK7yivF6O6BzeHKC3I6wDBsteHhxLPJBr6puSKPeYMvAmA3/7iCD/Lj0BeUyVEcjX0y85lYMyyS/Pa5uj7y0hm0ZtXGCbJbtOtoOtHUTTKTO7kSTrJr6veZXwQBkPliX9xPfHjmWb9MwFWs2gjRMnfaDP5vI6vDhYJcyVK0c/tsArsNDZJi8YeD1flGsMvC+8djkI5WTwER9iRIfoWrq6cePGS7pnJv2tt95aaqBCHSgP7ixQbu1zTGvG87y2eWh/23wBwys9sSFBW86Hc7eXXM6RJAYnY96NmmwPzOIdpwsN1hDzwB2DOZ+ldDT4jcsh9Blkh4u+Ln8tGdlF9O7bmplbyln6yAlt2R9u5H3jJVtiIM1gZcpFB+/ujfXk9B1L1g/v5d5m8tPcu8Usg2iu7NTR4eKtEjomdYJc7StcCb81VkO8lSZ0gnJZJIdcPHmxrzT49xkL0okzGHOW24/HdCDmVYvH6ZKff+XvJyqXqXSiL6Gw0qWflOSHI+2I8lN8lwWdcNz1UNIT6DiO8BHi6bjLQZ4adGh9p/C6BNeN61V+pOXIHnGukxKuJ5xQnl3ldqK9uC6FlwEX6bPfoXpQXFC4aH8xvuLgVG8c48MNwOxMVxoex/E045cFh/C0Y76gOiWPmjoRfBiEwRyzn8xUll4Nx0cmSif70rIPBoolf/zQGfINgfxe3qhL6h+HLShMV5mlF8JPxe2VQS966noNIO3l4sWLxeP4+5cQb9++3TkTTH1MuQtVqr9I11cMY7uK+PHYp3ShsOpHvP69bynVj/JTfK8H6pvjQ20fdBxXyod4alN94frkgaHyRAiHPdFmlK6I+7J19Of5qG1ov9Q3gOsJN4Ty4yFdZPS0lT+4P3gfh4vtXXLw7+2c/JQnrkZ/SYGmc1qKprLbF3g3HVO7r5fG4w/4s49rOvT2ZePAfnwhOy8M9xeBK57S6iLmKUhP6CXnyp9j7AulgUMG5EZeUFjtw5zl5rj8JKfSBZcFkM/3uyCMdBDT1T5O5WVb8hM3hnWZSnh6lMfTQA6vD8L5PuGlA/5dFtLztEEyKQ6oThQvpgPEV74K72koXf6FdBPTIh38HI57euDpKS1PX3K4fkuyg+usC+UhKHO0F9cDSMfKz8tMXMnLPmFxoHAup2SXE4rruI1Cn/49D+lMfiV9lfTAPv5DuFySyeNxTPtsez5DEBYX7USobK6DLpDB9RfhWNR5RLoTlN3jSG/440C6dX0Dx1Uu6U1xxqD0S3lEkK9PB+SvtLwOI5J3KL8hyK9PHpWNcKpj9qM9cFzylmywBGFw6IT0PA1kcrkI5/uElwz8c1y6kA0obZBMUW78XE72kUOQp/z4B6UlfUCfPDBUni6QX2UQxMUpP/KRH3LKX3HxB4WL9cK+l5k4vt8HZfCwKjuOdDxttiULaF/ySq844khOtl2mkv6TOup7/g5kRFJ+qTJUse5HBbrBK543EsVzvy4IJwMBtj1eTL+UtozV5RTRsInnYZW+x60pt8I45KNGAMQZSrdETIc4vi+Z3Q/wJ09HjW4I0iuFi2VEB+6HDK4Xtr1uSDOmy77XCWl4WVRH8lN5Y7qer8JE3ZbsJdoARF0Sz9MH4rif0iGsQ9k8HMejXCVivUt2UdKDZIj54ed5xnpQWlEu8vc8QX6ebyxjKT3J4fFIy8so+d2PfdepwrhfF1GHMW1k9v1om11QLsLq38suSuXtQml1gYzeRkrEMMgU45CP+0mXnncpr1JaNRAPPfh/F7FuIshIGFxfOqUyTYHy9uWj+vV8oi1wDD+HdKNfCdIuhcPfIT/3Q4cuA9tug6QZ02Xf67dkAyov/yLmDTHMkDxD5emiVA4grtuR7MH92I55xDBA+l6/pTrvgnJEHSrfGJ9wpbxdRuKwH/XveaisHiapY+nlHNxCa9Jpbx1yq8C/5R/R9+iF35phnWBTqZ237IZojOLIrbinT58eSQv5kBM/blv0rRUs3RqNzFVu5KTcDmvz/BYht5789VGSne/696F0dLunC24tO9SFbnfJNY2sTa+G+IGFePsLxy0rkC6QAT+OoU9ufw7ZAvkgl+A2ObdulYf0qjVm0penywMbXbdahyCdpsNqbxF3wcNRKpccuA10we1l6QlIq8Y20Yk/kMWaTGxVUL/gelBZSnJF+52K7Oz58+ft/zIM1bVs7uzZs+3/WIbazvnz5w/z51Zr7YM/Wg9NPdLGSGcq1BVy1thEH5RTbZvyuM05pdvW3gdRJxcuXFjsLQfroak79NO13rkG6u/69eut/TcDhLZs9C/bwND5cFf74GgnWhPOuc6J5z3wMH3y1JRnLqa863zqOXsIb+uUM/b1oGVgUQ8nT55cbCVzMsuaaAwcQ6HDa656Fr7rhQdrMFwMB1cyfK2vYr0hHeqyrLPcpM+JwN3QAysaADD4IvwYGFDF/Go78C5iejh1inQO7NOxquOc0hk2V+Uv5TH1BFwDHZY6bzr2eEECXOBFmWoGXbqQoh6n6OK4s8q6Hmo7fmGiwXQN1KnqlfjLPHTGGu+5Bq3Ij0Mm7HXT0NfQN8QLwLEweKG/B9LSQHoX2tOu9sFzUSOPl0NuaOC/Lqacs5PdY+lBNCcbDHxZA2HQy1XV1EZLg6Ox8QooXhETZWG2iA4IOf1qbiqrLrc/4EC5NHsoGLAhQxekxwmEhjx2YIFMXJBEps7gaMZFsweCMqrc2lYdqS77YNaRE42g82QmJCK5dSVekmMqqnvSoI6ibfHkvt4y4NTqkgENgwBsuvQGhhLojsGfQ5lVbl1gRvtB93EGa040C+M66nooZ4ihupbNjX2YEGrajvLBVnFQY0carEbnaPZ8aMa+660cDm1kSMdMLuhCbxmok5KtT6GkI1wJ+oDSGw1inwpqryXdym+uOy/LsMt9sCazhLZLEwx99MlTU55NMuWcPRZdLMSHYXV3W8eT1bL0IFq3X2S40XBqUefmMysyDgwiNpYS8fa3Q2P3205xkDGWOctN+bzcsRPgdiSzXa4DlhD03arWCUEDlxr9CcnkT+su0zHRmDlJxyUvDIK8obsOOPEzAHXonFUO/qlrzTLB1atX2zA+mKHT0i1zBm90+FEOPwGrc0ZvxK3RG2Xreo8xs47YnZ/8SDOWrQsGScQn/dqLP+wF3XiHjb0oPvWLHhgoql4JSz6aJZ0LrwvyY/Y44gMv3YqkjqR7Xfxg0zpJDtU1dhXLyIUI4OfxIrVtx9sH7UUXJ31o0B2dg+yk19enUCYGF0M2gW6HXulFOt6+pw6EZatu69ghfl0D4C5KOsKV4MKvJLP06OXHRvAr6Q19YzPeJ22KXe2DddHp6TDwRa+1/ZfTJU9teUpwHJuHqRclQ0w5Z0+BOyuxHyRf8k/WRNMxLQ3JyDWGfbjNgnecH3/w4EG7oF37bIumsz0SVmnhX4Pik0ekK20ci+l9H+d5Nh3AkWPC/ZYpN3jYeAyQ0cOwP4TLzbbyZ7smvT55IyX9Rvp0jL7QkY4ho8O+nMLU1nPEy4WLKA/FLdWlI7m9PBGP72XzMuPIK0L4kn8fUeZS/bouY/2Wyhztoc+GFN/TKZUh1pfSjDpW3p7G2LqW/ZV0EXHdsK10VHek5XVXynsZkFF5lUAPJX06km+IWI/SEw4dx3rvs9lSWsRZNeTV1f6i/F1wzG1D5eyrBwc9KB+5aMcxDMf75KuVHWrag9ctznWGbF63sdzsyylMLJ/QcVyUw+PjuuQekgf6ytOFp0n4Uv4eBocsMa8YJpYztgW3rRJRDhxpDpUx5uN1UpIh6r9U1qSeV/hpFJckW41mZPbxy03MMnArdMpMzqZAZmZEsnuZDjbPLKru0o1l2fi7BDPM3Lmc2j8sG38f2Oc+OEm6mOXBwiRJVge3KHdpAJ3MA4MVbpX3LT3pglvfLHPYhwE0UE7KO7T+uwT6Rc85OEySZCw5E50kSZLsNMwks9YdHjx4kBedSZKshZyJTpIk2SA8fMSDd1NmnPcRBszoi3+hVxHeuXPnpYfNkiRJVkUOopMkSTYIb4XIG4L18GYk9OUf5tBbIfRWiyRJknWQg+gkSZINMvdrBY87Q/qasi46SZJkCjmITpIk2TC8A7j0wZCkTNe7u3nvbz4gmCTJushBdJIkyYbJJQj1cMFRWvfMAFof20mSJFkH+XaOJEmSDcLyA762yhcP9+WVdMvAQ4U8QAjoi0F1vAjheOoySZJVkzPRSZIkG4IPWPBp41wXXQcXHDdu3DgyQOYzzswFucsBdJIk6yAH0UmSJBuA5QegN0sk/eQFR5Ik20YOopMkSdYMM6off/zxkYfg+GoeSxU0uE4+Bb2AX3DwcZV8v3aSJJsk10QnSZIkSZIkyUhyJjpJkiRJkiRJRpKD6CRJkiRJkiQZSQ6ikyRJkiRJkmQkKxtE8+BMPiCzXnhfKg/afPvb31747AarlnsOW1zVA0yUmbTRwTLsat1virn0HtlkPfD2CtxYJLPc3DoZgrZJG012m1W1qTGQ97psCbsd2863zda3oc52ndkG0T5IwUgeP3682EvWAY1gFV89W3bwOcSq5BZz2CKdzKrgfbY828u7bmtBZ3FAr3fl5vtxh+HEwZsd5mYZW6Y+lzmRMXj+6KOPFnv1SGZsB8f2qtu8p8/2e++9t9hbDehGFwglN+XCYx2gmyjrJi7OapnSl80JusF+v/vd77Z2HXUXXUmXpXBd8KaY999/v7q9rMPWx7CqfnDfmGUQHY0RI17lwCh5GTquuS9c1tFhr0JuZw5b5MSwTdy9e3exlUyBk72+eDcny9jyrVu3FlvT4FV5Fy9eXOzVgy15+6C9+Gv35obBDa/2EwxErly5sthbDZRH9UK964IBx4dbuPhgsPSd73ynDbMtoBv1PcjJdl4kl8GuGBBKX5pUQG/AtjtsnvBxVphj6huwGfb7wLbIu+Yu5TpsfQyr6gf3jVkG0Xk1czzJet0+6LDfeeedxV5yHOAEvMoLyT6ePHmy2FoPtbN264IPt0j3r732Wvuf7B7Y1ZgBIReLDLCp+2VtksExfTJ9c7J/LDWIxmh0u4PbFGzH2Uuu7nVbpHS1hgHr+FhjVprkqTSUP1eY8ivht/iiXCqXXLzdh5y6gu0KM4TrBRfLTno4l6WkHy/7o0ePFr71SP/8kxZy9dXrkG7mlNvrEOcoPvkonPC8a2YISnj9qOyRWIfCy4YjHEjXPvuBn+8DMnt85U86mjWk0/a0XR8OdeFpRWrrq4THUdoqI450wfXk8nnY2jwdt49Yz7FuatMvyQ/Kq1SmLvtwXFacID9dFFG3bguxDCV0bKz+VHe0bwYSbPfl3aVf/tEJ29H2IqTPrK/yK8ms/Fz3wtuVy7oszFpqxnLIjiJ9x6Vj1xFuSE/LMCSvy4FzWRTX5eW4yoH+vW8inEN9xnohHHG87krtxeWqaU8OcmBXY2fpuYCizdEGlqkT7Ie7QGPuEHo9lWxdx3BRp9Il/sgdbdb7sFIb8+PJDHwyA40hfnLlypXF3gvwwzXG1e43nRT3RdptwTH8oelY2+PaH4KwOM+bf+VLelBKk+PNVWu7/eDBgzaM9sHj4M8+4YA8lC//oDQUZgiVVXlKN5IZvSgPHEgOhQH5CcXzsvRBOOlOaXkZyDvWK2G6dDOn3PjhQPqK+eII4+kprGSSbmM5+lB9SmbVufIH8lQZATl8X2l4mcDlULoeT2VR3jFdKMmCn8cD4np+7Me02Sd95RHzr4G4Lo/KrjqAqH/yVhzyIrynMQThpVvJrPyUno6TLvteplI5XV8lmTiOEzEfKPl5vFK6JVnw83onfskOlI/ScPlqoLwx3Wj/2pdutK/8SvJ3UcpPfqQBSl/1CejLy0b4MWUt1Yuj457mUB30HXcdeVkIw/4YCF/TNobkjbon3aE6/du//dtDf+JKDsLgBOkojPB4yifKAJ6W6kGuBmSKaQqO9aWj424XKnuNPYs+GRzpSeWV3qUfcH3puORDJuUjXalOgHS1XzpOul5vHMeNKWtylJW9nQO4UtL6unPnzrX/uuLjn1sqXA2CruZq1wY2srf/Fy5caG+nwPnz59tZDtbckR40BnfkliVXgMilq9ZXX321DcMDAs6pU6fa/7Nnz7b/z549a/+1rol8JANpgMLUcvLkyfZfunn+/Hn77+sb0RFIDoUBlls0jWux90K2sag+0EfT2NrtIbp0M7fcqkP+vR59Ldebb77Z7qsuuMqmflQnmm0YA2miC+Vfkg97cZshDjah2RnyRxf3799v9wFdY6NCtlRCeb/++uttun24PoRmZ65evbrw+bQcN2/ebP9r62sI2uC9e/cWewcHJ06caP9v377d/sPp06cXW8u3fWZeqFO1Yf3H9tfVvkpEfUmmuZY7qD75j31SiSH7ws6Rz3WgulyW69evt3YpmbFl9jVjx35zcm+PYZ/kTftT+CkM9afYhrfDt956q60v9V+rYKgO+o67jvhXmfi0O4ydba1hSF7wWU3sRfrrqtMf//Efb9MAdK42Sxj1GVDqy1SfpXO0Qz0iK2BDpINsij8EfU+crV03nBNjubqg/WtchN4519C2HPWXspunT5+2/6B80JXqDIb6VeyAfLwdxfNGMp6VDqJLnapOZNzCxxh0WwFHY6o1ROEn5xoePnzY5uP5kqd3CDReOhGdrEuU/N3Q+0Av5EED4WTYtRav1DHoxKKOUYOeKRBXuuCWEo1PjbaLGt3MITedDI2d8KqjEjEdyuMD1bHQEZGXBl5dkA/1JhvS+nE/8XNi8M6R233org+dvAA9Tl3/jJ2Dt0F1qnHg0VdfNVy+fLnVh9KlnBp0AXWoCy9Ytu0zAI0yyy6htn05Kq/rSza4LLW27AzZFxMFDGRWAXnHfpU6hnghMtROaunrT9UmCSN9qF2Mudgby1AdDB0vURoUzcWQPNggtog+OU74El116m0Yattr6Rwd+yDXB+H9fHzckV6lEw2EtWzDoX/ydoA9adA81K9yTiCu93HJ8qx0ED0EFcrJLrpVo6tcd95oGSBigFDbUYxF65IY8PnV5DrRYAN90NiQR4PcLtahG9CAg1kxZCydZDcN9eY2hPNBsrbHzjoxQKTsDJRq7w5sEjpy6kdrAhnkatBFPdJ5x4udVbf9bWhfYqotD9nXvkF/E/UxdNFfi2w3XpgM1cGm64j+2OmTR4NnZobxp9/fBujj/C4U26UL+1WgfGsno9DhUH+u9u4uXjT0ocEzdzGppwhjFZ0XCOc2sKkx1T6zsUE0t1rpFKNxcfJbJVz1la7AlS8NgFkOZNMV3tzQSJilw7indri6ZT7lYUKBHJSX2QlkoVPlRN/FHLqplZsZFW41IdsY6ER8CcVYdJWumdwuyCeGQT+xg2VGllu4HKuZtcMOOYFQH8vMGGh2I8pDB7yKGUwt6SA/BtAaWLOkg0G1l2XZts9MVakN62QypX1p6Qf15PgJamp9TLHlIfvCRnwJzZyQtw9qQLOZcw1axyC9xz4DXcT6mgJ2qJlt79eG6qC2D3Bk83PM4MeyD8lDG2XwtW2zvOic/kADThjTVuLSklpo2+RLH13btlkKVxpwM5OO/oE2Qt/jri994nLuJQz2wR0ELoZKd8E4jtzojHSpT9nuUL/KOaF0PFmOWQbRdOiqmNoTIYaG4dCwnVVXMCdVjN2fiPU8dbLwZSdz47cpIXZ8NdDg0B8NTuloNgU/P/n3wQlekKY39livc+imRm75u55o/DWwbo8BlE4cnESIi1/pKegSdKp0TDpJKS38ZN9cbHgYYMAYO1gGlORP+JoBCGX1E0JpoIT9MjAlbFc9Y+dRz5QDWVZxcaglHazJVDlVF/F27rJtX/J7fVIPunCY0r4kk7cHiGuXffAuW0DHXQMnydBnyxrA065k/0P2xVIhZFH9E4d93LKzeNQhMnpfzoXgHHdFsAWVv/ZcAeTttgxua1NBfxr8xHoZqoPaPoBwgjJjZ2PljnYI2KpsHobkoXyejtvyJsFekU0DzrEDYi5IiO+2MQR5oivqomvJVlwqRBunP/NzpKCfpr+rAVlVR/wjh9bKK0+da70uhdcx9e+D975+VRMKflxr6EmjlFdSQWO0S3Nn8TSrktM2rqnUI8dx7AuO+7EaGiM8EqcZ9Bw+ZSvXXMkd2ScfpzGaw2NsO36MtH07yluSpYYYR9uUI8oWy0IYUZLV9dsH4TztqIdYr9Cnmznl9rQJK72zXaprx+MSjzj4jcHrWem5/OD6wXXpXTJHoi1B1JmXReVU+UnX92M48DwUXrDvx/rqqwbiuA7UNmL9iFL5a4ntjrQcP+Y6pEx9+nKd4CJ+TPWvMnfpz/MnfZXb60P5us6H7MvLoXSjHrqI+sN5G4ll8byH5OrD8yWPaAN9/WlfvfUR84jOdR4ZKmvfcenQ5a6tH3C76XLoy+mTJ+rP0++KV7LpKBcy1NRjV/111U9t/QJpeFlj/iUXdQelcO5K9ae8SumViHqIuD7YVt/ANnkQ38PEfP0YzpGscqrLWtmTl3mFn0aJSZIkSZLMBDN7zBY3g8GlZ8yPM9wNKN0d405T7bIOZlubwSYjxoXP+uDuAndYVnGHL9l+NvpgYZIkSZIk+0nXMigY8+wGSyy4WBn7EN+yaElZDqD3lxxEJ0mSJEmydngInDW+ceDL7O7Yhy+Z7WcmOq4JXhXIyLMJc7wGM9ldcjlHkiRJkiRJkowkZ6KTJEmSJEmSZCQ5iE6SJEmSJEmSkeQgOkmSJEmSJElGsrWDaBbtT/1oAK/M0ZePuiBt8thGauR3eCiD8H1POjuEW/dTzMn62GbbTraDVfQB29SvZBvYLNiBPgR0nBh7rl0nvCmk74NiU2QfSjOZYRBNpbhzhdOR+bHaQTGdH18Gmgqvm+l7XhI5HoevU5XgPZ8u/5DxeXmnduB0PDytXAsNg/djjoEvF6Gf0peXxhD109VpKtxQp6pGjhujP/Qe68bTwm1jp7cKam3bQTeuq1Kn6cdrGYpDHet4bd+QjIe2ENveXH2AwIb4ouCqiO25y8GUNpBMp9RXY1vH8bVvtBfKpq//bRO8U7vrvdpTxgmcB7blq5bbzNKDaAzqzp077TYdl1cin+/k3Y3Af+3nPHllzJUrVxZ784McNQalV+ZIFn0iswQnEXXcxJn62hs6nhsjPq9Lo97ECYNGyWdmKatsgMF/PFnTwcZPKZcgHHVC2cfoj3ix/MjGa44kG2nGTwYfV2pt2+GTs+hQ+oodMYMT6pdj/Guw0sdQHOqN+lCe1FcOpFeDPqu/ShhUUM+rBLuWvSgvt1uOY1NT2kAynWUmvJL1MGWcwHng4sWLi72ki1wTXQnGxFVZ1/flGYiscuC/bTx69OjIYIuTKDq6d+/ewucFDIY5wfXBgIqOmEY+ZvaCC5eugbFfsCnN58+ft//Jp6BDBrBdM5JcFFGvmnnhnwFK312FoTjUGfVNmxHYCfWPPMl8oOsxd7a2GS7a+3jrrbcWW8m6oO9Okn1m7YNoGh2zUjifeWIbv3hy9iUD8TazwuPY1gmYcDEsJxOlE/OoQSf8UkeOjAxE+PRnCcknVxqIIy/HumbjXH5c1+CxBuog5qN0ka2mY9QAyWEg1jUY64I6Y0DFnYqxcamT0ox1TAddMair/fSu60d6kT15XcYBn+pQNllTR7Fe3W51jHw832g/ngbhxoAO0T9xS/LeunXrpS+HMViJF0vOUBxdzJw4caL9F9QRH18YQroA9RuqL3TVpQv1PfwTrtQOhyAOafCv+pbePO/YvnQMf8K7bPixr3LhCO8oX7mIx1XexNGMLANpjqnMbuMudwzj6YHn47YaiWlKR0qzL24J2nSpz3G4WI5t3/XmOgdvN7iS/ZdAH7KhmK6nJz0K9ye+ID5pujwl/RBGx71OQOVU2grnZXJdgMsAsmdc1FUJwmsWmjiS2WUR0pnLoDw831gHHh43RAyvdkReMQ3PF+d5cwyn9KQr1XmU09PxcoP7R51H3AZw4PUZdYyTLGzHevN2yMRXCbcr6auE60tyQE3cY80nM3Dnzh2mGj95/PjxwudTmsFRe4x/ceXKldbPIe6NGzcWe5+GaU6s7b7SwR/Y1nHiNSeLVg72PR6QNn6SgfCe1hCEI67ieVmAvPy4I9mEwiCrIL5kkayejsoupBuhOJ5mF4rrMkl+bfuxMRAv6kaQJ2WPEEflJwxOuuiDOJR7qOwcH1Me1w//IP2Tp8pAOE8Xfx1THZP3EIRTPLUj8lO5cOSjMMiAEwonvSvvGh0SVzKq3DEeflG3yqOLoTjSZwwTy1ZC6eDQi8rAPnElf5SRvHTM9TwGya28lA75k5/Ljmza57hsRbISHpSe+0l2yUc+bmuk6/uUy/Mmrtejpw0cw8/T6KoTT4ftmK7vuz58P+rZ40wl5hWhbDjpRToVKq9AJt/vgjQJR7rE4Z+0Va+SR/lpn3DSA9vSvfLFyU9puZ445nUoOUBlURqqQ7Y9DYWHUvqK12ULJbr0ilN5JSv/sT5cL1Fe8sdPuN76kP5cX+Bps+1pEV7HyUey4fDHyeZwqlflpf3aeh+CcC6/9Kq0wMsjuTyO5BUql9erp6nwnod0IEplILynuW8M9xoVuHF1Oa8YwM8rnG0ZHkQjBxmoYNsrWJQq3g0OSDv6dUE4yR/zxF/7UT5gPxoYeats0p0T0yF/T4M8PV015JhPF1G3yO9xvXy1IFOfPpHP6xskN7Iof/5LYR3CKHxf2SkHx+RqoRwxPPsuU6wj9r38bLs9d0EaXWXRfsw31l3UO8f76qKLki7jPsSyR2riIGNMg/0auZWW6zeWmfw9DPtu16QR+6Qa1PaI75C/y+MySq+CNDx+KT38VB7kdllVNvyUth8nnpe1lD5h3I4g2hJpez2SjkNY94s6B9J0WZDT05xKKS+HssV8PTyyuxw6XiMbaUfdoV/XsepFfsjiabtspbpw3aqsjuRV+iX5ScPz4bjKzz/HgbgeDqL+uiD/Ltn4F1314WE47mHY9uPSg/t1EcsOXj/xeCnvWC6QDNIjaXq6Y+q9jyif0lWdgecLni+w73krDfnx7+lBzAMZSnJ4ujGNfWPW5RyNgrG6I64x+MXRozSKb2/7iidPngzezj916lT777dS4m3jEqxlPn/+/GJvORojPbI2muUdvrbTUZiTJ0+2/4Jb2+gKuH3dGGm73cXHH3/cPhin2yZ6UO/p06ft/7KgQ6XPLZmuJ3z7QA9TH6bkgU1fP4s+3DYc6h6dDd3aBcqBDWJrMHQrzWk6/cVWHefOnTtcFkE+6KJmeQryURZuw3XlKbsXsh2Y07aRF/uuWVKxLFqzLpvGwZiy1OhXnD17ttUV+XArklv/tct7SlDfApukTqg/lUXrkFm6gpx+jHyH1v7TBtTPITdtXvH1Foxnz54Vl8Zge1Pa8Jtvvnl4ex54IFHtTH2ZZMAprPfHEdJEfoW5fft2Vdudg5J9SF/L9qks33NYqqQlMzi1Zc5rMLaPVTtQfxf7BmwIP6Uv4rnGnw3BpmSHoP764cOHh21DDnv2uHMwpr1Cn90Pcfny5SN2h84vXbrUboPaCMdJm7CRofMyzF3vIspPW+Q8pjZHe4znBUftlX6vC+xK5yw5UJ4lqEP00veShX1jYw8WXr16tW2oVDauaz3xtqGTH4NGjI2OZpmTcS3Nld+RixPc0Im4FtLRwIwGH9fbDcGgZOoAukRfZ0tn4g1fnVaf3MhW0yEuAzZAnSCP5OvrjITW6oEPjjdF7JgpTxxYcILo02dtHLdlBu/EW9UAC5siH2TQgEEnmrmg/rxMOPUN9BOUEcg7rl0cgsmImPbculJ6fWsbowy4vvZKmtSr3hBS0ybWxdx9KvUb01O/uGwfOwcM4CgzUCc+qUC7iLLPPYiewlS710WG7I7+yO0UO6Qd6i1Oy5wfVlHvUX76TwbWQL/FhU/fALkWBuZR9qGBPpOGGuDTV8w1ibOrbGwQrSsaBqPMTvhVYhc0BOL0ddolMMY5Z9doNBgRDbD0oKHQ7BDlc3xAwcVD6SrYoeFF+THgsSfiLuhMNcigweP6TqQOcWlUY+sEiEPdRP1AnOkRdEre4NVBcXLo6/TJa5UXatQFdYIMyEW5bt68uThahs6QWQzKMPXkPbdt08a8Uyw9RMgsXt8doLFxsDX0sMrZDfJA35wgqB/1PXMg248P7ihPtVXZLv2HZqq7wI6kL+qYk6ZDuqSvPibmPbVv4KRKeyZ9n21XPvg7svs+sAfKi7zMTG8Dc/ep2EC0edBAdWwfSzuk3tVvET7qHr8xAxhkYACKDPSXXOxTZi6cS+cgyb4p+uy+BtmdyuhwfqEdLnuhMHe9O8hI2oRnAK2BNedLxhCk20VXv+BgV/TLkaF6lxyc3/wu8r6y0Vfc+a2+kkF4x8E/DaJr6UQfNCY6DBkvaZE2fv6UaS0+4OkzIMpEQ/W8KSv7KocuHlwOLWXQbCZhPQ3gCtVPcstAHmo4qod4W7AEcejENdsGyOhyDsFAhrJ5PbPPnYq5UJpTB6q1+MCfk/TQoF23JXWLua/D62JO2yYNOm23aXRGG1X6+u/T5Zg4yMgMDTNObkerwN9Xjp33nYTGQjunHLQlwQlGZaLvko1zQuck5PigmoEcdSh90UY8PnAiZSaKMjDwjXlzkhXkxT7HhwaJnKzJmzy9PpSP6xCwlyE9qo9Dxm054c7dp9JfYfOuX9L2ZRl9fSw6lyyEpb51kYcdUIeue/LBb4w+vbzKG1lIg7S8v3Bb6kODUy/fXPTZfQ2yO/rlqCf07W2kdBFRw7L13oeWdHg/ov5+6NxCXkwUeL+gWW38kBe7Qg9eb+i6ZrJJcgy1/b2guUJaCpJw11Tc4siLhwn8GPsR/Jqr4sXeUZoT05H4oqn4I/5N5744cjRPz48w8kdGjnm8EjzAoDgxPWST3FEenKcdy0F4J+YjWZ0YhjT7/Lug7B4eyI+y1KYBMR051xHEssfj4HnjXD/SbSke6LjbUMzTbXKIWK5Yt+hK9SNHGPL0uqjNk3Ipjqfr/jjSL+UL7k++xMVviKj3rjiug1I9SFbJA31x/NiYuoGSDnyf9GL9Uy+U1evHZVL4IZ1FfbnNQSlfQEaOuW25rtiPcSNDeXvaOE9faavMMWyEcMQpUdI/dJVdEK+kX8Xryq+E54OLNhSPdenO7QFXI4OHj/lGW/Tysu1yeF4cQ+f863isXyCMjnvepfKV6pj0PY9YR56+bKUGxSHfkiy+31UfXXmXwo6B/ChzJNqr66VLlhjH27f7e35sexnG2DkQx8usvLzu+vL3sqiMUYc6jnO76tKDwD/a0D7yCj+NMjYGV0FczeUVTZIkm4S+aM61/bVwx6k5uRZn6o8LmgHdlpnobQK747b6NqxBTpJauHNR+6DkcWajyzmAWw05gE6SZNPU3sJOxpNrJ5Pk+JDjtk/Z6CCadTnb8qBJkiT7Cf0Qs8E5q7IaWGeZJ9wkOT7wUKHeFrLvrH0QTYfKCQsHOTuRJMkmYRnFJla1MZujfpAHqPwBn+OA+nkeENvEMpldgFviPKD1+PHjQ1tIkm2EJVlq0/GFAvvMxtdEJ0mSJEmSJMmusfE10UmSJEmSJEmya+QgOkmSJEmSJElGkoPoJEmSJEmSJBnJSgbRWoC+7a+M4qty8WEe5PavDx0XeIDFv0i1KvSmg11kqo5WZe/rqrNVIVvAjSnHutrgpvS7DW1kV/rofaHWFjdls12sq63Oib/cYNPtcCy7fH49rsw+iKZz5rOS2w4DaJ6IjvCc5XH46IFfHNDpTv2s6Rho4P754l1ijI6wcX2KdlX2vq46WxXYAp/VpT09ePCgLQu6KuG2uq4TxKb0O2cbwQa7dOowUI4DHd6KRN3kq+c2T60tbkOfsIm2Oie0GV6ri+3rnQq79FacVb1JqLYvSV5m9kE0nfOdO3cWe9sLX4c6c+bMYu/4wWuTBO+/vXjx4mJvddDA+fLaLjJGR3w4QqzK3tdVZ6vi1q1bB6dPn263eRUSHX/pdZZ03j4buooTRIlN6XfONnL79u3FVj93795dbCXbSK0tbrpPiIOsdbXVOaHNMIEmKEO+frG+L0leJtdEH0N26cp61+BEssszxOuidJenRH5saRpcfPiFchdcoOzq3aFku9iFO8xD5PKll6ntS5Iysw2iGbj1rTHCeHUc58bMwETxdNyvFgW3s3Tcb08qbdLBX2EwDsdliLc3gfAxntZN61gpbixbLTFeHPyqLIAcbJf04qAjNQjCs+94nqXBtvLB1d7eUb10yeb1hkMG8Lwki8sX6y+icLiYN/tD9QY6VtJFhPg6kbz22mttvIiXNUIeY/ITY+rM9SD7wUWdu12QJv7KJ9pMRO1Vzu3EbZbBG9slvQPHGGxzUVIK5/lEW3Sd4FS+IRS+S/+uSxzEvCSLyzcFT9frTX6S0fPBntnH/gB7xL9UfsLqbpvqQm1KdS6UB+lEG/Y2VNKbh491qGP444bqSWG9zGyD103Ew3u5wOXHxTKUZORf4SWz8vf2wTZOeXjaShdHehEdK+l0DLF8jh8Dz0tllPzSc4TyKz7nFrZj2FJ9CdcDzm0AeVRfOk74GjzNWOcR6YG+Rv2N54McSks6ct0pfS+LwnnZayEdnHTr6bksyOAonlDepOOyiWizXibi4rr6EpfN/cGPsS3Z95JPZuDKlSufXLx4cbHX3uNpXXOCbPcfPHjQ7gvCa//GjRuH4T0M255mczL45M6dO+220mOfPBSXMKQHxPX4CkdcUL7IAkrTw5Ce0lVaiieUrmTz8ijtElEe7cd8cOTP8ZhXF1FGIF3SwQFpEIY0BfuSR8e13wXpqpySz/PmmPIEjrleXLeCvIfyJV6sO+mFY+x72iWdeByVN8oSUV4un+KSn/zZJ09ButqXnvx4CeKQJg6Uj9cZxzwd9gkjSnFIV+VEhxwnnmTUsRJKT2g/lqXkV8JlEZJH9RttSHUgVIYhCIO8ILk9b5cl1lFXnUnGMZAGaalMSttl8fJDtDvFUXn6iHJLX8pfupCf27DrROHclggvGSSj9slT+arMHjfCcZyXnX/2cYpLGC8P21F3sR5dJpeDfaUVj5XK6/rgn+PyQ1bJ3acX8H3lo3T7UF6CuOQlOOb7pCvYlnzIpHRK8pUgXcUXxHN//j3/uO8y8K/4/INk4b8Pz7NWfoj6A/ZlA7IX7UNMW/XldiFZaiA/4lMG6UZpckx5RVkVT36Ko7SkM/Zd/piOyqh84j5Ip0J1JchP5Wfb0983hs88A0jZblCqXPlRAaUKkp86L8c7NMLHSvKKkxG44bAtAwXCRkPnuPtJLv5FNJBYXpXVYd/LW4J8Y5lUZuXvOhDI4+UsoXgOebk+pDPlhbxRPxyPfk6p7DHvWE62S/p0nfflKaIeSMP3h+qtpP8oW4mSvNKD+5GOykGernvgePSLxDClOmPfkXzShcK4DcVyImcM00XUO5BWlMNl6CPKAsR1G4jl5Bh+QmV2v0hNfbPt+VJW36c8sZx+vJZSOrGeYt4qo+pettBXZkG4WBek7bal/JU+RP1EGfj340Ca8iNPLwPbQzZG+h4n6gWibtj349IvftKTy+z7fTKW8o46YZswzpBeyCMeZz/6lYjh2FZZINajy8+/ysrxWP+4PqLegfTdT/mLWNYoP8ej/tivkcVtmjjRxkvE/NGJ6wE47n7sexnRXcyvJm8n5oEcsdzkWZLN5Ze+3QaivDFOzKsrb99XmeWHXDHPfWXp5RwPHz5sbxn2PeX98ccfH94qwOn2wdOnT9v/EufOnWv/nz9/3uahWzByTcW3Dwc6p06dWmy9gDCC+HzvfQqlsiGX47c60Edf2QCdxHQvXbrU/j979qz9FzEcbz2YQumWl/K6f//+4a06OfByRYjTNJ7FXhke2uCBGN3+oR4cHjpDX3qwgXB6IK0P6p6HtHQ7q0RfvaH/119/vd2eixMnTiy2XiDdPXr0qLVF1y16cPvsYqjO0J0jfY61kaE2DJQHmWM707rmPlsZS8kGlP6U/qSmvrFT7FW3PGP9qH1ic/q/fPlyu70sJ0+ebP9jv7Juog33MdQv04erT+F2L7odsjGoaf9CNon9SgatAUeX5Nec59p2gQyyFTFVRif2gUN6mbPvIR/KpHy03Ex9BLJJN6CH6NAHMuFPH8NDv6UHf2voa6tD/T/EPgyGzp81/X8NNX0z/ZuWSALnqmaQ2T48Dcig8coYSn37VLrOPVMZ6mMvXLhwaHf0l9TxvrK2BwubK5i2M3NHI6iFziDGj4PoTUDHQyegJ+AxKBpgHGjsAnQMUcfLNg51njQ60isNuq9fv37YSaHHmg5JnScDSdLddrAR16tcMo1l+5MSGjxjj6QXT+4MrmgjejsL/wxG9pm+fhndSI8aqC57cu+CPjfKobrRGlMmUR48eND6iVXJuM7zFWWKeWlATP9NWwHK6GtXCYc9axCpi8M5qen/pzBn/z/UN0uX5El5uGjAltAb+8hwHPuBvj6WiyO1JQbTvk5731h6EM1gUcbUBVdcGJpD+NLDFkJX0hgneZSuYL1DGIKGEmWYC066enAHg7px48bgVT068atb0CzU2bNn2/91QsfA1WekT8fEKdWLQ+eJPvpOINIV9nDv3r3BDgnb4SqZRqyZlbGgf/JaB8wyltrIGPstgf5Jl8Gfg5/uumiGcw40Q+ev+ANmJ2hfY2fwpjKlP6mpb9ouJ46+C0dmnrF5Tqhzlpf+Dh3K9pF32xnql6kP6oW2zwmY8t28ebM9NheqA2YUHeqHdsE//Sz5l/rkPhmntp0hvczZ9yAvM9+Oyg3kSbkpG7aNLiivwtB/cozB9LVr19o4c1LT/48F+Zft/0Vt38zgnz6HSR7uSMmWduH1kVP6qaE+Fv3oApR6wN7jeWhfWHoQLWOisQidZGngKJbGSeNVw4bSjKOfBGkkND4gD9Lyq51o9EO89dZbR2RALhoPfstcRZEO5cWY5GpmxNT4PW8G43Rmy56cNQuOjmJn0AUyow8PT9lKt+qEbm97GXSLSzM6pOlLC7oG3ZSbC5Ga25y62NCF1pTGyy06ZJHNISv7uL4BjG6bkTe2VJM3nQ2dsLcRGGvDEeqMdqFbbUB58IsDBg0ykFnlnGL3nIhdb0C9YbtTwNZ1gq211dr+xBmqb9WFblfKdiPUJfqlf1p2KYfkJy/Sizr0i1otmaGusTn1EcjLvusigry0QfLxeluWmn7ZbR499/UnU+E8gf48b90l8PqEOOCEIRnHtp0hvUzte0poAsf7IZYbaCLG24kuCmQ7rjPKPJS3t5Patlrb/49hjv5f1PbNWtJBWaQ/nbOmLhNdJ6531R31j22U+pKhPpa+Sfarc+KYpWDHimbQtzRNQ2kXncs1xtX+4y+aq5UjYZqOb3Hk0wdB9B+Pi6ZjOjzONpTSVf5yksP9m4bTpoEfNIODI3HiPuFLYWLZ5SRfHzGul7lUBt9Hnj4UDhmj3vrqwv2H8oCYluQWXqd+3MOAyqe6GgLZlI7qUtvy136p3sBlIz5hcUMoD8oSy4c+or5FlK2PmEZfnXnYkvyuc7a9nGNkElGfyCaiPuLxiJcrlrFLvzEezvXRhaeFzlwP4HrScW07alMR6cXT7CLqMNp9bO8K77qUvEP5qdySWeWSK+mYsNpnO8rLvohhBel6PfXJGcvbVfe+7+mVwgr39zomzpCMHp5tjitMV7lF33GXl2Oebhdd6fXVDTJ7GaQX/gnnaQ7h+dTWV/RzWSivHyul2YXHZVvl6NKh17EcsokoS0SySX+gNDk2hliPUTbkKskT45X0HcMID6d6jHaCn+uvJJeI9e1p7Ruv8NMoYaNwRcMV3RaIMgnkj7PPXNEx61EzK528gJmbfX5AIdkNmKXiln/XrWT6A+7SaIYnSZIkOZ6s7cHC4woDPy1rcLi1oWUVyTBcdMz9towkWQXc1uxbysF61xxAJ0mSHH9yEL0krDUqrQ9jjVVcl5p0w9q+0sVIkmwbrPtnLWWE9YM8C7Dsw05JkiTJbrAVyzmSJEmSJEmSZJfImegkSZIkSZIkGUkOopMkSZIkSZJkJDmITpIkSZIkSZKR5CB6x9HDTPHl8JuAByzHfiygBG/qoEz+Ev250l4HJflL8GaXvo82HEeow9oPNWwz67LHbbf7XWqX28a26u64tNEx0F/r4yFD1PbvY9JMdhgeLNxVeMm5vwA8eQE6GfsC+G1GL4L3l8evG//QQJfjRfX+gnq2k+2l72MO28A22H0f2y5fMsy2t4Fto7Z/V5i5xyfbWF/7bkM7PRO9C9+t3wT69PZxgVeGNQ11sbc5kKFpM61juxk8HO43Fy3t56t59VnTuS5iJNuKf852W9kWu+9i2+VLhuHTzkk9tf0754S5YeZ7G+44O8jDJ8D3mZ0dRFN5fOUwOQq3jxjQJfNz9erVxdbL8HEN3g2e7AZvvPHGYitJ9pN9W7Kx67z55puLre0hbWimQbTWCMlFxeLHzA/+ChPXE7EOS8dwjtaOKh/CMgsIDKRL6dUwJk8vk8dbZkbL9Uaa7GsNlfyVr9Y+47ysHCeutnVhgX7kDx7f/ceCTkgDOXF+ZUy6rieOExY45nm7PCqz8Hh9SBY5l0V6wY9jhF0GPpwz9BW60kc2kKMkH+DnZXd7ANflGFRmOeXrsqgeVC84ga4I21dHoGM4t0nFI1+lD5KL44Lj5OVlV15ev1F3Hh7nkJ7Kp+Oqf8kAzMKx7fLUIrmF0sWx7cfGIp112Wwse6wb4uFcppI8ygc3RQeOyxR1HfOQfeDG4n2D5Fdde7ole+0rL/rRMZzbs5cNpEv5ad/zV3zpgGPKQ8eQx+ulr2wO+uWY0p0CaWgWmnRinXm6HMOV7EllwrnOINY/+6JLp1PweiVdlUX6xClvhVUYkCxRfsIofp98roOucDqOk226Dlw3JQjD5Bgfd/M0wMtfspchov4iLidOEA95kAv/Zepwp/lkSVh7SzKsGQXWALGPv47JKcyVxVo6jgNrSXGgOFpLhD/7CkNcHHi4sSyTp9ZCaZ1s39qoLohDXOlA+14e1hoqX1AY5ccx9n1NomRSukCaKisQ3vdrIR3Jx7/nQ5rsS14dl3yEk56lUw8nPJ5DeC9n3Ce80uSffZWTNKeUt4+Yv6N64rhsnm3JByojsgn2BdsevhblLaQLIfuQXPx7OdjmuOcf46geVfeqM/YVFofOta84OKWjvPhX/Sgt9mXnUXfEd5kJq32VV2lDbDcQ0xyD0vb4+EkfbE+1N+LKJtyOhPyUl/YlC/kqjuJ5HQj2pQ8dd/3UIn2rvFEe1bvbOUzRvWwDR9lIW+mTv9JUOIf9rvLG8KTt9efH2HbZCev7Kj//kg1Hep4v8TytvrIRT5COZPP0cWOJ5fb0lCd5sY9MOFA5VCZwuUB6EJTT9+O263AMyEQ5hMsJkpWyCZdVcuLYFp6uwqis2ld40i7JTxiXjf0YjuMuWx9Rx1AqP/nUQnjJ5LqQDimztoH8fZ+4vr+P1Gu7AzU8GVTJwNiXAYL8VPlUjBuXVyxwnPART2MsU/KkDB4GCBP9aiC/KHssT5Qp6hY47kZc6jQ47vvkEcPUQDyXh+2Yjx8v5RPDlORVPCeWk33qSLDt+xyfUsZaojyO6sltPsoLhPH6dnn5J85YiOP5lmSRfvGLMkGsI8BPYYnvciOrl0V16nmCwrm/pwslG+e4h2Hbjys/+SE7+04p3yn6FTE++1HmsaA/0nHIw/1IN8qt8sp2CONxpHfJhx5iGhyPfjVE+UD2JeI+TMkLlJbKCuTv6ckeFGaovFHvsUyeFv+eVswbHRNeupbuY/4Q43aVDX/Bcbdj4k+xNSjVi+T1PLrsKcrhYdj349KL/NhWOfkv6WeIWG8Q5SA/zwsoj+ss1pniOF4PHp6wUQZBGK+7kixjyh3lLskp2TzfLlSPKjeQvsvEvh9XnvKL+t5Hll7OwS3uJp12wT3T+a+99triSDfEaRR/8OTJk3b/ww8/bG+F67ZBU7mtv9NU5mJrHqbkef/+/cNbwHIwdCsmQnjyO3fu3MJndSgv9C2ZteTj+fPn7X8tyKvyU9fob2iJA9SEGQt5U4eUD3m4rRShzKvIu5aTJ08utl7Ag4d9YG+qJygtDxmChzxY76u6Vnt8+vRp+w9vv/12mxfhavNgvbfkv3fv3uEyKhwyg9qzOHv27GKrn7F1RF1TLuWv9c3Pnj1r/0EyOa6DuUE/kok+BdscC3odWldP2U+fPr3Ye8Hly5fbf2/Ppdu60s9c/VgXp06dav+V3qVLl9p/LRHgXzJPZYzNDJWX9iDb5piWOYg52uX58+cXW8PEssV25XaMLQz1K3MwdpnAUD80h05r2ssUkDH2H+gYO3Fu377d2lat/lkSCHohAv3EGLuIkHeUk3EYftFm+vB+E9vzfqCmr913ZlkTrTVBGERzhbLwrUcD2evXr7cD8mgYq2Bqns2VVxve3ZQT5rphIB3lpsGNgfDEQ1c6Kc114h0L+ZI/nSgyxQueXQQ7aq702210PHWNGWl4PePiCUAd/9ST0I0bN17KY8qJcCr0MzF/naQ2AWVX38dJx9dcbiPr7Mc4MZPf+++/3+7zP7bvWZah8tKX4PAnrDNXu5wD2p2/fYntKetg10FfP7RNOp0KFwqcB0vr77vw+mMQXjvRsApiuwTKEy/itq2v3TaWHkQzq4DixyqWQZ2uwjjp0KBW1YmXmJInV/2l17lM7QAePny42CozR+eoBvHo0aP2X1BvXEiMgc6CwSsDMOqbzu/mzZuLo+uFwR8d0jpmYdYFdkQbQrfYJu1q7EUKNsMMhUMa3tGTjy4eaYc1Az7S0IAbm2IWKLKuEyF2F9sOtuwPQ60byq6LTE46zOCMbV/odeh1UZQ9vsJSs0K1A9O5+7EIM3lc1PrJmJlndEIdxZP0qhkqL/avQXYJwnW1y3UPYBmE0mY16Id1njdrGeqH+nQ6hqH2Eu8G1sCdFHQc5YmDZQafnIO4K1fb93BXhrQJT/rLtAXsmrRiP4Nf7Qw3b5yiXcqeaLc+GbKNfe22sfQgWrdnZHBdA8Nr164ttl40ICqHRqR4ng5GUANpcNuCOGOuBqfmqQ7MTzYYFMY8BhoOnTaNT7J04Z2EXnHDBUDXCVqdBrd2NeiloXMbxvOaOhvkM5d0lGPLPhfUg9+yoiPYdTiRqHNSPY7tZGlnng5w+1BLh7AJOlhd8KJHdOc2DX5SIy3C6BV/6ni9zRFmmVuTY+ACgLbjbWDsrA62q/LFsk+Bdip9nDhx4sh/LQw0Y/9CPeCnARPtNoahzmnjtczVjwnSUl3wT914fw/0NfTX9EPLLuUYy1B5uRD3vjEOzIba5dg+ehmwW8rC4BO3zCSCL7uZow04Q/3QkE5rIA90UXPu1ySS+jJc1+QBfSO26uc6tw8H2+Jcjl3X1DdlZKBK+LGvrCOu6pv6Im/kxNYEutDYqgbCypZw8YJsqK/VQB7mtqGdoVHc0pCMXGNQh9tNx364eJ1t+TcVt4j5Ao/DscbIDrdxXfGUZvSvYWqeoGM44k3F85GjTEK6k2uu2Nv/B4tF/ZJZTihdT8v1j1MaYyAN4ikNL7unjb/rFxfLQpiSTKV4pXLGuB6vSy9Q8huDl1/O9aw6kmM/yhN1gezAv5djSh1BlFHyuc6QATwccgL2gxxunwovuspQqlMoyeT75F3SncvAtiiFhVpde/yxeHrSGel6mSTPWGK5VA9O1KXnFfXVZQvg/irHFGJddoGcsSxjQBeeT6xbytBlf+D+Xt6SzrVNHuy7n6cZZVBahBmj+5qyAf/uL+cyjUHxkbskb409RZlEX/n7dDqGWN+4aGOeD9vIK33GumdfuL/SLIUfqjuOO9LLWFyf5CG8jlSuWjyuO0+/T0de1ql1uOtMH01UIiW74pMy6Mk7miTZBHSsseNPkmXhXJB2tRxd54exg6fjCvYVB9HbyLbUV9rT8szyYGGSJEmS9MHt/HUv5ThO+NKIyOuvv77YSrYd6nEbvj7I0g8t6YmkPdWTg+gkSZJk5fBA5LrfynGc4EG90nM0rEVdx+tSk3lgLXftmuVVwsPhemWdw1pxvZYyqWAxI70S+tZEJZ/i64pwecsz2RRuh3lLL1kWbEj2lCxPaQ1wLpV8gdsabpvgnC65fL3xpnG55PZ1bfNUXuGnUVySJEmSJEmSJJXkco4kSZIkSZIkGUkOopMkSZIkSZJkJDmITpIkSZIkSZKRHItBNF/T4Yteq/hCVJIkSZIkSZJEjsUgmtcm8Xxk7euTVvF5St79mIP4zcJ7L7s+z5okSZIkSTIne7mcg+/2z83777+/2Eo2Be+hTZIkSZIkWQd7N4he1Sz0Rx99tNhLNgGz0I8fP17sJUmSJEmSrJbZBtF85YZ1ybjSQFXHcIR1GADhD5/73Ofabf4dhSEuyyb8E6geXygdwkke4moWWmkJpRH9AT+Ok5bCKH/89dWf1157rT1WgmUGiud5aQmI9mO5oU82rQeXi7ov6YF/hRel+OSFUx5+TOnipAtQWP49H8ovHeBiOcDDe16AX1cdEJYvecGZM2cOdej5sR3TTJIkSZIkmQwfW1mWZuBy+DVCfaXQv6JU2icO+BeY8ONrPvqCn+KQpr6eFtP3+IKv8Oi4vsgjFN4hDE5wXPtKG9nkx7/kB8nU9aUf/yKh64oy6StLEMsNfbLF8Cob/kC4MXrwMkouyYi/juGvspK+9qUHHGkpb7ZJQzor6Yvj0ovKpX1PUzLwr/RAcqjswHHtS4YkSZIkSZI5ODqKmoAGLw4DFg2A+PfBDiiOBlka0MUBkNJgsOVpEE9xQfEFAyzFBR88xbAQ94nvfmxr8AaxzKVBYSQODKFLFg8Tj7tsSlP5RjnG6gEdezkJH8NQdg8DhJGfZPD64ZjXn+RWGPajjZC3+3keEOtA+9GGvE5yEJ0kSZIkyVwsvZzj6dOn7S1057vf/e7B22+/3W7fu3fvpSUKX/rSl9p/4jqf/exnF1svePLkSfvPWzeawVF7W560iK80Spw/f769vU94lgB8+OGHiyMvE5dT4LTkgyUA4vTp04utT/HjtZw6dWqxNcyQbOirqcNWPyxVYDmJM0YPXTQDz8XWC+7fv9/K4DJB1MXJkycXW8M8evTosH7lWGOOnzO2Di5cuHC4xAZdTil/kiRJkiRJiZ15sJDB4pUrVw4HW74ON8IAm/CgQeQQhI8uDuo3RZ9sWkfMgPnBgwetn5iihxqohyjPsgNULsRimrhlePfddw91wmC6tA47SZIkSZJkCksPoplZZWAbZwSZ+QQGe8wq+nFtnzt3rv0fgllEBs0MihhYMYi7du3a4ujL6AEyH4hJnsiJEyfaf836CsL3zXKugyHZ0AmzwpSxNDPfp4cxM+IOs8Eff/zxYu9Tlnloj1nrkg0tkyYQX+8QZzCNHUZdJkmSJEmSTGHpQTSDN2YRuXUufDDEwBf8+M2bN9tlArUfRwHegKF0GcjFJSIRn3VEPg0a9U9aDLIY5DMoj0shWIZSOxOtwe6zZ8/age1cA7Uh2bQcRnp5+PBh++906UFIVgbXDGQZlPcNXlmmQzgPQxqlpRa1YAfYg9sIxEF1H1o+8vz588OLDAb7umhQHek/SZIkSZJkKT6ZCZKSawZrC99P8ePNwHDh++mDcnLNAO3IPg+D8XAYD46RrvxFKT5+xJGf5wfyjw+/yR9HOlEWwuiBPDnyAeRUmIjLgiONmF8M4+mUZBPu7+GUR58eJLPCo1+FcV0P1SfpgB7uk2Pf88CVdCFiWBhTB5JZacawXt8K67pMkiRJkiSp5RV+msFEkiRJkiRJkiSV7MyDhUmSJEmSJEmyLeQgOkmSJEmSJElGkoPoJEmSJEmSJBlJDqKTJEmSJEmSZCQ5iE6SJEmSJEmSkeQgOkmSJEmSJElGcXDw/wNDn62durgwrgAAAABJRU5ErkJggg==\\\"\\u003e\\u003cbr\\u003e\\u003c/p\\u003e\\n\\u003ch2\\u003e5.6 \\u0026nbsp; User Experience\\u003c/h2\\u003e\\n\\u003cp\\u003eUser experience (UX) plays a pivotal role in the testing phase of chatbot development for a variety of reasons. At its core, a positive user experience is synonymous with user satisfaction. When users find the interaction with a chatbot enjoyable and seamless, they are more likely to continue engaging with it and extracting value from its func- tionalities. The intuitive nature of a chatbot\\u0026rsquo;s design is a key factor in user adoption. An easily navigable and user-friendly interface encourages users to actively participate, making the chatbot a valuable tool for accomplishing its intended objectives.\\u003c/p\\u003e\\n\\u003cp\\u003eThe effectiveness of a chatbot is closely intertwined with user experience. A well- designed chatbot allows users to effortlessly navigate, access information, or receive assistance, contributing to the overall success of the chatbot in meeting user needs. Success\\u0026nbsp;is\\u0026nbsp;also\\u0026nbsp;measured\\u0026nbsp;by\\u0026nbsp;the\\u0026nbsp;users\\u0026rsquo;\\u0026nbsp;ability\\u0026nbsp;to\\u0026nbsp;complete\\u0026nbsp;tasks\\u0026nbsp;successfully\\u0026nbsp;and\\u0026nbsp;achieve their goals within the chatbot interface. A positive user experience fosters user reten- tion, as users are more likely to return for future interactions if their initial experience was positive.\\u003c/p\\u003e\\n\\u003cp\\u003eImportantly, a well-designed chatbot helps minimize user frustration. By identify- ing potential pain points and areas of confusion during testing, developers can make improvements\\u0026nbsp;that\\u0026nbsp;enhance\\u0026nbsp;the\\u0026nbsp;user\\u0026nbsp;experience,\\u0026nbsp;reducing\\u0026nbsp;frustration\\u0026nbsp;and\\u0026nbsp;ensuring a smoother interaction. Gathering feedback on the user experience during testing is invaluable for iterative improvements. Understanding user preferences, expectations, and challenges allows developers to refine the chatbot for better performance over time.\\u003c/p\\u003e\\n\\u003cp\\u003eThe user experience also contributes to the overall brand image associated with the chatbot. A positive experience enhances the brand perception and fosters a favorable impression among users. Additionally, considering the diversity of users \\u0026ndash; in terms of background, technical expertise, and communication preferences \\u0026ndash; is vital. A well- crafted user experience ensures that the chatbot is accessible and usable by a broad range of individuals.\\u003c/p\\u003e\\n\\u003cp\\u003eIn a competitive landscape where multiple chatbots may offer similar function- alities, a superior user experience becomes a significant differentiator. It can give a chatbot\\u0026nbsp;a\\u0026nbsp;distinct\\u0026nbsp;advantage\\u0026nbsp;over\\u0026nbsp;others\\u0026nbsp;in\\u0026nbsp;the\\u0026nbsp;same\\u0026nbsp;domain.\\u0026nbsp;Ultimately,\\u0026nbsp;prioritiz- ing user experience in chatbot testing is essential for creating a chatbot that not only functions correctly but is also well-received and embraced by its users. This user- centric approach contributes to the overall success and effectiveness of the chatbot in delivering a positive and satisfying user experience.\\u003c/p\\u003e\\n\\u003cp\\u003eThroughout this research, we have emphasized one of the key objectives of this study, which is to develop an additional system for international students to access general information about the university. A crucial aspect of this objective is to ensure the system is user-friendly and provides a positive user experience, irrespective of the user\\u0026rsquo;s\\u0026nbsp;background.\\u0026nbsp;The\\u0026nbsp;quality\\u0026nbsp;of\\u0026nbsp;user\\u0026nbsp;experience\\u0026nbsp;is\\u0026nbsp;influenced\\u0026nbsp;by\\u0026nbsp;three\\u0026nbsp;main\\u0026nbsp;factors:\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull;\\u0026nbsp;\\u0026nbsp;The\\u0026nbsp;ease-of-use\\u0026nbsp;of\\u0026nbsp;the\\u0026nbsp;User\\u0026nbsp;Interface\\u0026nbsp;(UI).\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull;\\u0026nbsp;\\u0026nbsp;The\\u0026nbsp;abilities\\u0026nbsp;of\\u0026nbsp;the\\u0026nbsp;chatbot\\u0026nbsp;in\\u0026nbsp;understanding\\u0026nbsp;and\\u0026nbsp;responding\\u0026nbsp;to\\u0026nbsp;questions.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull;\\u0026nbsp;\\u0026nbsp;The behavior of the chatbot towards the user, aiming to avoid any awkwardness or difficulty in understanding.\\u003c/p\\u003e\\n\\u003cp\\u003eEach of these factors is the responsibility of a different component of AEdBOT: its UI, Natural Language Understanding (NLU) component, and Natural Language Generation (NLG) unit.\\u003c/p\\u003e\\n\\u003cp\\u003eQuantifying these aspects can be challenging as they are not easily measurable in\\u0026nbsp;a quantitative manner. A common approach to assess these factors is to gather user feedback by asking individuals to grade their experience on a given scale. To achieve\\u0026nbsp;this, we conducted tests where a selected group of external individuals, representing the intended users (present and future international students), interacted with AEdBOT. We employed the System Usability Scale (SUS) [\\u003ca href=\\\"#_bookmark86\\\"\\u003e55\\u003c/a\\u003e], a widely used industry standard,\\u0026nbsp;to\\u0026nbsp;evaluate\\u0026nbsp;usability.\\u0026nbsp;In\\u0026nbsp;this\\u0026nbsp;test, users\\u0026nbsp;are\\u0026nbsp;asked\\u0026nbsp;to\\u0026nbsp;rate\\u0026nbsp;10\\u0026nbsp;statements\\u0026nbsp;on\\u0026nbsp;a\\u0026nbsp;scale\\u0026nbsp;from \\u0026rdquo;strongly disagree\\u0026rdquo; to \\u0026rdquo;strongly agree.\\u0026rdquo;\\u003c/p\\u003e\\n\\u003cp\\u003eThe scores obtained are typically divided into five steps and assigned numerical values (1 for \\u0026rdquo;strongly disagree\\u0026rdquo; and 5 for \\u0026rdquo;strongly agree\\u0026rdquo;). The results are then used to calculate a score out of 100. It\\u0026rsquo;s important to note that this score is not a percentage; rather, it indicates how convenient and user-friendly the system appears to first-time users, allowing for comparisons between different systems.\\u003c/p\\u003e\\n\\u003cp\\u003eWe conducted the test with several participants and aggregated their results by computing the mean and median scores for each question, ultimately deriving the final score described above.\\u003c/p\\u003e\\n\\u003cp\\u003eThe 10 questions asked in the standard SUS [55] test, as well as in our conducted test, are as follows:\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull;\\u0026nbsp;\\u0026nbsp;I\\u0026nbsp;think\\u0026nbsp;that\\u0026nbsp;I\\u0026nbsp;would\\u0026nbsp;like\\u0026nbsp;to\\u0026nbsp;use\\u0026nbsp;this\\u0026nbsp;system\\u0026nbsp;frequently.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; \\u0026nbsp;I found the system unnecessarily complex.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; \\u0026nbsp;I thought the system was easy to use.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull; \\u0026nbsp;I think that I would need the support of a technical person to be able to use this system.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull;\\u0026nbsp;\\u0026nbsp;I\\u0026nbsp;found\\u0026nbsp;the\\u0026nbsp;various\\u0026nbsp;functions\\u0026nbsp;in\\u0026nbsp;this\\u0026nbsp;system\\u0026nbsp;were\\u0026nbsp;well\\u0026nbsp;integrated.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull;\\u0026nbsp;\\u0026nbsp;I\\u0026nbsp;thought\\u0026nbsp;there\\u0026nbsp;was\\u0026nbsp;too\\u0026nbsp;much\\u0026nbsp;inconsistency\\u0026nbsp;in\\u0026nbsp;this\\u0026nbsp;system.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull;\\u0026nbsp;\\u0026nbsp;I\\u0026nbsp;would\\u0026nbsp;imagine\\u0026nbsp;that\\u0026nbsp;most\\u0026nbsp;people\\u0026nbsp;would\\u0026nbsp;learn\\u0026nbsp;to\\u0026nbsp;use\\u0026nbsp;this\\u0026nbsp;system\\u0026nbsp;very\\u0026nbsp;quickly.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull;\\u0026nbsp;\\u0026nbsp;I\\u0026nbsp;found\\u0026nbsp;the\\u0026nbsp;system\\u0026nbsp;very\\u0026nbsp;cumbersome\\u0026nbsp;to\\u0026nbsp;use.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull;\\u0026nbsp;\\u0026nbsp;I\\u0026nbsp;felt\\u0026nbsp;very\\u0026nbsp;confident\\u0026nbsp;using\\u0026nbsp;the\\u0026nbsp;system.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u0026bull;\\u0026nbsp;\\u0026nbsp;I\\u0026nbsp;needed\\u0026nbsp;to\\u0026nbsp;learn\\u0026nbsp;a\\u0026nbsp;lot\\u0026nbsp;of\\u0026nbsp;things\\u0026nbsp;before\\u0026nbsp;I\\u0026nbsp;could\\u0026nbsp;get\\u0026nbsp;going\\u0026nbsp;with\\u0026nbsp;this\\u0026nbsp;system.\\u003c/p\\u003e\\n\\u003cp\\u003eAfter interviewing 11 people, the average grades for each question, in the same order as above, are as follows in Table\\u0026nbsp;\\u003ca href=\\\"#_bookmark27\\\"\\u003e7\\u003c/a\\u003e:\\u003c/p\\u003e\\n\\u003cp\\u003eThis yields a total mean score of 77.5 out of 100 and a total median score of 85. Of course, interpreting those results heavily depends on the type of system at hand. Nevertheless, knowing that the mean score for this test is 68, this result is very good, and the system seems easy to use and understandable even to non-technical users.\\u003c/p\\u003e\"},{\"header\":\"6\\tComparative Analysis\",\"content\":\"\\u003cp\\u003eWe compared the proposed chatbot methodology with the developed approaches in the literature to identify areas where we can improve performance. By comparing the performance of our developed chatbot to other approaches, we can identify areas where we need to improve in the proposed approach and take steps to improve. We per- formed the theoretical and experimental comparison with the state-of-the-art chatbot concerning the performance as given in Section 6.1 and Section 6.2, respectively.\\u003c/p\\u003e\\n\\u003ch2\\u003e6.1 Theoretical Comparison\\u003c/h2\\u003e\\n\\u003cp\\u003eWe identified the various chatbots in the literature review (Section 2) that are built for the administrative area of the educational domain [22],[23],[18],[24],[19],[26],[15],[16], [29],[25],[30]. But unfortunately, their data set is private to perform comparative analysis. We got only one data set [20] that was available publicly. So, to perform a comparative analysis, we decided to compare our proposed methodology with the chatbot [56]. University students from Belgium developed chatbot [56] for their administrative support at the university level [56]. The produced chatbot, named ’JAQ’, is aimed to be a Proof of Concept of an additional way for those students to obtain general information about the university. The scope of topics relevant to chatbot being very extensive, the authors implemented a complete system on top of DialogFlow platform to aid with data management and maintenance. The theoretical comparison between our proposed methodology chatbot and the chatbot [56] is performed in Table 8.\\u003c/p\\u003e\\n\\u003ch2\\u003e6.2 Experimental Comparison\\u003c/h2\\u003e\\n\\u003cp\\u003eTo test the chatbot as mentioned above [56] with our proposed methodology, we settled on an environment to test the validity of our proposed approach. A Core i5 6th generation – 2.56 GHz Processor with 12 GB RAM having an Amd Radeon r5 430 -2GB graphic card is used on the hardware side. For testing purposes, the same NLP pipeline is used that we proposed to develop the AEdBOT. The action confusion matrix is shown in Fig. 22.\\u003c/p\\u003e\\n\\u003cp\\u003eWe ran the 121 examples of test data as mentioned in the dataset [20] on our proposed methodology that we adopted to develop AEdBOT and performed end-to- end level and action-level testing. To run the test examples on the Hybrid model to evaluate the performance of the proposed methodology, we used the ‘rasa evaluate’ command to generate a confusion matrix for the Hybrid model. This command will run the test data set on the Hybrid model and summarize the model’s performance, including the confusion matrix. Then we used the ‘rasa test’ command to get a detailed breakdown of the predictions made by the Hybrid model on a specific test set [20]. Out of 121 test examples, the Hybrid model predicts 78 test examples correctly at end-to-end level testing. At the action level testing, the Hybrid model predicted 182 stories correctly out of 239. The experimental result comparison graph and proposed framework’s performance evaluation results between the chatbot [20] and our proposed Deep Learning (DL)-based Hybrid model methodology chatbot are shown in Fig. 23 and Table 9, respectively\\u003c/p\\u003e\"},{\"header\":\"7\\tDiscussion and Future Work\",\"content\":\"\\u003cp\\u003eTo discuss whether AEdBOT is a good Proof of Concept for an assistant chatbot, we can look at the results of our different experiments; in terms of performance, we could argue that AEdBOT works well enough to be usable with 96% accuracy. More- over, upon misclassification, AEdBOT recommends something related to the topic that the user would like to know about based on conversation history, making that misclassification less annoying than if it had given information utterly unrelated to the question. During the conception of AEdBOT, we roughly tracked the evolution of the NLU performance and saw that it varied. The first challenge that stood out during those experiments is that the proposed NLP pipeline needs deterministic model train- ing. Indeed, running the same experiments twice on the default Rasa NLP pipeline yields similar results. However, training two models, such as the BERT and DIET Classifier, with the same training data and executing the same experiments produces different results. We performed a fine-tuning by comparing different configurations in the customized NLP pipeline and concluded that: 1) A small confidence threshold (0.3 compared to the default 0.6) improves the overall performances; 2) Augmenting the number of examples in the training set by only adding a question mark at the end of some examples does not improve performance; 3) Increasing the number of examples in the training set never decreases performance. Moreover, from a user perspective, capitalizing the first letter of a message or adding a question mark at the end does not influence its classification. Logically, defining many intents gives the model more classes to classify user messages as and, thus, more potential incorrect classifications. Therefore, we anticipated that activating the Smalltalk module would decrease the performance, as it would add many new intents. On the contrary, removing some of the intents we defined and whose examples were very close to examples of other intents improved the classification performances. Verifying the correctness of generated responses from a chatbot is a crucial step in ensuring the quality and reliability of the proposed system. A knowledge-based integration in AEdBOT relies on structured data and cross-reference responses with the data in the knowledge base to ensure that information is accurate and up to date.\\u003c/p\\u003e\\n\\u003cp\\u003eAnother challenge we identified during experiments was how well AEdBOT worked when users spoke poor English. It is extremely relevant since AEdBOT is meant to be used by people from abroad whose native language might not be English. Therefore, having a chatbot that behaves well in this situation is essential. When a user message contains a typo or slightly off grammar, the NLU component could still classify the intents with the same performance. However, when the question was phrased utterly incorrectly, the chatbot would either need help understanding it or completely misclassifying it. In other words, the poorer the quality of the English in the user messages, the worse the user experience was. From a user-friendliness perspective, AEdBOT is thus able to cope with users who speak bad English but not with users whose first language is not English. To make AEdBOT multilingual, AEdBOT\\u0026rsquo;s NLP pipeline can use multiple pre-trained language models, such as spaCy or Hugging Face\\u0026rsquo;s transformers, and machine translation services, such as Google Cloud Translation, and Microsoft Azure Translator, which support various languages. These models and services can process the user\\u0026rsquo;s input in their languages, and then the output can be passed to the AEdBOT.\\u003c/p\\u003e\\n\\u003cp\\u003eIn the future, we will add the principles of continuous learning and adaptation of the Never-ending learning (NEL) [57] approach in AEdBOT that enable it to proac- tively discover new user intents through active engagement, monitoring interactions, and topic modeling, ensuring its knowledge base remains current and relevant. To improve over time, the AEdBOT leverages feedback loops, version control, and adap- tive learning, allowing it to adjust responses to user expectations and enhance overall user satisfaction. Knowledge enrichment is fundamental through data enrichment, knowledge graph expansion, and integration of external data sources. It ensures the AEdBOT can respond accurately to a wide range of queries, making it a dynamic and valuable conversational partner. In addition, developing an advanced automated SQL query generation algorithm through NLP are other potential areas for improvement. However, these improvements would require further investigation to avoid overfitting and classification issues.\\u003c/p\\u003e\"},{\"header\":\"Conclusion\",\"content\":\"\\u003cp\\u003eIn this paper, we were interested in studying the modern chatbot Artificial Intelligence (AI) approaches and platforms for building an effective chatbot, AEdBOT, to assist local and international students of Educational Institutions. The difference with previ- ous works was developing a novel Deep Learning (DL)-based Hybrid Model approach on top of the RASA platform with a customized NLP pipeline. The developed NLP pipeline can generate a response by efficient prediction of intent and entity extraction from Natural Language (NL)-based text and process text as a whole by using the attention base mechanism of the transformer and provide the chatbot with greater predictive accuracy to act as a recommendation system. We performed several experi- ments and a comparative analysis to assess its performance and usability. During those experiments the Deep Learning (DL)-based Hybrid Model approach performed well on the generated dataset and achieved encouraging results on performance metrics like Precision, Accuracy, Recall, and F1-Score, which come out to be 94.7%, 96.0%, 96.0%, and 95.1%, respectively, with a response time from the mean and standard deviation is 216.43 ms and 50.09 ms, respectively on an average basis. We also performed the User experience (UX) testing in the testing phase of chatbot development and AEdBOT showed its user-friendliness with a total mean score of 77.5 out of 100 and a total median score of 85 on the System Usability Scale (SUS), this result is very good, even though there is room for improvement. The proposed methodology is compared with the state-of-the-art chatbot and outperformed as well on it with an accuracy of 76.2%. We concluded that the challenges faced in developing a chatbot with a large scope make it difficult to create a training set of good quality and can decrease the performance and user experience. The modeler design and structured implementation methodology enable further improvements to be incorporated, i.e., adding more questions, conversational flows, and examples of utterances and entity values easily. Despite these challenges, AEdBOT fulfills its intended purpose as a practical expert chatbot system and is already operational in helping students query information. The results proved that the chatbot outperformed well, and users perceived it as enjoyable and user-friendly.\\u003c/p\\u003e\"},{\"header\":\"Declarations\",\"content\":\"\\u003cp\\u003e\\u003cstrong\\u003eFunding \\u003c/strong\\u003eThis work is partly supported by the Higher Education Commission (HEC), Pakistan, the National University of Science and Technology (NUST), Pakistan, and Malardalen University, Sweden.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eConflict of interest \\u003c/strong\\u003eThe authors have no relevant financial or non-financial interests to disclose.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eAuthor contribution \\u003c/strong\\u003eMaterial Preparation, Data Collection, Methodology Design, Experimentation and Analysis were performed by Muhammad Shahroze Ali. The first draft of the manuscript was written by Muhammad Shahroze Ali and Muhammad Waseem Anwar commented on previous versions of the manuscript. All authors read and approved the final manuscript.\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eData availability \\u003c/strong\\u003eThe Project Data is publicly available for the online communi- ties on [51].\\u003c/p\\u003e\\n\\u003cp\\u003e\\u003cstrong\\u003eCode availability \\u003c/strong\\u003eThe Project Code is publicly available for the online communities on [45].\\u003c/p\\u003e\"},{\"header\":\"References\",\"content\":\"\\u003col\\u003e\\n \\u003cli\\u003eRoos, S.: CHATBOTS IN EDUCATION A passing trend or a valuable pedagog- ical tool ? In: Chatbots in Education A, p. 58 (2018).\\u003c/li\\u003e\\n \\u003cli\\u003eWu, E.H.K., Lin, C.H., Ou, Y.Y., Liu, C.Z., Wang, W.K., Chao, C.Y.: Advantages and constraints of a hybrid model K-12 E-Learning assistant chatbot. IEEE Access \\u003cstrong\\u003e8\\u003c/strong\\u003e, 77788\\u0026ndash;77801 (2020) https://doi.org/10.1109/ACCESS.2020.2988252\\u003c/li\\u003e\\n \\u003cli\\u003eFakhri, S.A., Lutfi, H.U., Wardana, W.K., ...: Aplikasi Chatbot Informasi Kam- pus Polban Menggunakan Aplikasi LINE Messenger. https://jurnal.polban.ac.id/ proceeding/article/view/1403 (2019)\\u003c/li\\u003e\\n \\u003cli\\u003eRanoliya, B.R., Raghuwanshi, N., Singh, S.: Chatbot for university related FAQs. In: 2017 International Conference on Advances in Computing, Communications and Informatics, ICACCI 2017, vol. 2017-January, pp. 1525\\u0026ndash;1530 (2017). https: //doi.org/10.1109/ICACCI.2017.8126057\\u003c/li\\u003e\\n \\u003cli\\u003eNguyen, T.T., Le, A.D., Hoang, H.T., Nguyen, T.: NEU-chatbot: Chatbot for admission of National Economics University. Elsevier Ltd (2021). https://doi. org/10.1016/j.caeai.2021.100036\\u003c/li\\u003e\\n \\u003cli\\u003eClarizia, F., Colace, F., Lombardi, M., Pascale, F., Santaniello, D.: Chatbot: An education support system for student. In: 2018 International Symposium on Cyberspace Safety and Security, Amalfi, pp. 291\\u0026ndash;302 (2018). https://doi.org/10. 1007/978-3-030-01689-0 23\\u003c/li\\u003e\\n \\u003cli\\u003eOkonkwo, C.W., Ade-Ibijola, A.: Python-bot: A chatbot for teaching python programming, vol. 29, pp. 25\\u0026ndash;34 (2021)\\u003c/li\\u003e\\n \\u003cli\\u003ePham, X.L., Pham, T., Nguyen, Q.M., Nguyen, T.H., Cao, T.T.H.: Chatbot as an intelligent personal assistant for mobile language learning. In: Proceedings of the 2018 2nd International Conference on Education and E-Learning, pp. 16\\u0026ndash;21 (2018)\\u003c/li\\u003e\\n \\u003cli\\u003eHien, H.T., Cuong, P.-N., Nam, L.N.H., Nhung, H.L.T.K., Thang, L.D.: Intel- ligent assistants in higher-education environments: The fit-ebot, a chatbot for administrative and learning support. In: Proceedings of the Ninth International Symposium on Information and Communication Technology (2018)\\u003c/li\\u003e\\n \\u003cli\\u003eGoodfellow, I., Bengio, Y., Courville, A.: Deep learning. In: The MIT Press, Cambridge, MA (2016)\\u0026nbsp;\\u003c/li\\u003e\\n \\u003cli\\u003eOpenAI: GPT-4 Technical Report. http://arxiv.org/abs/2303.08774 (2023)\\u003c/li\\u003e\\n \\u003cli\\u003eChan, A.: GPT-3 and InstructGPT: technological dystopianism, utopianism, and \\u0026ldquo;Contextual\\u0026rdquo; perspectives in AI ethics and industry. AI and Ethics \\u003cstrong\\u003e3\\u003c/strong\\u003e(1), 53\\u0026ndash;64 (2023) https://doi.org/10.1007/s43681-022-00148-6\\u003c/li\\u003e\\n \\u003cli\\u003eYenduri, G., M, R., G, C.S., Y, S., Srivastava, G., Maddikunta, P.K.R., G, D.R., Jhaveri, R.H., B, P., Wang, W., Vasilakos, A.V., Gadekallu, T.R.: Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions, 1\\u0026ndash;40 (2023) arXiv:2305.10435\\u003c/li\\u003e\\n \\u003cli\\u003eYang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., Le, Q.V.: XLNet: Generalized autoregressive pretraining for language understanding. In: Advances in Neural Information Processing Systems, vol. 32, pp. 1\\u0026ndash;18 (2019)\\u003c/li\\u003e\\n \\u003cli\\u003eWindiatmoko, Y., Hidayatullah, A.F.: Developing FB chatbot based on deep learning using RASA framework for university enquiries. https://arxiv.org/pdf/ 2009.12341. ArXiv Preprint ArXiv:2009.12341 (2020)\\u003c/li\\u003e\\n \\u003cli\\u003eNguyen, M.T., Tran-Tien, M., Viet, A.P., Vu, H.T., Nguyen, V.H.: Building a Chatbot for Supporting the Admission of Universities, vol. 2021-November (2021). https://doi.org/10.1109/KSE53942.2021.9648677\\u003c/li\\u003e\\n \\u003cli\\u003eZhang, Y., Sun, S., Galley, M., Chen, Y.C., Brockett, C., Gao, X., Gao, J., Liu, J., Dolan, B.: DIALOGPT: Large-scale generative pre-training for conversational\\u003c/li\\u003e\\n \\u003cli\\u003eresponse generation. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 270\\u0026ndash;278 (2020). https://doi.org/10.18653/v1/ 2020.acl-demos.30\\u003c/li\\u003e\\n \\u003cli\\u003eSai Vikas, G.S., Kumar, I.D., Shareef, S.A., Roy, B.R., Geetha, G.: Informa- tion chatbot for college management system using multinomial naive bayes. In: 2021 2nd International Conference on Smart Electronics and Communica- tion (ICOSEC), Trichy, India, pp. 1149\\u0026ndash;1153 (2021). https://doi.org/10.1109/ ICOSEC51865.2021.9591757\\u003c/li\\u003e\\n \\u003cli\\u003eRana, M.: EagleBot: A Chatbot-Based Multi-Tier Question Answering Sys- tem for Retrieving Answers from Heterogeneous Sources Using BERT. https: //digitalcommons.georgiasouthern.edu/etd (2019) gellens: Master thesis JAQ code. https://github.com/gellens/Master thesis JAQ code (2024)\\u003c/li\\u003e\\n \\u003cli\\u003eNouman, N., Shaikh, Z.A., Wasi, S.: A novel personalized learning framework with interactive e-mentoring. IEEE Access \\u003cstrong\\u003e12\\u003c/strong\\u003e(November 2023), 10428\\u0026ndash;10458 (2024) https://doi.org/10.1109/ACCESS.2024.3354167\\u003c/li\\u003e\\n \\u003cli\\u003eBala, K., Kumar, M., Hulawale, S., Pandita, S.: Chat-Bot for College Management System Using A.I. www.irjet.net. (September) (2017)\\u003c/li\\u003e\\n \\u003cli\\u003eVichare, A., Shrikhande, Y., Gyani, A., Rathod, N.: Derek: A chatbot that shows intelligence in behavior using nlp, pp. 232\\u0026ndash;235 (2016). (March)\\u003c/li\\u003e\\n \\u003cli\\u003eRaees, W., Ismail, M.A., Aziz, R., Afshan, A.: NED Chatbot for Admission Related Queries Using Prescriptive Analysis, vol. 19, pp. 133\\u0026ndash;138 (2019)\\u003c/li\\u003e\\n \\u003cli\\u003eMahanan, W., Thanyaphongphat, J., Sawadsitang, S., Sangamuang, S.: College agent: The machine learning chatbot for college tasks. In: 7th International Con- ference on Digital Arts, Media and Technology, DAMT 2022 and 5th ECTI Northern Section Conference on Electrical, Electronics, Computer and Telecom- munications Engineering, NCON 2022, pp. 329\\u0026ndash;332 (2022). https://doi.org/10. 1109//ECTI-DAMTNCON53731.2022.9720420\\u003c/li\\u003e\\n \\u003cli\\u003eThakkar, J., Raut, P., Doshi, Y., Parekh, K.: Erasmus ai chatbot, vol. 6, pp. 498\\u0026ndash;502 (2018). https://doi.org/10.26438/ijcse/v6i10.498502\\u003c/li\\u003e\\n \\u003cli\\u003eMeshram, S., Naik, N., Megha, V.R., More, T., Kharche, S.: College enquiry chatbot using rasa framework. In: 2021 Asian Conference on Innovation in Tech- nology, ASIANCON, pp. 1\\u0026ndash;8 (2021). https://doi.org/10.1109/ASIANCON51346.\\u003c/li\\u003e\\n \\u003cli\\u003eLee, K., Jo, J., Kim, J., Kang, Y.: Can chatbots help reduce the workload of administrative officers? - implementing and deploying faq chatbot service in a univerity. International Conference on Human-Computer Interaction (2019)\\u003c/li\\u003e\\n \\u003cli\\u003eGonsalves, X., Deshmukh, S.: Designing an interactive chatbot for educational assistance using rasa framework. In: IDCIoT 2023 - International Conference on Intelligent Data Communication Technologies and Internet of Things, Proceed- ings, pp. 68\\u0026ndash;74 (2023). https://doi.org/10.1109/IDCIoT56793.2023.10053457\\u003c/li\\u003e\\n \\u003cli\\u003eChannabasamma, Lakshmi Soumya, P., Indu, M., Swetha, N., Saritha, C.: Smart chatbot for college information enquiry using deep neural network. In: 2023 9th International Conference on Advanced Computing and Communica- tion Systems, ICACCS 2023, vol. 1, pp. 991\\u0026ndash;994 (2023). https://doi.org/10.1109/ ICACCS57279.2023.10112919\\u003c/li\\u003e\\n \\u003cli\\u003eBraun, D., Mendez, A.H., Matthes, F., Langen, M.: Evaluating natural lan- guage understanding services for conversational question answering systems. In: SIGDIAL 2017 - 18th Annual Meeting of the Special Interest Group on Dis- course and Dialogue, Proceedings of the Conference, pp. 174\\u0026ndash;185 (2017). https: //doi.org/10.18653/v1/w17-5522\\u003c/li\\u003e\\n \\u003cli\\u003eKang, A.: Understanding the Differences between Alexa, API.ai, Wit.ai, and LUIS/Cortana. https://medium.com/@abraham.kang/ understanding-the-differences-between-alexa-api-ai-wit-ai-and-luis cortana-2404ece0977c. Accessed: 2018-8-11 (2017)\\u003c/li\\u003e\\n \\u003cli\\u003eBobriakov, I.: A Comparative Analysis of Chatbots APIs. https://medium.com/activewizards-machinelearning-company/ a-comparative-analysis-of-Chatbots-APIs-f9d240263e1d. Accessed: 2018-8-13 (2018)\\u003c/li\\u003e\\n \\u003cli\\u003eLiu, G., Wang, S., Yu, J., Yin, J.: A survey on multimodal dialogue systems: Recent advances and new frontiers. In: Proceedings \\u0026ndash; 2022 5th International Con- ference on Advanced Electronic Materials, Computers and Software Engineering, AEMCSE 2022, pp. 845\\u0026ndash;853 (2022). https://doi.org/10.1109/AEMCSE55572.\\u003c/li\\u003e\\n \\u003cli\\u003eYan, R., Song, Y., Wu, H.: Learning to respond with deep neural networks for retrieval-based human-computer conversation system. In: Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR \\u0026rsquo;16), Pisa, Italy, pp. 55\\u0026ndash;64 (2016). ACM Press\\u003c/li\\u003e\\n \\u003cli\\u003eDevlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidi- rectional transformers for language understanding. In: NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, vol. 1(Mlm), pp. 4171\\u0026ndash;4186 (2019)\\u003c/li\\u003e\\n \\u003cli\\u003eDevlin, J., Chang, M.-W., Lee, K., Google, K.T., Language, A.I.: BERT: Pre- training of Deep Bidirectional Transformers for Language Understanding. Naacl- Hlt 2019 (Mlm), 4171\\u0026ndash;4186 (2018)\\u003c/li\\u003e\\n \\u003cli\\u003eRaffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text- to-text transformer. In: Journal of Machine Learning Research, vol. 21, pp. 1\\u0026ndash;67 (2020)\\u003c/li\\u003e\\n \\u003cli\\u003ePennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word rep- resentation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532\\u0026ndash;1543 (2014)\\u003c/li\\u003e\\n \\u003cli\\u003eHenderson, M., Casanueva, I., Mrkˇsi\\u0026acute;c, N., Su, P.-H., Wen, T.-H., Vulic, I.: Con- vert: Efficient and accurate conversational representations from transformers. In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2161\\u0026ndash;2174 (2020). https://doi.org/10.18653/v1/2020.findings-emnlp.196\\u003c/li\\u003e\\n \\u003cli\\u003eVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998\\u0026ndash;6008 (2017)\\u003c/li\\u003e\\n \\u003cli\\u003eBunk, T., Varshneya, D., Vlasov, V., Nichol, A.: DIET: Lightweight Language Understanding for Dialogue Systems. http://arxiv.org/abs/2004.09936 (2020)\\u003c/li\\u003e\\n \\u003cli\\u003eSharma, R.K.: An analytical study and review of open source chatbot framework, rasa. In: International Journal of Engineering Research And, vol. V9, pp. 1011\\u0026ndash; 1014 (2020). https://doi.org/10.17577/ijertv9is060723\\u003c/li\\u003e\\n \\u003cli\\u003eRasa: Connecting to Messaging and Voice Channels. https://rasa.com/docs/rasa/ messagingand-voice-channels. Accessed 5 June 2021 (2021)\\u003c/li\\u003e\\n \\u003cli\\u003eAli, M.S.: AEdBOT. https://github.com/MuhammadShahrozeAli/AEdBOT (2024)\\u003c/li\\u003e\\n \\u003cli\\u003ePimentel, R.: Chatito. https://github.com/rodrigopivi/Chatito. Consulted in April 2019 (2019)\\u003c/li\\u003e\\n \\u003cli\\u003eLeicher, J.: Chatl. https://github.com/atlassistant/chatl. Consulted in April 2019 (2019)\\u003c/li\\u003e\\n \\u003cli\\u003eBuck, M.: Expando. https://github.com/voxable-labs/expando. Consulted in April 2019 (2019)\\u003c/li\\u003e\\n \\u003cli\\u003eGalaxy, K.: Tracery. http://tracery.io/. Consulted in April 2019 (2019)\\u003c/li\\u003e\\n \\u003cli\\u003eGustin, S.: Chatette. https://github.com/SimGus/Chatette. Consulted in April 2019 (2019)\\u003c/li\\u003e\\n \\u003cli\\u003eAli, M.S.: AEdBOT - Data Repository. https://github.com/ MuhammadShahrozeAli/AEdBOT/tree/main/data (2024)\\u003c/li\\u003e\\n \\u003cli\\u003eRasa: Rasa Tutorial. https://rasa.com/docs/rasa/user-guide/rasa-tutorial/ (2024)\\u003c/li\\u003e\\n \\u003cli\\u003eAISB: Loebner Prize. http://aisb.org.uk/events/loebner-prize. Consulted in June 2019 (2019)\\u003c/li\\u003e\\n \\u003cli\\u003eShawar, B.A., Atwell, E.: Different measurements metrics to evaluate a chatbot system. In: Proceedings of the Workshop on Bridging the Gap: Academic and Industrial Research in Dialog Technologies, pp. 89\\u0026ndash;96 (2007). Association for Computational Linguistics\\u003c/li\\u003e\\n \\u003cli\\u003eBrooke, J., Jordan, P.W., Thomas, B., Weerdmeester, B.A., McClelland, I.L.: Sus: A quick and dirty usability scale. In: Usability Evaluation in Industry, vol. 189, pp. 4\\u0026ndash;7 (1996). https://doi.org/10.1016/B978-0-12-566251-8.50006-1\\u003c/li\\u003e\\n \\u003cli\\u003eGellens, A., Gustin, S.: Jaq: A chatbot for foreign students, Ecole polytechnique de Louvain, Universit\\u0026acute;e catholique de Louvain (2019)\\u003c/li\\u003e\\n \\u003cli\\u003eMitchell, T., Cohen, W., Hruschka, E., Talukdar, P., Yang, B., Betteridge, J., Carlson, A., Dalvi, B., Gardner, M., Kisiel, B., Krishnamurthy, J., Lao, N., Mazaitis, K., Mohamed, T., Nakashole, N., Platanios, E., Ritter, A., Samadi, M., Settles, B., Wang, R., Wijaya, D., Gupta, A., Chen, X., Saparov, A., Greaves, M., Welling, M.: Never-ending learning. In: Communications of the ACM, vol. 61, pp. 103\\u0026ndash;115 (2018). https://doi.org/10.1145/3191513\\u003c/li\\u003e\\n\\u003c/ol\\u003e\"},{\"header\":\"Tables\",\"content\":\"\\u003cp\\u003eTables 1 to 3 are available in the Supplementary Files section\\u003c/p\\u003e\"}],\"fulltextSource\":\"\",\"fullText\":\"\",\"funders\":[],\"hasAdminPriorityOnWorkflow\":false,\"hasManuscriptDocX\":true,\"hasOptedInToPreprint\":true,\"hasPassedJournalQc\":\"\",\"hasAnyPriority\":false,\"hideJournal\":true,\"highlight\":\"\",\"institution\":\"\",\"isAcceptedByJournal\":false,\"isAuthorSuppliedPdf\":false,\"isDeskRejected\":\"\",\"isHiddenFromSearch\":false,\"isInQc\":false,\"isInWorkflow\":false,\"isPdf\":false,\"isPdfUpToDate\":true,\"isWithdrawnOrRetracted\":false,\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"researchsquare\",\"isNatureJournal\":false,\"hasQc\":true,\"allowDirectSubmit\":true,\"externalIdentity\":\"\",\"sideBox\":\"\",\"snPcode\":\"\",\"submissionUrl\":\"/submission\",\"title\":\"Research Square\",\"twitterHandle\":\"researchsquare\",\"acdcEnabled\":true,\"dfaEnabled\":false,\"editorialSystem\":\"\",\"reportingPortfolio\":\"\",\"inReviewEnabled\":false,\"inReviewRevisionsEnabled\":true},\"keywords\":\"Artificial Intelligence (AI), Chatbot, Educational Institutions, Deep Learning (DL), Natural Language Processing (NLP)\",\"lastPublishedDoi\":\"10.21203/rs.3.rs-4257811/v1\",\"lastPublishedDoiUrl\":\"https://doi.org/10.21203/rs.3.rs-4257811/v1\",\"license\":{\"name\":\"CC BY 4.0\",\"url\":\"https://creativecommons.org/licenses/by/4.0/\"},\"manuscriptAbstract\":\"\\u003cp\\u003eChatbots substantially improve administrative support for educational institu- tions facing immense pressure during admissions. Chatbots not only automate repetitive tasks, handle large volumes of inquiries, collecting data from inter- actions but also provide an additional way for students to access information. Existing chatbots are built on traditional Artificial Intelligence (AI) approaches where the accuracy required for seamless real-time interactions is usually compro- mised. This article presents novel AEdBOT - an AI-based Educational ChatBOT system where a novel Deep Learning (DL)-based Hybrid model approach is proposed grounded on integrating informational retrieval and generative neural networks. Moreover, a novel Natural Language Processing (NLP) pipeline is developed on top of the open-source Rasa platform to aid with BERT (Bidi- rectional Encoder Representation Transformer) for dense feature extraction and DIET (Dual Intent and Entity Transformer) Classifier for intent classification and entity extraction from the natural language text. Furthermore, the customized dual fallback classifier algorithm is developed to provide the self-learning ability to\\u003c/p\\u003e\\n\\u003cp\\u003ea chatbot on out-of-scope inputs and acts as a recommendation system. The effec- tiveness of the proposed chatbot is established through two real-life datasets from educational institutes. For the first dataset, AEdBOT achieved 94.7%, 96.0%, 96.0%, and 95.1% precision, accuracy, recall, and F1-Score, respectively at an average mean response time of 216.43ms per query and a user-friendliness score of 77.5 on the System Usability Scale (SUS). The second dataset is used from the literature for comparative analysis, and AEdBOT attained 76.2%, 83.7%, 77.7%, and 79.1% accuracy, precision, F1-Score, and recall, respectively. Experiment results reveal that AEdBOT significantly improves response accuracy and outperforms state-of-the-art educational chatbots.\\u003c/p\\u003e\",\"manuscriptTitle\":\"Intelligent Agents in Educational Institutions: AEdBOT– A Chatbot for Administrative Assistance using Deep Learning Hybrid Model Approach\",\"msid\":\"\",\"msnumber\":\"\",\"nonDraftVersions\":[{\"code\":1,\"date\":\"2024-05-07 09:44:38\",\"doi\":\"10.21203/rs.3.rs-4257811/v1\",\"editorialEvents\":[{\"type\":\"communityComments\",\"content\":0}],\"status\":\"published\",\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"researchsquare\",\"isNatureJournal\":false,\"hasQc\":true,\"allowDirectSubmit\":true,\"externalIdentity\":\"\",\"sideBox\":\"\",\"snPcode\":\"\",\"submissionUrl\":\"/submission\",\"title\":\"Research Square\",\"twitterHandle\":\"researchsquare\",\"acdcEnabled\":true,\"dfaEnabled\":false,\"editorialSystem\":\"\",\"reportingPortfolio\":\"\",\"inReviewEnabled\":false,\"inReviewRevisionsEnabled\":true}}],\"origin\":\"\",\"ownerIdentity\":\"089ef53d-6d3b-46a7-9cb0-1ed8f62b65b3\",\"owner\":[],\"postedDate\":\"May 7th, 2024\",\"published\":true,\"recentEditorialEvents\":[],\"rejectedJournal\":[],\"revision\":\"\",\"amendment\":\"\",\"status\":\"posted\",\"subjectAreas\":[],\"tags\":[],\"updatedAt\":\"2024-05-17T11:44:28+00:00\",\"versionOfRecord\":[],\"versionCreatedAt\":\"2024-05-07 09:44:38\",\"video\":\"\",\"vorDoi\":\"\",\"vorDoiUrl\":\"\",\"workflowStages\":[]},\"version\":\"v1\",\"identity\":\"rs-4257811\",\"journalConfig\":\"researchsquare\"},\"__N_SSP\":true},\"page\":\"/article/[identity]/[[...version]]\",\"query\":{\"redirect\":\"/article/rs-4257811\",\"identity\":\"rs-4257811\",\"version\":[\"v1\"]},\"buildId\":\"qtupq5eGEP_6zYnWcrvyt\",\"isFallback\":false,\"isExperimentalCompile\":false,\"dynamicIds\":[84888],\"gssp\":true,\"scriptLoader\":[]}","source_license":"CC-BY-4.0","license_restricted":false}