Evaluating Domain Coverage in Low-Resource Generative Chatbots: A Comparative Study of Open-Domain and Closed-Domain Approaches Using BLEU Scores

Abstract Chatbots employ Natural Language Processing (NLP) and Artificial Intelligence (AI) to conduct automated chats and provide online assistance to users. Chatbots fall into two categories—generative models and retrieval models. E-commerce is the main application for retrieval-based chatbots—which provide pre-programmed responses and are easy to use with controlled interactions. While generative-based chatbots are more appropriate for research applications—they are more difficult to train and maintain. This is because they produce unique responses based on large conversational datasets. Therefore, in order to establish whether open-domain or closed-domain techniques work better—this study examines the domain coverage of low-resource generative chatbots. We compare their performance using Bilingual Evaluation Understudy (BLEU) scores in different domains to determine which method performs better.