Building a Robust Retrieval-Augmented Generation Chatbot for Immigration Knowledge Base Using Google Cloud Vertex AI and Open-Source LLMs

This paper details the end-to-end development and implementation of a Retrieval-Augmented Gener-ation (RAG) chatbot designed to provide accurate and grounded answers from a custom knowledge baseon immigration policies and procedures. Leveraging Google Cloud’s Vertex AI for robust text embeddingand vector search capabilities via Matching Engine, the system demonstrates an effective architecturefor combating the inherent limitations of large language models (LLMs), such as factual inaccuraciesand knowledge cut-off dates. The development journey highlights practical challenges encountered dur-ing LLM integration, including free-tier service access limitations with Google’s proprietary models andtask-specific API constraints with hosted open-source solutions like Hugging Face’s Inference API. Ulti-mately, the project pivots to a strategy of local LLM deployment, providing valuable insights into thefull lifecycle of a RAG system and the critical considerations for productionizing such applications withincloud environments, particularly concerning resource management and cost optimization.