Robust Framework for Scalable AI Inference Using Distributed Cloud Services and Event-driven Architecture

Abstract The deployment and scaling of AI models in production present challenges in ensuring efficient inference while maintaining cost-effectiveness and scalability. This paper proposes a robust framework using AWS services, including Elastic Beanstalk, SQS, EC2 instances with GPUs, EventBridge, Step Functions, API Gateway, and a hybrid approach with spot and on-demand instances. Elastic Beanstalk handles incoming requests and routes them to SQS for asynchronous processing. API Gateway manages rate limiting, while EventBridge and Step Functions dynamically scale the infrastructure. EC2 instances are pre-configured with AI scripts via EC2 Image Builder, requests are retrieved from SQS, models from Hugging Face are processed, and results are stored in Cloudflare. This architecture ensures high availability and responsiveness, optimizing performance and cost. Comprehensive experiments demonstrate the framework's effectiveness in handling varying inference demands, showing significant scalability, resource utilization, and operational efficiency improvements.