Software Engineering Challenges in the Deployment of Generative AI Models at Scale

Software Engineering Challenges in the Deployment of Generative AI Models at Scale | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Software Engineering Challenges in the Deployment of Generative AI Models at Scale Devisharan Mishra This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6875774/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Because generative AI has expanded quickly, today Large Language Models (LLMs) and diffusion-based generative models are transforming healthcare, financial, educational, and entertainment fields. Even though these models work well, getting them into use on a wide scale creates software engineering difficulties. Some of these issues include providing infrastructure, controlling inference latency, maintaining privacy, obeying ethics, building CI/CD processes, controlling versions, and keeping an eye on the models. The study explores the complicated aspects of putting generative AI models into real-world situations. In examining AI software engineering patterns, as well as its primary development and use processes, this study highlights the specific problems that still exist when employing traditional methods with AI systems. We share information about new methods and procedures made for AI operations (MLOps), highlight the value of joint efforts, and change the usual development steps. Next, the research paper demonstrates how a generative AI system was implemented in practice, points out problems that came up and reveals solutions, finishing with a guideline for proper and successful deployment. Generative AI Software Engineering Model Deployment MLOps Large Language Models Scalability CI/CD Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 1. Introduction Generative AI refers to tools that produce entirely new content, such as scripts, photos, sounds and coding. For example, OpenAI’sChatGPT, Google’s Gemini and Stability AI’s Stable Diffusion, along with other cutting-edge models, are now helping people automate some creative work, boosting their job productivity. Their power to produce useful and applicable results has attracted lots of interest and use in both entertainment and healthcare. Even though they have a lot of potential, bringing experimental robots to the production level remains a difficult and complicated job. [ 1 – 4 ] In traditional software, how the program behaves is clearly defined by its code, but since generative AI works with probabilities and data, the presence of version control, reproducibility, scaling and ethical issues is more noticeable. These models also mean engineers have to adopt updated ways to manage infrastructure, develop models and observe important data, as earlier approaches are unfit. They lead us to explore unique approaches to developing, using and maintaining generative AI so we can better link academic work with what happens in the industrial sector. 1.1. Importance of Software Engineering Challenges in the Deployment Complexity of Model Integration: Using generative AI models means integrating significant and complicated architectures with modern programs. They stand out from the usual applications because they require the use of large data sets, complex data preparation, and advanced inference systems. It is important to make sure AI applications can work well with existing systems and maintain their stability, so special software engineering techniques are essential. Scalability and Infrastructure Management: Such models are complex to run and commonly need the use of GPUs or TPUs. Making these models available to millions of users at once is not easy because of the tough infrastructure needed. Because resources, tasks, and money need to be managed effectively, it is essential to have reliable orchestration and automation solutions. Using standard approaches may not be enough, which makes this a significant engineering difficulty. Reproducibility and Version Control: It is important to keep the training and production environments similar to guarantee consistent behavior of the model. Still, these models can be affected by even small variations in data, code and the environment. Thus, there may be version drift, where mistakes happen when what is created in development does not work properly in production. The best way to address reproducibility is to use advanced systems for managing data, code and models, as code in software alone is often not enough. Monitoring and Error Handling: Since generative models offer probabilities, a single input can get a different result, which complicates finding mistakes and fixing them. System monitoring should include looking at things such as how sensitive chatbots respond to prompts and whether they make up fictitious responses since these details are often ignored by most software systems. Detection of anomalies and proper observability are necessary to ensure a system is dependable and safe. Ethical and Privacy Considerations : Generative AI systems raise new concerns related to ethics and privacy. Model training can lead to unintentional reflection of biases in the data or result in memorization of private information, both with the risk of exposing data. All software engineering efforts should include ways to find bias, secure privacy, and follow regulations like GDPR and HIPAA. It makes the deployment process more complicated. 1.2. Relevance to Modern Software Systems Many modern software systems now depend on generative AI, which has a major effect on fields where smart decision-making and creation processes are needed. Thanks to generative AI, clinicians can access helpful tools for diagnosing illnesses, selecting treatment plans, and tailoring care for patients. Because applications can significantly influence human lives, it is essential that the software used is highly reliable, secure, and open. In the field of finance, generative models are being introduced to handle tasks such as detecting fraud, estimating risk, and reporting. They must meet tough legal requirements and remain strong against abusive attempts, all while using architectures that guarantee safety, privacy and a clear trail of transactions. [ 5 , 6 ] Generative AI in these industries is employed to produce images, videos and written texts that appear realistic, making it possible for creators to produce lots of new content in many formats. Relying on these models makes it important to keep their content authentic and ethical, so strict safeguards and methods to interpret what they mean are needed. Such important and unique applications make it clear that special software engineering practices for generative AI are needed right now. Most of these development pipelines are not equipped with the right tools to handle the complexity and unpredictability inherent in these models. Such systems are not able to judge when the changes to the model inputs are normal or if they are signs of harmful biases or hallucinations. In addition, because computations are heavy and models can change a lot, it is necessary to have architectures that can expand with no interruptions. Making sure security, compliance and ethical standards are met makes things more difficult for most existing computer systems. Thus, embracing generative AI in modern software programs calls for a thorough change in how software is built. These changes include choosing particular lifecycle management, deployment and observability strategies that all enhance the secure and easy use of generative AI in essential applications. If we manage these issues, generative AI can truly transform the world and mitigate risks in real-life settings. 2. Literature Survey 2.1. Traditional Software Engineering Approaches For years, traditional software engineering practices have employed two structural methods: the Waterfall and Agile models. These frameworks rely on building independent modules, clearly explaining requirements, conducting multiple tests, and striving for high-quality code. They contribute to maintaining, tracking and strengthening software. They fail to cover the special traits associated with AI and machine learning systems. The standard way of looking at models fails to accommodate the need for repeated experimentation, large datasets and the random changes AI requires. [ 7 – 10 ] While software engineers rely on unit testing and code coverage, validating machine learning behavior may call for methods that are quite different from them. It makes clear that software engineers must use more precise ways to combine AI into their systems. 2.2. Evolution of MLOps MLOps, which is a blend of machine learning, DevOps, and data engineering, has become increasingly important in the industry. It wants to make it easier to develop, deploy and oversee machine learning models as part of real-life operations. Thanks to DevOps, MLOps helps solve challenges in model deployment by using CI/CD for model codes, setting up automated data channels, as well as tracking experiments. The workflows now feature tools such as MLflow, Kubeflow, and Amazon SageMaker, which ensure that models can be reused, easily updated, and monitored for performance. Still, these tools are often unable to support the special requirements of generative models, which may involve numerous billions of parameters and require a lot of computing power. Because of these problems, there is a need for enhanced MLOps frameworks to serve AI models that are more advanced. 2.3. Prior Work on AI Deployment Challenges Many leading studies have looked at the challenges that arise when machine learning systems are used at a large scale. They pointed out Sculley et al. (2015) that machine learning projects depend on data, which can lead to further complications and issues when reproducing or updating results. Zaharia et al. (2020) underlined how making ML pipelines robust is essential for use in production. Even though these articles explain many generic ML challenges, they do not tackle the detailed issues involved in deploying generative models. Some examples include swift sensitivity to small changes in the input, amplification of biases present in the data, and the generation of false or misleading details. These issues demand specific approaches that go past what is commonly studied in research. 2.4. Scalability Constraints in Generative Models Large language models like GPT-3 and GPT-4 are huge and take a lot of computing power, which is a major problem when scaling these models. Models of this type can only be used in environments with powerful GPUs or TPUs because running inference is very demanding. Also, chatbots, virtual assistants and content generators need to respond quickly to users, and many existing systems do not have the necessary setup for speedy responses. Table 1 clearly shows that the memory used by GPT-3 and similar models is very high, so special methods for managing these resources are necessary. Not being able to scale easily results in more expenses and can make it challenging to deploy properly, mainly for organizations with few resources. 2.5. Security and Privacy Concerns Making generative models secure and private is very important. When training these models, huge, varied datasets are used, and occasionally, PII might be present in them. It is possible that the model will remember and reproduce this data, which might break the laws set by GDPR and HIPAA. In addition, users with bad intentions could try to get hold of training data by providing cleverly designed inputs. Various research efforts are being made to avoid these dangers using model watermarking, differential privacy and robust access logging. They intend to follow how models are applied, protect user data and meet regulations, even though they are only now getting accepted in practice. 3. Methodology 3.1. Research Design Literature Review : It is important to discover what previous studies have found for traditional software engineering, MLOps and the special deployment situation of generative models. [ 11 – 14 ] During this stage, key issues and recent trends are identified, which serve as the base for more studies moving forward. The review also guides the choice of the right framework and evaluation standards in the experiment. Case Studies (Industry Use) : To follow the literature review, the research studies industry case studies to understand how organizations make use of generative models in real life. Case studies discuss issues that come up in real life, such as sharing data, handling increases in activity and keeping an eye on artificial intelligence models. Companies relying on GPT-3, Stable Diffusion or proprietary LLMs offer useful information about the problems and victories they experience when sending their creations into the wild. Deployment Architecture Evaluation : In the final part, researchers examine deployment structures that can support generative models. This covers the evaluation of various ways of storing data (such as in the cloud versus on-site), different tools for managing workflows and the possible gains in performance. An evaluation is made to see if these architectures are scalable, reliable and follow all compliance rules to ensure large AI systems are deployed with efficiency and security. 3.2. Deployment Architecture Patterns Monolithic vs. Microservices : All parts of an AI system, such as data preparation, prediction and post-processing, are brought together in a single deployable form by monolithic architectures. Even though this way makes it easy to develop and deploy the application, it can cause major issues with scaling, maintenance and identifying faults. Unlike monolithic, microservices subdivide the system into smaller features which teams can deploy and update apart from each other. The advantage is that we can separately develop and monitor each AI processing step, but it becomes difficult to keep data and inferences flowing synchronized among the services. Frequently, using generative models in a microservices system needs close cooperation between the API gateway, the model server and the backend service processors. Serverless and Edge Deployments : For AI models, deploying to the edge or serverless is an alternate choice to typical hosting methods. Since they are on the edge, these deployments help processes needing fast and responsive results. Generative models are commonly too big to fit on the small computers found in many edge devices. On the other side, platforms like AWS Lambda and Google Cloud Functions handle scaling automatically and run on events, which is helpful for work that changes often over time. Even though serverless functions provide many benefits, they can’t be used effectively for complex tasks in inference because they take time to initialize and aren’t efficient with every use. 3.3. Model Lifecycle Management Model Training : Training a generative model forms the key beginning step in its development. The process requires putting together large data and carrying out training on powerful hardware such as GPUs or TPUs. Here, the learning rate, batch size, and model network settings are set to their optimal values to improve its results. Training generative models are very demanding due to their size and layer complexity, and it is often necessary to perform this training on multiple machines. Managing the memory used in training is made possible through mixed-precision training and gradient checkpointing. Model Evaluation : After training, the model is put to rigorous tests to see if accuracy, robustness, and ethical behavior are present. Standard tools, such as BLEU and ROUGE, are applied to evaluate how well texts have been generated for translation and summarization. Apart from numbers, the evaluation process also makes sure the model is not influenced by harmful assumptions or unfair decisions. One more thing to do is adversarial testing, where intentionally confusing inputs are inputted to see how the model reacts and becomes more robust before deployment. Model Deployment : Making the trained model accessible to users or applications in a secure way is what deployment is all about. Canary deployments, A/B tests and the blue-green method are often put in place to limit the risks as you move to the next version. With a canary deployment, just a few users come across the model to check for odd behavior, and A/B testing compares versions to find out which does the job best. With this approach, there are always two systems ready: one live and one in a staged environment, allowing problems to be quickly fixed in case something goes wrong. As a result, moving to production is easier, and possible delays are reduced. Important aspects of the program are closely monitored. 3.4. Observability and Monitoring For generative AI systems to be dependable and consistent in use, close observation is essential. To reach this, the monitoring stack consisted of Prometheus and Grafana, supporting the immediate collection, display and notification of system metrics. [ 15 – 18 ] They provide useful information on how the system works under different conditions, making it easy to find delays, issues with memory or unusual results. By including monitoring as part of the deployment process, teams can handle failures, boost performance and keep users’ experiences uniform. Latency : The model requires to respond after getting input. Such metrics matter a lot in chatbots or content generation tools, as the slow response can negatively impact the user’s experience. A latency that is too high might suggest that the computer is overloaded, resources are being shared, or the network is carrying extra traffic. By monitoring latency, developers can improve the infrastructure, use batching and move some of the computation to enhance responsiveness. Memory Utilization : The amount of memory taken by the model during inference is recorded through memory utilization. Many large language models are memory-intensive and may not function properly if proper limits are not set. Tracking this metric ensures that the system stays within the boundaries of your hardware and indicates whether the model loads correctly. It also supports decisions about when to use sharding and quantization in order to free up more memory. Error Rate : It shows what portion of API calls or inferences did not go as expected. Some of these failures are due to timeouts, insufficient memory or server problems inside. When the error rate goes up, it may be because of infrastructure faults or update errors, and this needs to be fixed immediately to ensure people keep using the service. With this measure, teams can spot errors, learn what caused them and stop new deployments before anyone sees the problem. Prompt Sensitivity : How prompt sensitivity works is this: the more different input prompts are from the main one, the more the outputs vary. When slightly changing how something is phrased, generative models can deliver outputs that are very different. Although a bit of creativity is good, being too sensitive can reduce the trustworthiness of your outcome and make it harder to repeat it. Keeping an eye on prompt sensitivity is useful for testing a model’s reliability, guiding prompt engineering, and deciding whether and when to improve it. 3.5. Toolchain Docker and Kubernetes for Container Orchestration : Packaging and orchestrating our AI services became easier using the base setup of Docker and Kubernetes. Because Docker was used to package the application components—model server, API gateway and a set of monitoring tools—they performed the same in every environment. Because of Kubernetes, these containers were deployed, scaled and managed automatically across many servers in a cluster. In this architecture, errors could be corrected, the service could expand horizontally, and systems could be easily rolled back, which were necessary for generative AI to succeed. TensorRT for Model Optimization : Inference performance was boosted by using TensorRT as the high-performance deep learning inference optimizer from NVIDIA. Bringing model training into an efficient runtime form, TensorRT cut down latency and raised the speed of execution, especially for models deployed to GPUs. This helped a lot when applying large generative models since the sheer amount of calculations needed is often the main limitation. Because of TensorRT, the system answered questions more promptly and efficiently, utilising the GPU, and accuracy was preserved. DVC for Data Versioning : With DVC, data could be handled, and any shifts in datasets during development could be tracked. Since it is important for machine learning to reuse results, DVC enabled us to version both the code and any data or model artefacts. Because of this, we could easily go back to any previous point, observe the experiments and verify that the data being used was what was intended for training and testing, which is required for correcting errors and auditing. Weights & Biases for Experiment Tracking : Weights & Biases helped us manage all the information related to our model’s training experiment, hyperparameters and measurements. Thanks to W&B, we could manage our experiments better and check model performance and differences between runs in real-time. Because the data and results were open to all, team members could pick the right training settings and regularly enhance how accurate and reliable the model was. 4. Results and Discussion 4.1. Case Study: Text Generation Platform In this case, a transformer-based text assistant is used to support users by summarizing information, answering questions and making content suggestions. The application could automatically expand and shrink its resources, depending on traffic and workload, because horizontal pod auto-scaling was set up on the Kubernetes cluster. As a result, the system could cope with a lot of users and save resources when there were only a few. One important change during deployment was quantizing the model, which lowered the accuracy with which the model’s weights and calculations were recorded. By using this method, the average time it takes to process user requests dropped by 34%, reaching 620 milliseconds, without negatively impacting the language of the text. The faster answer speed boosted user interest, as the assistant felt more responsive in real time. Simultaneously, Prometheus began handling data measurement, while Grafana did the job of creating dashboards and issuing alerts. It offered an up-to-date look at functions such as system memory usage, error rates on the API and how sensitive the NLP model was. Early detection of anything unusual occurred because the framework made it possible for alerts to be set off automatically when the system’s behavior changed. So, the team could take steps beforehand to fix issues which brought system downtime down to 6 hours for the entire month. As downtime decreased, the service became even more accessible, and people’s faith in and satisfaction with the product grew as well. In short, the case study shows that using practices like Kubernetes orchestration and making models smaller while using observability can make generative AI perform better, work with more reliability and be more scalable when deployed at scale. 4.2. Performance Metrics Table 1 Performance Metrics Metric % Improvement Inference Latency 34% API Downtime 70% User Engagement 15% Inference Latency (34% Improvement) : Because inference latency was reduced by 34%, the responsiveness of the generative text assistant greatly increased. By making the model efficient and placing it on auto-scalable Kubernetes, the system managed to provide faster answers and maintain its accuracy. With this upgrade, users noticed shorter delays, making the platform more suitable for use with quick chatbots and assistants. API Downtime (70% Improvement) : The significant reduction in API downtime means systems are more reliable and can be utilised more frequently. Because of real-time monitoring, active incident handling and the system’s ability to handle errors, several possible outages were prevented before they affected users. As a result of reducing downtime, people working with the system felt more secure, and the business experienced fewer disruptions to its everyday activities. User Engagement (15% Improvement) : Higher user engagement by 15% means the system’s improvements have improved how users interact on the website. This made the assistant more useful and made people continue to use it and spend more time on it. It demonstrates that raising backend efficiency and user satisfaction usually come from technical improvements. 4.3. Challenges Encountered During deployment and operations, several challenges were identified: Dependency Hell : The process of deploying the model pointed out that it uses many third-party libraries. Since a large number of these libraries were either unstable, deprecated or required specific versions, this caused frequent problems when trying to use them in containerized and integrated processes. Because of these problems, the continuous integration and deployment pipelines had unexpected failures, which caused longer release delays and made maintenance more complex. Therefore, in order to solve this, the system was set up so that all applications used the same versions of required libraries. Frequent checks were also made on the packages being used to fix and remove those that may cause instability in the system. Version Drift : This issue happened when the differences between training and production environments caused the model’s behavior to vary. Updating the framework affected the models and, on some occasions, caused the program to fail. The changes caused the system to be unreliable and not able to be reused. Docker containers helped ensure that the training environment was the same in production, which reduced the possibility of problems. Also, MLflow was included to keep track of environment information together with model artifacts, allowing us to easily see how things are connected and solve issues. Monitoring Noise : It was a hard task to tell apart genuine innovative content and the occasional ‘hallucinations’ when using generative models. Given that generative AI works in an unpredictable way, the traditional approach to finding mistakes in AI failed. We installed more layers focused on examination to ensure the output is both relevant and coherent. Adding better logging and manual reviews by people helped classify the outputs, making it simpler to identify anomalies and avoid wrongly triggered alarms. 5. Conclusion Scaling generative AI models requires businesses to make major adjustments to the usual software engineering process. Unlike classic software, generative models require complex infrastructure that can handle a great amount of computations and memory. Besides, these models increase the chance of ethical and operational issues such as biased results, wrong predictions and issues related to privacy, which add to the difficulties of deploying and managing them. Through this paper, the gaps found in existing generation AI methodologies have been shown, and a practical route for addressing the complexity of implementing generative AI systems has been offered. As the study employs a modular architecture, handles lifecycle tasks, and includes observability, the information is particularly helpful for engineers and researchers working in industrial AI. This work helps by outlining the main issues regarding deployment that are important for generative AI. Such issues include infrastructure capacity problems, unpredictable results, complex data versions, and dangers due to increased bias and data exposure. Listing these specific concerns enables the paper to provide a better framework for developing systems that can address them. Moreover, the suggestion of using a modular tool framework that works with containers enhances models and provides a strong monitoring unit, uniting theory and practice. Validation of the framework comes from a case study with a transformer platform used for text generation, which shows great improvements in inference, reliability and the level of engagement. This process shows that the recommendations work as expected in practical situations. Looking forward, the use of explainability in generative AI is expected to play a larger role in making its decisions easier to understand and explain. It will play a key role in establishing users’ trust and obeying regulations in sensitive sectors. Today, another useful approach is to use bias corrections when the model predicts, thereby helping it avoid producing any unfair or harmful outcomes during use. Also, considering federated and decentralized deployment systems could improve privacy and flexibility since training and deployment of models could take place at different edge devices without sharing data with a central server. With the new advancements, generative AI will be able to work effectively on a large scale and in many industries and use cases. Declarations Author Contribution D.M. Authored the manuscript with details and diagrams References Beck K, Beedle M, Van Bennekum A, Cockburn A, Cunningham W, Fowler M, Thomas D (2001) Manifesto for agile software development. Amershi S, Begel A, Bird C, DeLine R, Gall H, Kamar E, Zimmermann T (2019), May Software engineering for machine learning: A case study. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) (pp. 291–300). IEEE Sculley D, Holt G, Golovin D, Davydov E, Phillips T, Ebner D, Dennison D (2015) Hidden technical debt in machine learning systems. Adv Neural Inf Process Syst, 28 Zaharia M, Chen A, Davidson A, Ghodsi A, Hong SA, Konwinski A, Zumar C (2018) Accelerating the machine learning lifecycle with MLflow. IEEE Data Eng Bull 41(4):39–45 Breck E, Cai S, Nielsen E, Salib M, Sculley D (2017), December The ML test score: A rubric for ML production readiness and technical debt reduction. In 2017 IEEE international conference on big data (big data) (pp. 1123–1132). IEEE Touvron H, Lavril T, Izacard G, Martinet X, Lachaux MA, Lacroix T, Lample G (2023) Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, Amodei D (2020) Language models are few-shot learners. Adv Neural Inf Process Syst 33:1877–1901 Patterson D, Gonzalez J, Le Q, Liang C, Munguia LM, Rothchild D, Dean J (2021) Carbon emissions and large neural network training. arXiv preprint arXiv :210410350 Carlini N, Liu C, Erlingsson Ú, Kos J, Song D (2019) The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX security symposium (USENIX security 19) (pp. 267–284) Fredrikson M, Jha S, Ristenpart T (2015), October Model inversion attacks that exploit confidence information and basic countermeasures in Proceedings of the 22nd ACM SIGSAC conference on computer and communications security (pp. 1322–1333) Zhang C, Bengio S, Hardt M, Recht B, Vinyals O (2016) Understanding deep learning requires rethinking generalization. arXiv preprint arXiv :161103530 Shokri R, Stronati M, Song C, Shmatikov V (2017), May Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP) (pp. 3–18). IEEE Benala TR, Dehuri S (2025) Transforming Software Development: From Traditional Methods to Generative Artificial Intelligence. Boosting Software Development Using Machine Learning. Springer, Cham, pp 1–13 Mäntylä MV, Vanhanen J (2011), March Software deployment activities and challenges-a case study of four software product companies. In 2011 15th European Conference on Software Maintenance and Reengineering (pp. 131–140). IEEE Casale G, Chesta C, Deussen P, Di Nitto E, Gouvas P, Koussouris S, Zhao Z (2016) Current and future challenges of software engineering for services and applications. Procedia Comput Sci 97:34–42 Forward A, Lethbridge TC (2002), November The relevance of software documentation, tools and technologies: a survey. In Proceedings of the 2002 ACM symposium on Document engineering (pp. 26–33) Aitken A, Ilango V (2013), January A comparative analysis of traditional software engineering and agile software development. In 2013 46th Hawaii International Conference on System Sciences (pp. 4751–4760). IEEE He M, Li Z, Liu C, Shi D, Tan Z (2020) Deployment of artificial intelligence in real-world practice: opportunity and challenge. Asia-Pacific J Ophthalmol 9(4):299–307 Shaw J, Rudzicz F, Jamieson T, Goldfarb A (2019) Artificial intelligence and the implementation challenges. J Med Internet Res, 21(7), e13659 Wang Z, Duan Z, Zhang H, Wang C, Tian L, Chen B, Zhou M (2020), November Friendly topic assistant for transformer-based abstractive summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 485–497) Sauvola J, Tarkoma S, Klemettinen M, Riekki J, Doermann D (2024) Future of software development with generative AI. Automated Softw Eng 31(1):26 Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6875774","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":470134741,"identity":"4d1aba02-2e84-4e97-8dee-1aac671ef7e0","order_by":0,"name":"Devisharan Mishra","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAxUlEQVRIiWNgGAWjYDACZhDB818ORB14QLwWGWZjsJYE4q2yYU5sANFEaTFnZ7/46UYOW/r8sMMPgbbYyek2ENBi2cxTLJ1zhid34+00A6CWZGOzAwS0GBzmSZDO7ZHI3Tg7AaTlQOI2IrQk/879Z5BuODv9A7Fa2I9J5/AkJMhL5xBvC5t1Ds8Bww3SOQUHEgyI8cv5449vA7XIy89O3/zhQ4WdHEEtwJg3gOgFqzQgqBwE2B+AKfkGolSPglEwCkbBSAQA7D1E3YeVb7sAAAAASUVORK5CYII=","orcid":"","institution":"","correspondingAuthor":true,"prefix":"","firstName":"Devisharan","middleName":"","lastName":"Mishra","suffix":""}],"badges":[],"createdAt":"2025-06-12 02:38:04","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-6875774/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6875774/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":84795438,"identity":"37b65b42-df0a-4692-bd3e-41617a5507ad","added_by":"auto","created_at":"2025-06-17 12:18:57","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":156981,"visible":true,"origin":"","legend":"\u003cp\u003eImportance of Software Engineering Challenges in the Deployment\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-6875774/v1/547aa0957b86c31429b87892.png"},{"id":84795443,"identity":"723f1bce-3340-491e-839a-2caf8564500b","added_by":"auto","created_at":"2025-06-17 12:18:57","extension":"jpeg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":49619,"visible":true,"origin":"","legend":"\u003cp\u003eResearch Design\u003c/p\u003e","description":"","filename":"floatimage2.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-6875774/v1/c2fe3b65dfc9ae5df8b6d9e5.jpeg"},{"id":84797048,"identity":"e5aa2a37-6fd4-4d93-8f4e-fae76a86496b","added_by":"auto","created_at":"2025-06-17 12:34:57","extension":"jpeg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":42260,"visible":true,"origin":"","legend":"\u003cp\u003eModel Lifecycle Management\u003c/p\u003e","description":"","filename":"floatimage3.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-6875774/v1/f6578b27c936e78b01e236aa.jpeg"},{"id":84796645,"identity":"2a9dd0ec-ae8e-4372-a707-e8b43aab7e9f","added_by":"auto","created_at":"2025-06-17 12:26:57","extension":"jpeg","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":59503,"visible":true,"origin":"","legend":"\u003cp\u003eObservability and Monitoring\u003c/p\u003e","description":"","filename":"floatimage4.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-6875774/v1/1120a6283bcd7b1a664967e8.jpeg"},{"id":84795447,"identity":"98039130-5087-4b07-a307-c996d1da7542","added_by":"auto","created_at":"2025-06-17 12:18:57","extension":"jpeg","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":49254,"visible":true,"origin":"","legend":"\u003cp\u003eToolchain\u003c/p\u003e","description":"","filename":"floatimage5.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-6875774/v1/7c382136794fe1652489deb4.jpeg"},{"id":84796642,"identity":"473ec1ce-1562-4470-ba52-42f852f042f5","added_by":"auto","created_at":"2025-06-17 12:26:57","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":11610,"visible":true,"origin":"","legend":"\u003cp\u003eGraph representing Performance Metrics\u003c/p\u003e","description":"","filename":"6.png","url":"https://assets-eu.researchsquare.com/files/rs-6875774/v1/85aa30f30ffdac02f6a19ce5.png"},{"id":85363525,"identity":"d9cb1511-33ba-4fe4-ba4e-d5551646531c","added_by":"auto","created_at":"2025-06-25 06:24:20","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1952126,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6875774/v1/ae98222a-e69c-4f59-9cde-b788ba98ae47.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Software Engineering Challenges in the Deployment of Generative AI Models at Scale","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eGenerative AI refers to tools that produce entirely new content, such as scripts, photos, sounds and coding. For example, OpenAI\u0026rsquo;sChatGPT, Google\u0026rsquo;s Gemini and Stability AI\u0026rsquo;s Stable Diffusion, along with other cutting-edge models, are now helping people automate some creative work, boosting their job productivity. Their power to produce useful and applicable results has attracted lots of interest and use in both entertainment and healthcare. Even though they have a lot of potential, bringing experimental robots to the production level remains a difficult and complicated job. [\u003cspan additionalcitationids=\"CR2 CR3\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e] In traditional software, how the program behaves is clearly defined by its code, but since generative AI works with probabilities and data, the presence of version control, reproducibility, scaling and ethical issues is more noticeable. These models also mean engineers have to adopt updated ways to manage infrastructure, develop models and observe important data, as earlier approaches are unfit. They lead us to explore unique approaches to developing, using and maintaining generative AI so we can better link academic work with what happens in the industrial sector.\u003c/p\u003e \u003cp\u003e \u003cb\u003e1.1. Importance of Software Engineering Challenges in the Deployment\u003c/b\u003e \u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eComplexity of Model Integration: Using generative AI models means integrating significant and complicated architectures with modern programs. They stand out from the usual applications because they require the use of large data sets, complex data preparation, and advanced inference systems. It is important to make sure AI applications can work well with existing systems and maintain their stability, so special software engineering techniques are essential.\u003c/b\u003e \u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eScalability and Infrastructure Management: Such models are complex to run and commonly need the use of GPUs or TPUs. Making these models available to millions of users at once is not easy because of the tough infrastructure needed. Because resources, tasks, and money need to be managed effectively, it is essential to have reliable orchestration and automation solutions. Using standard approaches may not be enough, which makes this a significant engineering difficulty.\u003c/b\u003e \u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eReproducibility and Version Control: It is important to keep the training and production environments similar to guarantee consistent behavior of the model. Still, these models can be affected by even small variations in data, code and the environment. Thus, there may be version drift, where mistakes happen when what is created in development does not work properly in production. The best way to address reproducibility is to use advanced systems for managing data, code and models, as code in software alone is often not enough.\u003c/b\u003e \u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eMonitoring and Error Handling: Since generative models offer probabilities, a single input can get a different result, which complicates finding mistakes and fixing them. System monitoring should include looking at things such as how sensitive chatbots respond to prompts and whether they make up fictitious responses since these details are often ignored by most software systems. Detection of anomalies and proper observability are necessary to ensure a system is dependable and safe.\u003c/b\u003e \u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eEthical and Privacy Considerations\u003c/b\u003e: Generative AI systems raise new concerns related to ethics and privacy. Model training can lead to unintentional reflection of biases in the data or result in memorization of private information, both with the risk of exposing data. All software engineering efforts should include ways to find bias, secure privacy, and follow regulations like GDPR and HIPAA. It makes the deployment process more complicated.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cdiv id=\"Sec2\" class=\"Section2\"\u003e \u003ch2\u003e1.2. Relevance to Modern Software Systems\u003c/h2\u003e \u003cp\u003eMany modern software systems now depend on generative AI, which has a major effect on fields where smart decision-making and creation processes are needed. Thanks to generative AI, clinicians can access helpful tools for diagnosing illnesses, selecting treatment plans, and tailoring care for patients. Because applications can significantly influence human lives, it is essential that the software used is highly reliable, secure, and open. In the field of finance, generative models are being introduced to handle tasks such as detecting fraud, estimating risk, and reporting. They must meet tough legal requirements and remain strong against abusive attempts, all while using architectures that guarantee safety, privacy and a clear trail of transactions. [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e] Generative AI in these industries is employed to produce images, videos and written texts that appear realistic, making it possible for creators to produce lots of new content in many formats. Relying on these models makes it important to keep their content authentic and ethical, so strict safeguards and methods to interpret what they mean are needed. Such important and unique applications make it clear that special software engineering practices for generative AI are needed right now. Most of these development pipelines are not equipped with the right tools to handle the complexity and unpredictability inherent in these models. Such systems are not able to judge when the changes to the model inputs are normal or if they are signs of harmful biases or hallucinations. In addition, because computations are heavy and models can change a lot, it is necessary to have architectures that can expand with no interruptions. Making sure security, compliance and ethical standards are met makes things more difficult for most existing computer systems. Thus, embracing generative AI in modern software programs calls for a thorough change in how software is built. These changes include choosing particular lifecycle management, deployment and observability strategies that all enhance the secure and easy use of generative AI in essential applications. If we manage these issues, generative AI can truly transform the world and mitigate risks in real-life settings.\u003c/p\u003e \u003c/div\u003e"},{"header":"2. Literature Survey","content":"\u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.1. Traditional Software Engineering Approaches\u003c/h2\u003e \u003cp\u003eFor years, traditional software engineering practices have employed two structural methods: the Waterfall and Agile models. These frameworks rely on building independent modules, clearly explaining requirements, conducting multiple tests, and striving for high-quality code. They contribute to maintaining, tracking and strengthening software. They fail to cover the special traits associated with AI and machine learning systems. The standard way of looking at models fails to accommodate the need for repeated experimentation, large datasets and the random changes AI requires. [\u003cspan additionalcitationids=\"CR8 CR9\" citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e–\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e] While software engineers rely on unit testing and code coverage, validating machine learning behavior may call for methods that are quite different from them. It makes clear that software engineers must use more precise ways to combine AI into their systems.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.2. Evolution of MLOps\u003c/h2\u003e \u003cp\u003eMLOps, which is a blend of machine learning, DevOps, and data engineering, has become increasingly important in the industry. It wants to make it easier to develop, deploy and oversee machine learning models as part of real-life operations. Thanks to DevOps, MLOps helps solve challenges in model deployment by using CI/CD for model codes, setting up automated data channels, as well as tracking experiments. The workflows now feature tools such as MLflow, Kubeflow, and Amazon SageMaker, which ensure that models can be reused, easily updated, and monitored for performance. Still, these tools are often unable to support the special requirements of generative models, which may involve numerous billions of parameters and require a lot of computing power. Because of these problems, there is a need for enhanced MLOps frameworks to serve AI models that are more advanced.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e2.3. Prior Work on AI Deployment Challenges\u003c/h2\u003e \u003cp\u003eMany leading studies have looked at the challenges that arise when machine learning systems are used at a large scale. They pointed out Sculley et al. (2015) that machine learning projects depend on data, which can lead to further complications and issues when reproducing or updating results. Zaharia et al. (2020) underlined how making ML pipelines robust is essential for use in production. Even though these articles explain many generic ML challenges, they do not tackle the detailed issues involved in deploying generative models. Some examples include swift sensitivity to small changes in the input, amplification of biases present in the data, and the generation of false or misleading details. These issues demand specific approaches that go past what is commonly studied in research.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e2.4. Scalability Constraints in Generative Models\u003c/h2\u003e \u003cp\u003eLarge language models like GPT-3 and GPT-4 are huge and take a lot of computing power, which is a major problem when scaling these models. Models of this type can only be used in environments with powerful GPUs or TPUs because running inference is very demanding. Also, chatbots, virtual assistants and content generators need to respond quickly to users, and many existing systems do not have the necessary setup for speedy responses. Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e clearly shows that the memory used by GPT-3 and similar models is very high, so special methods for managing these resources are necessary. Not being able to scale easily results in more expenses and can make it challenging to deploy properly, mainly for organizations with few resources.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003e2.5. Security and Privacy Concerns\u003c/h2\u003e \u003cp\u003eMaking generative models secure and private is very important. When training these models, huge, varied datasets are used, and occasionally, PII might be present in them. It is possible that the model will remember and reproduce this data, which might break the laws set by GDPR and HIPAA. In addition, users with bad intentions could try to get hold of training data by providing cleverly designed inputs. Various research efforts are being made to avoid these dangers using model watermarking, differential privacy and robust access logging. They intend to follow how models are applied, protect user data and meet regulations, even though they are only now getting accepted in practice.\u003c/p\u003e "},{"header":"3. Methodology","content":"\u003ch2\u003e3.1. Research Design\u003c/h2\u003e\u003cul\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eLiterature Review\u003c/b\u003e: It is important to discover what previous studies have found for traditional software engineering, MLOps and the special deployment situation of generative models. [\u003cspan additionalcitationids=\"CR12 CR13\" citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e–\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e] During this stage, key issues and recent trends are identified, which serve as the base for more studies moving forward. The review also guides the choice of the right framework and evaluation standards in the experiment.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eCase Studies (Industry Use)\u003c/b\u003e: To follow the literature review, the research studies industry case studies to understand how organizations make use of generative models in real life. Case studies discuss issues that come up in real life, such as sharing data, handling increases in activity and keeping an eye on artificial intelligence models. Companies relying on GPT-3, Stable Diffusion or proprietary LLMs offer useful information about the problems and victories they experience when sending their creations into the wild.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eDeployment Architecture Evaluation\u003c/b\u003e: In the final part, researchers examine deployment structures that can support generative models. This covers the evaluation of various ways of storing data (such as in the cloud versus on-site), different tools for managing workflows and the possible gains in performance. An evaluation is made to see if these architectures are scalable, reliable and follow all compliance rules to ensure large AI systems are deployed with efficiency and security.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e\u003cp\u003e \u003cb\u003e3.2. Deployment Architecture Patterns\u003c/b\u003e \u003c/p\u003e\u003cul\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eMonolithic vs. Microservices\u003c/b\u003e: All parts of an AI system, such as data preparation, prediction and post-processing, are brought together in a single deployable form by monolithic architectures. Even though this way makes it easy to develop and deploy the application, it can cause major issues with scaling, maintenance and identifying faults. Unlike monolithic, microservices subdivide the system into smaller features which teams can deploy and update apart from each other. The advantage is that we can separately develop and monitor each AI processing step, but it becomes difficult to keep data and inferences flowing synchronized among the services. Frequently, using generative models in a microservices system needs close cooperation between the API gateway, the model server and the backend service processors.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eServerless and Edge Deployments\u003c/b\u003e: For AI models, deploying to the edge or serverless is an alternate choice to typical hosting methods. Since they are on the edge, these deployments help processes needing fast and responsive results. Generative models are commonly too big to fit on the small computers found in many edge devices. On the other side, platforms like AWS Lambda and Google Cloud Functions handle scaling automatically and run on events, which is helpful for work that changes often over time. Even though serverless functions provide many benefits, they can’t be used effectively for complex tasks in inference because they take time to initialize and aren’t efficient with every use.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e\u003ch2\u003e3.3. Model Lifecycle Management\u003c/h2\u003e\u003cul\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eModel Training\u003c/b\u003e: Training a generative model forms the key beginning step in its development. The process requires putting together large data and carrying out training on powerful hardware such as GPUs or TPUs. Here, the learning rate, batch size, and model network settings are set to their optimal values to improve its results. Training generative models are very demanding due to their size and layer complexity, and it is often necessary to perform this training on multiple machines. Managing the memory used in training is made possible through mixed-precision training and gradient checkpointing.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eModel Evaluation\u003c/b\u003e: After training, the model is put to rigorous tests to see if accuracy, robustness, and ethical behavior are present. Standard tools, such as BLEU and ROUGE, are applied to evaluate how well texts have been generated for translation and summarization. Apart from numbers, the evaluation process also makes sure the model is not influenced by harmful assumptions or unfair decisions. One more thing to do is adversarial testing, where intentionally confusing inputs are inputted to see how the model reacts and becomes more robust before deployment.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eModel Deployment\u003c/b\u003e: Making the trained model accessible to users or applications in a secure way is what deployment is all about. Canary deployments, A/B tests and the blue-green method are often put in place to limit the risks as you move to the next version. With a canary deployment, just a few users come across the model to check for odd behavior, and A/B testing compares versions to find out which does the job best. With this approach, there are always two systems ready: one live and one in a staged environment, allowing problems to be quickly fixed in case something goes wrong. As a result, moving to production is easier, and possible delays are reduced. Important aspects of the program are closely monitored.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e\u003ch2\u003e3.4. Observability and Monitoring\u003c/h2\u003e\u003cp\u003eFor generative AI systems to be dependable and consistent in use, close observation is essential. To reach this, the monitoring stack consisted of Prometheus and Grafana, supporting the immediate collection, display and notification of system metrics. [\u003cspan additionalcitationids=\"CR16 CR17\" citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e–\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e] They provide useful information on how the system works under different conditions, making it easy to find delays, issues with memory or unusual results. By including monitoring as part of the deployment process, teams can handle failures, boost performance and keep users’ experiences uniform.\u003c/p\u003e\u003cul\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eLatency\u003c/b\u003e: The model requires to respond after getting input. Such metrics matter a lot in chatbots or content generation tools, as the slow response can negatively impact the user’s experience. A latency that is too high might suggest that the computer is overloaded, resources are being shared, or the network is carrying extra traffic. By monitoring latency, developers can improve the infrastructure, use batching and move some of the computation to enhance responsiveness.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eMemory Utilization\u003c/b\u003e: The amount of memory taken by the model during inference is recorded through memory utilization. Many large language models are memory-intensive and may not function properly if proper limits are not set. Tracking this metric ensures that the system stays within the boundaries of your hardware and indicates whether the model loads correctly. It also supports decisions about when to use sharding and quantization in order to free up more memory.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eError Rate\u003c/b\u003e: It shows what portion of API calls or inferences did not go as expected. Some of these failures are due to timeouts, insufficient memory or server problems inside. When the error rate goes up, it may be because of infrastructure faults or update errors, and this needs to be fixed immediately to ensure people keep using the service. With this measure, teams can spot errors, learn what caused them and stop new deployments before anyone sees the problem.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003ePrompt Sensitivity\u003c/b\u003e: How prompt sensitivity works is this: the more different input prompts are from the main one, the more the outputs vary. When slightly changing how something is phrased, generative models can deliver outputs that are very different. Although a bit of creativity is good, being too sensitive can reduce the trustworthiness of your outcome and make it harder to repeat it. Keeping an eye on prompt sensitivity is useful for testing a model’s reliability, guiding prompt engineering, and deciding whether and when to improve it.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e\u003ch2\u003e3.5. Toolchain\u003c/h2\u003e\u003cul\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eDocker and Kubernetes for Container Orchestration\u003c/b\u003e: Packaging and orchestrating our AI services became easier using the base setup of Docker and Kubernetes. Because Docker was used to package the application components—model server, API gateway and a set of monitoring tools—they performed the same in every environment. Because of Kubernetes, these containers were deployed, scaled and managed automatically across many servers in a cluster. In this architecture, errors could be corrected, the service could expand horizontally, and systems could be easily rolled back, which were necessary for generative AI to succeed.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eTensorRT for Model Optimization\u003c/b\u003e: Inference performance was boosted by using TensorRT as the high-performance deep learning inference optimizer from NVIDIA. Bringing model training into an efficient runtime form, TensorRT cut down latency and raised the speed of execution, especially for models deployed to GPUs. This helped a lot when applying large generative models since the sheer amount of calculations needed is often the main limitation. Because of TensorRT, the system answered questions more promptly and efficiently, utilising the GPU, and accuracy was preserved.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eDVC for Data Versioning\u003c/b\u003e: With DVC, data could be handled, and any shifts in datasets during development could be tracked. Since it is important for machine learning to reuse results, DVC enabled us to version both the code and any data or model artefacts. Because of this, we could easily go back to any previous point, observe the experiments and verify that the data being used was what was intended for training and testing, which is required for correcting errors and auditing.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eWeights \u0026amp; Biases for Experiment Tracking\u003c/b\u003e: Weights \u0026amp; Biases helped us manage all the information related to our model’s training experiment, hyperparameters and measurements. Thanks to W\u0026amp;B, we could manage our experiments better and check model performance and differences between runs in real-time. Because the data and results were open to all, team members could pick the right training settings and regularly enhance how accurate and reliable the model was.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e"},{"header":"4. Results and Discussion","content":"\u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003e4.1. Case Study: Text Generation Platform\u003c/h2\u003e \u003cp\u003eIn this case, a transformer-based text assistant is used to support users by summarizing information, answering questions and making content suggestions. The application could automatically expand and shrink its resources, depending on traffic and workload, because horizontal pod auto-scaling was set up on the Kubernetes cluster. As a result, the system could cope with a lot of users and save resources when there were only a few. One important change during deployment was quantizing the model, which lowered the accuracy with which the model\u0026rsquo;s weights and calculations were recorded. By using this method, the average time it takes to process user requests dropped by 34%, reaching 620 milliseconds, without negatively impacting the language of the text. The faster answer speed boosted user interest, as the assistant felt more responsive in real time. Simultaneously, Prometheus began handling data measurement, while Grafana did the job of creating dashboards and issuing alerts. It offered an up-to-date look at functions such as system memory usage, error rates on the API and how sensitive the NLP model was. Early detection of anything unusual occurred because the framework made it possible for alerts to be set off automatically when the system\u0026rsquo;s behavior changed. So, the team could take steps beforehand to fix issues which brought system downtime down to 6 hours for the entire month. As downtime decreased, the service became even more accessible, and people\u0026rsquo;s faith in and satisfaction with the product grew as well. In short, the case study shows that using practices like Kubernetes orchestration and making models smaller while using observability can make generative AI perform better, work with more reliability and be more scalable when deployed at scale.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003e4.2. Performance Metrics\u003c/h2\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003ePerformance Metrics\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMetric\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003e% Improvement\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInference Latency\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e34%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAPI Downtime\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e70%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eUser Engagement\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e15%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eInference Latency (34% Improvement)\u003c/b\u003e: Because inference latency was reduced by 34%, the responsiveness of the generative text assistant greatly increased. By making the model efficient and placing it on auto-scalable Kubernetes, the system managed to provide faster answers and maintain its accuracy. With this upgrade, users noticed shorter delays, making the platform more suitable for use with quick chatbots and assistants.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eAPI Downtime (70% Improvement)\u003c/b\u003e: The significant reduction in API downtime means systems are more reliable and can be utilised more frequently. Because of real-time monitoring, active incident handling and the system\u0026rsquo;s ability to handle errors, several possible outages were prevented before they affected users. As a result of reducing downtime, people working with the system felt more secure, and the business experienced fewer disruptions to its everyday activities.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eUser Engagement (15% Improvement)\u003c/b\u003e: Higher user engagement by 15% means the system\u0026rsquo;s improvements have improved how users interact on the website. This made the assistant more useful and made people continue to use it and spend more time on it. It demonstrates that raising backend efficiency and user satisfaction usually come from technical improvements.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003e4.3. Challenges Encountered\u003c/h2\u003e \u003cp\u003eDuring deployment and operations, several challenges were identified:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eDependency Hell\u003c/b\u003e: The process of deploying the model pointed out that it uses many third-party libraries. Since a large number of these libraries were either unstable, deprecated or required specific versions, this caused frequent problems when trying to use them in containerized and integrated processes. Because of these problems, the continuous integration and deployment pipelines had unexpected failures, which caused longer release delays and made maintenance more complex. Therefore, in order to solve this, the system was set up so that all applications used the same versions of required libraries. Frequent checks were also made on the packages being used to fix and remove those that may cause instability in the system.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eVersion Drift\u003c/b\u003e: This issue happened when the differences between training and production environments caused the model\u0026rsquo;s behavior to vary. Updating the framework affected the models and, on some occasions, caused the program to fail. The changes caused the system to be unreliable and not able to be reused. Docker containers helped ensure that the training environment was the same in production, which reduced the possibility of problems. Also, MLflow was included to keep track of environment information together with model artifacts, allowing us to easily see how things are connected and solve issues.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eMonitoring Noise\u003c/b\u003e: It was a hard task to tell apart genuine innovative content and the occasional \u0026lsquo;hallucinations\u0026rsquo; when using generative models. Given that generative AI works in an unpredictable way, the traditional approach to finding mistakes in AI failed. We installed more layers focused on examination to ensure the output is both relevant and coherent. Adding better logging and manual reviews by people helped classify the outputs, making it simpler to identify anomalies and avoid wrongly triggered alarms.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"5. Conclusion","content":"\u003cp\u003eScaling generative AI models requires businesses to make major adjustments to the usual software engineering process. Unlike classic software, generative models require complex infrastructure that can handle a great amount of computations and memory. Besides, these models increase the chance of ethical and operational issues such as biased results, wrong predictions and issues related to privacy, which add to the difficulties of deploying and managing them. Through this paper, the gaps found in existing generation AI methodologies have been shown, and a practical route for addressing the complexity of implementing generative AI systems has been offered. As the study employs a modular architecture, handles lifecycle tasks, and includes observability, the information is particularly helpful for engineers and researchers working in industrial AI.\u003c/p\u003e \u003cp\u003eThis work helps by outlining the main issues regarding deployment that are important for generative AI. Such issues include infrastructure capacity problems, unpredictable results, complex data versions, and dangers due to increased bias and data exposure. Listing these specific concerns enables the paper to provide a better framework for developing systems that can address them. Moreover, the suggestion of using a modular tool framework that works with containers enhances models and provides a strong monitoring unit, uniting theory and practice. Validation of the framework comes from a case study with a transformer platform used for text generation, which shows great improvements in inference, reliability and the level of engagement. This process shows that the recommendations work as expected in practical situations.\u003c/p\u003e \u003cp\u003eLooking forward, the use of explainability in generative AI is expected to play a larger role in making its decisions easier to understand and explain. It will play a key role in establishing users\u0026rsquo; trust and obeying regulations in sensitive sectors. Today, another useful approach is to use bias corrections when the model predicts, thereby helping it avoid producing any unfair or harmful outcomes during use. Also, considering federated and decentralized deployment systems could improve privacy and flexibility since training and deployment of models could take place at different edge devices without sharing data with a central server. With the new advancements, generative AI will be able to work effectively on a large scale and in many industries and use cases.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eD.M. Authored the manuscript with details and diagrams\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eBeck K, Beedle M, Van Bennekum A, Cockburn A, Cunningham W, Fowler M, Thomas D (2001) Manifesto for agile software development.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAmershi S, Begel A, Bird C, DeLine R, Gall H, Kamar E, Zimmermann T (2019), May Software engineering for machine learning: A case study. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) (pp. 291\u0026ndash;300). IEEE\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSculley D, Holt G, Golovin D, Davydov E, Phillips T, Ebner D, Dennison D (2015) Hidden technical debt in machine learning systems. Adv Neural Inf Process Syst, 28\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZaharia M, Chen A, Davidson A, Ghodsi A, Hong SA, Konwinski A, Zumar C (2018) Accelerating the machine learning lifecycle with MLflow. IEEE Data Eng Bull 41(4):39\u0026ndash;45\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBreck E, Cai S, Nielsen E, Salib M, Sculley D (2017), December The ML test score: A rubric for ML production readiness and technical debt reduction. In 2017 IEEE international conference on big data (big data) (pp. 1123\u0026ndash;1132). IEEE\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTouvron H, Lavril T, Izacard G, Martinet X, Lachaux MA, Lacroix T, Lample G (2023) Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBrown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, Amodei D (2020) Language models are few-shot learners. Adv Neural Inf Process Syst 33:1877\u0026ndash;1901\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePatterson D, Gonzalez J, Le Q, Liang C, Munguia LM, Rothchild D, Dean J (2021) Carbon emissions and large neural network training. arXiv preprint arXiv :210410350\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCarlini N, Liu C, Erlingsson \u0026Uacute;, Kos J, Song D (2019) The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX security symposium (USENIX security 19) (pp. 267\u0026ndash;284)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFredrikson M, Jha S, Ristenpart T (2015), October Model inversion attacks that exploit confidence information and basic countermeasures in Proceedings of the 22nd ACM SIGSAC conference on computer and communications security (pp. 1322\u0026ndash;1333)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang C, Bengio S, Hardt M, Recht B, Vinyals O (2016) Understanding deep learning requires rethinking generalization. arXiv preprint arXiv :161103530\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShokri R, Stronati M, Song C, Shmatikov V (2017), May Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP) (pp. 3\u0026ndash;18). IEEE\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBenala TR, Dehuri S (2025) Transforming Software Development: From Traditional Methods to Generative Artificial Intelligence. Boosting Software Development Using Machine Learning. Springer, Cham, pp 1\u0026ndash;13\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eM\u0026auml;ntyl\u0026auml; MV, Vanhanen J (2011), March Software deployment activities and challenges-a case study of four software product companies. In 2011 15th European Conference on Software Maintenance and Reengineering (pp. 131\u0026ndash;140). IEEE\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCasale G, Chesta C, Deussen P, Di Nitto E, Gouvas P, Koussouris S, Zhao Z (2016) Current and future challenges of software engineering for services and applications. Procedia Comput Sci 97:34\u0026ndash;42\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eForward A, Lethbridge TC (2002), November The relevance of software documentation, tools and technologies: a survey. In Proceedings of the 2002 ACM symposium on Document engineering (pp. 26\u0026ndash;33)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAitken A, Ilango V (2013), January A comparative analysis of traditional software engineering and agile software development. In 2013 46th Hawaii International Conference on System Sciences (pp. 4751\u0026ndash;4760). IEEE\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHe M, Li Z, Liu C, Shi D, Tan Z (2020) Deployment of artificial intelligence in real-world practice: opportunity and challenge. Asia-Pacific J Ophthalmol 9(4):299\u0026ndash;307\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShaw J, Rudzicz F, Jamieson T, Goldfarb A (2019) Artificial intelligence and the implementation challenges. J Med Internet Res, 21(7), e13659\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang Z, Duan Z, Zhang H, Wang C, Tian L, Chen B, Zhou M (2020), November Friendly topic assistant for transformer-based abstractive summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 485\u0026ndash;497)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSauvola J, Tarkoma S, Klemettinen M, Riekki J, Doermann D (2024) Future of software development with generative AI. Automated Softw Eng 31(1):26\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":true,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Generative AI, Software Engineering, Model Deployment, MLOps, Large Language Models, Scalability, CI/CD","lastPublishedDoi":"10.21203/rs.3.rs-6875774/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6875774/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eBecause generative AI has expanded quickly, today Large Language Models (LLMs) and diffusion-based generative models are transforming healthcare, financial, educational, and entertainment fields. Even though these models work well, getting them into use on a wide scale creates software engineering difficulties. Some of these issues include providing infrastructure, controlling inference latency, maintaining privacy, obeying ethics, building CI/CD processes, controlling versions, and keeping an eye on the models. The study explores the complicated aspects of putting generative AI models into real-world situations. In examining AI software engineering patterns, as well as its primary development and use processes, this study highlights the specific problems that still exist when employing traditional methods with AI systems. We share information about new methods and procedures made for AI operations (MLOps), highlight the value of joint efforts, and change the usual development steps. Next, the research paper demonstrates how a generative AI system was implemented in practice, points out problems that came up and reveals solutions, finishing with a guideline for proper and successful deployment.\u003c/p\u003e","manuscriptTitle":"Software Engineering Challenges in the Deployment of Generative AI Models at Scale","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-06-17 12:18:52","doi":"10.21203/rs.3.rs-6875774/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"2808ea43-f949-46b2-9b99-9283e5494e08","owner":[],"postedDate":"June 17th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-06-25T06:24:02+00:00","versionOfRecord":[],"versionCreatedAt":"2025-06-17 12:18:52","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6875774","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6875774","identity":"rs-6875774","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}