EADF: An Environment-Aware Deployment Design Pattern for Multi-Cloud Data Engineering CI/CD Pipelines

EADF: An Environment-Aware Deployment Design Pattern for Multi-Cloud Data Engineering CI/CD Pipelines | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article EADF: An Environment-Aware Deployment Design Pattern for Multi-Cloud Data Engineering CI/CD Pipelines Chiara Rucco, Motaz Saad, Enrique Puig, Antonella Longo This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7659174/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Continuous Integration and Continuous Deployment (CI/CD) have revolutionized software engineering by enabling rapid, reliable, and automated delivery of code. However, data engineering projects face unique challenges, such as environment-specific configurations, data versioning, schema drift, and multi-cloud heterogeneity, that render traditional CI/CD pipelines insufficient. Existing tools for data engineering offer partial solutions but lack a standardized, reusable approach to promote data components (e.g., pipelines, models, transformation logics) consistently across development, testing, and production environments. This paper introduces the Environment-Aware Deployment Framework (EADF), a novel CI/CD design pattern for data engineering that decouples deployment logic from environment and cloud specifics using minimal, parameterized configurations. EADF enables teams to "write once, reuse everywhere" by late-binding configurations at runtime, ensuring reproducible, auditable, and portable deployments across multi-clouds platforms (e.g., Azure, AWS, GCP). Unlike ad-hoc scripts or vendor-specific pipelines, EADF fills a critical gap in the literature: the absence of a standardized, composable, and environment-aware design pattern for CI/CD for data projects. Our framework supports incremental adoption, reduces configuration duplication, and enhances governance, making it suitable for both data engineering and MLOps workflows. We provide open, reusable specifications and implementation guidance to facilitate real-world adoption. Design Patterns CICD Continuous Integration Continuous Deployment Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7659174","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":517992253,"identity":"0bf42c70-aaac-427a-9669-86aa94064e84","order_by":0,"name":"Chiara Rucco","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABA0lEQVRIiWNgGAWjYDACCcYGBOcDAwMPAwMzRISNGC2MM8BaQCIJQC249EggsZl5IDohWnBZwz+7ufFzRQ1DvsHxww8f27bdkZFvP9j4uPCHDQOffANWLRJ3DjZLnjnGYLnhTJqxcW7bMx6DM4nNxjMS0nA6zEAisUGygY3BwOAGD5t0btthHgOGxDZpnoTD+LQ0/2z4B9ViCdQi3/+w/TcBLW2SjW1QLYxALQw3EtuY8WmRACqwbOyTMJAE+sWw5xzQYTceNkvzpKXxsLElYA+xGemPbzZ8szHgA4bYgx9lh+3l+5MPfuaxsZGTbz6A3RqoZZhCPPjUj4JRMApGwSjADwAhhlQ1sycZXwAAAABJRU5ErkJggg==","orcid":"","institution":"University of Salento","correspondingAuthor":true,"prefix":"","firstName":"Chiara","middleName":"","lastName":"Rucco","suffix":""},{"id":517992255,"identity":"5dd11d86-603c-4439-ae09-2d8525fc4e41","order_by":1,"name":"Motaz Saad","email":"","orcid":"","institution":"University of Salento","correspondingAuthor":false,"prefix":"","firstName":"Motaz","middleName":"","lastName":"Saad","suffix":""},{"id":517992256,"identity":"397fd249-44c7-4f87-a1ae-16681dd5211e","order_by":2,"name":"Enrique Puig","email":"","orcid":"","institution":"University of Alicante","correspondingAuthor":false,"prefix":"","firstName":"Enrique","middleName":"","lastName":"Puig","suffix":""},{"id":517992259,"identity":"057bd77a-dd5f-4078-83fe-7bb691e29b1c","order_by":3,"name":"Antonella Longo","email":"","orcid":"","institution":"University of Salento","correspondingAuthor":false,"prefix":"","firstName":"Antonella","middleName":"","lastName":"Longo","suffix":""}],"badges":[],"createdAt":"2025-09-19 13:53:14","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7659174/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7659174/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":92065142,"identity":"b136043d-7ff4-4dcf-8629-7b57ee8caf68","added_by":"auto","created_at":"2025-09-24 08:52:14","extension":"json","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":6330,"visible":true,"origin":"","legend":"","description":"","filename":"47257a9f658941d693ce3c8fb797031a.json","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/7f515b8c8b363afc502ad64b.json"},{"id":92065113,"identity":"55a34702-cc76-437d-a1b4-09aec8e6217c","added_by":"auto","created_at":"2025-09-24 08:52:00","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":677977,"visible":true,"origin":"","legend":"","description":"","filename":"CICDJournal.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/4ae22494778f30324ab626b7.pdf"},{"id":92065127,"identity":"cff2ab6c-3b81-4a8f-8a1a-017e475c24e1","added_by":"auto","created_at":"2025-09-24 08:52:06","extension":"pdf","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":64750,"visible":true,"origin":"","legend":"","description":"","filename":"CoverLetterKSI.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/cfb4b84a1724315aba3f24b5.pdf"},{"id":92065126,"identity":"2cab0fcd-33a4-4bfd-9b15-0aea7067af77","added_by":"auto","created_at":"2025-09-24 08:52:05","extension":"png","order_by":3,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":174553,"visible":true,"origin":"","legend":"","description":"","filename":"cicd.png","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/1f1455895bd609a73c858552.png"},{"id":92065129,"identity":"f1bb1098-af1c-43fc-9098-161c9b934bd2","added_by":"auto","created_at":"2025-09-24 08:52:06","extension":"png","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":68837,"visible":true,"origin":"","legend":"","description":"","filename":"cicdpattern.png","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/1919faf486b1eea9786dbcd1.png"},{"id":92065138,"identity":"58859822-6f84-42fc-bc3b-b34fa87c1cf2","added_by":"auto","created_at":"2025-09-24 08:52:12","extension":"png","order_by":5,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":99363,"visible":true,"origin":"","legend":"","description":"","filename":"comparison.png","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/baa44bd8bf9f5f3244ea265f.png"},{"id":92065118,"identity":"ea5168f8-0904-4bec-91d2-a1e36afe3c27","added_by":"auto","created_at":"2025-09-24 08:52:03","extension":"eps","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":2890,"visible":true,"origin":"","legend":"","description":"","filename":"empty.eps","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/419cb376553b3ff4a3d0c319.eps"},{"id":92065116,"identity":"b3f5150e-6d19-4538-aa46-77f7251b5c3e","added_by":"auto","created_at":"2025-09-24 08:52:02","extension":"eps","order_by":7,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":91593,"visible":true,"origin":"","legend":"","description":"","filename":"fig.eps","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/505b4e8e0826f46a20cdae70.eps"},{"id":92065125,"identity":"5bf28f23-5872-4af3-aa07-6c841e2e9069","added_by":"auto","created_at":"2025-09-24 08:52:05","extension":"bst","order_by":8,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":146013,"visible":true,"origin":"","legend":"","description":"","filename":"snapacite.bst","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/077adbea09606cc71a6a6de2.bst"},{"id":92065115,"identity":"e3263c19-01d1-4bb7-b5f0-3b3280044e40","added_by":"auto","created_at":"2025-09-24 08:52:01","extension":"bst","order_by":9,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":29828,"visible":true,"origin":"","legend":"","description":"","filename":"snaps.bst","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/db67e9cd87901e92b48af0e0.bst"},{"id":92065139,"identity":"6bb70a50-064f-4f6f-9917-c41978c90c4e","added_by":"auto","created_at":"2025-09-24 08:52:12","extension":"pdf","order_by":10,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":421391,"visible":true,"origin":"","legend":"","description":"","filename":"snarticle.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/6474d26a74f355f2c1d3b86b.pdf"},{"id":92065134,"identity":"cb1751f2-71ee-46c5-b982-7672797fa7d9","added_by":"auto","created_at":"2025-09-24 08:52:10","extension":"bst","order_by":11,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":35515,"visible":true,"origin":"","legend":"","description":"","filename":"snbasic.bst","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/4ef01bc3491d67949be06e6e.bst"},{"id":92065137,"identity":"0dfc56e1-658c-42da-89d6-24a9e2828bb7","added_by":"auto","created_at":"2025-09-24 08:52:12","extension":"bst","order_by":12,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":33968,"visible":true,"origin":"","legend":"","description":"","filename":"snchicago.bst","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/26705aa55b01c5d18a0d1a89.bst"},{"id":92065144,"identity":"625e7c6e-e1c8-4e31-b946-b0c7d1e1b2c8","added_by":"auto","created_at":"2025-09-24 08:52:15","extension":"cls","order_by":13,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":52815,"visible":true,"origin":"","legend":"","description":"","filename":"snjnl.cls","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/830f0b3a41ba86389b5d1dcf.cls"},{"id":92065123,"identity":"b9401a42-cea4-4377-a42f-6cd3280384d0","added_by":"auto","created_at":"2025-09-24 08:52:04","extension":"bst","order_by":14,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":64023,"visible":true,"origin":"","legend":"","description":"","filename":"snmathphysay.bst","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/0aea692d16a3018bd0873eaf.bst"},{"id":92065119,"identity":"343e6531-1b97-4996-882b-2f9c5d99abd4","added_by":"auto","created_at":"2025-09-24 08:52:03","extension":"bst","order_by":15,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":64166,"visible":true,"origin":"","legend":"","description":"","filename":"snmathphysnum.bst","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/60dc55599065bfb6a2c78686.bst"},{"id":92065128,"identity":"a11a6edb-d1c2-4f0a-8181-3dce43ce7678","added_by":"auto","created_at":"2025-09-24 08:52:06","extension":"bst","order_by":16,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":37333,"visible":true,"origin":"","legend":"","description":"","filename":"snnature.bst","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/46d50909f9ae7db5b2edecf6.bst"},{"id":92065143,"identity":"e01212ec-f14d-428b-94ee-03cb1ba808d8","added_by":"auto","created_at":"2025-09-24 08:52:15","extension":"bst","order_by":17,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":39951,"visible":true,"origin":"","legend":"","description":"","filename":"snvancouveray.bst","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/7f7abaa80d36db135de413bb.bst"},{"id":92065114,"identity":"e9b7ba70-012b-401b-a74a-07ceb527f84f","added_by":"auto","created_at":"2025-09-24 08:52:01","extension":"bst","order_by":18,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":40758,"visible":true,"origin":"","legend":"","description":"","filename":"snvancouvernum.bst","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/45f8ae5c2ba87ac6187c4be8.bst"},{"id":92065122,"identity":"5c919b1b-8bcd-49f7-a6f1-0c3abcdbec11","added_by":"auto","created_at":"2025-09-24 08:52:04","extension":"png","order_by":19,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":163113,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinecicd.png","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/8e7915d743615f9bd430978f.png"},{"id":92065136,"identity":"606404e3-bf92-4381-a585-c464b62559ec","added_by":"auto","created_at":"2025-09-24 08:52:11","extension":"png","order_by":20,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":63100,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinecicdpattern.png","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/2baafc679e37253069e96933.png"},{"id":92065124,"identity":"ad414156-a7b0-478d-b27a-ad9980784d6f","added_by":"auto","created_at":"2025-09-24 08:52:05","extension":"png","order_by":21,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":97394,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinecomparison.png","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/18d762813e5f70b6ec0daac9.png"},{"id":92065120,"identity":"efd7cb7d-4387-4643-8fee-898f5e2eaea2","added_by":"auto","created_at":"2025-09-24 08:52:04","extension":"xml","order_by":22,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":92918,"visible":true,"origin":"","legend":"","description":"","filename":"47257a9f658941d693ce3c8fb797031a1structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1/eddce08e9089ebb473395857.xml"},{"id":92193283,"identity":"2ab7d7e6-ff76-4abb-8f75-20b11f8c1932","added_by":"auto","created_at":"2025-09-25 15:30:13","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":722016,"visible":true,"origin":"","legend":"","description":"","filename":"CICDJournal.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7659174/v1_covered_f45a7d43-38f1-4d74-bc48-1498b6bf3d34.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"EADF: An Environment-Aware Deployment Design Pattern for Multi-Cloud Data Engineering CI/CD Pipelines","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Design Patterns, CICD, Continuous Integration, Continuous Deployment","lastPublishedDoi":"10.21203/rs.3.rs-7659174/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7659174/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\nContinuous Integration and Continuous Deployment (CI/CD) have revolutionized software engineering by enabling rapid, reliable, and automated delivery of code. However, data engineering projects face unique challenges, such as environment-specific configurations, data versioning, schema drift, and multi-cloud heterogeneity, that render traditional CI/CD pipelines insufficient. Existing tools for data engineering offer partial solutions but lack a standardized, reusable approach to promote data components (e.g., pipelines, models, transformation logics) consistently across development, testing, and production environments.\n\nThis paper introduces the Environment-Aware Deployment Framework (EADF), a novel CI/CD design pattern for data engineering that decouples deployment logic from environment and cloud specifics using minimal, parameterized configurations. EADF enables teams to \"write once, reuse everywhere\" by late-binding configurations at runtime, ensuring reproducible, auditable, and portable deployments across multi-clouds platforms (e.g., Azure, AWS, GCP).\n\nUnlike ad-hoc scripts or vendor-specific pipelines, EADF fills a critical gap in the literature: the absence of a standardized, composable, and environment-aware design pattern for CI/CD for data projects. Our framework supports incremental adoption, reduces configuration duplication, and enhances governance, making it suitable for both data engineering and MLOps workflows. We provide open, reusable specifications and implementation guidance to facilitate real-world adoption.","manuscriptTitle":"EADF: An Environment-Aware Deployment Design Pattern for Multi-Cloud Data Engineering CI/CD Pipelines","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-09-24 08:51:20","doi":"10.21203/rs.3.rs-7659174/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"f5540286-5255-4a03-8d13-d66b95f184fa","owner":[],"postedDate":"September 24th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-09-25T15:13:41+00:00","versionOfRecord":[],"versionCreatedAt":"2025-09-24 08:51:20","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7659174","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7659174","identity":"rs-7659174","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}