AnVILWorkflow: A runnable workflow package for Cloud-implemented bioinformatics analysis pipelines

AnVILWorkflow: A runnable workflow package for Cloud-implemented bioinformatics analysis pipelines | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF software AnVILWorkflow: A runnable workflow package for Cloud-implemented bioinformatics analysis pipelines Sehyun Oh, Kai Gravel-Pucillo, Marcel Ramos, Sean Davis, Vince Carey, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4370115/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Advancements in sequencing technologies and the development of new data collection methods produce large volumes of biological data. The Genomic Data Science Analysis, Visualization, and Informatics Lab-space (AnVIL) provides a cloud-based platform for democratizing access to large-scale genomics data and analysis tools. However, utilizing the full capabilities of AnVIL can be challenging for researchers without extensive bioinformatics expertise, especially for executing complex workflows. Here we present the AnVILWorkflow R package, which enables the convenient execution of bioinformatics workflows hosted on AnVIL directly from an R environment. AnVILWorkflowsimplifies the setup of the cloud computing environment, input data formatting, workflow submission, and retrieval of results through intuitive functions. We demonstrate the utility of AnVILWorkflowfor three use cases: bulk RNA-seq analysis with Salmon, metagenomics analysis with bioBakery, and digital pathology image processing with PathML. The key features of AnVILWorkflow include user-friendly browsing of available data and workflows, seamless integration of R and non-R tools within a reproducible analysis pipeline, and accessibility to scalable computing resources without direct management overhead. While some limitations exist around workflow customization, AnVILWorkflowlowers the barrier to taking advantage of AnVIL's resources, especially for exploratory analyses or bulk processing with established workflows. This empowers a broader community of researchers to leverage the latest genomics tools and datasets using familiar R syntax. This package is distributed through the Bioconductor project (https://bioconductor.org/packages/AnVILWorkflow), and the source code is available through GitHub (https://github.com/shbrief/AnVILWorkflow). Figures Figure 1 1. Introduction The NHGRI’s Genomic Data Science An alysis, V isualization, and I nformatics L ab-space (AnVIL) consortium was launched in 2018, aiming to democratize genomics data[1]. AnVIL enables easy sharing of genomics data by organizing databases, bioinformatics pipelines for large-scale data processing, and interactive downstream analysis in one Cloud-based platform. AnVIL[2], also the name of the platform from the AnVIL project, implements the FAIR data-sharing philosophy and provides a graphical user interface (GUI, supported by Terra[3]), making it more accessible for researchers without programming backgrounds. However, a GUI tends to be less efficient and slower than a command line interface (CLI), especially for bulk analyses, still requires learning a new platform, and does not support version control and text-based workflows, often included as best practices for reproducible computational research[4] Bioconductor’s AnVIL package is an AnVIL API wrapper that provides R-friendly, programming-based functionalities to leverage flexible and scalable cloud-based resources implemented in the AnVIL platform. With the AnVIL package, users can easily access workflows, data, and Cloud-based computing resources managed by AnVIL. However, the AnVIL package is not customized for workflow execution tasks. Instead, AnVIL covers all the resources related to the AnVIL platform, such as interaction with the repository for Docker-based genomic analysis tools and workflows (Dockstore[5]), leveraging cloud resources (Leonardo[6]), and data search and digestion (Gen3[7]). Many AnVIL functions also expose API commands directly, requiring a deep understanding of the underlying AnVIL workspace structures and data models to use for workflow execution. Also, it is a general package without individual support on any workspace and provides no metadata curation. Because the majority of Bioconductor users focus on data analysis, a convenient R-friendly way of accessing and utilizing AnVIL resources is needed. Here, we present the AnVILWorkflow package to meet this need. AnVILWorkflow package is a convenient, fit-for-purpose wrapper around the AnVIL package with the following features optimized for workflow execution: Intuitive function names and returned values Support workflow-specific documentations Enable to set up a Cloud environment with a single function call Return error messages that are easy to interpret and actionable Essential metadata curation for more efficient data browsing Users can apply AnVILWorkflow on any workspace they can access, including 347 public workspaces (snapshot on 8.28.23) available to anyone with an AnVIL account. We present the three use cases where we ran non-R-based bioinformatics analysis tools using conventional R syntax: Salmon [8] , bioBakery [9] , and PathML [10] . Salmon is a widely used RNA sequencing analysis tool for quantifying the expression of transcripts and is based on the command-line interface. Its downstream analysis involves many R/Bioconductor packages, such as DESeq2 , edgeR , and limma . bioBakery is a widely used whole metagenomic shotgun (WMS) sequencing data analysis environment, mainly relying on Python. PathML is a general-purpose research toolkit for computational pathology, including many functionalities in digital pathology data analysis, such as strain normalization, nucleus segmentation, and tissue detection. PathML takes raw image files and returns the processed image data in an hdf5 format for further downstream analysis, including machine learning methods. 2. Overview AnVIL provides comprehensive resources for biomedical data analysis, including data (e.g., genomics), workflows for bulk analysis, and interactive analysis apps (i.e., Galaxy, Jupyter Notebooks, and RStudio) under the workspace. Among them, workflows are often a limiting factor in bioinformatics analysis due to computing demands and bioinformatics expertise required. Thus, the AnVILWorkflow package makes the workflow-related resources from AnVIL more accessible and easier to use, especially for R users (Fig. 1 ). While AnVIL manages workflow orchestration and workspace metadata and provides default setups simplifying decision-making for users, users still need to manage the storage of their data and cloud cost. Genomics data, especially their raw and intermediate forms, are very large, so data storage can be costly if the sample size increases. Storage costs incur and can be managed in two ways - storage itself and transfer. For example, using regional storage instead of multi-region, cleaning up intermediate results, and storing infrequently accessed data in low-cost storage (e.g., nearline or coldline storage from Google Cloud) can reduce per-sample costs. Analyzing data stored in one region using Virtual Machine (VM) compute resources in a different region incurs data transfer charges, so centralizing all storage and computing in a single region can be more cost-efficient by not only reducing the storage cost but also avoiding data transfer charges. Currently, the AnVIL workspaces use the us-central1 as a default region and any artifacts generated from the workflow execution, unless specified, are saved in the same-region bucket linked to the workspace. If users use the default region configured by AnVIL, bringing their data stored in the default region, us-central1, will save the data transfer charge. Additionally, open and controlled access genomic datasets hosted in AnVIL are stored in the us-multi-region, so there are no storage and transfer charges for users using the default workspace configuration. Downloading data to the user’s workstation or laptop is subject to charges, currently $ 0.08 to $ 0.12 per GB, depending on the amount of data[ 11 ] and geography of the transfer, and transfer from the US to another continent is more expensive than within the US transfer. While browsing existing resources through AnVILWorkflow is free, running workflows charge computing costs. AnVILWorkflow is designed to use existing workflows which usually predefine computing resources optimized for the types of analyses, simplifying computing-related cost management. You can further reduce the run cost using call caching and preemptive instances. For example, if your workflow runs in fewer than 24 hours since a preemptible VM lasts 24 hours at most, you can save up to 80% by using preemptible VMs. The cost management for a group of users can be efficiently managed through the AnVIL billing project. One billing account can be shared with others by simply adding email addresses under the billing project. The billing project offers details on each workspace, including workspace owner and spent reports, so we can easily identify ‘who’ uses ‘how much’ for ‘what’. In addition to the workspace-level expense reports, users can further enhance cost monitoring by configuring spending reporting[ 12 ]. This allows users to closely monitor the expenditure associated with each workflow execution. 3. Major functions Browse AnVIL resources. The AnVILBrowse function allows users to browse AnVIL resources using keywords. This function runs instantaneously because the AnVILWorkflow package includes the snapshot of metadata on all the publicly accessible AnVIL workspaces and their workflows and data. It performs basic metadata harmonization, allowing more efficient browsing and filtering, such as selecting workspaces based on the study size or participants’ ages. Users can also browse non-public workspaces they have access to using the getMetaTables function; however, this process can take a while depending on the number of workspaces a user has access to. Run AnVIL workflows. AnVILWorkflow package provides all the functionalities required to run workflows available in AnVIL from the local R session - from the environment setup to the output download. One prerequisite is to create an AnVIL account from the AnVIL web portal. AnVIL account provides two required inputs to run workflows remotely: 1) the email address associated with the user’s account and 2) the billing project name to cover the computing cost. AnVIL-hosted workflows can be run using four main functions: setCloudEnv, cloneWorkspace, runWorkflow, and getOutput. The setCloudEnv function accepts the AnVIL account email and billing project name and sets up your local R environment ready to access AnVIL and Cloud-computing resources. The cloneWorkspace function creates the user’s copy of a ‘template’ workspace and the runWorkflow executes the workflow. The getOutput function can check the outputs from successfully executed workflows and download user-specified files to a local computer. User input can be provided through the updateInput function, which accepts two different forms of tables depending on the workflows - AnVIL’s data model or URLs pointing to data files stored in Google Cloud buckets. The input data formats are already specified in the workflow scripts (Workflow Description Language, WDL[ 13 ]). Other accessory functions are available to monitor submission progress (monitorWorkflow), stop submitted workflow (stopWorkflow), and get Dashboard content (getDashboard). 4. Use cases The use cases demonstrated below include demo input data in the template workspaces, so the R scripts below can run the listed use cases from the local computer. Ready-to-run examples that can be used to test the process on the user’s own AnVIL account are available in the AnVILWorkflow package vignette. GATK best practice pipelines[ 14 ] are not demonstrated here, but they are also available as AnVIL workspaces. ## Setup the account setCloudEnv (accountEmail = {AnVIL account email}, billingProjectName = {AnVIL billing project name}) ## Clone the workspace of your interest newName <- {Unique name for your copy of workspace} cloneWorkspace (workspaceName = newName, templateName = templateName) ## Run workflow runWorkflow (workspaceName = newName, workflowName = {name of the workflow if there is more than one in the workspace of your interest}) ## Get workflow outputs getOutput (workspaceName = newName) The main features of the demo workspaces and their workflow-specific input data preparation process are described below. 4.1. Bulk RNA sequencing data analysis Salmon workflow uses AnVIL’s data model and requires four essential inputs - fastq1, fastq2, fasta, and transcriptome index name . This workflow can be easily applied to the consortium data hosted in AnVIL, which follows AnVIL’s data model. With the default runtime environment configured for this workflow (1 CPU, 2GB memory, and 10GB SSD disk), processing 16 demo samples (32 fastq files, ~1GB per file) took about 30 minutes and cost $ 0.12. 4.2. Whole metagenomic shotgun data analysis bioBakery is a metagenome analysis environment composed of Python-based tools, reference databases, and command-line-based workflows. It processes raw shotgun sequencing data into microbial community feature profiles, summary reports, and figures[ 9 ]. bioBakery ’s whole metagenome shotgun (wmgx) and visualization (wmgx_vis) workflows are implemented as an AnVIL workspace. The current version of the AnVILWorkflow supports bioBakery version 3[ 15 ]. While users can customize this workflow to a great degree, only six inputs are sufficient to run a standard, optimized version of this workflow. Those six inputs are: - Name of the Trimmomatic adaptor type (for demo data, NexteraPE ) - Your project name - Extension of input files (for demo data, .fastq.gz) - A table of your sequencing file (fastq) names stored in the Google Cloud Storage bucket - Input file identifier for paired-end sequencing (for demo data, _R1 and _R2) The seven required databases are already linked to this workflow and nine additional optional inputs are available for further customization. Optional inputs are for workflow customization, such as bypassing functional profiling (default is false) and maximum memory usage for different tasks (default is 32GB for functional profiling by HUMAnN , 8GB for quality control by Kneaddata , and 24GB for taxonomic profiling by MetaPhlAn ). This workflow uses call caching and preemptive instances by default for cost efficiency. Processing six paired-end demo samples (mean file size ~ 380MB) with the optimized default setting without using preemptive instances took about 5 hours and cost around $ 6.50. With the preemptive instances, it can take longer but cost less. Compared to the existing options such as Nephele[ 16 ], AnVILWorkflow allows a programmatic approach and more flexible customization options. 3.3. Histopathology image processing using PathML We implemented the hematoxylin-eosin (HE) stain normalization process of PathML as an AnVIL workspace. This workflow accepts an SVS file as input and returns original and normalized images as PNG files. There are two required inputs - Google Cloud Storage URI where the input SVS image file is stored and the sample name. Processing one publicly available image (CMU-1_Small_Region.svs, 1.8MB)[ 17 ] with the default runtime (4 CPU, 16GB memory) took about 8 minutes and cost $ 0.01. This simple but robust analysis setup can support clinical use cases, such as pathologists who process a large number of images in a short time, by offering guidance and cross-validation options. 5. Conclusions The AnVILWorkflow package enables users to conduct complex and computationally intense analyses with minimal bioinformatics expertise, through well-established workflows within AnVIL and versatile cloud resources directly from standard laptops using the familiar R syntax. The major advantages AnVILWorkflow provides over the existing approaches include 1) a minimal entry barrier, negating the need for software installations, preparation of properly versioned reference data, or construction and oversight of workflows, 2) leveraging flexible cloud computing resources without the need to learn or handle them directly, 3) user-friendly functions that provide enhanced information, and 4) greatly improved reproducibility and interoperability by seamlessly linking multiple analysis steps, conducted in both R and non-R based tools, within a single R vignette. However, there are still some limitations. For instance, certain customizations of the workflows are limited or require a more profound understanding of the workflows. Despite not being inherently more costly than an in-house server, the pay-per-use structure requires careful planning and management. The absence of an integrated versioning system in AnVIL workspaces requires users to manually monitor new versions. In conclusion, AnVILWorkflow proves most advantages for analyzing a bulk of samples on relatively simple workflows (i.e., single-stage workflow procedure) or for exploratory data analysis for non-technical users, particularly when employing well-established analysis workflows. Abbreviations AnVIL: Analysis, Visualization, and Informatics Lab-space GUI: Graphical User Interface CLI: Command Line Interface WMS: whole metagenomic shotgun VM: Virtual Machine WDL: Workflow Description Language HE: hematoxylin-eosin HUMAnN: HMP Unified Metabolic Analysis Network Declarations Ethics approval and consent to participate Not applicable Consent for publication Not applicable Availability of data and materials All data generated or analyzed during this study are included in this published article and available on the AnVIL platform (https://anvil.terra.bio/#). AnVILWorkflow package is distributed through the Bioconductor project (https://bioconductor.org/packages/AnVILWorkflow), and the source code is available through GitHub (https://github.com/shbrief/AnVILWorkflow). Competing interests The authors declare that they have no competing interests. Funding SO, KG, MR, VC, MM, and LW were supported by the National Human Genome Research Institute (U24HG010263). Authors’ contributions SO, LW, and VC conceived the research idea. SO, KG, MR, and MM developed the software. SO and KG performed the benchmarking analyses. SO, LW, and KG wrote the manuscript. LW, SD, and MR reviewed the manuscript. Acknowledgments Not applicable References Schatz MC, Philippakis AA, Afgan E, Banks E, Carey VJ, Carroll RJ, et al. Inverting the model of genomics data sharing with the NHGRI Genomic Data Science Analysis, Visualization, and Informatics Lab-space. Cell Genom [Internet]. 2022;2. Available from: http://dx.doi.org/10.1016/j.xgen.2021.100085 Terra [Internet]. [cited 2024 Jan 31]. Available from: https://anvil.terra.bio/ Terra [Internet]. [cited 2024 Jan 31]. Available from: https://app.terra.bio/ Sandve GK, Nekrutenko A, Taylor J, Hovig E. Ten simple rules for reproducible computational research. PLoS Comput Biol. 2013;9:e1003285. Yuen D, Cabansay L, Duncan A, Luu G, Hogue G, Overbeck C, et al. The Dockstore: enhancing a community platform for sharing reproducible and accessible computational protocols. Nucleic Acids Res. 2021;49:W624–32. leonardo: Notebook service [Internet]. Github; [cited 2024 Mar 6]. Available from: https://github.com/DataBiosphere/leonardo Hughes L, Grossman RL, Flamig Z, Prokhorenkov A, Lukowski M, Fitzsimons M, et al. Harmonization of clinical data across Gen3 data commons. J Clin Orthod. 2019;37:e18094–e18094. Patro R, Duggal G, Love MI, Irizarry RA, Kingsford C. Salmon provides fast and bias-aware quantification of transcript expression. Nat Methods. 2017;14:417–9. McIver LJ, Abu-Ali G, Franzosa EA, Schwager R, Morgan XC, Waldron L, et al. bioBakery: a meta’omic analysis environment. Bioinformatics. 2018;34:1235–7. Rosenthal J, Carelli R, Omar M, Brundage D, Halbert E, Nyman J, et al. Building Tools for Machine Learning and Artificial Intelligence in Cancer Research: Best Practices and a Case Study with the PathML Toolkit for Computational Pathology. Mol Cancer Res. 2022;20:202–6. Pricing [Internet]. Google Cloud. [cited 2024 Feb 26]. Available from: https://cloud.google.com/storage/pricing How much did my workflow cost? [Internet]. Terra Support. [cited 2024 Feb 20]. Available from: https://support.terra.bio/hc/en-us/articles/360037862771-How-much-did-my-workflow-cost Voss K, Gentry J, Van der Auwera G. Full-stack genomics pipelining with GATK4 + WDL + Cromwell [Internet]. F1000Research; 2017. Available from: https://f1000research.com/posters/6-1379 Van der Auwera GA, O’Connor BD. Genomics in the Cloud: Using Docker, GATK, and WDL in Terra. “O’Reilly Media, Inc.”; 2020. Beghini F, McIver LJ, Blanco-Míguez A, Dubois L, Asnicar F, Maharjan S, et al. Integrating taxonomic, functional, and strain-level profiling of diverse microbial communities with bioBakery 3. Elife [Internet]. 2021;10. Available from: http://dx.doi.org/10.7554/eLife.65088 Weber N, Liou D, Dommer J, MacMenamin P, Quiñones M, Misner I, et al. Nephele: a cloud platform for simplified, standardized and reproducible microbiome data analysis. Bioinformatics. 2018;34:1411–3. Aperio SVS [Internet]. [cited 2024 Feb 21]. Available from: https://openslide.cs.cmu.edu/download/openslide-testdata/Aperio/ Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4370115","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"software","associatedPublications":[],"authors":[{"id":302287717,"identity":"0218b25c-2339-4b2b-aaff-082e197bac0b","order_by":0,"name":"Sehyun Oh","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA/ElEQVRIiWNgGAWjYBAC+QYeMC1nwN7AwAwX5sGjxeAARNbYgOcAsVqgsokbJBKI1cLee0yadwdD4nbJ588eFzDYJW6XSGB88LYNtxb5nnNp0rxnGBJ3zs4xN57BkJy4c0YCs+FcPFoYbuSYSfO2AR12O4dNmofhQOKGGwlsIBHc4P4bqJabx5/BtLD/xqvlBg9Uyw0GM7gtzPi0GJzJMbac2yZhDGSYG/MYJBtvOPOwWXLOOTzebz9jeONtm42cwfHjzx7zVNjJbjiefPDDmzI8DmNgYJFgYJAAMdhA0cTAIJDYgFc9EDB/gDLYIBT/AUI6RsEoGAWjYIQBAEiCUGw5KymaAAAAAElFTkSuQmCC","orcid":"","institution":"City University of New York School of Public Health","correspondingAuthor":true,"prefix":"","firstName":"Sehyun","middleName":"","lastName":"Oh","suffix":""},{"id":302287718,"identity":"78017363-74ed-45b8-afca-7749d34be679","order_by":1,"name":"Kai Gravel-Pucillo","email":"","orcid":"","institution":"City University of New York School of Public Health","correspondingAuthor":false,"prefix":"","firstName":"Kai","middleName":"","lastName":"Gravel-Pucillo","suffix":""},{"id":302287719,"identity":"9d474a6d-7e3a-4a00-9d2b-ac101a6f49db","order_by":2,"name":"Marcel Ramos","email":"","orcid":"","institution":"City University of New York School of Public Health","correspondingAuthor":false,"prefix":"","firstName":"Marcel","middleName":"","lastName":"Ramos","suffix":""},{"id":302287720,"identity":"64b4e83e-18d8-4986-ad29-2f34b16c760a","order_by":3,"name":"Sean Davis","email":"","orcid":"","institution":"University of Colorado Anschutz School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Sean","middleName":"","lastName":"Davis","suffix":""},{"id":302287721,"identity":"c48955aa-9c93-4860-a926-546332efca46","order_by":4,"name":"Vince Carey","email":"","orcid":"","institution":"Harvard Medical School","correspondingAuthor":false,"prefix":"","firstName":"Vince","middleName":"","lastName":"Carey","suffix":""},{"id":302287722,"identity":"6f6d52f3-5e68-4bb9-962b-3a3a81f6ed9b","order_by":5,"name":"Martin Morgan","email":"","orcid":"","institution":"Roswell Park Comprehensive Cancer Center","correspondingAuthor":false,"prefix":"","firstName":"Martin","middleName":"","lastName":"Morgan","suffix":""},{"id":302287723,"identity":"58ef41f9-1423-4056-9d6d-2ebbc04bd4cd","order_by":6,"name":"Levi Waldron","email":"","orcid":"","institution":"City University of New York School of Public Health","correspondingAuthor":false,"prefix":"","firstName":"Levi","middleName":"","lastName":"Waldron","suffix":""}],"badges":[],"createdAt":"2024-05-05 03:09:20","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4370115/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4370115/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":56541286,"identity":"d77c5d8e-2724-4a56-9964-48b822833f7f","added_by":"auto","created_at":"2024-05-15 14:25:47","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":73557,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eOverview of \u003c/strong\u003e\u003cem\u003e\u003cstrong\u003eAnVILWorkflow\u003c/strong\u003e\u003c/em\u003e\u003cstrong\u003epackage.\u003c/strong\u003e AnVIL’s workflow description language (WDL) specified the runtime environment, which includes proper docker containers for existing analysis tools and computing resources. Cromwell, a scientific workflow execution and management system, runs WDL workflows on the cloud. \u003cem\u003eAnVILWorkflow\u003c/em\u003e users can easily run established workflows developed by experts and utilize the cloud resources without configuring or taking maintenance responsibilities.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-4370115/v1/7038043f0a4d3840380a5676.png"},{"id":56541326,"identity":"3993b248-84d7-467f-9db7-0e46d39e43db","added_by":"auto","created_at":"2024-05-15 14:25:52","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":432681,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4370115/v1/7c518d59-7ccf-42fc-a6e7-c0f1ff9cc56b.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"AnVILWorkflow: A runnable workflow package for Cloud-implemented bioinformatics analysis pipelines","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eThe NHGRI’s Genomic Data Science \u003cem\u003eAn\u003c/em\u003ealysis, \u003cem\u003eV\u003c/em\u003eisualization, and \u003cem\u003eI\u003c/em\u003enformatics \u003cem\u003eL\u003c/em\u003eab-space (AnVIL) consortium was launched in 2018, aiming to democratize genomics data[1]. AnVIL enables easy sharing of genomics data by organizing databases, bioinformatics pipelines for large-scale data processing, and interactive downstream analysis in one Cloud-based platform. AnVIL[2], also the name of the platform from the AnVIL project, implements the FAIR data-sharing philosophy and provides a graphical user interface (GUI, supported by Terra[3]), making it more accessible for researchers without programming backgrounds. However, a GUI tends to be less efficient and slower than a command line interface (CLI), especially for bulk analyses, still requires learning a new platform, and does not support version control and text-based workflows, often included as best practices for reproducible computational research[4]\u003c/p\u003e\n\u003cp\u003eBioconductor’s \u003cem\u003eAnVIL\u003c/em\u003e package is an AnVIL API wrapper that provides R-friendly, programming-based functionalities to leverage flexible and scalable cloud-based resources implemented in the AnVIL platform. With the \u003cem\u003eAnVIL\u003c/em\u003e package, users can easily access workflows, data, and Cloud-based computing resources managed by AnVIL. However, the AnVIL package is not customized for workflow execution tasks. Instead, \u003cem\u003eAnVIL\u003c/em\u003e covers all the resources related to the AnVIL platform, such as interaction with the repository for Docker-based genomic analysis tools and workflows (Dockstore[5]), leveraging cloud resources (Leonardo[6]), and data search and digestion (Gen3[7]). Many \u003cem\u003eAnVIL\u003c/em\u003e functions also expose API commands directly, requiring a deep understanding of the underlying \u003cem\u003eAnVIL\u003c/em\u003e workspace structures and data models to use for workflow execution. Also, it is a general package without individual support on any workspace and provides no metadata curation. Because the majority of Bioconductor users focus on data analysis, a convenient R-friendly way of accessing and utilizing AnVIL resources is needed. Here, we present the \u003cem\u003eAnVILWorkflow\u003c/em\u003e package to meet this need. \u003cem\u003eAnVILWorkflow\u003c/em\u003e package is a convenient, fit-for-purpose wrapper around the \u003cem\u003eAnVIL\u003c/em\u003e package with the following features optimized for workflow execution:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eIntuitive function names and returned values\u003c/li\u003e\n\u003cli\u003eSupport workflow-specific documentations \u003c/li\u003e\n\u003cli\u003eEnable to set up a Cloud environment with a single function call\u003c/li\u003e\n\u003cli\u003eReturn error messages that are easy to interpret and actionable \u003c/li\u003e\n\u003cli\u003eEssential metadata curation for more efficient data browsing\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eUsers can apply \u003cem\u003eAnVILWorkflow\u003c/em\u003e on any workspace they can access, including 347 public workspaces (snapshot on 8.28.23) available to anyone with an AnVIL account. We present the three use cases where we ran non-R-based bioinformatics analysis tools using conventional R syntax: \u003cem\u003eSalmon\u003c/em\u003e\u003cem\u003e[8]\u003c/em\u003e, \u003cem\u003ebioBakery\u003c/em\u003e\u003cem\u003e[9]\u003c/em\u003e, and \u003cem\u003ePathML\u003c/em\u003e\u003cem\u003e[10]\u003c/em\u003e. \u003cem\u003eSalmon\u003c/em\u003e is a widely used RNA sequencing analysis tool for quantifying the expression of transcripts and is based on the command-line interface. Its downstream analysis involves many R/Bioconductor packages, such as \u003cem\u003eDESeq2\u003c/em\u003e, \u003cem\u003eedgeR\u003c/em\u003e, and \u003cem\u003elimma\u003c/em\u003e. \u003cem\u003ebioBakery \u003c/em\u003eis a widely used whole metagenomic shotgun (WMS) sequencing data analysis environment, mainly relying on Python. \u003cem\u003ePathML \u003c/em\u003eis a general-purpose research toolkit for computational pathology, including many functionalities in digital pathology data analysis, such as strain normalization, nucleus segmentation, and tissue detection. \u003cem\u003ePathML\u003c/em\u003e takes raw image files and returns the processed image data in an hdf5 format for further downstream analysis, including machine learning methods. \u003c/p\u003e"},{"header":"2. Overview","content":"\u003cp\u003eAnVIL provides comprehensive resources for biomedical data analysis, including data (e.g., genomics), workflows for bulk analysis, and interactive analysis apps (i.e., Galaxy, Jupyter Notebooks, and RStudio) under the workspace. Among them, workflows are often a limiting factor in bioinformatics analysis due to computing demands and bioinformatics expertise required. Thus, the \u003cem\u003eAnVILWorkflow\u003c/em\u003e package makes the workflow-related resources from AnVIL more accessible and easier to use, especially for R users (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eWhile AnVIL manages workflow orchestration and workspace metadata and provides default setups simplifying decision-making for users, users still need to manage the storage of their data and cloud cost. Genomics data, especially their raw and intermediate forms, are very large, so data storage can be costly if the sample size increases. Storage costs incur and can be managed in two ways - storage itself and transfer. For example, using regional storage instead of multi-region, cleaning up intermediate results, and storing infrequently accessed data in low-cost storage (e.g., nearline or coldline storage from Google Cloud) can reduce per-sample costs. Analyzing data stored in one region using Virtual Machine (VM) compute resources in a different region incurs data transfer charges, so centralizing all storage and computing in a single region can be more cost-efficient by not only reducing the storage cost but also avoiding data transfer charges. Currently, the AnVIL workspaces use the us-central1 as a default region and any artifacts generated from the workflow execution, unless specified, are saved in the same-region bucket linked to the workspace. If users use the default region configured by AnVIL, bringing their data stored in the default region, us-central1, will save the data transfer charge. Additionally, open and controlled access genomic datasets hosted in AnVIL are stored in the us-multi-region, so there are no storage and transfer charges for users using the default workspace configuration. Downloading data to the user\u0026rsquo;s workstation or laptop is subject to charges, currently \u003cspan\u003e$\u003c/span\u003e0.08 to \u003cspan\u003e$\u003c/span\u003e0.12 per GB, depending on the amount of data[\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e] and geography of the transfer, and transfer from the US to another continent is more expensive than within the US transfer.\u003c/p\u003e \u003cp\u003eWhile browsing existing resources through \u003cem\u003eAnVILWorkflow\u003c/em\u003e is free, running workflows charge computing costs. \u003cem\u003eAnVILWorkflow\u003c/em\u003e is designed to use existing workflows which usually predefine computing resources optimized for the types of analyses, simplifying computing-related cost management. You can further reduce the run cost using call caching and preemptive instances. For example, if your workflow runs in fewer than 24 hours since a preemptible VM lasts 24 hours at most, you can save up to 80% by using preemptible VMs.\u003c/p\u003e \u003cp\u003eThe cost management for a group of users can be efficiently managed through the AnVIL billing project. One billing account can be shared with others by simply adding email addresses under the billing project. The billing project offers details on each workspace, including workspace owner and spent reports, so we can easily identify \u0026lsquo;who\u0026rsquo; uses \u0026lsquo;how much\u0026rsquo; for \u0026lsquo;what\u0026rsquo;. In addition to the workspace-level expense reports, users can further enhance cost monitoring by configuring spending reporting[\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. This allows users to closely monitor the expenditure associated with each workflow execution.\u003c/p\u003e"},{"header":"3. Major functions","content":"\u003cp\u003e \u003cb\u003eBrowse AnVIL resources.\u003c/b\u003e The \u003cem\u003eAnVILBrowse\u003c/em\u003e function allows users to browse AnVIL resources using keywords. This function runs instantaneously because the \u003cem\u003eAnVILWorkflow\u003c/em\u003e package includes the snapshot of metadata on all the publicly accessible AnVIL workspaces and their workflows and data. It performs basic metadata harmonization, allowing more efficient browsing and filtering, such as selecting workspaces based on the study size or participants\u0026rsquo; ages. Users can also browse non-public workspaces they have access to using the \u003cem\u003egetMetaTables\u003c/em\u003e function; however, this process can take a while depending on the number of workspaces a user has access to.\u003c/p\u003e \u003cp\u003e \u003cb\u003eRun AnVIL workflows.\u003c/b\u003e \u003cem\u003eAnVILWorkflow\u003c/em\u003e package provides all the functionalities required to run workflows available in AnVIL from the local R session - from the environment setup to the output download. One prerequisite is to create an AnVIL account from the AnVIL web portal. AnVIL account provides two required inputs to run workflows remotely: 1) the email address associated with the user\u0026rsquo;s account and 2) the billing project name to cover the computing cost.\u003c/p\u003e \u003cp\u003eAnVIL-hosted workflows can be run using four main functions: setCloudEnv, cloneWorkspace, runWorkflow, and getOutput. The setCloudEnv function accepts the AnVIL account email and billing project name and sets up your local R environment ready to access AnVIL and Cloud-computing resources. The cloneWorkspace function creates the user\u0026rsquo;s copy of a \u0026lsquo;template\u0026rsquo; workspace and the runWorkflow executes the workflow. The getOutput function can check the outputs from successfully executed workflows and download user-specified files to a local computer.\u003c/p\u003e \u003cp\u003eUser input can be provided through the updateInput function, which accepts two different forms of tables depending on the workflows - AnVIL\u0026rsquo;s data model or URLs pointing to data files stored in Google Cloud buckets. The input data formats are already specified in the workflow scripts (Workflow Description Language, WDL[\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]). Other accessory functions are available to monitor submission progress (monitorWorkflow), stop submitted workflow (stopWorkflow), and get Dashboard content (getDashboard).\u003c/p\u003e"},{"header":"4. Use cases","content":"\u003cp\u003eThe use cases demonstrated below include demo input data in the template workspaces, so the R scripts below can run the listed use cases from the local computer. Ready-to-run examples that can be used to test the process on the user\u0026rsquo;s own AnVIL account are available in the \u003cem\u003eAnVILWorkflow\u003c/em\u003e package vignette. GATK best practice pipelines[\u003cspan class=\"CitationRef\"\u003e14\u003c/span\u003e] are not demonstrated here, but they are also available as AnVIL workspaces.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e## Setup the account\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003esetCloudEnv\u003c/strong\u003e(accountEmail = {AnVIL account email},\u003c/p\u003e\n\u003cdiv class=\"BlockQuote\"\u003e\n \u003cp\u003ebillingProjectName = {AnVIL billing project name})\u003c/p\u003e\n\u003c/div\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e## Clone the workspace of your interest\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003enewName \u0026lt;- {Unique name for your copy of workspace}\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ecloneWorkspace\u003c/strong\u003e(workspaceName\u0026thinsp;=\u0026thinsp;newName, templateName\u0026thinsp;=\u0026thinsp;templateName)\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e## Run workflow\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003erunWorkflow\u003c/strong\u003e(workspaceName\u0026thinsp;=\u0026thinsp;newName,\u003c/p\u003e\n\u003cdiv class=\"BlockQuote\"\u003e\n \u003cp\u003eworkflowName = {name of the workflow if there is more than one in the workspace of your interest})\u003c/p\u003e\n\u003c/div\u003e\n\u003cp\u003e\u003cem\u003e## Get workflow outputs\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003egetOutput\u003c/strong\u003e(workspaceName\u0026thinsp;=\u0026thinsp;newName)\u003c/p\u003e\n\u003cp\u003eThe main features of the demo workspaces and their workflow-specific input data preparation process are described below.\u003c/p\u003e\n\u003cdiv id=\"Sec5\" class=\"Section2\"\u003e\n \u003ch2\u003e4.1. Bulk RNA sequencing data analysis\u003c/h2\u003e\n \u003cp\u003e\u003cem\u003eSalmon\u003c/em\u003e workflow uses AnVIL\u0026rsquo;s data model and requires four essential inputs - fastq1, fastq2, fasta, and \u003cem\u003etranscriptome index name\u003c/em\u003e. This workflow can be easily applied to the consortium data hosted in AnVIL, which follows AnVIL\u0026rsquo;s data model. With the default runtime environment configured for this workflow (1 CPU, 2GB memory, and 10GB SSD disk), processing 16 demo samples (32 fastq files, ~1GB per file) took about 30 minutes and cost \u003cspan\u003e$\u003c/span\u003e0.12.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec6\" class=\"Section2\"\u003e\n \u003ch2\u003e4.2. Whole metagenomic shotgun data analysis\u003c/h2\u003e\n \u003cp\u003e\u003cem\u003ebioBakery\u003c/em\u003e is a metagenome analysis environment composed of Python-based tools, reference databases, and command-line-based workflows. It processes raw shotgun sequencing data into microbial community feature profiles, summary reports, and figures[\u003cspan class=\"CitationRef\"\u003e9\u003c/span\u003e]. \u003cem\u003ebioBakery\u003c/em\u003e\u0026rsquo;s whole metagenome shotgun (wmgx) and visualization (wmgx_vis) workflows are implemented as an AnVIL workspace. The current version of the \u003cem\u003eAnVILWorkflow\u003c/em\u003e supports \u003cem\u003ebioBakery\u003c/em\u003e version 3[\u003cspan class=\"CitationRef\"\u003e15\u003c/span\u003e]. While users can customize this workflow to a great degree, only six inputs are sufficient to run a standard, optimized version of this workflow. Those six inputs are:\u003c/p\u003e\n \u003cp\u003e- Name of the Trimmomatic adaptor type (for demo data, \u003cem\u003eNexteraPE\u003c/em\u003e)\u003c/p\u003e\n \u003cp\u003e- Your project name\u003c/p\u003e\n \u003cp\u003e- Extension of input files (for demo data, .fastq.gz)\u003c/p\u003e\n \u003cp\u003e- A table of your sequencing file (fastq) names stored in the Google Cloud Storage bucket\u003c/p\u003e\n \u003cp\u003e- Input file identifier for paired-end sequencing (for demo data, _R1 and _R2)\u003c/p\u003e\n \u003cp\u003eThe seven required databases are already linked to this workflow and nine additional optional inputs are available for further customization. Optional inputs are for workflow customization, such as bypassing functional profiling (default is false) and maximum memory usage for different tasks (default is 32GB for functional profiling by \u003cem\u003eHUMAnN\u003c/em\u003e, 8GB for quality control by \u003cem\u003eKneaddata\u003c/em\u003e, and 24GB for taxonomic profiling by \u003cem\u003eMetaPhlAn\u003c/em\u003e). This workflow uses call caching and preemptive instances by default for cost efficiency. Processing six paired-end demo samples (mean file size\u0026thinsp;~\u0026thinsp;380MB) with the optimized default setting without using preemptive instances took about 5 hours and cost around \u003cspan\u003e$\u003c/span\u003e6.50. With the preemptive instances, it can take longer but cost less. Compared to the existing options such as Nephele[\u003cspan class=\"CitationRef\"\u003e16\u003c/span\u003e], \u003cem\u003eAnVILWorkflow\u003c/em\u003e allows a programmatic approach and more flexible customization options.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec7\" class=\"Section2\"\u003e\n \u003ch2\u003e3.3. Histopathology image processing using PathML\u003c/h2\u003e\n \u003cp\u003eWe implemented the hematoxylin-eosin (HE) stain normalization process of \u003cem\u003ePathML\u003c/em\u003e as an AnVIL workspace. This workflow accepts an SVS file as input and returns original and normalized images as PNG files. There are two required inputs - Google Cloud Storage URI where the input SVS image file is stored and the sample name. Processing one publicly available image (CMU-1_Small_Region.svs, 1.8MB)[\u003cspan class=\"CitationRef\"\u003e17\u003c/span\u003e] with the default runtime (4 CPU, 16GB memory) took about 8 minutes and cost \u003cspan\u003e$\u003c/span\u003e0.01. This simple but robust analysis setup can support clinical use cases, such as pathologists who process a large number of images in a short time, by offering guidance and cross-validation options.\u003c/p\u003e\n\u003c/div\u003e"},{"header":"5. Conclusions","content":"\u003cp\u003eThe \u003cem\u003eAnVILWorkflow\u003c/em\u003e package enables users to conduct complex and computationally intense analyses with minimal bioinformatics expertise, through well-established workflows within AnVIL and versatile cloud resources directly from standard laptops using the familiar R syntax. The major advantages \u003cem\u003eAnVILWorkflow\u003c/em\u003e provides over the existing approaches include 1) a minimal entry barrier, negating the need for software installations, preparation of properly versioned reference data, or construction and oversight of workflows, 2) leveraging flexible cloud computing resources without the need to learn or handle them directly, 3) user-friendly functions that provide enhanced information, and 4) greatly improved reproducibility and interoperability by seamlessly linking multiple analysis steps, conducted in both R and non-R based tools, within a single R vignette. However, there are still some limitations. For instance, certain customizations of the workflows are limited or require a more profound understanding of the workflows. Despite not being inherently more costly than an in-house server, the pay-per-use structure requires careful planning and management. The absence of an integrated versioning system in AnVIL workspaces requires users to manually monitor new versions. In conclusion, \u003cem\u003eAnVILWorkflow\u003c/em\u003e proves most advantages for analyzing a bulk of samples on relatively simple workflows (i.e., single-stage workflow procedure) or for exploratory data analysis for non-technical users, particularly when employing well-established analysis workflows.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cp\u003e\u003cstrong\u003e\u003cem\u003eAnVIL:\u003c/em\u003e\u003c/strong\u003e Analysis, Visualization, and Informatics Lab-space\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eGUI:\u003c/em\u003e\u003c/strong\u003e Graphical User Interface\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eCLI:\u003c/em\u003e\u003c/strong\u003e Command Line Interface\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eWMS:\u003c/em\u003e\u003c/strong\u003e whole metagenomic shotgun\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eVM:\u003c/em\u003e\u003c/strong\u003e Virtual Machine\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eWDL:\u003c/em\u003e\u003c/strong\u003e Workflow Description Language\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eHE:\u003c/em\u003e\u003c/strong\u003e hematoxylin-eosin\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eHUMAnN:\u003c/em\u003e\u003c/strong\u003e HMP Unified Metabolic Analysis Network\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch3\u003eEthics approval and consent to participate\u003c/h3\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e\n\u003ch3\u003eConsent for publication\u003c/h3\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e\n\u003ch3\u003eAvailability of data and materials\u003c/h3\u003e\n\u003cp\u003eAll data generated or analyzed during this study are included in this published article and available on the AnVIL platform (https://anvil.terra.bio/#). AnVILWorkflow package is distributed through the Bioconductor project (https://bioconductor.org/packages/AnVILWorkflow), and the source code is available through GitHub (https://github.com/shbrief/AnVILWorkflow).\u003c/p\u003e\n\u003ch3\u003eCompeting interests\u003c/h3\u003e\n\u003cp\u003eThe authors declare that they have no competing interests.\u003c/p\u003e\n\u003ch3\u003eFunding \u003c/h3\u003e\n\u003cp\u003eSO, KG, MR, VC, MM, and LW were supported by the National Human Genome Research Institute (U24HG010263).\u003c/p\u003e\n\u003ch3\u003eAuthors’ contributions \u003c/h3\u003e\n\u003cp\u003eSO, LW, and VC conceived the research idea. SO, KG, MR, and MM developed the software. SO and KG performed the benchmarking analyses. SO, LW, and KG wrote the manuscript. LW, SD, and MR reviewed the manuscript. \u003c/p\u003e\n\u003ch3\u003eAcknowledgments\u003c/h3\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n \u003cli\u003eSchatz MC, Philippakis AA, Afgan E, Banks E, Carey VJ, Carroll RJ, et al. Inverting the model of genomics data sharing with the NHGRI Genomic Data Science Analysis, Visualization, and Informatics Lab-space. Cell Genom [Internet]. 2022;2. Available from: http://dx.doi.org/10.1016/j.xgen.2021.100085\u003c/li\u003e\n \u003cli\u003eTerra [Internet]. [cited 2024 Jan 31]. Available from: https://anvil.terra.bio/\u003c/li\u003e\n \u003cli\u003eTerra [Internet]. [cited 2024 Jan 31]. Available from: https://app.terra.bio/\u003c/li\u003e\n \u003cli\u003eSandve GK, Nekrutenko A, Taylor J, Hovig E. Ten simple rules for reproducible computational research. PLoS Comput Biol. 2013;9:e1003285.\u003c/li\u003e\n \u003cli\u003eYuen D, Cabansay L, Duncan A, Luu G, Hogue G, Overbeck C, et al. The Dockstore: enhancing a community platform for sharing reproducible and accessible computational protocols. Nucleic Acids Res. 2021;49:W624\u0026ndash;32.\u003c/li\u003e\n \u003cli\u003eleonardo: Notebook service [Internet]. Github; [cited 2024 Mar 6]. Available from: https://github.com/DataBiosphere/leonardo\u003c/li\u003e\n \u003cli\u003eHughes L, Grossman RL, Flamig Z, Prokhorenkov A, Lukowski M, Fitzsimons M, et al. Harmonization of clinical data across Gen3 data commons. J Clin Orthod. 2019;37:e18094\u0026ndash;e18094.\u003c/li\u003e\n \u003cli\u003ePatro R, Duggal G, Love MI, Irizarry RA, Kingsford C. Salmon provides fast and bias-aware quantification of transcript expression. Nat Methods. 2017;14:417\u0026ndash;9.\u003c/li\u003e\n \u003cli\u003eMcIver LJ, Abu-Ali G, Franzosa EA, Schwager R, Morgan XC, Waldron L, et al. bioBakery: a meta\u0026rsquo;omic analysis environment. Bioinformatics. 2018;34:1235\u0026ndash;7.\u003c/li\u003e\n \u003cli\u003eRosenthal J, Carelli R, Omar M, Brundage D, Halbert E, Nyman J, et al. Building Tools for Machine Learning and Artificial Intelligence in Cancer Research: Best Practices and a Case Study with the PathML Toolkit for Computational Pathology. Mol Cancer Res. 2022;20:202\u0026ndash;6.\u003c/li\u003e\n \u003cli\u003ePricing [Internet]. Google Cloud. [cited 2024 Feb 26]. Available from: https://cloud.google.com/storage/pricing\u003c/li\u003e\n \u003cli\u003eHow much did my workflow cost? [Internet]. Terra Support. [cited 2024 Feb 20]. Available from: https://support.terra.bio/hc/en-us/articles/360037862771-How-much-did-my-workflow-cost\u003c/li\u003e\n \u003cli\u003eVoss K, Gentry J, Van der Auwera G. Full-stack genomics pipelining with GATK4 + WDL + Cromwell [Internet]. F1000Research; 2017. Available from: https://f1000research.com/posters/6-1379\u003c/li\u003e\n \u003cli\u003eVan der Auwera GA, O\u0026rsquo;Connor BD. Genomics in the Cloud: Using Docker, GATK, and WDL in Terra. \u0026ldquo;O\u0026rsquo;Reilly Media, Inc.\u0026rdquo;; 2020.\u003c/li\u003e\n \u003cli\u003eBeghini F, McIver LJ, Blanco-M\u0026iacute;guez A, Dubois L, Asnicar F, Maharjan S, et al. Integrating taxonomic, functional, and strain-level profiling of diverse microbial communities with bioBakery 3. Elife [Internet]. 2021;10. Available from: http://dx.doi.org/10.7554/eLife.65088\u003c/li\u003e\n \u003cli\u003eWeber N, Liou D, Dommer J, MacMenamin P, Qui\u0026ntilde;ones M, Misner I, et al. Nephele: a cloud platform for simplified, standardized and reproducible microbiome data analysis. Bioinformatics. 2018;34:1411\u0026ndash;3.\u003c/li\u003e\n \u003cli\u003eAperio SVS [Internet]. [cited 2024 Feb 21]. Available from: https://openslide.cs.cmu.edu/download/openslide-testdata/Aperio/\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-4370115/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4370115/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eAdvancements in sequencing technologies and the development of new data collection methods produce large volumes of biological data. The Genomic Data Science Analysis, Visualization, and Informatics Lab-space (AnVIL) provides a cloud-based platform for democratizing access to large-scale genomics data and analysis tools. However, utilizing the full capabilities of AnVIL can be challenging for researchers without extensive bioinformatics expertise, especially for executing complex workflows. Here we present the AnVILWorkflow R package, which enables the convenient execution of bioinformatics workflows hosted on AnVIL directly from an R environment. AnVILWorkflowsimplifies the setup of the cloud computing environment, input data formatting, workflow submission, and retrieval of results through intuitive functions. We demonstrate the utility of AnVILWorkflowfor three use cases: bulk RNA-seq analysis with Salmon, metagenomics analysis with bioBakery, and digital pathology image processing with PathML. The key features of AnVILWorkflow include user-friendly browsing of available data and workflows, seamless integration of R and non-R tools within a reproducible analysis pipeline, and accessibility to scalable computing resources without direct management overhead. While some limitations exist around workflow customization, AnVILWorkflowlowers the barrier to taking advantage of AnVIL's resources, especially for exploratory analyses or bulk processing with established workflows. This empowers a broader community of researchers to leverage the latest genomics tools and datasets using familiar R syntax. This package is distributed through the Bioconductor project (https://bioconductor.org/packages/AnVILWorkflow), and the source code is available through GitHub (https://github.com/shbrief/AnVILWorkflow).\u003c/p\u003e","manuscriptTitle":"AnVILWorkflow: A runnable workflow package for Cloud-implemented bioinformatics analysis pipelines","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-05-15 14:25:41","doi":"10.21203/rs.3.rs-4370115/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"3d6db8fe-5b7b-4bba-aa2e-b1edf47a7b48","owner":[],"postedDate":"May 15th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-05-15T14:25:42+00:00","versionOfRecord":[],"versionCreatedAt":"2024-05-15 14:25:41","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4370115","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4370115","identity":"rs-4370115","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}