Toward Best Practice in Identifying Subtle Differential Expression with RNA-seq: A Real-World Multi-Center Benchmarking Study Using Quartet and MAQC Reference Materials

Toward Best Practice in Identifying Subtle Differential Expression with RNA-seq: A Real-World Multi-Center Benchmarking Study Using Quartet and MAQC Reference Materials | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Toward Best Practice in Identifying Subtle Differential Expression with RNA-seq: A Real-World Multi-Center Benchmarking Study Using Quartet and MAQC Reference Materials Rui Zhang, Duo Wang, Yaqing Liu, Yuanfeng Zhang, Qingwang Chen, and 10 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3718519/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 22 Jul, 2024 Read the published version in Nature Communications → Version 1 posted You are reading this latest preprint version Abstract Translating RNA-seq into clinical diagnostics requires ensuring the reliability of detecting clinically relevant subtle differential expressions, such as those between different disease subtypes or stages. Moreover, cross-laboratory reproducibility and consistency under diverse experimental and bioinformatics workflows urgently need to be addressed. As part of the Quartet project, we presented a comprehensive RNA-seq benchmarking study utilizing Quartet and MAQC RNA reference samples spiked with ERCC controls in 45 independent laboratories, each employing their in-house RNA-seq workflows. We assessed the data quality, accuracy and reproducibility of gene expression and differential gene expression and compared over 40 experimental processes and 140 combined differential analysis pipelines based on multiple 'ground truths'. Here we show that real-world RNA-seq exhibited greater inter-laboratory variations when detecting subtle differential expressions between Quartet samples. Experimental factors including mRNA enrichment methods and strandedness, and each bioinformatics step, particularly normalization, emerged as primary sources of variations in gene expression and have a more pronounced impact on the subtle differential expression measurement. We underscored the pivotal role of experimental execution over the choice of experimental protocols, the importance of strategies for filtering low-expression genes, and optimal gene annotation and analysis tools. In summary, this study provided best practice recommendations for the development, optimization, and quality control of RNA-seq for clinical diagnostic purposes. Biological sciences/Biotechnology/Sequencing/RNA sequencing Biological sciences/Computational biology and bioinformatics/Quality control RNA sequencing Subtle differential expression Quartet MAQC ERCC Performance assessment Real world Quality control Full Text Additional Declarations There is NO Competing Interest. Supplementary Files SupplementaryTable1.xlsx SupplementaryTable2.xlsx SupplementaryTable3.xlsx SupplementaryTable4.xlsx SupplementaryTable5.xlsx SupplementaryTable6.xlsx Supplementarymaterial.pdf Supplementary Materials Cite Share Download PDF Status: Published Journal Publication published 22 Jul, 2024 Read the published version in Nature Communications → Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3718519","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":258905137,"identity":"69f7ee07-595c-4698-87b3-82964c0dca14","order_by":0,"name":"Rui Zhang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA2UlEQVRIiWNgGAWjYFACxgZmEMUPIh6QpEWyAUgkEGsPWIvBAWK1mM9Ibv5cUHPHbvPt5ocfEhhs8uUdmJ/hdaDMjcQG4xnHniVvu3PMWCKBIc1y4wE2cwN8WiQkEhuSedgOJ5vdyGEDOuywgWEDD5sEIS2Hef4dTjaeQYKWxmbetsN2BhJQLfIMhLTwPGxm5u07nCBxIw3oF4M0AwNmNjP8WtjTH3/m+XbYnn9G8sMPHypsDOTbm5/h1QIDiQ1gChhUBoeJUQ8E9nCWfAORWkbBKBgFo2DEAABefUItm98oWAAAAABJRU5ErkJggg==","orcid":"https://orcid.org/0000-0003-4660-2042","institution":"National Center for Clinical Laboratories, Institute of Geriatric Medicine, Chinese Academy of Medical Sciences, Beijing Hospital, Beijing, China.","correspondingAuthor":true,"prefix":"","firstName":"Rui","middleName":"","lastName":"Zhang","suffix":""},{"id":258905138,"identity":"8779f77c-b1c0-41d7-b650-3a44f805b8be","order_by":1,"name":"Duo Wang","email":"","orcid":"","institution":"National Center for Clinical Laboratories, Institute of Geriatric Medicine, Chinese Academy of Medical Sciences, Beijing Hospital, Beijing, China.","correspondingAuthor":false,"prefix":"","firstName":"Duo","middleName":"","lastName":"Wang","suffix":""},{"id":258905139,"identity":"9c2a63c8-4fc9-4697-95ce-88ca9453dd9b","order_by":2,"name":"Yaqing Liu","email":"","orcid":"https://orcid.org/0000-0001-9356-9164","institution":"State Key Laboratory of Genetic Engineering, School of Life Sciences and Human Phenome Institute, Fudan University, Shanghai, China.","correspondingAuthor":false,"prefix":"","firstName":"Yaqing","middleName":"","lastName":"Liu","suffix":""},{"id":258905140,"identity":"917b287e-269b-428c-a3cc-01f6cb73a6ad","order_by":3,"name":"Yuanfeng Zhang","email":"","orcid":"","institution":"National Center for Clinical Laboratories, Institute of Geriatric Medicine, Chinese Academy of Medical Sciences, Beijing Hospital, Beijing, China.","correspondingAuthor":false,"prefix":"","firstName":"Yuanfeng","middleName":"","lastName":"Zhang","suffix":""},{"id":258905141,"identity":"31b4098b-338f-4a38-860f-3a36b30cf651","order_by":4,"name":"Qingwang Chen","email":"","orcid":"","institution":"Fudan University","correspondingAuthor":false,"prefix":"","firstName":"Qingwang","middleName":"","lastName":"Chen","suffix":""},{"id":258905142,"identity":"5ed966d7-ba48-4921-a00b-9098ed15994e","order_by":5,"name":"Yanxi Han","email":"","orcid":"","institution":"National Center for Clinical Laboratories, Institute of Geriatric Medicine, Chinese Academy of Medical Sciences, Beijing Hospital, Beijing, China.","correspondingAuthor":false,"prefix":"","firstName":"Yanxi","middleName":"","lastName":"Han","suffix":""},{"id":258905143,"identity":"6654bd3c-7563-4fca-a30e-db92cbbdc096","order_by":6,"name":"Wanwan Hou","email":"","orcid":"","institution":"Center for Pharmacogenomics, School of Life Sciences and Shanghai Cancer Center, Fudan University, Shanghai, China, 200438","correspondingAuthor":false,"prefix":"","firstName":"Wanwan","middleName":"","lastName":"Hou","suffix":""},{"id":258905144,"identity":"611b18e1-05c7-4eeb-aecb-ba70131ce21a","order_by":7,"name":"Cong Liu","email":"","orcid":"","institution":"National Center for Clinical Laboratories, Institute of Geriatric Medicine, Chinese Academy of Medical Sciences, Beijing Hospital, Beijing, China.","correspondingAuthor":false,"prefix":"","firstName":"Cong","middleName":"","lastName":"Liu","suffix":""},{"id":258905145,"identity":"b256465c-98ca-4967-a687-45176aea74f4","order_by":8,"name":"Ying Yu","email":"","orcid":"https://orcid.org/0000-0002-4084-908X","institution":"Fudan University","correspondingAuthor":false,"prefix":"","firstName":"Ying","middleName":"","lastName":"Yu","suffix":""},{"id":258905146,"identity":"9aab278c-6930-407d-b5ca-3726b197a24a","order_by":9,"name":"Ziyang Li","email":"","orcid":"","institution":"Department of Laboratory Medicine, The Second Xiangya Hospital, Central South University, Changsha, Hunan, P.R. China","correspondingAuthor":false,"prefix":"","firstName":"Ziyang","middleName":"","lastName":"Li","suffix":""},{"id":258905147,"identity":"cccbcba3-092b-4c0f-bec4-6c0064198979","order_by":10,"name":"Ziqiang Li","email":"","orcid":"","institution":"National Center for Clinical Laboratories, Institute of Geriatric Medicine, Chinese Academy of Medical Sciences, Beijing Hospital, Beijing, China.","correspondingAuthor":false,"prefix":"","firstName":"Ziqiang","middleName":"","lastName":"Li","suffix":""},{"id":258905148,"identity":"d4a4f161-54f4-4412-9e30-9df9e0a3fd49","order_by":11,"name":"Jiaxin Zhao","email":"","orcid":"","institution":"National Center for Clinical Laboratories, Institute of Geriatric Medicine, Chinese Academy of Medical Sciences, Beijing Hospital, Beijing, China.","correspondingAuthor":false,"prefix":"","firstName":"Jiaxin","middleName":"","lastName":"Zhao","suffix":""},{"id":258905149,"identity":"e5a35132-c82f-4c86-af3d-ad9fd29b3735","order_by":12,"name":"Yuanting Zheng","email":"","orcid":"https://orcid.org/0000-0003-4480-8303","institution":"State Key Laboratory of Genetic Engineering, School of Life Sciences and Human Phenome Institute, Shanghai Cancer Center, Fudan University, Shanghai, China.","correspondingAuthor":false,"prefix":"","firstName":"Yuanting","middleName":"","lastName":"Zheng","suffix":""},{"id":258905150,"identity":"53822536-1520-45ef-8b68-95d312b30987","order_by":13,"name":"Leming Shi","email":"","orcid":"https://orcid.org/0000-0002-2981-4150","institution":"State Key Laboratory of Genetic Engineering, Human Phenome Institute, School of Life Sciences and Shanghai Cancer Center, Fudan University, Shanghai 200438, China","correspondingAuthor":false,"prefix":"","firstName":"Leming","middleName":"","lastName":"Shi","suffix":""},{"id":258905151,"identity":"63eb67d8-dc7e-4098-ae15-6639274cb66a","order_by":14,"name":"Jinming Li","email":"","orcid":"","institution":"National Center for Clinical Laboratories, Institute of Geriatric Medicine, Chinese Academy of Medical Sciences, Beijing Hospital, Beijing, China.","correspondingAuthor":false,"prefix":"","firstName":"Jinming","middleName":"","lastName":"Li","suffix":""}],"badges":[],"createdAt":"2023-12-07 05:55:46","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3718519/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3718519/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41467-024-50420-y","type":"published","date":"2024-07-22T04:00:00+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":60880965,"identity":"c56fc7a3-2eac-4cd5-8ad9-8515eb76a13f","added_by":"auto","created_at":"2024-07-23 07:07:13","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1827266,"visible":true,"origin":"","legend":"","description":"","filename":"Manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3718519/v1_covered_009b9351-74fb-45ac-956a-e719060f8054.pdf"},{"id":49359134,"identity":"cf92dbce-e824-4994-b4d6-d3ef736ba3c9","added_by":"auto","created_at":"2024-01-09 09:02:20","extension":"xlsx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":18724,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryTable1.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-3718519/v1/c57047ca4a4de732a8c7bc13.xlsx"},{"id":49359443,"identity":"729b4f12-e4dc-431a-bc58-9a70f17de45d","added_by":"auto","created_at":"2024-01-09 09:10:20","extension":"xlsx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":135030,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryTable2.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-3718519/v1/2011dd1e7a25f79557c0d072.xlsx"},{"id":49359131,"identity":"753d07de-7f6e-48fd-aa0d-f7ee5ac01bee","added_by":"auto","created_at":"2024-01-09 09:02:20","extension":"xlsx","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":12597,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryTable3.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-3718519/v1/4371a95ba38d6e42ab1815a2.xlsx"},{"id":49359136,"identity":"ae15abc3-d8fe-4bd2-b86f-eca9d78993df","added_by":"auto","created_at":"2024-01-09 09:02:20","extension":"xlsx","order_by":4,"title":"","display":"","copyAsset":false,"role":"supplement","size":147650,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryTable4.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-3718519/v1/e901d09274e1decf55e402dd.xlsx"},{"id":49359133,"identity":"5c039e92-ca2c-46f8-84fc-c0280f799c08","added_by":"auto","created_at":"2024-01-09 09:02:20","extension":"xlsx","order_by":5,"title":"","display":"","copyAsset":false,"role":"supplement","size":18381,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryTable5.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-3718519/v1/766fd6451aaa29af9e222080.xlsx"},{"id":49359132,"identity":"5ab3d2af-069a-4cd0-83ab-aeaafcb77bdb","added_by":"auto","created_at":"2024-01-09 09:02:20","extension":"xlsx","order_by":6,"title":"","display":"","copyAsset":false,"role":"supplement","size":11459,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryTable6.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-3718519/v1/cea1137405e01fb3bb72452b.xlsx"},{"id":49359137,"identity":"8ab4e936-a17b-4987-95b1-fcbdaaf25db4","added_by":"auto","created_at":"2024-01-09 09:02:20","extension":"pdf","order_by":7,"title":"","display":"","copyAsset":false,"role":"supplement","size":8890991,"visible":true,"origin":"","legend":"\u003cp\u003eSupplementary Materials\u003c/p\u003e","description":"","filename":"Supplementarymaterial.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3718519/v1/09daf6108be6770d44798d10.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"Toward Best Practice in Identifying Subtle Differential Expression with RNA-seq: A Real-World Multi-Center Benchmarking Study Using Quartet and MAQC Reference Materials","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"nature-portfolio","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"","title":"Nature Portfolio","twitterHandle":"","acdcEnabled":false,"dfaEnabled":false,"editorialSystem":"ejp","reportingPortfolio":"","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"RNA sequencing, Subtle differential expression, Quartet, MAQC, ERCC, Performance assessment, Real world, Quality control","lastPublishedDoi":"10.21203/rs.3.rs-3718519/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3718519/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Translating RNA-seq into clinical diagnostics requires ensuring the reliability of detecting clinically relevant subtle differential expressions, such as those between different disease subtypes or stages. Moreover, cross-laboratory reproducibility and consistency under diverse experimental and bioinformatics workflows urgently need to be addressed. As part of the Quartet project, we presented a comprehensive RNA-seq benchmarking study utilizing Quartet and MAQC RNA reference samples spiked with ERCC controls in 45 independent laboratories, each employing their in-house RNA-seq workflows. We assessed the data quality, accuracy and reproducibility of gene expression and differential gene expression and compared over 40 experimental processes and 140 combined differential analysis pipelines based on multiple 'ground truths'. Here we show that real-world RNA-seq exhibited greater inter-laboratory variations when detecting subtle differential expressions between Quartet samples. Experimental factors including mRNA enrichment methods and strandedness, and each bioinformatics step, particularly normalization, emerged as primary sources of variations in gene expression and have a more pronounced impact on the subtle differential expression measurement. We underscored the pivotal role of experimental execution over the choice of experimental protocols, the importance of strategies for filtering low-expression genes, and optimal gene annotation and analysis tools. In summary, this study provided best practice recommendations for the development, optimization, and quality control of RNA-seq for clinical diagnostic purposes.","manuscriptTitle":"Toward Best Practice in Identifying Subtle Differential Expression with RNA-seq: A Real-World Multi-Center Benchmarking Study Using Quartet and MAQC Reference Materials","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-01-09 09:02:15","doi":"10.21203/rs.3.rs-3718519/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"nature-communications","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"NCOMMS","sideBox":"Learn more about [Nature Communications](http://www.nature.com/ncomms/)","snPcode":"","submissionUrl":"https://mts-ncomms.nature.com/","title":"Nature Communications","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Nature Communications","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"db1e9551-f504-427e-902c-327394436fe5","owner":[],"postedDate":"January 9th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":27336727,"name":"Biological sciences/Biotechnology/Sequencing/RNA sequencing"},{"id":27336728,"name":"Biological sciences/Computational biology and bioinformatics/Quality control"}],"tags":[],"updatedAt":"2024-07-23T07:07:01+00:00","versionOfRecord":{"articleIdentity":"rs-3718519","link":"https://doi.org/10.1038/s41467-024-50420-y","journal":{"identity":"nature-communications","isVorOnly":false,"title":"Nature Communications"},"publishedOn":"2024-07-22 04:00:00","publishedOnDateReadable":"July 22nd, 2024"},"versionCreatedAt":"2024-01-09 09:02:15","video":"","vorDoi":"10.1038/s41467-024-50420-y","vorDoiUrl":"https://doi.org/10.1038/s41467-024-50420-y","workflowStages":[]},"version":"v1","identity":"rs-3718519","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3718519","identity":"rs-3718519","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}