Advancing Climate Science Data Efficiency by Dual-stage Extreme Compression with an Efficient VAEformer

Advancing Climate Science Data Efficiency by Dual-stage Extreme Compression with an Efficient VAEformer | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Advancing Climate Science Data Efficiency by Dual-stage Extreme Compression with an Efficient VAEformer Tao Han, Zhenghao Chen, Song Guo, Wanghan Xu, Wanli Ouyang, Lei Bai This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6452190/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 24 Nov, 2025 Read the published version in Communications Earth & Environment → Version 1 posted You are reading this latest preprint version Abstract The growing threat of climate change highlights the urgent need for deeper understanding and precise weather prediction. Despite advancements in atmospheric research, the ever-growing volume of weather data challenges efficient storage and broadcasting, slowing efforts to mitigate climate change. To this end, we present Aeolus, a pioneering deep learning-based framework for efficient atmospheric data compression. Aeolus achieves a remarkable 470 times compression ratio that reduces the 400 TB ERA5 dataset to just 0.85 TB, significantly lowering storage requirements and improving transmission efficiency. This outperforms classical methods such as JPEG2000, which achieves compression ratios typically below 10 times on atmospheric datasets. Furthermore, Aeolus ensures practical compression and decompression speeds exceeding 1 GB/s, providing a substantial computational cost benefit compared to traditional techniques. Extensive comparative experiments validate the high numerical accuracy of the compressed data, with a mean absolute error of 0.17°K for temperature, consistent climatology, and comparable power spectral density. Such a level of precision enables efficient data-driven climate research, including accurate weather forecasting and extreme event reconstructions. Overall, our findings highlight the significant transformative impact of Aeolus on climate science, expanding access to large-scale atmospheric data across disciplines and empowering researchers with the data-driven insights needed to combat climate change. Earth and environmental sciences/Climate sciences Earth and environmental sciences/Climate sciences/Atmospheric science Earth and environmental sciences/Climate sciences/Climate change/Climate and Earth system modelling Climate Data Compression Data-Effciency Meteorological Science Deep Learning Weather Forecasting Extreme Weather Analysis Artificial Intelligence Full Text Additional Declarations There is NO Competing Interest. Cite Share Download PDF Status: Published Journal Publication published 24 Nov, 2025 Read the published version in Communications Earth & Environment → Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6452190","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":495345085,"identity":"178ef54a-e9ee-45cd-82b8-1ecc7034172f","order_by":0,"name":"Tao Han","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA70lEQVRIiWNgGAWjYBACAwglwcPPzNj4AMji4SNCC2MDA4OFnGR782GQfh42IrVUGBucOZYmARIhqMWc/ezxBz8qJBIbbuSYVX7NsZNhY2B++OgGHi2WPXmJjT1nJBIbZ+SY3Zbdlgx0GJuxcQ4+hx3IMWzgbZNIbJYAapHcxgzUwsMmjVfL+TeGjX+BWtqAWoolt9UToeVGjmEz0BZjHp5jaYwftx0mRssbw9kyZyTkJNibD0szbjvOw8ZMyC/ncww+vqmo47E/zNj48ee2ant+9uaHj/FpQQHMPGCSWOUgwPiDFNWjYBSMglEwYgAAi7pIBTCdtF8AAAAASUVORK5CYII=","orcid":"https://orcid.org/0000-0003-1255-3270","institution":"The Hong Kong University of Science and Technology","correspondingAuthor":true,"prefix":"","firstName":"Tao","middleName":"","lastName":"Han","suffix":""},{"id":495345086,"identity":"73ba835d-e36e-49f3-929f-b026cadbfbd6","order_by":1,"name":"Zhenghao Chen","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Zhenghao","middleName":"","lastName":"Chen","suffix":""},{"id":495345087,"identity":"6a14dca8-62a8-463e-94a8-20cdf7b0d3ee","order_by":2,"name":"Song Guo","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Song","middleName":"","lastName":"Guo","suffix":""},{"id":495345088,"identity":"fe24a262-d449-4b15-ad6b-23bf27f6b55c","order_by":3,"name":"Wanghan Xu","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Wanghan","middleName":"","lastName":"Xu","suffix":""},{"id":495345089,"identity":"322aea32-b1ce-4ef1-bb62-0495bc846048","order_by":4,"name":"Wanli Ouyang","email":"","orcid":"","institution":"Shanghai Artificial Intelligence Laboratory","correspondingAuthor":false,"prefix":"","firstName":"Wanli","middleName":"","lastName":"Ouyang","suffix":""},{"id":495345090,"identity":"98d0ffad-eedd-4a6c-ae2b-f04b23697f5c","order_by":5,"name":"Lei Bai","email":"","orcid":"https://orcid.org/0000-0003-3378-7201","institution":"Shanghai AI Laboratory","correspondingAuthor":false,"prefix":"","firstName":"Lei","middleName":"","lastName":"Bai","suffix":""}],"badges":[],"createdAt":"2025-04-15 07:41:09","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6452190/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6452190/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s43247-025-02903-z","type":"published","date":"2025-11-24T05:00:00+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":96698298,"identity":"bf74492e-c581-4d50-841c-b7ed1f738815","added_by":"auto","created_at":"2025-11-25 08:06:27","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":23060885,"visible":true,"origin":"","legend":"Article File","description":"","filename":"manuscriptnomark.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6452190/v1_covered_1e41af59-65c1-4ebb-8306-deb43b7bd003.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"Advancing Climate Science Data Efficiency by Dual-stage Extreme Compression with an Efficient VAEformer","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"nature-portfolio","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"","title":"Nature Portfolio","twitterHandle":"","acdcEnabled":false,"dfaEnabled":false,"editorialSystem":"ejp","reportingPortfolio":"","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Climate Data Compression, Data-Effciency Meteorological Science, Deep Learning, Weather Forecasting, Extreme Weather Analysis, Artificial Intelligence","lastPublishedDoi":"10.21203/rs.3.rs-6452190/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6452190/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"The growing threat of climate change highlights the urgent need for deeper understanding and precise weather prediction. Despite advancements in atmospheric research, the ever-growing volume of weather data challenges efficient storage and broadcasting, slowing efforts to mitigate climate change. To this end, we present Aeolus, a pioneering deep learning-based framework for efficient atmospheric data compression. Aeolus achieves a remarkable 470 times compression ratio that reduces the 400 TB ERA5 dataset to just 0.85 TB, significantly lowering storage requirements and improving transmission efficiency. This outperforms classical methods such as JPEG2000, which achieves compression ratios typically below 10 times on atmospheric datasets. Furthermore, Aeolus ensures practical compression and decompression speeds exceeding 1 GB/s, providing a substantial computational cost benefit compared to traditional techniques. Extensive comparative experiments validate the high numerical accuracy of the compressed data, with a mean absolute error of 0.17°K for temperature, consistent climatology, and comparable power spectral density. Such a level of precision enables efficient data-driven climate research, including accurate weather forecasting and extreme event reconstructions. Overall, our findings highlight the significant transformative impact of Aeolus on climate science, expanding access to large-scale atmospheric data across disciplines and empowering researchers with the data-driven insights needed to combat climate change.","manuscriptTitle":"Advancing Climate Science Data Efficiency by Dual-stage Extreme Compression with an Efficient VAEformer","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-08-21 05:04:26","doi":"10.21203/rs.3.rs-6452190/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"communications-earth-and-environment","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"commsenv","sideBox":"Learn more about [Communications Earth and Environment](https://www.nature.com/commsenv/)","snPcode":"","submissionUrl":"","title":"Communications Earth \u0026 Environment","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Communications Series","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"97eacc44-6e8e-4933-828c-9b6ff3a4afd9","owner":[],"postedDate":"August 21st, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":52604924,"name":"Earth and environmental sciences/Climate sciences"},{"id":52604925,"name":"Earth and environmental sciences/Climate sciences/Atmospheric science"},{"id":52604926,"name":"Earth and environmental sciences/Climate sciences/Climate change/Climate and Earth system modelling"}],"tags":[],"updatedAt":"2025-11-25T08:06:17+00:00","versionOfRecord":{"articleIdentity":"rs-6452190","link":"https://doi.org/10.1038/s43247-025-02903-z","journal":{"identity":"communications-earth-and-environment","isVorOnly":false,"title":"Communications Earth \u0026 Environment"},"publishedOn":"2025-11-24 05:00:00","publishedOnDateReadable":"November 24th, 2025"},"versionCreatedAt":"2025-08-21 05:04:26","video":"","vorDoi":"10.1038/s43247-025-02903-z","vorDoiUrl":"https://doi.org/10.1038/s43247-025-02903-z","workflowStages":[]},"version":"v1","identity":"rs-6452190","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6452190","identity":"rs-6452190","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}