Performance of Five Machine Learning-based Global Weather Prediction Models in the East Asia Region

Performance of Five Machine Learning-based Global Weather Prediction Models in the East Asia Region | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Physical Sciences - Article Performance of Five Machine Learning-based Global Weather Prediction Models in the East Asia Region Kathryn Hsu, Cheng-Chin Liu, Melinda Peng, Der-Song Chen, Pao-Liang Chang, and 7 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4250353/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 28 Sep, 2024 Read the published version in npj Climate and Atmospheric Science → Version 1 posted You are reading this latest preprint version Abstract Recently, the development of artificial intelligence (AI) has led to the fruition of machine-learning (ML) based weather prediction (MLWP) systems. Five global models have emerged as notable contenders: Pangu-Weather, FourCastNet v2 (FCN2), GraphCast, FuXi, and FengWu. Despite utilizing different AI/ML methodologies, all five systems rely on the ECMWF Reanalysis v5 (ERA5), as their training dataset. This study evaluates the performance of these five models in the Eastern Asia region from June to November 2023 with our own simulations using identical initial conditions from ERA5. The evaluation comprises three key metrics: 1) Synoptic-scale prediction assessment with Root Mean Square Error and Anomaly Correlation Coefficients, 2) Typhoon predictions as extreme weather cases, and 3) Local rainfalls induced by a typhoon. Results indicate that FengWu emerges as the best-performing model, followed by FuXi and GraphCast, with FCN2 and Pangu-Weather ranking lower. Notably, performance of the ECMWF IFS lies within the MLWP model range. A multi-model ensemble, formed by averaging predictions from all five models, demonstrates superior performance, rivaling that of FengWu. For the 11 typhoons in 2023, FengWu demonstrates the most accurate track prediction. While Pangu-Weather displays the largest overall track error, it still excels in predicting the tracks of 3 typhoons. Meanwhile, FengWu has the largest intensity errors. In addition, both GraphCast and FuXi demonstrate some encouraging precipitation predictions associated with typhoons. Suggestions for future improvements in MLWP models are discussed. Earth and environmental sciences/Planetary science/Atmospheric dynamics Earth and environmental sciences/Natural hazards Full Text Additional Declarations There is NO Competing Interest. Cite Share Download PDF Status: Published Journal Publication published 28 Sep, 2024 Read the published version in npj Climate and Atmospheric Science → Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4250353","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Physical Sciences - Article","associatedPublications":[],"authors":[{"id":290269307,"identity":"50b58d5a-f02b-4cdc-99dc-8859ff11f7c9","order_by":0,"name":"Kathryn Hsu","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA6UlEQVRIiWNgGAWjYDAC9obEBx8qJHj42RuAPAMLIrTwHHhsOOOMjYxkzwGQFgkitEgkPpPmbUuzMbiRAOYS1mE+IzlBgrftMI/kzOdXN/wokGDgb+9OwKtF5syzBAOJc4d5+KVzym72AB0mcebsBvzuYs9JSDAoA9oyOyftBg9Qi4FELgEtDPkfDiSwHeYxuHkm7eYforRwJCQ2HGhL4zG4wX7sNnG28BxIZmw4Y8Mj2ZPDdlvGQIKHCL80pP/+UyFhz89+/NnNN39s5Pjbe/FrQQI8BmCSWOUgwP6AFNWjYBSMglEwggAAu8FK8ARiZQoAAAAASUVORK5CYII=","orcid":"","institution":"Central Weather Administration","correspondingAuthor":true,"prefix":"","firstName":"Kathryn","middleName":"","lastName":"Hsu","suffix":""},{"id":290269308,"identity":"3fcf7cae-fc86-47c4-9ca5-3a2aeda6da7e","order_by":1,"name":"Cheng-Chin Liu","email":"","orcid":"","institution":"Central Weather Administration","correspondingAuthor":false,"prefix":"","firstName":"Cheng-Chin","middleName":"","lastName":"Liu","suffix":""},{"id":290269309,"identity":"42bc1d7d-26d4-4f30-bae6-19d8968db495","order_by":2,"name":"Melinda Peng","email":"","orcid":"","institution":"University of Colorado Colorado Springs","correspondingAuthor":false,"prefix":"","firstName":"Melinda","middleName":"","lastName":"Peng","suffix":""},{"id":290269310,"identity":"71d85028-0ba4-4b90-8c60-2cbea4cf711e","order_by":3,"name":"Der-Song Chen","email":"","orcid":"","institution":"Central Weather Administration","correspondingAuthor":false,"prefix":"","firstName":"Der-Song","middleName":"","lastName":"Chen","suffix":""},{"id":290269311,"identity":"faaff1bf-e7e2-4e56-8b7f-e71f65a95f45","order_by":4,"name":"Pao-Liang Chang","email":"","orcid":"","institution":"Central Weather Administration","correspondingAuthor":false,"prefix":"","firstName":"Pao-Liang","middleName":"","lastName":"Chang","suffix":""},{"id":290269312,"identity":"c4c1e6b0-e818-478e-be70-18584002c41d","order_by":5,"name":"Ling-Feng Hsiao","email":"","orcid":"","institution":"Central Weather Administration","correspondingAuthor":false,"prefix":"","firstName":"Ling-Feng","middleName":"","lastName":"Hsiao","suffix":""},{"id":290269313,"identity":"f04bb536-6e54-4b82-93ad-5e6b5dd37951","order_by":6,"name":"Chin-Tzu Fong","email":"","orcid":"","institution":"Central Weather Administration","correspondingAuthor":false,"prefix":"","firstName":"Chin-Tzu","middleName":"","lastName":"Fong","suffix":""},{"id":290269314,"identity":"bd9f494c-5950-4800-a10f-7014402c7ac1","order_by":7,"name":"Jing-Shan Hong","email":"","orcid":"","institution":"Central Weather Administration","correspondingAuthor":false,"prefix":"","firstName":"Jing-Shan","middleName":"","lastName":"Hong","suffix":""},{"id":290269315,"identity":"79b41c2c-1e0f-4abd-9ce6-ffcf17e4bd69","order_by":8,"name":"Chia-Ping Cheng","email":"","orcid":"","institution":"Central Weather Administration","correspondingAuthor":false,"prefix":"","firstName":"Chia-Ping","middleName":"","lastName":"Cheng","suffix":""},{"id":290269316,"identity":"ae07d82a-1037-4f2f-afe8-28fc5f4c25c2","order_by":9,"name":"Kuo-Chen Lu","email":"","orcid":"","institution":"Central Weather Administration","correspondingAuthor":false,"prefix":"","firstName":"Kuo-Chen","middleName":"","lastName":"Lu","suffix":""},{"id":290269317,"identity":"2d8fc9da-3108-4040-ae87-12c1e6283a1e","order_by":10,"name":"Chia-Rong Chen","email":"","orcid":"","institution":"Central Weather Administration","correspondingAuthor":false,"prefix":"","firstName":"Chia-Rong","middleName":"","lastName":"Chen","suffix":""},{"id":290269318,"identity":"25661742-8911-4855-a863-1cd445d0456d","order_by":11,"name":"Hung-Chi Kuo","email":"","orcid":"","institution":"National Taiwan University","correspondingAuthor":false,"prefix":"","firstName":"Hung-Chi","middleName":"","lastName":"Kuo","suffix":""}],"badges":[],"createdAt":"2024-04-11 05:30:24","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4250353/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4250353/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41612-024-00769-0","type":"published","date":"2024-09-28T04:00:00+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":65523855,"identity":"f2f0058e-5188-4fad-b4a2-ae9c17d13417","added_by":"auto","created_at":"2024-09-29 07:06:41","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1157405,"visible":true,"origin":"","legend":"Article File","description":"","filename":"MaunscriptFigNature04112024.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4250353/v1_covered_b093e6c3-a7b3-4fa8-bd3f-6f26092eb4a6.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"Performance of Five Machine Learning-based Global Weather Prediction Models in the East Asia Region","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"npj-climate-and-atmospheric-science","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"npjclimatsci","sideBox":"Learn more about [npj Climate and Atmospheric Science](http://www.nature.com/npjclimatsci/)","snPcode":"41612","submissionUrl":"https://submission.springernature.com/new-submission/41612/3","title":"npj Climate and Atmospheric Science","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-4250353/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4250353/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Recently, the development of artificial intelligence (AI) has led to the fruition of machine-learning (ML) based weather prediction (MLWP) systems. Five global models have emerged as notable contenders: Pangu-Weather, FourCastNet v2 (FCN2), GraphCast, FuXi, and FengWu. Despite utilizing different AI/ML methodologies, all five systems rely on the ECMWF Reanalysis v5 (ERA5), as their training dataset. This study evaluates the performance of these five models in the Eastern Asia region from June to November 2023 with our own simulations using identical initial conditions from ERA5. The evaluation comprises three key metrics: 1) Synoptic-scale prediction assessment with Root Mean Square Error and Anomaly Correlation Coefficients, 2) Typhoon predictions as extreme weather cases, and 3) Local rainfalls induced by a typhoon. Results indicate that FengWu emerges as the best-performing model, followed by FuXi and GraphCast, with FCN2 and Pangu-Weather ranking lower. Notably, performance of the ECMWF IFS lies within the MLWP model range. A multi-model ensemble, formed by averaging predictions from all five models, demonstrates superior performance, rivaling that of FengWu. For the 11 typhoons in 2023, FengWu demonstrates the most accurate track prediction. While Pangu-Weather displays the largest overall track error, it still excels in predicting the tracks of 3 typhoons. Meanwhile, FengWu has the largest intensity errors. In addition, both GraphCast and FuXi demonstrate some encouraging precipitation predictions associated with typhoons. Suggestions for future improvements in MLWP models are discussed.","manuscriptTitle":"Performance of Five Machine Learning-based Global Weather Prediction Models in the East Asia Region","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-04-19 02:12:27","doi":"10.21203/rs.3.rs-4250353/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"npj-climate-and-atmospheric-science","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"npjclimatsci","sideBox":"Learn more about [npj Climate and Atmospheric Science](http://www.nature.com/npjclimatsci/)","snPcode":"41612","submissionUrl":"https://submission.springernature.com/new-submission/41612/3","title":"npj Climate and Atmospheric Science","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"0e77c2f0-fba6-4a75-b936-f36c6cd58ab3","owner":[],"postedDate":"April 19th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":30575376,"name":"Earth and environmental sciences/Planetary science/Atmospheric dynamics"},{"id":30575377,"name":"Earth and environmental sciences/Natural hazards"}],"tags":[],"updatedAt":"2024-09-29T07:06:36+00:00","versionOfRecord":{"articleIdentity":"rs-4250353","link":"https://doi.org/10.1038/s41612-024-00769-0","journal":{"identity":"npj-climate-and-atmospheric-science","isVorOnly":false,"title":"npj Climate and Atmospheric Science"},"publishedOn":"2024-09-28 04:00:00","publishedOnDateReadable":"September 28th, 2024"},"versionCreatedAt":"2024-04-19 02:12:27","video":"","vorDoi":"10.1038/s41612-024-00769-0","vorDoiUrl":"https://doi.org/10.1038/s41612-024-00769-0","workflowStages":[]},"version":"v1","identity":"rs-4250353","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4250353","identity":"rs-4250353","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}