Learning spatiotemporal dynamics with a pretrained generative model

Learning spatiotemporal dynamics with a pretrained generative model | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Learning spatiotemporal dynamics with a pretrained generative model Lijun Yang, Zeyu Li, Wang Han, Yue Zhang, Qingfei Fu, Jingxuan Li, and 4 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4183330/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 06 Dec, 2024 Read the published version in Nature Machine Intelligence → Version 1 posted You are reading this latest preprint version Abstract Reconstructing spatiotemporal dynamics with sparse sensor measurement is an outstanding problem, commonly encountered in a wide spectrum of scientific and engineering applications. Such a problem is particularly challenging when the number and/or types of sensors (e.g., randomly placed) are extremely insufficient. Existing end-to-end learning models ordinarily suffer from the generalization issue for full-field reconstruction of spatiotemporal dynamics, especially in sparse data regimes typically seen in real-world applications. To this end, we propose a sparse-sensor-assisted score-based generative model (S 3 GM) to reconstruct and predict full-field spatiotemporal dynamics based on sparse measurements. Instead of learning directly the mapping between input and output pairs, an unconditioned generative model is firstly pretrained capturing the joint distribution of a vast group of pretraining data in a self-supervised manner, followed then by a sampling process conditioned on unseen sparse measurement. The efficacy of S 3 GM has been verified on multiple dynamical systems with various synthetic, real-world, and lab-test datasets (ranging from turbulent flow modeling to weather/climate forecasting). The results demonstrate the excellent performance of S 3 GM in zero-shot reconstruction and prediction of spatiotemporal dynamics even with high levels of data sparsity and noise. We find that S 3 GM exhibits high accuracy, generalizability, and robustness when handling different reconstruction tasks. Physical sciences/Mathematics and computing/Applied mathematics Physical sciences/Mathematics and computing/Computational science Physical sciences/Physics/Fluid dynamics Full Text Additional Declarations There is NO Competing Interest. Supplementary Files SupplementaryInformation.pdf SupplementaryVideo1.mp4 Supplementary Video 1 SupplementaryVideo2.mp4 Supplementary Video 2 SupplementaryVideo3.mp4 Supplementary Video 3 SupplementaryVideo4.mp4 Supplementary Video 4 Cite Share Download PDF Status: Published Journal Publication published 06 Dec, 2024 Read the published version in Nature Machine Intelligence → Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4183330","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":300240806,"identity":"3f15774d-96bb-4bf5-8dd2-0d91e6495f9c","order_by":0,"name":"Lijun Yang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA70lEQVRIiWNgGAWjYLCCBwZsDAzsPVDeAWK0JIC08JwhSQuIkMghUovB8bOHXyQU8CVul3x7TOJHDYMc340Exs8F+LScyUuzADoscefsvDTJnmMMxpI3EpilZ+DTciDHzACkZcPtHDMJHjaGxA03EtiYefBpOf8GquXmGTPJP/8Y6glruZFj/ACs5QaPmTRvGzD0CGmRvPHGDBTIxhvO5CVby/ZJGM4887BZGp8WvvM5xh8+/Dkmu+H42YM333yzkec7nnzwMz4tCgcY2CQYGI7B+EA2A2MDHg0MDPINDMwfGBhq8CoaBaNgFIyCEQ4A6JdQMYHgVx8AAAAASUVORK5CYII=","orcid":"","institution":"Beihang University","correspondingAuthor":true,"prefix":"","firstName":"Lijun","middleName":"","lastName":"Yang","suffix":""},{"id":300240808,"identity":"c0089863-0dc0-4ca4-a566-b4894763c7ef","order_by":1,"name":"Zeyu Li","email":"","orcid":"","institution":"Beihang University","correspondingAuthor":false,"prefix":"","firstName":"Zeyu","middleName":"","lastName":"Li","suffix":""},{"id":300240809,"identity":"0f099609-f9b6-46b3-96cc-5bb6034dde9f","order_by":2,"name":"Wang Han","email":"","orcid":"","institution":"Beihang University","correspondingAuthor":false,"prefix":"","firstName":"Wang","middleName":"","lastName":"Han","suffix":""},{"id":300240810,"identity":"72affacf-32ac-4e05-bb4b-e5f8c6f0c660","order_by":3,"name":"Yue Zhang","email":"","orcid":"","institution":"Beihang University","correspondingAuthor":false,"prefix":"","firstName":"Yue","middleName":"","lastName":"Zhang","suffix":""},{"id":300240811,"identity":"d17360e7-377c-42de-92a5-fb22fd569386","order_by":4,"name":"Qingfei Fu","email":"","orcid":"","institution":"Beihang University","correspondingAuthor":false,"prefix":"","firstName":"Qingfei","middleName":"","lastName":"Fu","suffix":""},{"id":300240812,"identity":"9f802e2d-e282-4883-8841-fe922b91bf63","order_by":5,"name":"Jingxuan Li","email":"","orcid":"","institution":"Beihang University","correspondingAuthor":false,"prefix":"","firstName":"Jingxuan","middleName":"","lastName":"Li","suffix":""},{"id":300240813,"identity":"958814b6-2467-441d-8da8-03bf7dc1b335","order_by":6,"name":"Lizi Qin","email":"","orcid":"","institution":"Beihang University","correspondingAuthor":false,"prefix":"","firstName":"Lizi","middleName":"","lastName":"Qin","suffix":""},{"id":300240814,"identity":"5a86f37c-f41e-4423-b73e-4f71b70210be","order_by":7,"name":"Ruoyu Dong","email":"","orcid":"","institution":"Beihang University","correspondingAuthor":false,"prefix":"","firstName":"Ruoyu","middleName":"","lastName":"Dong","suffix":""},{"id":300240815,"identity":"66d24906-854e-48ce-ba8b-fce03ebf7459","order_by":8,"name":"Hao Sun","email":"","orcid":"https://orcid.org/0000-0002-5145-3259","institution":"Renmin University of China","correspondingAuthor":false,"prefix":"","firstName":"Hao","middleName":"","lastName":"Sun","suffix":""},{"id":300240816,"identity":"22d9f464-90cf-4ea5-b331-f95d376c1101","order_by":9,"name":"Yue Deng","email":"","orcid":"https://orcid.org/0000-0003-2871-8922","institution":"Beihang University","correspondingAuthor":false,"prefix":"","firstName":"Yue","middleName":"","lastName":"Deng","suffix":""}],"badges":[],"createdAt":"2024-03-28 15:42:31","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4183330/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4183330/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s42256-024-00938-z","type":"published","date":"2024-12-06T05:00:00+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":70817105,"identity":"1c80fe6f-9203-4efb-bbe6-ab8223856c38","added_by":"auto","created_at":"2024-12-07 08:07:32","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":11201299,"visible":true,"origin":"","legend":"","description":"","filename":"ManuscriptS3GM.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4183330/v1_covered_ea1d3e8e-388a-4e41-90ae-9de4b8054a0e.pdf"},{"id":56131476,"identity":"f378d252-2f05-4732-9712-ba010ee54f6d","added_by":"auto","created_at":"2024-05-09 02:03:44","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":10590658,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryInformation.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4183330/v1/cf87e0a0f4bccbfda863a5e8.pdf"},{"id":56140212,"identity":"32d072f3-5bc3-469b-9204-8e6be9369771","added_by":"auto","created_at":"2024-05-09 04:09:30","extension":"mp4","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":23958350,"visible":true,"origin":"","legend":"Supplementary Video 1","description":"","filename":"SupplementaryVideo1.mp4","url":"https://assets-eu.researchsquare.com/files/rs-4183330/v1/b45c7029ef0cc112b75518e0.mp4"},{"id":56140260,"identity":"d9a3392c-054f-4342-93e5-5144eb78ecb4","added_by":"auto","created_at":"2024-05-09 04:10:09","extension":"mp4","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":93494839,"visible":true,"origin":"","legend":"\u003cp\u003eSupplementary Video 2\u003c/p\u003e","description":"","filename":"SupplementaryVideo2.mp4","url":"https://assets-eu.researchsquare.com/files/rs-4183330/v1/58af0429807d3a21df0e51f5.mp4"},{"id":56140156,"identity":"bd826ffd-0f91-413b-b0e5-d1c530ffa924","added_by":"auto","created_at":"2024-05-09 04:06:01","extension":"mp4","order_by":4,"title":"","display":"","copyAsset":false,"role":"supplement","size":34321813,"visible":true,"origin":"","legend":"\u003cp\u003eSupplementary Video 3\u003c/p\u003e","description":"","filename":"SupplementaryVideo3.mp4","url":"https://assets-eu.researchsquare.com/files/rs-4183330/v1/089067f4004c0ecdffb48361.mp4"},{"id":56140237,"identity":"d49f45d9-a9d5-45f0-9f1c-b435826f01e3","added_by":"auto","created_at":"2024-05-09 04:09:56","extension":"mp4","order_by":5,"title":"","display":"","copyAsset":false,"role":"supplement","size":50573302,"visible":true,"origin":"","legend":"\u003cp\u003eSupplementary Video 4\u003c/p\u003e","description":"","filename":"SupplementaryVideo4.mp4","url":"https://assets-eu.researchsquare.com/files/rs-4183330/v1/93d9e2161de8d6d108531c71.mp4"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"Learning spatiotemporal dynamics with a pretrained generative model","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"nature-portfolio","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"","title":"Nature Portfolio","twitterHandle":"","acdcEnabled":false,"dfaEnabled":false,"editorialSystem":"ejp","reportingPortfolio":"","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-4183330/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4183330/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Reconstructing spatiotemporal dynamics with sparse sensor measurement is an outstanding problem, commonly encountered in a wide spectrum of scientific and engineering applications. Such a problem is particularly challenging when the number and/or types of sensors (e.g., randomly placed) are extremely insufficient. Existing end-to-end learning models ordinarily suffer from the generalization issue for full-field reconstruction of spatiotemporal dynamics, especially in sparse data regimes typically seen in real-world applications. To this end, we propose a sparse-sensor-assisted score-based generative model (S\u003csup\u003e3\u003c/sup\u003eGM) to reconstruct and predict full-field spatiotemporal dynamics based on sparse measurements. Instead of learning directly the mapping between input and output pairs, an unconditioned generative model is firstly pretrained capturing the joint distribution of a vast group of pretraining data in a self-supervised manner, followed then by a sampling process conditioned on unseen sparse measurement. The efficacy of S\u003csup\u003e3\u003c/sup\u003eGM has been verified on multiple dynamical systems with various synthetic, real-world, and lab-test datasets (ranging from turbulent flow modeling to weather/climate forecasting). The results demonstrate the excellent performance of S\u003csup\u003e3\u003c/sup\u003eGM in zero-shot reconstruction and prediction of spatiotemporal dynamics even with high levels of data sparsity and noise. We find that S\u003csup\u003e3\u003c/sup\u003eGM exhibits high accuracy, generalizability, and robustness when handling different reconstruction tasks.","manuscriptTitle":"Learning spatiotemporal dynamics with a pretrained generative model","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-05-09 02:03:33","doi":"10.21203/rs.3.rs-4183330/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"nature-machine-intelligence","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"natmachintell","sideBox":"Learn more about [Nature Machine Intelligence](http://www.nature.com/natmachintell/)","snPcode":"","submissionUrl":"","title":"Nature Machine Intelligence","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Nature Research","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"e5e6087d-b9fa-49df-b9e8-f606d7756c9c","owner":[],"postedDate":"May 9th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":31679768,"name":"Physical sciences/Mathematics and computing/Applied mathematics"},{"id":31679769,"name":"Physical sciences/Mathematics and computing/Computational science"},{"id":31679770,"name":"Physical sciences/Physics/Fluid dynamics"}],"tags":[],"updatedAt":"2024-12-07T08:07:16+00:00","versionOfRecord":{"articleIdentity":"rs-4183330","link":"https://doi.org/10.1038/s42256-024-00938-z","journal":{"identity":"nature-machine-intelligence","isVorOnly":false,"title":"Nature Machine Intelligence"},"publishedOn":"2024-12-06 05:00:00","publishedOnDateReadable":"December 6th, 2024"},"versionCreatedAt":"2024-05-09 02:03:33","video":"","vorDoi":"10.1038/s42256-024-00938-z","vorDoiUrl":"https://doi.org/10.1038/s42256-024-00938-z","workflowStages":[]},"version":"v1","identity":"rs-4183330","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4183330","identity":"rs-4183330","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}