Multi-agent reinforcement learning for the control of three-dimensional Rayleigh-Bénard convection

Multi-agent reinforcement learning for the control of three-dimensional Rayleigh-Bénard convection | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Multi-agent reinforcement learning for the control of three-dimensional Rayleigh-Bénard convection Joel Vasanth, Jean Rabault, Francisco Alcántara-Ávila, Mikael Mortensen, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4813370/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 14 Dec, 2024 Read the published version in Flow, Turbulence and Combustion → Version 1 posted 9 You are reading this latest preprint version Abstract Deep reinforcement learning (DRL) has found application in numerous use-cases pertaining to flow control. Multi-agent RL (MARL), a variant of DRL, has shown to be more effective than single-agent RL in controlling flows exhibiting locality and translational invariance. We present, for the first time, an implementation of MARL-based control of three-dimensional Rayleigh-Bénard convection (RBC). Control is executed by modifying the temperature distribution along the bottom wall divided into multiple control segments, each of which acts as an independent agent. Two regimes of RBC are considered at Rayleigh numbers Ra = 500 and 750. Evaluation of the learned control policy reveals a reduction in convection intensity by 23.5% and 8.7% at Ra = 500 and 750, respectively. The MARL controller converts irregularly shaped convective patterns to regular straight rolls with lower convection that resemble flow in a relatively more stable regime. We draw comparisons with proportional control at both Ra and show that MARL is able to outperform the proportional controller. The learned control strategy is complex, featuring different non-linear segment-wise actuator delays and actuation magnitudes. We also perform successful evaluations on a larger domain than used for training, demonstrating that the invariant property of MARL allows direct transfer of the learnt policy. Reinforcement Learning Active Flow Control Rayleigh–B´enard Convection Multi-agent Reinforcement Learning Machine Learning Full Text Additional Declarations No competing interests reported. Supplementary Files supplementarymaterialvideos.zip Cite Share Download PDF Status: Published Journal Publication published 14 Dec, 2024 Read the published version in Flow, Turbulence and Combustion → Version 1 posted Editorial decision: Revision requested 14 Oct, 2024 Reviews received at journal 14 Oct, 2024 Reviews received at journal 30 Sep, 2024 Reviewers agreed at journal 16 Sep, 2024 Reviewers agreed at journal 26 Aug, 2024 Reviewers invited by journal 01 Aug, 2024 Editor assigned by journal 31 Jul, 2024 Submission checks completed at journal 31 Jul, 2024 First submitted to journal 27 Jul, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4813370","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":343085065,"identity":"a30b6de3-aeb9-4aec-a74f-df10550629eb","order_by":0,"name":"Joel Vasanth","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAvUlEQVRIiWNgGAWjYBACxgYIncDPw8BGkhaDBMkeZiK1QIFBgsEZYrUwz0h+/OHnjj95xmfOH3vA8MeGCIfNSDOT7D1jUGx2tpndgLEtjQgtPQfMGHjbDBK3nWdmk2BsOEyMluOfP/4FatncD9TC8Oc/EVraewykQbZs4G0GamE7QJSWMmnZNuPEGWcOm0kktiUT1mLYzL7549s2ucT+nsRnEh/+2BGhpQGZl0BYAwODPDGKRsEoGAWjYIQDABUoOIWQ/eF7AAAAAElFTkSuQmCC","orcid":"","institution":"KTH Royal Institute of Technology","correspondingAuthor":true,"prefix":"","firstName":"Joel","middleName":"","lastName":"Vasanth","suffix":""},{"id":343085066,"identity":"e63d1360-6f09-42dc-8f14-36eba9243299","order_by":1,"name":"Jean Rabault","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Jean","middleName":"","lastName":"Rabault","suffix":""},{"id":343085067,"identity":"d40c2544-1966-4d50-8b7b-8d32cb4df778","order_by":2,"name":"Francisco Alcántara-Ávila","email":"","orcid":"","institution":"KTH Royal Institute of Technology","correspondingAuthor":false,"prefix":"","firstName":"Francisco","middleName":"","lastName":"Alcántara-Ávila","suffix":""},{"id":343085069,"identity":"b668f596-bb27-48f7-b56d-709fb0b6bac8","order_by":3,"name":"Mikael Mortensen","email":"","orcid":"","institution":"University of Oslo","correspondingAuthor":false,"prefix":"","firstName":"Mikael","middleName":"","lastName":"Mortensen","suffix":""},{"id":343085070,"identity":"4964c9f8-2245-48f2-9a54-13ce306aecb9","order_by":4,"name":"Ricardo Vinuesa","email":"","orcid":"","institution":"KTH Royal Institute of Technology","correspondingAuthor":false,"prefix":"","firstName":"Ricardo","middleName":"","lastName":"Vinuesa","suffix":""}],"badges":[],"createdAt":"2024-07-27 13:51:30","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4813370/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4813370/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s10494-024-00619-2","type":"published","date":"2024-12-14T15:57:19+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":71552428,"identity":"65fb8161-0c19-47cf-a107-64e87dff3118","added_by":"auto","created_at":"2024-12-16 16:06:01","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1406785,"visible":true,"origin":"","legend":"","description":"","filename":"drlcontrol3drbc.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4813370/v1_covered_140f937b-46aa-47cf-80d6-78c099678c5c.pdf"},{"id":63104547,"identity":"df31f06a-1d3b-47c3-90b6-1944550b8ea3","added_by":"auto","created_at":"2024-08-23 07:26:15","extension":"zip","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":44933620,"visible":true,"origin":"","legend":"","description":"","filename":"supplementarymaterialvideos.zip","url":"https://assets-eu.researchsquare.com/files/rs-4813370/v1/0a495b96f96711c2f06877d4.zip"}],"financialInterests":"No competing interests reported.","formattedTitle":"Multi-agent reinforcement learning for the control of three-dimensional Rayleigh-Bénard convection","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"flow-turbulence-and-combustion","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"appl","sideBox":"Learn more about [Flow, Turbulence and Combustion](http://link.springer.com/journal/10494)","snPcode":"10494","submissionUrl":"https://submission.nature.com/new-submission/10494/3","title":"Flow, Turbulence and Combustion","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Reinforcement Learning, Active Flow Control, Rayleigh–B´enard Convection, Multi-agent Reinforcement Learning, Machine Learning","lastPublishedDoi":"10.21203/rs.3.rs-4813370/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4813370/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Deep reinforcement learning (DRL) has found application in numerous use-cases pertaining to flow control. Multi-agent RL (MARL), a variant of DRL, has shown to be more effective than single-agent RL in controlling flows exhibiting locality and translational invariance. We present, for the first time, an implementation of MARL-based control of three-dimensional Rayleigh-Bénard convection (RBC). Control is executed by modifying the temperature distribution along the bottom wall divided into multiple control segments, each of which acts as an independent agent. Two regimes of RBC are considered at Rayleigh numbers Ra = 500 and 750. Evaluation of the learned control policy reveals a reduction in convection intensity by 23.5% and 8.7% at Ra = 500 and 750, respectively. The MARL controller converts irregularly shaped convective patterns to regular straight rolls with lower convection that resemble flow in a relatively more stable regime. We draw comparisons with proportional control at both Ra and show that MARL is able to outperform the proportional controller. The learned control strategy is complex, featuring different non-linear segment-wise actuator delays and actuation magnitudes. We also perform successful evaluations on a larger domain than used for training, demonstrating that the invariant property of MARL allows direct transfer of the learnt policy.","manuscriptTitle":"Multi-agent reinforcement learning for the control of three-dimensional Rayleigh-Bénard convection","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-08-23 07:26:09","doi":"10.21203/rs.3.rs-4813370/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-10-14T19:42:28+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-10-14T07:07:33+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-09-30T14:37:48+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"49968947540008711006654494114306748696","date":"2024-09-16T07:45:27+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"193624895376485442143524018481326957029","date":"2024-08-26T08:11:52+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-08-01T21:31:29+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-07-31T16:32:43+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-07-31T14:31:31+00:00","index":"","fulltext":""},{"type":"submitted","content":"Flow, Turbulence and Combustion","date":"2024-07-27T13:49:12+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"flow-turbulence-and-combustion","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"appl","sideBox":"Learn more about [Flow, Turbulence and Combustion](http://link.springer.com/journal/10494)","snPcode":"10494","submissionUrl":"https://submission.nature.com/new-submission/10494/3","title":"Flow, Turbulence and Combustion","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"794fc700-9449-40b2-9380-34365f0eb075","owner":[],"postedDate":"August 23rd, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2024-12-16T16:00:35+00:00","versionOfRecord":{"articleIdentity":"rs-4813370","link":"https://doi.org/10.1007/s10494-024-00619-2","journal":{"identity":"flow-turbulence-and-combustion","isVorOnly":false,"title":"Flow, Turbulence and Combustion"},"publishedOn":"2024-12-14 15:57:19","publishedOnDateReadable":"December 14th, 2024"},"versionCreatedAt":"2024-08-23 07:26:09","video":"","vorDoi":"10.1007/s10494-024-00619-2","vorDoiUrl":"https://doi.org/10.1007/s10494-024-00619-2","workflowStages":[]},"version":"v1","identity":"rs-4813370","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4813370","identity":"rs-4813370","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}