Ventilator Treatment Policy Control based on BCQ off-line Deep Reinforcement Learning

Ventilator Treatment Policy Control based on BCQ off-line Deep Reinforcement Learning | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Ventilator Treatment Policy Control based on BCQ off-line Deep Reinforcement Learning Jingkun MAO, Fengxi LI, Chunxin LIU, Pixuan ZHOU This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4485071/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Ventilator plays a crucial role in treating cardiopulmonary disorders, and proper parameter settings are essential for the successful recovery of patients. Traditional ventilator control relies on the expertise of physicians, leading to delayed treatment responses. Although some machine learning methods have made improvements in this scenario, they are inadequate to adapt to dynamic changes of patient conditions. This paper proposes a dynamic ventilator control method based on the BCQ offline deep reinforcement learning algorithm, achieving real-time adjustment of ventilator treatment policies based on changes in the patient’s medical condition. In the experiments, the Double DQN and SAC algorithms are used as baseline algorithms. During the training phase, the algorithms’ optimal models under different hyperparameter combinations are determined using temporal difference error and average action values. In the testing phase, the model’s therapeutic efficacy is evaluated using the FQE method, while the safety of the treatment is assessed by statistically analyzing the predicted action distribution. Additionally, the algorithm’s generalization ability is further evaluated on an OOD test set. The experimental results demonstrate that the BCQ algorithm outperforms both in terms of treatment effectiveness, safety, and generalization ability, indicating its promising application prospects in medical scenarios. Health sciences/Health care Physical sciences/Engineering Offline Reinforcement Learning Ventilator Control FQE BCQ Double DQN SAC Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4485071","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":312239063,"identity":"b6fe4213-415d-4fcf-b6c9-114bdd32abd7","order_by":0,"name":"Jingkun MAO","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABBElEQVRIiWNgGAWjYBACxmYgbmBgkANxDjxgYGAGC/MQocUYrCWBwYCZgY2AFrA2oJbEBhALqIWBoBbmdt7DL2e23UnvFzv8EGjLH3aD+w2MD962Mcib43QYX5rlxrZnuTNnpxmAHWZwjIHZcG4bg+HOBlxaeMwMH7Ydzt1wOwGuhU2at40ByMWvJd3+dvoHmBb23wS0GD/c2HY4wUA6B2ELMyFbGGecO2w443ZOwYEEA2NmyWOJzZJzzkkYbsChxbD/jPHHnrLD8vyz0zd/+FAhl8x3+PDBD2/KbORx2WLYwMAmgeAaMCRD4olBArt6IJAHRs0HZAE7nEpHwSgYBaNgxAIAgC1b51Lt6tkAAAAASUVORK5CYII=","orcid":"","institution":"Tianjin University of Technology","correspondingAuthor":true,"prefix":"","firstName":"Jingkun","middleName":"","lastName":"MAO","suffix":""},{"id":312239064,"identity":"916bb8b7-b210-4c56-b1a3-cdaf40d77c02","order_by":1,"name":"Fengxi LI","email":"","orcid":"","institution":"Tianjin University of Technology","correspondingAuthor":false,"prefix":"","firstName":"Fengxi","middleName":"","lastName":"LI","suffix":""},{"id":312239065,"identity":"16ddc085-3f4b-42a6-9585-3a4d7a7e01b4","order_by":2,"name":"Chunxin LIU","email":"","orcid":"","institution":"Sun Yat-sen University","correspondingAuthor":false,"prefix":"","firstName":"Chunxin","middleName":"","lastName":"LIU","suffix":""},{"id":312239066,"identity":"7e66ff87-e206-44bf-abd9-10b3e91cb919","order_by":3,"name":"Pixuan ZHOU","email":"","orcid":"","institution":"Guangzhou Thearay Medical Co., Ltd","correspondingAuthor":false,"prefix":"","firstName":"Pixuan","middleName":"","lastName":"ZHOU","suffix":""}],"badges":[],"createdAt":"2024-05-27 12:40:30","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4485071/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4485071/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":87791187,"identity":"40f820a8-054a-49e0-9705-f0f78f2b0a25","added_by":"auto","created_at":"2025-07-29 05:40:59","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":831899,"visible":true,"origin":"","legend":"","description":"","filename":"PaperBCQver05engderivedforScientificReportssubmission02.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4485071/v1_covered_bad2b540-25a2-4d12-80db-3003e538875b.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Ventilator Treatment Policy Control based on BCQ off-line Deep Reinforcement Learning","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Offline Reinforcement Learning, Ventilator Control, FQE, BCQ, Double DQN, SAC","lastPublishedDoi":"10.21203/rs.3.rs-4485071/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4485071/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Ventilator plays a crucial role in treating cardiopulmonary disorders, and proper parameter settings are essential for the successful recovery of patients. Traditional ventilator control relies on the expertise of physicians, leading to delayed treatment responses. Although some machine learning methods have made improvements in this scenario, they are inadequate to adapt to dynamic changes of patient conditions. This paper proposes a dynamic ventilator control method based on the BCQ offline deep reinforcement learning algorithm, achieving real-time adjustment of ventilator treatment policies based on changes in the patient’s medical condition. In the experiments, the Double DQN and SAC algorithms are used as baseline algorithms. During the training phase, the algorithms’ optimal models under different hyperparameter combinations are determined using temporal difference error and average action values. In the testing phase, the model’s therapeutic efficacy is evaluated using the FQE method, while the safety of the treatment is assessed by statistically analyzing the predicted action distribution. Additionally, the algorithm’s generalization ability is further evaluated on an OOD test set. The experimental results demonstrate that the BCQ algorithm outperforms both in terms of treatment effectiveness, safety, and generalization ability, indicating its promising application prospects in medical scenarios.","manuscriptTitle":"Ventilator Treatment Policy Control based on BCQ off-line Deep Reinforcement Learning","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-06-10 11:46:39","doi":"10.21203/rs.3.rs-4485071/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"6b5d1bc7-da4e-4440-9f2f-3b70d85a57f6","owner":[],"postedDate":"June 10th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":33002568,"name":"Health sciences/Health care"},{"id":33002569,"name":"Physical sciences/Engineering"}],"tags":[],"updatedAt":"2025-07-29T05:08:51+00:00","versionOfRecord":[],"versionCreatedAt":"2024-06-10 11:46:39","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4485071","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4485071","identity":"rs-4485071","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}