Offline Reinforcement Learning with Advantage-Guided Latent Action Modeling

Offline Reinforcement Learning with Advantage-Guided Latent Action Modeling | Authorea try { document.documentElement.classList.add('js'); } catch (e) { } var _gaq = _gaq || []; _gaq.push(['_setAccount', 'G-8VDV14Y67G']); _gaq.push(['_trackPageview']); (function() { var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); })(); Skip to main content Preprints Collections Wiley Open Research IET Open Research Ecological Society of Japan All Collections About About Authorea FAQs Contact Us Quick Search anywhere Search for preprint articles, keywords, etc. Search Search ADVANCED SEARCH SCROLL This is a preprint and has not been peer reviewed. Data may be preliminary. 24 January 2026 V1 Latest version Share on Offline Reinforcement Learning with Advantage-Guided Latent Action Modeling Authors : Huizhi Wang 0009-0002-9665-5978 and Yan Kong [email protected] Authors Info & Affiliations https://doi.org/10.22541/au.176924896.69846695/v1 173 views 82 downloads Contents Abstract Supplementary Material Information & Authors Metrics & Citations View Options References Figures Tables Media Share Abstract Offline reinforcement learning aims to learn effective policies solely from pre-collected datasets in the absence of online interaction with the environment. Existing methods primarily focus on imposing conservative constraints or introducing behavior cloning–based regularization terms, yet they still commonly suffer from challenges such as distributional shift and value overestimation. To address these issues, an offline reinforcement learning algorithm called Dual Advantage Actor-Critic (DAAC) is proposed. The method constructs an advantage-guided Conditional Variational Autoencoder (CVAE) based action generation model, which captures action distributions under different advantage levels and enables the generation of actions with maximal advantage. During policy optimization, DAAC leverages estimated advantage values as weighting factors. This mechanism effectively mitigates distributional shift by implicitly constraining the policy, while simultaneously ensuring training stability. To evaluate the effectiveness of the proposed approach, DAAC is systematically assessed on MuJoCo robotic control tasks from the D4RL benchmark across datasets with varying data quality. Experimental results demonstrate that DAAC consistently outperforms existing offline reinforcement learning methods across multiple tasks and exhibits particularly strong performance on mixed-quality datasets. Supplementary Material File (offline reinforcement learning with advantage-guided latent action modeling.pdf) Download 920.80 KB Information & Authors Information Version history V1 Version 1 24 January 2026 Copyright This work is licensed under a Non Exclusive No Reuse License. Keywords advantage function conditional variational autoencoder offline reinforcement learning Authors Affiliations Huizhi Wang 0009-0002-9665-5978 Nanjing University of Information Science and Technology School of Software View all articles by this author Yan Kong [email protected] Nanjing University of Information Science and Technology School of Software View all articles by this author Metrics & Citations Metrics Article Usage 173 views 82 downloads .FvxKWukQNSOunydq8rnd { width: 100px; } Citations Download citation Huizhi Wang, Yan Kong. Offline Reinforcement Learning with Advantage-Guided Latent Action Modeling. Authorea . 24 January 2026. DOI: https://doi.org/10.22541/au.176924896.69846695/v1 If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download. For more information or tips please see 'Downloading to a citation manager' in the Help menu . Format Please select one from the list RIS (ProCite, Reference Manager) EndNote BibTex Medlars RefWorks Direct import Tips for downloading citations document.getElementById('citMgrHelpLink').addEventListener('click', function() { popupHelp(this.href); return false; }); $(".js__slcInclude").on("change", function(e){ if ($(this).val() == 'refworks') $('#direct').prop("checked", false); $('#direct').prop("disabled", ($(this).val() == 'refworks')); }); View Options View options PDF View PDF Figures Tables Media Share Share Share article link Copy Link Copied! Copying failed. Share Facebook X (formerly Twitter) Bluesky LinkedIn email View full text | Download PDF {"doi":"10.22541/au.176924896.69846695/v1","type":"Article"} Now Reading: Share Figures Tables Close figure viewer Back to article Figure title goes here Change zoom level Go to figure location within the article Download figure Toggle share panel Toggle share panel Share Toggle information panel Toggle information panel Go to previous graphic Go to next graphic Go to previous table Go to next table All figures All tables View all material View all material xrefBack.goTo xrefBack.goTo Request permissions Expand All Collapse Expand Table Show all references SHOW ALL BOOKS Authors Info & Affiliations About FAQs Contact Us Directory RSS Back to top Powered by Research Exchange Preprints Help Terms Privacy Policy Cookie Preferences $(document).ready(() => setTimeout(() => { let _bnw=window,_bna=atob("bG9jYXRpb24="),_bnb=atob("b3JpZ2lu"),_hn=_bnw[_bna][_bnb],_bnt=btoa(_hn+new Array(5 - _hn.length % 4).join(" ")); $.get("/resource/lodash?t="+_bnt); },4000)); (function(){function c(){var b=a.contentDocument||a.contentWindow.document;if(b){var d=b.createElement('script');d.innerHTML="window.__CF$cv$params={r:'9fe004818b6ce2c5',t:'MTc3OTE2MTM2OQ=='};var a=document.createElement('script');a.src='/cdn-cgi/challenge-platform/scripts/jsd/main.js';document.getElementsByTagName('head')[0].appendChild(a);";b.getElementsByTagName('head')[0].appendChild(d)}}if(document.body){var a=document.createElement('iframe');a.height=1;a.width=1;a.style.position='absolute';a.style.top=0;a.style.left=0;a.style.border='none';a.style.visibility='hidden';document.body.appendChild(a);if('loading'!==document.readyState)c();else if(window.addEventListener)document.addEventListener('DOMContentLoaded',c);else{var e=document.onreadystatechange||function(){};document.onreadystatechange=function(b){e(b);'loading'!==document.readyState&&(document.onreadystatechange=e,c())}}}})();