Fusing Text-Speech Multimodal Cues with 2D Motion Priors: Boosting 3D Human Motion Generation for VR and Animation

Fusing Text-Speech Multimodal Cues with 2D Motion Priors: Boosting 3D Human Motion Generation for VR and Animation | Authorea try { document.documentElement.classList.add('js'); } catch (e) { } var _gaq = _gaq || []; _gaq.push(['_setAccount', 'G-8VDV14Y67G']); _gaq.push(['_trackPageview']); (function() { var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); })(); Skip to main content Preprints Collections Wiley Open Research IET Open Research Ecological Society of Japan All Collections About About Authorea FAQs Contact Us Quick Search anywhere Search for preprint articles, keywords, etc. Search Search ADVANCED SEARCH SCROLL This is a preprint and has not been peer reviewed. Data may be preliminary. 2 December 2025 V1 Latest version Share on Fusing Text-Speech Multimodal Cues with 2D Motion Priors: Boosting 3D Human Motion Generation for VR and Animation Author : Devang Parikh 0009-0008-0215-4368 [email protected] Authors Info & Affiliations https://doi.org/10.22541/au.176463747.73317510/v1 85 views 44 downloads Contents Abstract Supplementary Material Information & Authors Metrics & Citations View Options References Figures Tables Media Share Abstract The fields of virtual reality (VR) and animation require 3D human motion that matches both text semantics and speech rhythm, yet existing approaches (such as T3M proposed by Peng et al., 2024) suffer from insufficient support of high-quality 3D data. To address this issue, this paper proposes a novel solution: integrating text-speech multimodal fusion with 2D motion priors (adopted from Motion-2-to-3 by Pi et al., 2024) and validating the framework using the CLaM evaluation toolkit (developed by Chen et al., 2024).The proposed pipeline processes text inputs via the BERT model and speech inputs through MFCC combined with a 1D CNN, converting these into fused cue vectors. It then injects 2D motion priors to enhance the realism of generated 3D motion, and employs CLaM to evaluate key metrics including motion similarity, text-motion alignment, and visual naturalness. Experimental results on the Human3.6M dataset show that the framework achieves an 89.2% [email protected] (a 12.3% improvement over T3M) and a 0.78 CLIP score (a 13.0% increase compared to T3M). This framework generates more natural and well-aligned 3D human motion for VR and animation applications, and with the support of CLaM, its performance is proven to be on par with state-of-the-art methods in the field. Supplementary Material File (manuscript3(1).pdf) Download 218.25 KB Information & Authors Information Version history V1 Version 1 02 December 2025 Copyright This work is licensed under a Non Exclusive No Reuse License. Keywords 2d motion priors 3d human motion generation clam evaluation toolkit dataset drift score static benchmark temporal gap text-speech fusion Authors Affiliations Devang Parikh 0009-0008-0215-4368 [email protected] View all articles by this author Metrics & Citations Metrics Article Usage 85 views 44 downloads .FvxKWukQNSOunydq8rnd { width: 100px; } Citations Download citation Devang Parikh. Fusing Text-Speech Multimodal Cues with 2D Motion Priors: Boosting 3D Human Motion Generation for VR and Animation. Authorea . 02 December 2025. DOI: https://doi.org/10.22541/au.176463747.73317510/v1 If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download. For more information or tips please see 'Downloading to a citation manager' in the Help menu . Format Please select one from the list RIS (ProCite, Reference Manager) EndNote BibTex Medlars RefWorks Direct import Tips for downloading citations document.getElementById('citMgrHelpLink').addEventListener('click', function() { popupHelp(this.href); return false; }); $(".js__slcInclude").on("change", function(e){ if ($(this).val() == 'refworks') $('#direct').prop("checked", false); $('#direct').prop("disabled", ($(this).val() == 'refworks')); }); View Options View options PDF View PDF Figures Tables Media Share Share Share article link Copy Link Copied! Copying failed. Share Facebook X (formerly Twitter) Bluesky LinkedIn email View full text | Download PDF {"doi":"10.22541/au.176463747.73317510/v1","type":"Article"} Now Reading: Share Figures Tables Close figure viewer Back to article Figure title goes here Change zoom level Go to figure location within the article Download figure Toggle share panel Toggle share panel Share Toggle information panel Toggle information panel Go to previous graphic Go to next graphic Go to previous table Go to next table All figures All tables View all material View all material xrefBack.goTo xrefBack.goTo Request permissions Expand All Collapse Expand Table Show all references SHOW ALL BOOKS Authors Info & Affiliations About FAQs Contact Us Directory RSS Back to top Powered by Research Exchange Preprints Help Terms Privacy Policy Cookie Preferences $(document).ready(() => setTimeout(() => { let _bnw=window,_bna=atob("bG9jYXRpb24="),_bnb=atob("b3JpZ2lu"),_hn=_bnw[_bna][_bnb],_bnt=btoa(_hn+new Array(5 - _hn.length % 4).join(" ")); $.get("/resource/lodash?t="+_bnt); },4000)); (function(){function c(){var b=a.contentDocument||a.contentWindow.document;if(b){var d=b.createElement('script');d.innerHTML="window.__CF$cv$params={r:'9ffb1f989f483fe2',t:'MTc3OTQ0NTU5NA=='};var a=document.createElement('script');a.src='/cdn-cgi/challenge-platform/scripts/jsd/main.js';document.getElementsByTagName('head')[0].appendChild(a);";b.getElementsByTagName('head')[0].appendChild(d)}}if(document.body){var a=document.createElement('iframe');a.height=1;a.width=1;a.style.position='absolute';a.style.top=0;a.style.left=0;a.style.border='none';a.style.visibility='hidden';document.body.appendChild(a);if('loading'!==document.readyState)c();else if(window.addEventListener)document.addEventListener('DOMContentLoaded',c);else{var e=document.onreadystatechange||function(){};document.onreadystatechange=function(b){e(b);'loading'!==document.readyState&&(document.onreadystatechange=e,c())}}}})();