A Comparative Study of Large Language Models for Gesture Selection in Virtual Agents

A Comparative Study of Large Language Models for Gesture Selection in Virtual Agents | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A Comparative Study of Large Language Models for Gesture Selection in Virtual Agents Parisa Ghanad Torshizi, Laura B. Hensel, Ari Shapiro, Stacy C. Marsella This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8768097/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 4 You are reading this latest preprint version Abstract Co-speech gestures convey a wide variety of meanings and play an important role in face-to-face human interaction, influencing addressees’ engagement, recall, comprehension, and attitudes toward speaker. Similar effects have been observed in interactions with embodied virtual agents, making selection and animation of meaningful gestures a key focus in agent design. Automating this gesture selection process, however, remains challenging. Prior approaches range from fully data-driven techniques—which often struggle to produce contextually meaningful gestures—to more manual methods that rely on handcrafted expertise and lack generalizability. In this paper, we leverage semantic capabilities of Large Language Models (LLMs) to realize an automated gesture selection. We first illustrate information on gestures encoded into LLM, using GPT4 as a high-quality baseline. Building on this, we evaluate alternative prompting approaches for their ability to select meaningful, contextually appropriate gestures aligned to co-speech utterance. While GPT-4 demonstrates strong gesture selection performance, its inference latency makes it unsuitable for real-time interaction. To assess feasibility under real-time constraints, we then evaluate two additional models relative to this baseline: locally deployable Llama3 and a new high-capacity cloud model (GPT-5.2). Our comparative analysis highlights differences in gesture appropriateness and alignment, including GPT5.2’s tendency to over-generate gestures by selecting multiple, inappropriate gestures for a single utterance. In contrast, Llama3 achieves a more favorable balance between gesture quality and inference speed, informing model selection for interactive virtual humans. Finally, we demonstrate how this approach is integrated into a virtual agent system, enabling automated gesture selection and animation during human–agent interaction. gesture selection virtual agents large language models multimodal behavior embodied agents Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Reviewers invited by journal 17 Mar, 2026 Editor assigned by journal 03 Feb, 2026 Submission checks completed at journal 03 Feb, 2026 First submitted to journal 02 Feb, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8768097","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":607850973,"identity":"d73fd68a-45f8-45b0-87a5-d2121fcb1b33","order_by":0,"name":"Parisa Ghanad Torshizi","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABR0lEQVRIie3Rv2vCQBTA8SeCU0rWJ/7Iv/DkQBdr/5AuCQdxCSIIod0CQqbSrlc6+C8ECtLxQiCTOgt1KBQ6dRGhOFhsEijqdXDtcN/huAv3yQsEQKf7j2G+VPLFAAkEYOb7UnC4cbw/InQg1fxGHPy+6AwpInmGWNfjj7etv2t2HuZxvB3CgL2O0836pWtZk2wzhG4jkiektUo7rbsFsfpyYEskGLVXKRfxzG1FaYU/CXCZSoTdxouQHIEGSSJwpkuPQRwmpahisLIBifOH9L+q3zkxZyTtjDyLguyvJqG5ycheJRZ67VoxBTySMiMRFkQ6QWqUMyJVQuj5tfqCMUSP4oBwhEuXwzzkPErd7MOIs0dliuhPq59+s4nmjK23u+7AFDyB27B3ORkn72Xjpte4V6acHgHt03P+c5WsQHmgEp1Op9PBDxyuen5f733cAAAAAElFTkSuQmCC","orcid":"","institution":"Northeastern University","correspondingAuthor":true,"prefix":"","firstName":"Parisa","middleName":"Ghanad","lastName":"Torshizi","suffix":""},{"id":607850974,"identity":"fe0984a1-eafe-424f-a9b2-9be41f33098f","order_by":1,"name":"Laura B. Hensel","email":"","orcid":"","institution":"University of Glasgow","correspondingAuthor":false,"prefix":"","firstName":"Laura","middleName":"B.","lastName":"Hensel","suffix":""},{"id":607850975,"identity":"34bbeb1e-2fd8-4344-8534-b787a1563dfc","order_by":2,"name":"Ari Shapiro","email":"","orcid":"","institution":"Flawless","correspondingAuthor":false,"prefix":"","firstName":"Ari","middleName":"","lastName":"Shapiro","suffix":""},{"id":607850976,"identity":"ed21c9e1-4c59-4c48-ae8c-e9bef59460ab","order_by":3,"name":"Stacy C. Marsella","email":"","orcid":"","institution":"Northeastern University","correspondingAuthor":false,"prefix":"","firstName":"Stacy","middleName":"C.","lastName":"Marsella","suffix":""}],"badges":[],"createdAt":"2026-02-02 18:23:26","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8768097/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8768097/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":105035632,"identity":"9c570ae8-115c-4949-a7f7-10dfbc48f1ef","added_by":"auto","created_at":"2026-03-20 07:26:21","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2450994,"visible":true,"origin":"","legend":"","description":"","filename":"JAAMASTorshizi.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8768097/v1_covered_f32452b8-80bb-4bfa-8c83-8be94d440f9a.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"A Comparative Study of Large Language Models for Gesture Selection in Virtual Agents","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"autonomous-agents-and-multi-agent-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"agnt","sideBox":"Learn more about [Autonomous Agents and Multi-Agent Systems](http://link.springer.com/journal/10458)","snPcode":"10458","submissionUrl":"https://submission.nature.com/new-submission/10458/3","title":"Autonomous Agents and Multi-Agent Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"gesture selection, virtual agents, large language models, multimodal behavior, embodied agents","lastPublishedDoi":"10.21203/rs.3.rs-8768097/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8768097/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Co-speech gestures convey a wide variety of meanings and play an important role in face-to-face human interaction, influencing addressees’ engagement, recall, comprehension, and attitudes toward speaker. Similar effects have been observed in interactions with embodied virtual agents, making selection and animation of meaningful gestures a key focus in agent design. Automating this gesture selection process, however, remains challenging. Prior approaches range from fully data-driven techniques—which often struggle to produce contextually meaningful gestures—to more manual methods that rely on handcrafted expertise and lack generalizability.\nIn this paper, we leverage semantic capabilities of Large Language Models (LLMs) to realize an automated gesture selection. We first illustrate information on gestures encoded into LLM, using GPT4 as a high-quality baseline. Building on this, we evaluate alternative prompting approaches for their ability to select meaningful, contextually appropriate gestures aligned to co-speech utterance. While GPT-4 demonstrates strong gesture selection performance, its inference latency makes it unsuitable for real-time interaction.\nTo assess feasibility under real-time constraints, we then evaluate two additional models relative to this baseline: locally deployable Llama3 and a new high-capacity cloud model (GPT-5.2). Our comparative analysis highlights differences in gesture appropriateness and alignment, including GPT5.2’s tendency to over-generate gestures by selecting multiple, inappropriate gestures for a single utterance. In contrast, Llama3 achieves a more favorable balance between gesture quality and inference speed, informing model selection for interactive virtual humans. Finally, we demonstrate how this approach is integrated into a virtual agent system, enabling automated gesture selection and animation during human–agent interaction.","manuscriptTitle":"A Comparative Study of Large Language Models for Gesture Selection in Virtual Agents","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-03-20 05:58:49","doi":"10.21203/rs.3.rs-8768097/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"reviewersInvited","content":"","date":"2026-03-17T21:01:46+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-02-04T03:32:41+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-02-03T08:31:34+00:00","index":"","fulltext":""},{"type":"submitted","content":"Autonomous Agents and Multi-Agent Systems","date":"2026-02-02T18:18:50+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"autonomous-agents-and-multi-agent-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"agnt","sideBox":"Learn more about [Autonomous Agents and Multi-Agent Systems](http://link.springer.com/journal/10458)","snPcode":"10458","submissionUrl":"https://submission.nature.com/new-submission/10458/3","title":"Autonomous Agents and Multi-Agent Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"26d98cd2-88c6-416a-ba38-9d2d5b22a03c","owner":[],"postedDate":"March 20th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-03-20T05:58:49+00:00","versionOfRecord":[],"versionCreatedAt":"2026-03-20 05:58:49","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8768097","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8768097","identity":"rs-8768097","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}