Bridging the Semantic Gaps: Improving MVQA Consistency with LLM-Augmented Question Sets

Bridging the Semantic Gaps: Improving MVQA Consistency with LLM-Augmented Question Sets | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Bridging the Semantic Gaps: Improving MVQA Consistency with LLM-Augmented Question Sets Yongpei Ma, Pengyu Wang, Zhuoran Duan, Adam G Dunn, Usman Naseem, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6867575/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Purpose: We confront a critical yet under-studied weakness of Medical Visual Question Answering (MVQA): models often flip their answers when clinicians phrase the same diagnostic query differently. We ask whether large-language model-driven data augmentation can deliver paraphrase-proof MVQA. Methods: Our Semantically Equivalent Question Augmentation (SEQA) pipeline prompts a foundation LLM to spin each question into ten meaning preserving variants, enriching linguistic diversity while freezing the linked image and ground-truth answer. Three new diversity indices (ANQI, ANQA, ANQS) quantify dataset breadth, and a joint metric, TAR-SC, scores models on both accuracy and across-paraphrase agreement. Results: Augmenting SLAKE, VQA-RAD and PathVQA multiplied question–answer coverage by ×1.86, ×1.85 and ×1.46, respectively. Fine-tuning three representative backbones—M2I2, MUMC and BiomedGPT—on the enriched data raised mean answer accuracy by 19.4% and TAR-SC by 11.6% versus their original protocols, with gains persisting in zero-shot tests. Conclusion: Injecting linguistically diverse yet semantically tethered questions turns off the “paraphrase trap,” delivering MVQA systems that are markedly more stable, accurate, and thus safer for clinical deployment. Medical Visual Question Answering (MVQA) Semantically Equivalent Question Augmentation (SEQA) Paraphrase Robustness Model Consistency Clinical Artificial Intelligence Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6867575","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":475074672,"identity":"f2263ba7-adc6-4f91-a814-ee891f3bc2ad","order_by":0,"name":"Yongpei Ma","email":"","orcid":"","institution":"The University of Sydney","correspondingAuthor":false,"prefix":"","firstName":"Yongpei","middleName":"","lastName":"Ma","suffix":""},{"id":475074673,"identity":"77d457ed-83d0-4fea-9906-be14582d3a95","order_by":1,"name":"Pengyu Wang","email":"","orcid":"","institution":"The University of Sydney","correspondingAuthor":false,"prefix":"","firstName":"Pengyu","middleName":"","lastName":"Wang","suffix":""},{"id":475074674,"identity":"6df9e405-7170-4b4f-a42e-70491b4711b6","order_by":2,"name":"Zhuoran Duan","email":"","orcid":"","institution":"The University of Sydney","correspondingAuthor":false,"prefix":"","firstName":"Zhuoran","middleName":"","lastName":"Duan","suffix":""},{"id":475074675,"identity":"78efdb6d-67ce-414f-a2df-0d8820c735ff","order_by":3,"name":"Adam G Dunn","email":"","orcid":"","institution":"The University of Sydney","correspondingAuthor":false,"prefix":"","firstName":"Adam","middleName":"G","lastName":"Dunn","suffix":""},{"id":475074676,"identity":"440e3dc6-2e46-464c-a561-7b8656f82dd3","order_by":4,"name":"Usman Naseem","email":"","orcid":"","institution":"Macquarie University","correspondingAuthor":false,"prefix":"","firstName":"Usman","middleName":"","lastName":"Naseem","suffix":""},{"id":475074677,"identity":"cd97489f-ec7a-47b6-a4df-f48bf73a5e2b","order_by":5,"name":"Jinman Kim","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABBElEQVRIiWNgGAWjYBACxmYIbcDGwHyAgYENwpMgUgtbAnFaYMCAgYHHgDgtzO28Rzf83MFgzCd25vNnnrI7cgYHmA/e5mGwS2zA6TC+tJu9ZxjM2KRzt0nznHtmbHCALdmahyEZjxYesxu8bQw2IC3MvG2HEzcc4DGT5mFgxqvl5l+wlpzHn4Fa6jcc4P8G1FKPV8ttoC1Ah+UwSAO1JBgc4GEDajmMX4tsm4Qxm3SameScc4cNZx5mM7acY3DcGJcWw/4zZjffttkYzp+d/PjDm7LD8nzHmx/eeFNRLYtTC0QCEhFMPEBC4TCIaYBDPRDIo7jyB0gEl+mjYBSMglEwYgEAeClReWajyscAAAAASUVORK5CYII=","orcid":"","institution":"The University of Sydney","correspondingAuthor":true,"prefix":"","firstName":"Jinman","middleName":"","lastName":"Kim","suffix":""}],"badges":[],"createdAt":"2025-06-11 03:53:24","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6867575/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6867575/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":88121962,"identity":"32377add-5486-43e5-a019-28df31558e65","added_by":"auto","created_at":"2025-08-01 16:01:51","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1484916,"visible":true,"origin":"","legend":"","description":"","filename":"MultimediaSystemsBridgingtheSemanticGapsImprovingMVQAConsistencywithLLMAugmentedQuestionSets.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6867575/v1_covered_531b690d-c085-4c02-8918-b141b0479a79.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Bridging the Semantic Gaps: Improving MVQA Consistency with LLM-Augmented Question Sets","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Medical Visual Question Answering (MVQA), Semantically Equivalent Question Augmentation (SEQA), Paraphrase Robustness, Model Consistency, Clinical Artificial Intelligence","lastPublishedDoi":"10.21203/rs.3.rs-6867575/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6867575/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003ePurpose:\u003c/strong\u003e We confront a critical yet under-studied weakness of Medical Visual Question Answering (MVQA): models often flip their answers when clinicians phrase the same diagnostic query differently. We ask whether large-language model-driven data augmentation can deliver paraphrase-proof MVQA.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods: \u003c/strong\u003eOur Semantically Equivalent Question Augmentation (SEQA) pipeline prompts a foundation LLM to spin each question into ten meaning preserving variants, enriching linguistic diversity while freezing the linked image and ground-truth answer. Three new diversity indices (ANQI, ANQA, ANQS) quantify dataset breadth, and a joint metric, TAR-SC, scores models on both accuracy and across-paraphrase agreement.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults:\u003c/strong\u003e Augmenting SLAKE, VQA-RAD and PathVQA multiplied question–answer coverage by ×1.86, ×1.85 and ×1.46, respectively. Fine-tuning three representative backbones—M2I2, MUMC and BiomedGPT—on the\u003c/p\u003e\n\u003cp\u003eenriched data raised mean answer accuracy by 19.4% and TAR-SC by 11.6% versus their original protocols, with gains persisting in zero-shot tests.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusion:\u003c/strong\u003e Injecting linguistically diverse yet semantically tethered questions turns off the “paraphrase trap,” delivering MVQA systems that are markedly more stable, accurate, and thus safer for clinical deployment.\u003c/p\u003e","manuscriptTitle":"Bridging the Semantic Gaps: Improving MVQA Consistency with LLM-Augmented Question Sets","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-06-24 03:12:06","doi":"10.21203/rs.3.rs-6867575/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"aafd17d2-73ab-407a-84c6-c307a313a522","owner":[],"postedDate":"June 24th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-08-01T15:53:43+00:00","versionOfRecord":[],"versionCreatedAt":"2025-06-24 03:12:06","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6867575","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6867575","identity":"rs-6867575","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}