GFSNet: Gaussian Fourier with Sparse Attention Network for Visual Question Answering

GFSNet: Gaussian Fourier with Sparse Attention Network for Visual Question Answering | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article GFSNet: Gaussian Fourier with Sparse Attention Network for Visual Question Answering Xiang Shen, Dezhi Han, Chin-Chen Chang Chang, Ammar Oad, Huafeng Wu This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3852848/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 15 Mar, 2025 Read the published version in Artificial Intelligence Review → Version 1 posted 11 You are reading this latest preprint version Abstract A profound understanding and reasoning of the relationship between images and question are crucial in Visual Question Answering (VQA) tasks. However, traditional self-attention mechanisms exhibit limitations , primarily confined to spatial domain modeling of images, lacking 20 the capability to adequately model and analyze visual information at different scales in the frequency domain. Additionally, the traditional self-attention-based image feature modeling introduces noise when capturing long-distance dependencies, causing the model to overly focus on irrelevant details, thereby reducing robustness. To address these issues, 25 this paper proposes a novel Gaussian Fourier with Sparse Attention Network (GFSNet). GFSNet utilizes Fourier transform techniques to represent image attention weights obtained through self-attention in the frequency domain, facilitating the effective modeling of different scale information by analyzing attention weights in the frequency domain. 30 Recognizing that different scale information in images often manifests as distinct frequency components, the model can better capture and 1 Springer Nature 2021 L A T E X template Gaussian Fourier with Sparse Attention Network for VQA adapt to the complex structures and correlations of these various scale details. To mitigate high-frequency noise in the frequency domain, we design an adaptive Gaussian filter to effectively suppress or filter noise in 35 the images. Finally, a novel sparse attention mechanism is introduced to select optimized key frequency domain features. This enables the model to more effectively focus on critical image regions, reducing the processing of irrelevant or redundant information, while enhancing interpretability and robustness. The proposed GFSNet model aims to achieve effective 40 modeling of visual information at different scales without increasing model parameters or altering computational complexity. Extensive experiments on the VQAv2 and GQA benchmark datasets unequivocally demonstrate the superiority and effectiveness of the GFSNet approach. Source code is available at https://github.com/shenxiang-vqa/GFSNet . Visual question answering Adaptive fusion Visual relationship modeling Attention mechanisms Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 15 Mar, 2025 Read the published version in Artificial Intelligence Review → Version 1 posted Editorial decision: Revision requested 17 Nov, 2024 Reviews received at journal 25 Oct, 2024 Reviewers agreed at journal 19 Sep, 2024 Reviewers agreed at journal 19 Sep, 2024 Reviewers agreed at journal 18 Sep, 2024 Reviews received at journal 20 Feb, 2024 Reviewers agreed at journal 07 Feb, 2024 Reviewers invited by journal 07 Feb, 2024 Editor assigned by journal 17 Jan, 2024 Submission checks completed at journal 11 Jan, 2024 First submitted to journal 11 Jan, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3852848","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":266581875,"identity":"ed008dd9-0f07-4ee7-a4d4-c0a114b491af","order_by":0,"name":"Xiang Shen","email":"","orcid":"","institution":"Shanghai Maritime University","correspondingAuthor":false,"prefix":"","firstName":"Xiang","middleName":"","lastName":"Shen","suffix":""},{"id":266581876,"identity":"f74eaab2-b2fe-4473-8d28-230b917743bb","order_by":1,"name":"Dezhi Han","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAxUlEQVRIiWNgGAWjYDCCAzxAooIhAcJjI1rLGaAWNpK0MLaRooXv/NmDnwvn1eXpzu8xYPhQdpiBf3YDfi2SB84lS8/cdrjY7BiPAeOMc4cZJO4cwK/F4GCPgTTvtgOJ24BamHnbDjMYSCQQ0HKYx/g375w6iJa/RGk5xmMmzdvADNHCSIwWyTM8ZtY8xw4DtaQVHOw5l84jcYOAFr7zZ4xv89QAHXb48MYHP8qs5fhnENCCAg4AMQ8J6kfBKBgFo2AU4AIAuH5EIzVCeuYAAAAASUVORK5CYII=","orcid":"","institution":"Shanghai Maritime University","correspondingAuthor":true,"prefix":"","firstName":"Dezhi","middleName":"","lastName":"Han","suffix":""},{"id":266581877,"identity":"9f1e0d90-0972-41a2-823e-56ed2cb63eb6","order_by":2,"name":"Chin-Chen Chang Chang","email":"","orcid":"","institution":"Feng Chia University","correspondingAuthor":false,"prefix":"","firstName":"Chin-Chen","middleName":"Chang","lastName":"Chang","suffix":""},{"id":266581878,"identity":"cff7673d-75b7-4942-83c4-463b0b1af3f4","order_by":3,"name":"Ammar Oad","email":"","orcid":"","institution":"Shaoyang University","correspondingAuthor":false,"prefix":"","firstName":"Ammar","middleName":"","lastName":"Oad","suffix":""},{"id":266581879,"identity":"f4778adb-cbab-4954-98a3-0cd29ebdb5c9","order_by":4,"name":"Huafeng Wu","email":"","orcid":"","institution":"Shanghai Maritime University","correspondingAuthor":false,"prefix":"","firstName":"Huafeng","middleName":"","lastName":"Wu","suffix":""}],"badges":[],"createdAt":"2024-01-11 08:44:11","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3852848/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3852848/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s10462-025-11163-4","type":"published","date":"2025-03-15T15:57:59+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":78689174,"identity":"5e9cd1bb-334d-4be9-ad7e-264a9d5ff341","added_by":"auto","created_at":"2025-03-17 16:12:15","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":3204007,"visible":true,"origin":"","legend":"","description":"","filename":"Manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3852848/v1_covered_64ae8b9d-b78d-4195-bfe4-ae319370560e.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"GFSNet: Gaussian Fourier with Sparse Attention Network for Visual Question Answering","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"artificial-intelligence-review","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"aire","sideBox":"Learn more about [Artificial Intelligence Review](http://link.springer.com/journal/10462)","snPcode":"10462","submissionUrl":"https://submission.nature.com/new-submission/10462/3","title":"Artificial Intelligence Review","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Visual question answering, Adaptive fusion, Visual relationship modeling, Attention mechanisms","lastPublishedDoi":"10.21203/rs.3.rs-3852848/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3852848/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"A profound understanding and reasoning of the relationship between images and question are crucial in Visual Question Answering (VQA) tasks. However, traditional self-attention mechanisms exhibit limitations , primarily confined to spatial domain modeling of images, lacking 20 the capability to adequately model and analyze visual information at different scales in the frequency domain. Additionally, the traditional self-attention-based image feature modeling introduces noise when capturing long-distance dependencies, causing the model to overly focus on irrelevant details, thereby reducing robustness. To address these issues, 25 this paper proposes a novel Gaussian Fourier with Sparse Attention Network (GFSNet). GFSNet utilizes Fourier transform techniques to represent image attention weights obtained through self-attention in the frequency domain, facilitating the effective modeling of different scale information by analyzing attention weights in the frequency domain. 30 Recognizing that different scale information in images often manifests as distinct frequency components, the model can better capture and 1 Springer Nature 2021 L A T E X template Gaussian Fourier with Sparse Attention Network for VQA adapt to the complex structures and correlations of these various scale details. To mitigate high-frequency noise in the frequency domain, we design an adaptive Gaussian filter to effectively suppress or filter noise in 35 the images. Finally, a novel sparse attention mechanism is introduced to select optimized key frequency domain features. This enables the model to more effectively focus on critical image regions, reducing the processing of irrelevant or redundant information, while enhancing interpretability and robustness. The proposed GFSNet model aims to achieve effective 40 modeling of visual information at different scales without increasing model parameters or altering computational complexity. Extensive experiments on the VQAv2 and GQA benchmark datasets unequivocally demonstrate the superiority and effectiveness of the GFSNet approach. Source code is available at https://github.com/shenxiang-vqa/GFSNet.","manuscriptTitle":"GFSNet: Gaussian Fourier with Sparse Attention Network for Visual Question Answering","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-01-15 02:42:40","doi":"10.21203/rs.3.rs-3852848/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-11-17T23:29:43+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-10-25T12:10:34+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"61470471362512791285427092447636573969","date":"2024-09-20T01:00:08+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"111257784390001988179240813966221128342","date":"2024-09-19T12:01:41+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"207013816105486532246100996756992866131","date":"2024-09-18T07:18:07+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-02-20T09:21:26+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"10e2da41-79fe-4db1-98d3-4845b3e392cd","date":"2024-02-07T18:00:20+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-02-07T15:33:33+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-01-17T08:54:20+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-01-11T23:53:07+00:00","index":"","fulltext":""},{"type":"submitted","content":"Artificial Intelligence Review","date":"2024-01-11T08:29:46+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"artificial-intelligence-review","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"aire","sideBox":"Learn more about [Artificial Intelligence Review](http://link.springer.com/journal/10462)","snPcode":"10462","submissionUrl":"https://submission.nature.com/new-submission/10462/3","title":"Artificial Intelligence Review","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"132c53dc-963f-4783-aaec-b93392a8a333","owner":[],"postedDate":"January 15th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2025-03-17T16:07:19+00:00","versionOfRecord":{"articleIdentity":"rs-3852848","link":"https://doi.org/10.1007/s10462-025-11163-4","journal":{"identity":"artificial-intelligence-review","isVorOnly":false,"title":"Artificial Intelligence Review"},"publishedOn":"2025-03-15 15:57:59","publishedOnDateReadable":"March 15th, 2025"},"versionCreatedAt":"2024-01-15 02:42:40","video":"","vorDoi":"10.1007/s10462-025-11163-4","vorDoiUrl":"https://doi.org/10.1007/s10462-025-11163-4","workflowStages":[]},"version":"v1","identity":"rs-3852848","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3852848","identity":"rs-3852848","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}