A Conditioned Joint-Modality Attention Fusion Approach for Multimodal Aspect-Level Sentiment Analysis

A Conditioned Joint-Modality Attention Fusion Approach for Multimodal Aspect-Level Sentiment Analysis | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A Conditioned Joint-Modality Attention Fusion Approach for Multimodal Aspect-Level Sentiment Analysis Ying Yang, Xinyu Qian, Si Tang, Qinna Zhao This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3166760/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 14 Jun, 2024 Read the published version in The Journal of Supercomputing → Version 1 posted 15 You are reading this latest preprint version Abstract Multimodal aspect-level sentiment analysis (MALSA) aims to predict the sentiment polarity of each given aspect in multimodal contexts. Previous studies usually developed deep neural networks to capture the impacts that a given aspect brings to text and images. However, the dynamic interaction between the intra-modality and inter-modality relations is seldom investigated before fusing the textual and visual representations. This paper presents a conditioned joint-modality attention fusion approach for the MALSA task, which can iteratively delivery useful information flow between and across textual and visual modalities under the guidance of aspect information for sentiment polarity prediction. The point is the dual conditioned-attention mechanism, which calculates intra-modality attention flows dynamically modulated by the other modality. Experiments are conducted on three public datasets including Twitter-2015, Twitter-2017 and Multi-ZOL. Results show that the proposed model outperforms the state-of-the-art models, and demonstrate the effectiveness of the proposed approach. aspect-level sentiment analysis multimodal fusion dynamic attention flow dual conditioned-attention Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 14 Jun, 2024 Read the published version in The Journal of Supercomputing → Version 1 posted Editorial decision: Revision requested 09 May, 2024 Reviews received at journal 07 May, 2024 Reviews received at journal 06 May, 2024 Reviews received at journal 06 May, 2024 Reviews received at journal 22 Apr, 2024 Reviewers agreed at journal 20 Apr, 2024 Reviewers agreed at journal 19 Apr, 2024 Reviewers agreed at journal 19 Apr, 2024 Reviewers agreed at journal 19 Apr, 2024 Reviewers agreed at journal 19 Apr, 2024 Reviewers agreed at journal 19 Apr, 2024 Reviewers invited by journal 19 Apr, 2024 Editor assigned by journal 13 Jul, 2023 Submission checks completed at journal 13 Jul, 2023 First submitted to journal 13 Jul, 2023 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3166760","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":218326668,"identity":"d0858d73-7472-4f9c-b70b-50fdfa2f91d2","order_by":0,"name":"Ying Yang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA60lEQVRIie3OMWvCQBTA8RcObjp0PS1SP8KDg1ioxK9SOcgUakfHQHbnCH6IQKHzwUFdLK4HOghClmaIWwfBJmY/MwreHx7c8H68A3C57jYE6HLvUD+9uDXppQRBtSe1Mm0Jrn/0L/vYB5+7hD6VMB5kiuQHK9m8h68Mc/m115QrCEWm6AitREW+YKilb2bfFdHTTDHKrWRbNESksr5yaUFMJI4VCZBfibpNeqbwvRXqN24kedmgFEtNfSvpbCNRFmc96abSM/N5MFisk9xKhgponwFM4+af1RDbftVzDOT0BzC5sedyuVyP3D/PJ05etAbVNgAAAABJRU5ErkJggg==","orcid":"","institution":"Hefei University of Technology","correspondingAuthor":true,"prefix":"","firstName":"Ying","middleName":"","lastName":"Yang","suffix":""},{"id":218326669,"identity":"38f6f178-88ed-40f2-b198-50b9df91ceb8","order_by":1,"name":"Xinyu Qian","email":"","orcid":"","institution":"Hefei University of Technology","correspondingAuthor":false,"prefix":"","firstName":"Xinyu","middleName":"","lastName":"Qian","suffix":""},{"id":218326670,"identity":"5064a882-09b4-45fa-919b-f9cec53ee684","order_by":2,"name":"Si Tang","email":"","orcid":"","institution":"Hefei University of Technology","correspondingAuthor":false,"prefix":"","firstName":"Si","middleName":"","lastName":"Tang","suffix":""},{"id":218326671,"identity":"85835d12-3850-4e6f-97f1-033658ef2adc","order_by":3,"name":"Qinna Zhao","email":"","orcid":"","institution":"Hefei University of Technology","correspondingAuthor":false,"prefix":"","firstName":"Qinna","middleName":"","lastName":"Zhao","suffix":""}],"badges":[],"createdAt":"2023-07-13 10:18:15","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3166760/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3166760/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s11227-024-06290-8","type":"published","date":"2024-06-14T14:54:16+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":58822439,"identity":"8b465831-6bcf-4caf-8f9a-52a8ba167ec9","added_by":"auto","created_at":"2024-06-21 16:43:37","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":432954,"visible":true,"origin":"","legend":"","description":"","filename":"Manuscript2023.7.3clean.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3166760/v1_covered_00f682f5-3234-48ba-8b9a-f21ba4b16c15.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"A Conditioned Joint-Modality Attention Fusion Approach for Multimodal Aspect-Level Sentiment Analysis","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"the-journal-of-supercomputing","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [The Journal of Supercomputing](https://www.springer.com/journal/11227)","snPcode":"11227","submissionUrl":"https://submission.nature.com/new-submission/11227/3","title":"The Journal of Supercomputing","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"aspect-level sentiment analysis, multimodal fusion, dynamic attention flow, dual conditioned-attention","lastPublishedDoi":"10.21203/rs.3.rs-3166760/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3166760/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eMultimodal aspect-level sentiment analysis (MALSA) aims to predict the sentiment polarity of each given aspect in multimodal contexts. Previous studies usually developed deep neural networks to capture the impacts that a given aspect brings to text and images. However, the dynamic interaction between the intra-modality and inter-modality relations is seldom investigated before fusing the textual and visual representations. This paper presents a conditioned joint-modality attention fusion approach for the MALSA task, which can iteratively delivery useful information flow between and across textual and visual modalities under the guidance of aspect information for sentiment polarity prediction. The point is the dual conditioned-attention mechanism, which calculates intra-modality attention flows dynamically modulated by the other modality. Experiments are conducted on three public datasets including Twitter-2015, Twitter-2017 and Multi-ZOL. Results show that the proposed model outperforms the state-of-the-art models, and demonstrate the effectiveness of the proposed approach.\u003c/p\u003e","manuscriptTitle":"A Conditioned Joint-Modality Attention Fusion Approach for Multimodal Aspect-Level Sentiment Analysis","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2023-07-18 19:33:38","doi":"10.21203/rs.3.rs-3166760/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-05-09T17:09:16+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-05-07T12:56:28+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-05-07T00:41:53+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-05-06T19:58:27+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-04-22T06:36:09+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"0c69b21e-2720-4614-87e1-a74ec47c6446","date":"2024-04-21T01:25:52+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"b6c4ed98-d5fa-437a-b489-634603f74bd7","date":"2024-04-20T02:04:37+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"73d1a3d4-13fe-48b3-a153-d1e124041fd4","date":"2024-04-20T00:14:38+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"80958f85-c95c-430b-83a8-83cbab50d589","date":"2024-04-19T23:48:22+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"63e6ab96-48e9-43fa-b11c-9fac30175d4b","date":"2024-04-19T20:31:48+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"38ab9a18-a404-4711-857b-131206fe2432","date":"2024-04-19T20:11:33+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-04-19T20:08:13+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2023-07-13T13:32:35+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2023-07-13T13:32:34+00:00","index":"","fulltext":""},{"type":"submitted","content":"The Journal of Supercomputing","date":"2023-07-13T10:17:08+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"the-journal-of-supercomputing","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [The Journal of Supercomputing](https://www.springer.com/journal/11227)","snPcode":"11227","submissionUrl":"https://submission.nature.com/new-submission/11227/3","title":"The Journal of Supercomputing","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"d4ecbae3-7fd2-48ed-bcd9-b1db119b5cad","owner":[],"postedDate":"July 18th, 2023","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2024-06-21T14:54:16+00:00","versionOfRecord":{"articleIdentity":"rs-3166760","link":"https://doi.org/10.1007/s11227-024-06290-8","journal":{"identity":"the-journal-of-supercomputing","isVorOnly":false,"title":"The Journal of Supercomputing"},"publishedOn":"2024-06-14 14:54:16","publishedOnDateReadable":"June 14th, 2024"},"versionCreatedAt":"2023-07-18 19:33:38","video":"","vorDoi":"10.1007/s11227-024-06290-8","vorDoiUrl":"https://doi.org/10.1007/s11227-024-06290-8","workflowStages":[]},"version":"v1","identity":"rs-3166760","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3166760","identity":"rs-3166760","version":["v1"]},"buildId":"_2-kVJe1T_tPrBINL-cwx","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}