A Dynamic Weighted Fusion Model for Multimodal Sentiment Analysis

A Dynamic Weighted Fusion Model for Multimodal Sentiment Analysis | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A Dynamic Weighted Fusion Model for Multimodal Sentiment Analysis Liang Yang This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5950230/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 27 May, 2025 Read the published version in Signal, Image and Video Processing → Version 1 posted 8 You are reading this latest preprint version Abstract Multimodal sentiment analysis (MSA) tasks leverage diverse data sources, including text, audio, and visual data, to infer users' sentiment states. Previous research has mainly focused on capturing the differences and consistency of sentiment information between different modalities, emphasizing cross-modal interaction, while neglecting the in-depth exploration of sentiment information within individual modalities. Additionally, existing MSA methods rarely examine the contribution of each modality to model performance. To address these issues, our paper proposes a dynamic weighted fusion model for multimodal sentiment analysis. Specifically, we first design a multi-level semantic enhancement module (MLSE) for each single mode, which replicates three copies of each mode, captures local and global emotional information using convolutional neural networks and attention mechanisms, and aims to extract semantic information from multiple perspectives and levels in a single mode. Subsequently, we design a genetic algorithm module suitable for multimodal sentiment analysis tasks, which dynamically calculates the optimal weight of each modality during model training and selects the modality with the maximum weight as the primary modality, while the other two are considered as auxiliary modalities. We conduct extensive experiments on three benchmark datasets (CMU-MOSI, CMU-MOSEI and CH-SIMS), and the results demonstrate that our proposed model outperforms state-of-the-art models across various metrics. Multimodal sentiment analysis Multimodal interaction Deep learning Modalities fusion Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 27 May, 2025 Read the published version in Signal, Image and Video Processing → Version 1 posted Editorial decision: Revision requested 28 Feb, 2025 Reviews received at journal 27 Feb, 2025 Reviewers agreed at journal 24 Feb, 2025 Reviewers agreed at journal 24 Feb, 2025 Reviewers invited by journal 19 Feb, 2025 Editor assigned by journal 04 Feb, 2025 Submission checks completed at journal 04 Feb, 2025 First submitted to journal 03 Feb, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5950230","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":410935649,"identity":"5d3f415c-eff1-4966-a3d4-22c7d153eac3","order_by":0,"name":"Liang Yang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABEUlEQVRIiWNgGAWjYLCCBwZAgpn58IOECiBDghgtCSAt7GxpBg/OwLQkENICIvh5DCQfthGhxeD42cMvEgrs8uSdGQwMEucdluOf3cD24OMPPFrO5KVZJBgkFxseZkh4kLjtsLHEnQPshjPw2XIgx8wgwYA5cWMzwwEDoJbEhhsJbNI8+LScfwPSUg/UwtggkTjncP18kJY/+LTcyDF+kGBwOHE+MzODRGLD4QQDkBZ83pe88cYMGMjHEzcws7EZJBxLN9x4I7FNsicNtxa+8znGHz78qU6c33/+88MfNdbycjeSj0n8sMGtReEAA5sEJBzA/GYgZmzArR4I5BsYmD9AGSBQh1f1KBgFo2AUjEwAAFPPWu+ICzVmAAAAAElFTkSuQmCC","orcid":"","institution":"Guangxi Normal University","correspondingAuthor":true,"prefix":"","firstName":"Liang","middleName":"","lastName":"Yang","suffix":""}],"badges":[],"createdAt":"2025-02-03 10:23:33","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5950230/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5950230/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s11760-025-04201-9","type":"published","date":"2025-05-27T15:57:59+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":83783535,"identity":"1c7987e8-960f-4df3-b5a6-880307ca758b","added_by":"auto","created_at":"2025-06-02 16:11:34","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":666432,"visible":true,"origin":"","legend":"","description":"","filename":"Manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5950230/v1_covered_2caaed06-3f4e-47ad-910b-f8468f89d088.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"A Dynamic Weighted Fusion Model for Multimodal Sentiment Analysis","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"signal-image-and-video-processing","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"sivp","sideBox":"Learn more about [Signal, Image and Video Processing](http://link.springer.com/journal/11760)","snPcode":"11760","submissionUrl":"https://submission.nature.com/new-submission/11760/3","title":"Signal, Image and Video Processing","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Multimodal sentiment analysis, Multimodal interaction, Deep learning, Modalities fusion","lastPublishedDoi":"10.21203/rs.3.rs-5950230/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5950230/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eMultimodal sentiment analysis (MSA) tasks leverage diverse data sources, including text, audio, and visual data, to infer users' sentiment states. Previous research has mainly focused on capturing the differences and consistency of sentiment information between different modalities, emphasizing cross-modal interaction, while neglecting the in-depth exploration of sentiment information within individual modalities. Additionally, existing MSA methods rarely examine the contribution of each modality to model performance. To address these issues, our paper proposes a dynamic weighted fusion model for multimodal sentiment analysis. Specifically, we first design a multi-level semantic enhancement module (MLSE) for each single mode, which replicates three copies of each mode, captures local and global emotional information using convolutional neural networks and attention mechanisms, and aims to extract semantic information from multiple perspectives and levels in a single mode. Subsequently, we design a genetic algorithm module suitable for multimodal sentiment analysis tasks, which dynamically calculates the optimal weight of each modality during model training and selects the modality with the maximum weight as the primary modality, while the other two are considered as auxiliary modalities. We conduct extensive experiments on three benchmark datasets (CMU-MOSI, CMU-MOSEI and CH-SIMS), and the results demonstrate that our proposed model outperforms state-of-the-art models across various metrics.\u003c/p\u003e","manuscriptTitle":"A Dynamic Weighted Fusion Model for Multimodal Sentiment Analysis","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-02-06 06:39:55","doi":"10.21203/rs.3.rs-5950230/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-02-28T18:13:49+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-02-28T04:26:37+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"82934065354087755061884970448431090052","date":"2025-02-24T12:02:24+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"206416105183569432049194408574556242314","date":"2025-02-24T07:05:35+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-02-19T06:57:28+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-02-04T08:40:06+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-02-04T08:39:45+00:00","index":"","fulltext":""},{"type":"submitted","content":"Signal, Image and Video Processing","date":"2025-02-03T10:18:34+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"signal-image-and-video-processing","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"sivp","sideBox":"Learn more about [Signal, Image and Video Processing](http://link.springer.com/journal/11760)","snPcode":"11760","submissionUrl":"https://submission.nature.com/new-submission/11760/3","title":"Signal, Image and Video Processing","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"8539a008-6bd8-4eab-b491-79fef3b29032","owner":[],"postedDate":"February 6th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2025-06-02T16:09:31+00:00","versionOfRecord":{"articleIdentity":"rs-5950230","link":"https://doi.org/10.1007/s11760-025-04201-9","journal":{"identity":"signal-image-and-video-processing","isVorOnly":false,"title":"Signal, Image and Video Processing"},"publishedOn":"2025-05-27 15:57:59","publishedOnDateReadable":"May 27th, 2025"},"versionCreatedAt":"2025-02-06 06:39:55","video":"","vorDoi":"10.1007/s11760-025-04201-9","vorDoiUrl":"https://doi.org/10.1007/s11760-025-04201-9","workflowStages":[]},"version":"v1","identity":"rs-5950230","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5950230","identity":"rs-5950230","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}