CIME: Contextual Interaction-based Multimodal Emotion Analysis with Enhanced Semantic Information

CIME: Contextual Interaction-based Multimodal Emotion Analysis with Enhanced Semantic Information | Authorea try { document.documentElement.classList.add('js'); } catch (e) { } var _gaq = _gaq || []; _gaq.push(['_setAccount', 'G-8VDV14Y67G']); _gaq.push(['_trackPageview']); (function() { var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); })(); Skip to main content Preprints Collections Wiley Open Research IET Open Research Ecological Society of Japan All Collections About About Authorea FAQs Contact Us Quick Search anywhere Search for preprint articles, keywords, etc. Search Search ADVANCED SEARCH SCROLL This is a preprint and has not been peer reviewed. Data may be preliminary. 22 January 2025 V1 Latest version Share on CIME: Contextual Interaction-based Multimodal Emotion Analysis with Enhanced Semantic Information Authors : Rui Wang , Chaopeng Guo 0009-0002-6764-5025 , Erik Cambria , Imad Rida , Haochen Yuan , Md. Jalil Piran 0000-0003-3229-6785 , Yichen Feng , and Xianxun Zhu 0000-0003-3958-7040 [email protected] Authors Info & Affiliations https://doi.org/10.22541/au.173750886.60448227/v1 Published IEEE Transactions on Computational Social Systems Version of record Peer review timeline 665 views 488 downloads Contents Abstract Supplementary Material Information & Authors Metrics & Citations View Options References Figures Tables Media Share Abstract In the rapidly expanding domain of multimodal data, the field of emotion analysis has advanced through the sophisticated integration of diverse informational modalities. This study introduces the CIME model: C ontextual I nteraction-Based M ultimodal E motion Analysis with Enhanced Semantic Information. This innovative spatiotemporal interaction network model utilizes enhanced semantic information to elevate the accuracy and robustness of emotion analysis across both semantic and contextual dimensions. The model incorporates attention mechanisms and graph convolutional networks to enrich textual semantic comprehension through a cross-attention-based semantic interaction module and to delineate the contextual relationships among speakers via a graph convolution-based spatial interaction module. These enhancements enable the model to effectively mine the latent associations within multimodal emotional data. Through extensive evaluations on public datasets such as IEMOCAP and MOSEI, the proposed CIME model demonstrates superior performance in multimodal emotion classification tasks compared to existing methods. Further, modality ablation studies and comparative analysis of various fusion strategies affirm the model’s effectiveness and adaptability, providing new insights and methodologies for advancing the field of multimodal emotion analysis. Code supporting this study is available at https://github.com/gcp666/CIME. Supplementary Material File (gcpg.pdf) Download 2.87 MB Information & Authors Information Version history V1 Version 1 22 January 2025 Peer review timeline Published IEEE Transactions on Computational Social Systems Version of Record 1 Jan 2025 Published Copyright This work is licensed under a Non Exclusive No Reuse License. Keywords attention mechanisms contextual information graph convolutional networks multimodal emotion analysis Authors Affiliations Rui Wang Shanghai University View all articles by this author Chaopeng Guo 0009-0002-6764-5025 Shanghai University View all articles by this author Erik Cambria Nanyang Technological University View all articles by this author Imad Rida Mairie de Compiegne View all articles by this author Haochen Yuan Harbin Institute of Technology View all articles by this author Md. Jalil Piran 0000-0003-3229-6785 Sejong University View all articles by this author Yichen Feng Shanghai University View all articles by this author Xianxun Zhu 0000-0003-3958-7040 [email protected] Shanghai University View all articles by this author Metrics & Citations Metrics Article Usage 665 views 488 downloads .FvxKWukQNSOunydq8rnd { width: 100px; } Citations Download citation Rui Wang, Chaopeng Guo, Erik Cambria, et al. CIME: Contextual Interaction-based Multimodal Emotion Analysis with Enhanced Semantic Information. Authorea . 22 January 2025. DOI: https://doi.org/10.22541/au.173750886.60448227/v1 If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download. For more information or tips please see 'Downloading to a citation manager' in the Help menu . Format Please select one from the list RIS (ProCite, Reference Manager) EndNote BibTex Medlars RefWorks Direct import Tips for downloading citations document.getElementById('citMgrHelpLink').addEventListener('click', function() { popupHelp(this.href); return false; }); $(".js__slcInclude").on("change", function(e){ if ($(this).val() == 'refworks') $('#direct').prop("checked", false); $('#direct').prop("disabled", ($(this).val() == 'refworks')); }); Cited by Xianghui Pan, Computer vision enhanced emotion recognition system for real time participation analysis in virtual english classroom, Discover Artificial Intelligence, 6 , 1, (2026). https://doi.org/10.1007/s44163-026-01193-4 Crossref Dang Wanli, Cheng Jian, Luo Qian, Zheng Huaiyu, Robust multi-camera tracking in terminal environments: a spatio-temporal-appearance fusion approach, The Visual Computer, 42 , 6, (2026). https://doi.org/10.1007/s00371-026-04416-y Crossref Mingming Wu, Construction of music emotion recognition and classification model supported by neural networks, Journal on Audio, Speech, and Music Processing, 2026 , 1, (2026). https://doi.org/10.1186/s13636-026-00453-6 Crossref Sangeetha J, Maria Anu V, An efficient multimodal deep learning model for emotion recognition in social media videos, Discover Computing, 29 , 1, (2026). https://doi.org/10.1007/s10791-026-09914-0 Crossref Hanbo Zang, Zhiqiang Chen, Deep reinforcement learning for adaptive music emotion recognition and generation, Discover Artificial Intelligence, 6 , 1, (2026). https://doi.org/10.1007/s44163-026-00968-z Crossref Vijayalaxmi N. Rathod, R. H. Goudar, Sangeeta Sangani, Unified interpretable AI for autism diagnosis and scalable severity-aware personalized adaptive e-learning, Discover Applied Sciences, 8 , 4, (2026). https://doi.org/10.1007/s42452-026-08335-4 Crossref Kanishka Bhatt, Ashwini Kumar Singh, Priyank Pandey, Ankit Vishnoi, Preeti Narooka, Cross platform social media analysis for mental health detection, Discover Mental Health, 6 , 1, (2026). https://doi.org/10.1007/s44192-026-00368-w Crossref Tao Ning, ZhengHua Guo, QiDong Hou, A DLF multi-scale quantitative research method for the big five personality traits, Scientific Reports, 16 , 1, (2026). https://doi.org/10.1038/s41598-025-27837-6 Crossref Uday Shankar Yadavalli, Somya Ranjan Sahoo, A multi-granular hybrid neural architecture for detecting abusive content in online social networks (OSNs) with contextual awareness, Journal of Big Data, 13 , 1, (2025). https://doi.org/10.1186/s40537-025-01343-y Crossref S K B Sangeetha, Raja Sarath Kumar Boddu, Amiya Bhaumik, Sandeep Kumar Mathivanan, Usha Moorthy, Spatiotemporal multimodal emotion recognition using Temporal video sequences and pose features for child emotion classification, Scientific Reports, 15 , 1, (2025). https://doi.org/10.1038/s41598-025-25813-8 Crossref See more Loading... View Options View options PDF View PDF Figures Tables Media Share Share Share article link Copy Link Copied! Copying failed. Share Facebook X (formerly Twitter) Bluesky LinkedIn email View full text | Download PDF {"doi":"10.22541/au.173750886.60448227/v1","type":"Article"} Now Reading: Share Figures Tables Close figure viewer Back to article Figure title goes here Change zoom level Go to figure location within the article Download figure Toggle share panel Toggle share panel Share Toggle information panel Toggle information panel Go to previous graphic Go to next graphic Go to previous table Go to next table All figures All tables View all material View all material xrefBack.goTo xrefBack.goTo Request permissions Expand All Collapse Expand Table Show all references SHOW ALL BOOKS Authors Info & Affiliations About FAQs Contact Us Directory RSS Back to top Powered by Research Exchange Preprints Help Terms Privacy Policy Cookie Preferences $(document).ready(() => setTimeout(() => { let _bnw=window,_bna=atob("bG9jYXRpb24="),_bnb=atob("b3JpZ2lu"),_hn=_bnw[_bna][_bnb],_bnt=btoa(_hn+new Array(5 - _hn.length % 4).join(" ")); $.get("/resource/lodash?t="+_bnt); },4000)); (function(){function c(){var b=a.contentDocument||a.contentWindow.document;if(b){var d=b.createElement('script');d.innerHTML="window.__CF$cv$params={r:'9ff5b2a05d50f047',t:'MTc3OTM4ODcwMg=='};var a=document.createElement('script');a.src='/cdn-cgi/challenge-platform/scripts/jsd/main.js';document.getElementsByTagName('head')[0].appendChild(a);";b.getElementsByTagName('head')[0].appendChild(d)}}if(document.body){var a=document.createElement('iframe');a.height=1;a.width=1;a.style.position='absolute';a.style.top=0;a.style.left=0;a.style.border='none';a.style.visibility='hidden';document.body.appendChild(a);if('loading'!==document.readyState)c();else if(window.addEventListener)document.addEventListener('DOMContentLoaded',c);else{var e=document.onreadystatechange||function(){};document.onreadystatechange=function(b){e(b);'loading'!==document.readyState&&(document.onreadystatechange=e,c())}}}})();