UniC: a Dataset for Emotion Analysis of Videos with Multimodal and Unimodal Labels

UniC: a Dataset for Emotion Analysis of Videos with Multimodal and Unimodal Labels | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article UniC: a Dataset for Emotion Analysis of Videos with Multimodal and Unimodal Labels Quanqi Du, Sofie Labat, Thomas Demeester, Veronique Hoste This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4443808/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 21 May, 2025 Read the published version in Language Resources and Evaluation → Version 1 posted 11 You are reading this latest preprint version Abstract As one of the important features that differentiates humans from machines, emotion is complicated not only in its wide varieties but also in its expression channels, including both verbal and non-verbal language. Different modalities contribute in unique ways to the integrated expression of emotion. However, in most of the existing multimodal datasets, there is only one unified emotion label for the various modalities, ignoring the heterogeneity and complementarity of the different modalities. For instance, the text ''I love the test" may be labelled as love or joy, but if it is expressed with a low dejected tone and a mournful expression, the overall emotion might be more of sadness or even disgust. To bridge this gap, we present in this paper UniC, a novel multimodal emotion dataset featuring both integrated multimodal labels and independent unimodal labels. UniC is comprised of 965 emotion-rich video clips selected from YouTube, annotated in text, audio, (silent) video and multimodal setups with both categorical and dimensional labels. We present the dataset construction steps and an analysis of different modality perspectives based on UniC. It is found that although in most cases the modality of text shares more emotional resemblance with the multimodal setup, other modalities may have different and even opposite emotions which might contribute more to the overall emotion states. This dataset contributes a modality-specific perspective to multimodal emotion analysis, and has the potential to offer more insights for further research in human-machine interaction and emotion modelling for robots. Unimodal Multimodal Text Speech Video Sentiment and Emotion modelling Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 21 May, 2025 Read the published version in Language Resources and Evaluation → Version 1 posted Editorial decision: Revision requested 01 Apr, 2025 Reviewers agreed at journal 03 Feb, 2025 Reviews received at journal 08 Dec, 2024 Reviewers agreed at journal 05 Nov, 2024 Reviews received at journal 22 Oct, 2024 Reviewers agreed at journal 25 Sep, 2024 Reviewers agreed at journal 24 Sep, 2024 Reviewers invited by journal 16 Jun, 2024 Editor assigned by journal 11 Jun, 2024 Submission checks completed at journal 21 May, 2024 First submitted to journal 19 May, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4443808","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":317219863,"identity":"4afb24e4-4a6e-4c11-94fc-07a1678ec27a","order_by":0,"name":"Quanqi Du","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA1UlEQVRIiWNgGAWjYNACAwbGBvYGGJtoLTwHICwitTAAtUgkEKmF70byswdvChhkN9x8Y/zi444/DOYwF+ICkjfSzA3nGDAYb7idY2Y584wBg2XPAfxaDG4kmEnzGDAkgrQY87YZgEQIaUn/BtFy84yZ8V+QlvsPCGnJgdpyg8f4MSPYFvw6GCTPvCmTnGMgYTzzTFoZY2+bMY9lDwGH8R1P3ybx5o+NbN/xw5s//GyTkzNnP0DAGpA8D4MEiMkmAWYTBAcQypg/EFY+CkbBKBgFIxEAAAZ7RT37yDJmAAAAAElFTkSuQmCC","orcid":"","institution":"Ghent University","correspondingAuthor":true,"prefix":"","firstName":"Quanqi","middleName":"","lastName":"Du","suffix":""},{"id":317219864,"identity":"19589718-60cd-4d39-9dec-bf43f7b7c9ea","order_by":1,"name":"Sofie Labat","email":"","orcid":"","institution":"Ghent University","correspondingAuthor":false,"prefix":"","firstName":"Sofie","middleName":"","lastName":"Labat","suffix":""},{"id":317219865,"identity":"99119b8e-c8e9-41d7-9587-5b981b09c89d","order_by":2,"name":"Thomas Demeester","email":"","orcid":"","institution":"Ghent University","correspondingAuthor":false,"prefix":"","firstName":"Thomas","middleName":"","lastName":"Demeester","suffix":""},{"id":317219866,"identity":"416feeab-b4cf-409f-9ce6-39fffc7346f7","order_by":3,"name":"Veronique Hoste","email":"","orcid":"","institution":"Ghent University","correspondingAuthor":false,"prefix":"","firstName":"Veronique","middleName":"","lastName":"Hoste","suffix":""}],"badges":[],"createdAt":"2024-05-19 09:53:23","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4443808/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4443808/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s10579-025-09837-0","type":"published","date":"2025-05-21T15:57:52+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":83460043,"identity":"5f2df499-b7ca-4fa0-aa33-3f98e7b5d34d","added_by":"auto","created_at":"2025-05-26 16:09:27","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":7477801,"visible":true,"origin":"","legend":"","description":"","filename":"LREDufinalNewtemplate2023December.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4443808/v1_covered_7ea7b305-1182-4e5f-885f-07de38dad495.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"UniC: a Dataset for Emotion Analysis of Videos with Multimodal and Unimodal Labels","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"language-resources-and-evaluation","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"lrev","sideBox":"Learn more about [Language Resources and Evaluation](http://link.springer.com/journal/10579)","snPcode":"10579","submissionUrl":"https://submission.nature.com/new-submission/10579/3","title":"Language Resources and Evaluation","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Unimodal, Multimodal, Text, Speech, Video, Sentiment and Emotion modelling","lastPublishedDoi":"10.21203/rs.3.rs-4443808/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4443808/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"As one of the important features that differentiates humans from machines, emotion is complicated not only in its wide varieties but also in its expression channels, including both verbal and non-verbal language. Different modalities contribute in unique ways to the integrated expression of emotion. However, in most of the existing multimodal datasets, there is only one unified emotion label for the various modalities, ignoring the heterogeneity and complementarity of the different modalities. For instance, the text ''I love the test\" may be labelled as love or joy, but if it is expressed with a low dejected tone and a mournful expression, the overall emotion might be more of sadness or even disgust. To bridge this gap, we present in this paper UniC, a novel multimodal emotion dataset featuring both integrated multimodal labels and independent unimodal labels. UniC is comprised of 965 emotion-rich video clips selected from YouTube, annotated in text, audio, (silent) video and multimodal setups with both categorical and dimensional labels. We present the dataset construction steps and an analysis of different modality perspectives based on UniC. It is found that although in most cases the modality of text shares more emotional resemblance with the multimodal setup, other modalities may have different and even opposite emotions which might contribute more to the overall emotion states. This dataset contributes a modality-specific perspective to multimodal emotion analysis, and has the potential to offer more insights for further research in human-machine interaction and emotion modelling for robots.","manuscriptTitle":"UniC: a Dataset for Emotion Analysis of Videos with Multimodal and Unimodal Labels","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-06-27 02:26:08","doi":"10.21203/rs.3.rs-4443808/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-04-01T12:08:07+00:00","index":"","fulltext":""},{"type":"reviewerAgreed","content":"328747127953108607820528550824187018795","date":"2025-02-03T09:15:10+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-12-08T18:56:36+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"326213785917353685182697671316140645114","date":"2024-11-05T14:57:45+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-10-22T16:39:38+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"62879740980443009669092652323827046893","date":"2024-09-25T15:30:08+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"292546983444070398624086970794965054269","date":"2024-09-24T15:29:44+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-06-16T08:26:04+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-06-11T10:30:44+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-05-21T12:08:14+00:00","index":"","fulltext":""},{"type":"submitted","content":"Language Resources and Evaluation","date":"2024-05-19T09:49:59+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"language-resources-and-evaluation","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"lrev","sideBox":"Learn more about [Language Resources and Evaluation](http://link.springer.com/journal/10579)","snPcode":"10579","submissionUrl":"https://submission.nature.com/new-submission/10579/3","title":"Language Resources and Evaluation","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"86fcced5-977e-4d2e-95be-90c4ba5d07e7","owner":[],"postedDate":"June 27th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2025-05-26T16:02:20+00:00","versionOfRecord":{"articleIdentity":"rs-4443808","link":"https://doi.org/10.1007/s10579-025-09837-0","journal":{"identity":"language-resources-and-evaluation","isVorOnly":false,"title":"Language Resources and Evaluation"},"publishedOn":"2025-05-21 15:57:52","publishedOnDateReadable":"May 21st, 2025"},"versionCreatedAt":"2024-06-27 02:26:08","video":"","vorDoi":"10.1007/s10579-025-09837-0","vorDoiUrl":"https://doi.org/10.1007/s10579-025-09837-0","workflowStages":[]},"version":"v1","identity":"rs-4443808","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4443808","identity":"rs-4443808","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}