Method of Multi-Label Visual Emotion Recognition Fusing Fore-Background Features

Method of Multi-Label Visual Emotion Recognition Fusing Fore-Background Features | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Method of Multi-Label Visual Emotion Recognition Fusing Fore-Background Features Yuehua Feng, Ruoyan Wei This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4752870/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract To address the issues that visual-based multi-label emotion recognition often overlooks the impact of background that the person is placed and foreground such as social interactions among different individuals on emotion recognition, simplifies multi-label recognition tasks into multiple binary classification tasks, and ignores the global correlations among different emotion labels, this paper proposes a method of multi-label visual emotion recognition fusing fore-background features. This method consists of two components: Fore-Background-Aware Emotion Recognition model (FB-ER) and Multi-Label Emotion Recognition Classifier model (ML-ERC). FB-ER is a three-branch multi-feature hybrid fusion network. It effectively extracts body features through the design of a Core Region Unit (CR-Unit), represents background features as background keywords, and extracts depth map information to simulate social interactions among different individuals as foreground features. These three features are fused both at the feature level and the decision level. ML-ERC captures the relationships among different emotion labels by designing label co-occurrence probability matrix and cosine similarity matrix, and utilizing Graph Convolutional Networks (GCN) to learn the correlations between different emotion labels, thereby generating a classifier that accounts for emotion relevance. Finally, the visual features are combined with the object classifier to enable multi-label recognition of 26 different emotions. The proposed method is evaluated on the Emotic dataset, demonstrating significant improvements over the state-of-the-art methods. Specifically, the mAP increased by 0.732% and the Jaccard Coefficient increased by 0.007. Multi-Label Emotion Recognition Fore-Background Features GCN Correlation Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4752870","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":331658422,"identity":"aafa786a-08f3-4d2d-aa56-04dd3680e264","order_by":0,"name":"Yuehua Feng","email":"","orcid":"","institution":"School of Management Science and Information Engineering，Hebei University of Economics and Business","correspondingAuthor":false,"prefix":"","firstName":"Yuehua","middleName":"","lastName":"Feng","suffix":""},{"id":331658423,"identity":"f4747c9c-1cc8-4e0c-85b9-070d37974d72","order_by":1,"name":"Ruoyan Wei","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA7klEQVRIiWNgGAWjYFACxgYIzcx88IFEhYScPPFa2NmSDSzOWBgbNhBtGz+PmURlW0UiwwECCg2ON7d95qk4LG/OzGNscHOeRAJjA/PDRzfwaTlzsHk2z5nDhjub2QofztwmkcfOwGZsnINHi9mNxGZm3rbDjBsOM282ltwmUczYwMMmjVfL/YdgLfYbDjOYSf+dI5HYcICQlhuMYC2JGw6zmElINhChxf5MYjPjnDPpyRsOAwNZ4piEsWEzAb9Ith9/zPCmwtp2w/nDwKisqZOTZ29++BifFihoRmIzE1YOAnXEKRsFo2AUjIKRCQApHU0gslmSUgAAAABJRU5ErkJggg==","orcid":"","institution":"School of Management Science and Information Engineering，Hebei University of Economics and Business","correspondingAuthor":true,"prefix":"","firstName":"Ruoyan","middleName":"","lastName":"Wei","suffix":""}],"badges":[],"createdAt":"2024-07-17 01:57:42","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4752870/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4752870/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":62313413,"identity":"db057e55-73b8-48aa-a68a-a6811c01b1f7","added_by":"auto","created_at":"2024-08-12 21:56:27","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":3104676,"visible":true,"origin":"","legend":"","description":"","filename":"Manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4752870/v1_covered_407eed50-b1fa-4eca-a950-c2c012aba293.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Method of Multi-Label Visual Emotion Recognition Fusing Fore-Background Features","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Multi-Label, Emotion Recognition, Fore-Background Features, GCN, Correlation","lastPublishedDoi":"10.21203/rs.3.rs-4752870/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4752870/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"To address the issues that visual-based multi-label emotion recognition often overlooks the impact of background that the person is placed and foreground such as social interactions among different individuals on emotion recognition, simplifies multi-label recognition tasks into multiple binary classification tasks, and ignores the global correlations among different emotion labels, this paper proposes a method of multi-label visual emotion recognition fusing fore-background features. This method consists of two components: Fore-Background-Aware Emotion Recognition model (FB-ER) and Multi-Label Emotion Recognition Classifier model (ML-ERC). FB-ER is a three-branch multi-feature hybrid fusion network. It effectively extracts body features through the design of a Core Region Unit (CR-Unit), represents background features as background keywords, and extracts depth map information to simulate social interactions among different individuals as foreground features. These three features are fused both at the feature level and the decision level. ML-ERC captures the relationships among different emotion labels by designing label co-occurrence probability matrix and cosine similarity matrix, and utilizing Graph Convolutional Networks (GCN) to learn the correlations between different emotion labels, thereby generating a classifier that accounts for emotion relevance. Finally, the visual features are combined with the object classifier to enable multi-label recognition of 26 different emotions. The proposed method is evaluated on the Emotic dataset, demonstrating significant improvements over the state-of-the-art methods. Specifically, the mAP increased by 0.732% and the Jaccard Coefficient increased by 0.007.","manuscriptTitle":"Method of Multi-Label Visual Emotion Recognition Fusing Fore-Background Features","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-08-09 18:25:33","doi":"10.21203/rs.3.rs-4752870/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"7135a87d-3edf-4a35-a53f-35b96b90af9a","owner":[],"postedDate":"August 9th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-08-12T21:48:18+00:00","versionOfRecord":[],"versionCreatedAt":"2024-08-09 18:25:33","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4752870","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4752870","identity":"rs-4752870","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}