Modelling Human-Like Emotion Perception Using Multimodal Deep Learning

Modelling Human-Like Emotion Perception Using Multimodal Deep Learning | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Modelling Human-Like Emotion Perception Using Multimodal Deep Learning Aarushi Shanker, Dr. Subhash Chandra Pandey This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9572322/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Unlike humans who can see and hear emotion, computers do not have the capability for innate perception (or the ability to "see" and "hear"), therefore, they must rely on data generated from sensors (such as visual and auditory), to interpret the emotional state of an individual. This research presents a multi-modal system for accurate and reliable classification of emotion, through facial expression and speech; using state of the art techniques (deep learning methods) such as Hybrid CNN-ViT for visual feature extraction and BiLSTM for analysing auditory features, the approach enables real time detection of emotion regardless of environment, speakers, or expressions. Enhanced robustness and performance have been created through systematic preprocessing and feature extraction from the multi-modal sources, followed by late fusion of the multi-modal predictions, to allow for better accuracy of emotion recognition. Additionally, the research supports the use of a multi-modal approach for enhancing emotional accuracy and provides practical implications for many areas of application, such as: adaptive learning, health monitoring, safety for drivers, delivery of personalised content, marketing, surveillance, and human-robot interaction. Artificial Intelligence and Machine Learning Multimodal Deep Learning Human Computer Interaction Emotion Recognition Facial Emotion Recognition Speech Emotion Recognition Full Text Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9572322","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":632170785,"identity":"ad74dd4b-d94d-4abd-95ee-34c7fdf09995","order_by":0,"name":"Aarushi Shanker","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABKUlEQVRIie3RMWvCQBTA8XcE0uWJW0lI0a9wElAES75KQyBToY4ODieBuLVroKWfIZOl25ODdPEbNNCUgrMScFGkiVMKJ5ROHe4PN939uDsegE73D+PVMoAfsQ3Eis2EsLHJhJKYNRlTx56R0UtWhPg7siGXR2Q6rZgAVceaDbrRshzz3E9NAqv1nF95l1KWkyl028KICwUZxmbgJHztvyABtxdrRCcM7VUGvYTYnKselmHfQG74rwnBTW8hK3LbZ8IElgKLLSVp704k/SiA/Kea3JVbcQTvPEGzItLlREwsxekWsGcx+OfIMA5dB3nYscUyYiKr/vIe9u3ZvRUkUk0Ghvws8TCqRinf9odp7l08Bl9bsRtdP8znaxVp9HNwVj0vnU6n0/2xb1BXZh8RVovyAAAAAElFTkSuQmCC","orcid":"","institution":"Birla Institute of Technology Mesra, Off Campus Patna","correspondingAuthor":true,"prefix":"","firstName":"Aarushi","middleName":"","lastName":"Shanker","suffix":""},{"id":632170786,"identity":"c38fa127-db24-435e-a33b-e5bc7d541f7b","order_by":1,"name":"Dr. Subhash Chandra Pandey","email":"","orcid":"","institution":"Birla Institute of Technology Mesra, Off Campus Patna","correspondingAuthor":false,"prefix":"Dr.","firstName":"Subhash","middleName":"Chandra","lastName":"Pandey","suffix":""}],"badges":[],"createdAt":"2026-04-30 05:32:54","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-9572322/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9572322/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":108385493,"identity":"dab0da1b-94da-446f-9b86-34bfc2977258","added_by":"auto","created_at":"2026-05-04 06:03:48","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":441624,"visible":true,"origin":"","legend":"","description":"","filename":"MultimodalEmotionRecognitionResearchPaper.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9572322/v1_covered_7298ebfe-4168-4edb-89d0-4d0871c83b23.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003eModelling Human-Like Emotion Perception Using Multimodal Deep Learning\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"Birla Institute of Technology, Mesra","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Multimodal Deep Learning, Human Computer Interaction, Emotion Recognition, Facial Emotion Recognition, Speech Emotion Recognition","lastPublishedDoi":"10.21203/rs.3.rs-9572322/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9572322/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eUnlike humans who can see and hear emotion, computers do not have the capability for innate perception (or the ability to \"see\" and \"hear\"), therefore, they must rely on data generated from sensors (such as visual and auditory), to interpret the emotional state of an individual. This research presents a multi-modal system for accurate and reliable classification of emotion, through facial expression and speech; using state of the art techniques (deep learning methods) such as Hybrid CNN-ViT for visual feature extraction and BiLSTM for analysing auditory features, the approach enables real time detection of emotion regardless of environment, speakers, or expressions. Enhanced robustness and performance have been created through systematic preprocessing and feature extraction from the multi-modal sources, followed by late fusion of the multi-modal predictions, to allow for better accuracy of emotion recognition. Additionally, the research supports the use of a multi-modal approach for enhancing emotional accuracy and provides practical implications for many areas of application, such as: adaptive learning, health monitoring, safety for drivers, delivery of personalised content, marketing, surveillance, and human-robot interaction.\u003c/p\u003e","manuscriptTitle":"Modelling Human-Like Emotion Perception Using Multimodal Deep Learning","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-05-04 06:03:38","doi":"10.21203/rs.3.rs-9572322/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"10733263-1ed0-4bb1-bcac-1e23a0ce9069","owner":[],"postedDate":"May 4th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":67455376,"name":"Artificial Intelligence and Machine Learning"}],"tags":[],"updatedAt":"2026-05-04T06:03:41+00:00","versionOfRecord":[],"versionCreatedAt":"2026-05-04 06:03:38","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9572322","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9572322","identity":"rs-9572322","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}