Emotion Recognition for Mental Health Assessment Using Transformer-Based Multimodal Learning

Emotion Recognition for Mental Health Assessment Using Transformer-Based Multimodal Learning | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Emotion Recognition for Mental Health Assessment Using Transformer-Based Multimodal Learning Rahat Malik, Bushra Siddique, Erum Ashraf, Hafiz Ishfaq This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8613173/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract The complex dynamics of artificial intelligence, the mental health detection tools and frameworks are increasingly mounting different intelligent approaches and strategies to support early assessment and intervention through emotion detection. However, a lot of these existing system uses approaches of unimodal feature extraction and only early fusion strategies which often limits robustness and generalization in real world mental health scenarios. This paper presents \textit{MindMed AI}, a deep learning framework that employs multimodal feature extraction and transformer based analysis of emotion detection to assist in proactive mental health support. This AI framework is a compilation of three individual unimodal models basing HUBERT and OpenSMILE for voice based analysis, a Data efficient Image Transformers (DeiT) for facial emotion recognition and BERT for text based emotion analysis. The paper carries a comparative study of efficiency of unimodel models with early and intermediate fusion. The results show that the individual model analysis demonstrates the hightest accuracy of 91.22% in case of acoustic evaluation, while the use of intermediate fusion outperformed in accuracy in comparison to both individual unimodal evaluations and early fusion with highest score of 91.89%. The statistical compilation and comparison of results defined the significance of structured multimodal integration in enhanced emotion recognition accuracy and robustness, highlighting the potential of transformer based intermediate fusion for scalable and reliable mental health monitoring applications. Emotion Recognition Multimodal Learning Mental Health Assessment Transformers Deep Learning Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8613173","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":586424816,"identity":"43f215c5-6633-4494-a115-a3ba74bd6697","order_by":0,"name":"Rahat Malik","email":"","orcid":"","institution":"Bahria University","correspondingAuthor":false,"prefix":"","firstName":"Rahat","middleName":"","lastName":"Malik","suffix":""},{"id":586424817,"identity":"cc1cca63-2d12-4c61-b580-364b2653759c","order_by":1,"name":"Bushra Siddique","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Bushra","middleName":"","lastName":"Siddique","suffix":""},{"id":586424818,"identity":"ecc60aea-e078-4867-9f63-2f1d3ede9b18","order_by":2,"name":"Erum Ashraf","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA+ElEQVRIiWNgGAWjYLACxgYGHgb2HgZmBgYJEN8AiBOI0MJzhkQtQMU5IC0MhLXwS58x3cC447AM/8y3Bz8X7rGQZ2Bv3ibBuCMNpxbJvhyzG4xnDvNI3M5Llp7xTMKwgedYmQTjmRycWgzO8AC1tB3mMZDOMZDmOSDB2CCRYybB2FaBU4s9XIvkGePfQC32DfJv8Gsx4IFpkeAxA9mS2ABkALXgdpjEGbayG4ln0nkkzuSlWQO1JLfxpBVbJJ7B7X3+HuZtNz7usLbnbz97+DbPgTrbfvbDG4EiyTi1gEECMocNRCQ24NeBBTCSrmUUjIJRMAqGLwAAjhdMVPkK++kAAAAASUVORK5CYII=","orcid":"","institution":"Bahria University","correspondingAuthor":true,"prefix":"","firstName":"Erum","middleName":"","lastName":"Ashraf","suffix":""},{"id":586424819,"identity":"d5f6121d-7e34-4634-877b-1705b359897d","order_by":3,"name":"Hafiz Ishfaq","email":"","orcid":"","institution":"Bahria University","correspondingAuthor":false,"prefix":"","firstName":"Hafiz","middleName":"","lastName":"Ishfaq","suffix":""}],"badges":[],"createdAt":"2026-01-15 19:08:13","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8613173/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8613173/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":109164183,"identity":"76930819-8b95-409d-b961-cdc53cdf05ab","added_by":"auto","created_at":"2026-05-13 08:02:29","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":305793,"visible":true,"origin":"","legend":"","description":"","filename":"EmotionRecognitionforMentalHealthAssessmentUsingTransformerBasedMultimodalLearning2.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8613173/v1_covered_82f6a2d9-a2b3-4641-85a2-1111a62e354f.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Emotion Recognition for Mental Health Assessment Using Transformer-Based Multimodal Learning","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Emotion Recognition, Multimodal Learning, Mental Health Assessment, Transformers, Deep Learning","lastPublishedDoi":"10.21203/rs.3.rs-8613173/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8613173/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"The complex dynamics of artificial intelligence, the mental health detection tools and frameworks are increasingly mounting different intelligent approaches and strategies to support early assessment and intervention through emotion detection. However, a lot of these existing system uses approaches of unimodal feature extraction and only early fusion strategies which often limits robustness and generalization in real world mental health scenarios. This paper presents \\textit{MindMed AI}, a deep learning framework that employs multimodal feature extraction and transformer based analysis of emotion detection to assist in proactive mental health support. This AI framework is a compilation of three individual unimodal models basing HUBERT and OpenSMILE for voice based analysis, a Data efficient Image Transformers (DeiT) for facial emotion recognition and BERT for text based emotion analysis. The paper carries a comparative study of efficiency of unimodel models with early and intermediate fusion. The results show that the individual model analysis demonstrates the hightest accuracy of 91.22\\% in case of acoustic evaluation, while the use of intermediate fusion outperformed in accuracy in comparison to both individual unimodal evaluations and early fusion with highest score of 91.89\\%. The statistical compilation and comparison of results defined the significance of structured multimodal integration in enhanced emotion recognition accuracy and robustness, highlighting the potential of transformer based intermediate fusion for scalable and reliable mental health monitoring applications.","manuscriptTitle":"Emotion Recognition for Mental Health Assessment Using Transformer-Based Multimodal Learning","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-02-09 17:11:51","doi":"10.21203/rs.3.rs-8613173/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"e4070d3a-2b6b-43ee-b13b-1d6cc5b947ce","owner":[],"postedDate":"February 9th, 2026","published":true,"recentEditorialEvents":[{"type":"decision","content":"Withdrawn","date":"2026-05-13T07:46:51+00:00","index":"","fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-05-13T08:00:26+00:00","versionOfRecord":[],"versionCreatedAt":"2026-02-09 17:11:51","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8613173","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8613173","identity":"rs-8613173","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}