A Dual-Branch CNN–Vision Transformer Model with Feature Fusion for Robust Facial Expression Recognition
preprint
OA: closed
AI-generated deep summary
The paper studied facial emotion recognition on the in-the-wild FER2013 dataset, using a hybrid deep-learning approach that combines a ResNet-18 CNN branch for local facial feature extraction with a Vision Transformer branch for global contextual representation learning. Using standard training/validation/test splits, the authors trained the dual-branch model with class-weighted loss to address class imbalance, Mixup augmentation, and cosine annealing optimization, and reported that the fused architecture achieved 71.48% test accuracy and 69.96% macro F1-score, with improved performance on underrepresented classes such as disgust. An ablation study indicated that the hybrid design yields consistent gains over using a CNN-only or transformer-only model, but the work’s evaluation is limited to a single dataset and relies on pre-defined FER2013 splits. This paper does not explicitly discuss endometriosis or adenomyosis; it was included in the corpus via a keyword match in the upstream search index.
Read from the paper's body, not the abstract. Not a substitute for reading the paper. No clinical advice. How this works
Abstract
Abstract Facial emotion recognition (FER) on in-the-wild datasets such as FER2013 remains a challenging task due to label noise, significant class imbalance, and variations in lighting, pose, and occlusion. Convolutional neural networks (CNNs) are effective at extracting local facial features but often struggle to capture long-range dependencies across facial regions. In contrast, Vision Transformers (ViTs) model global contextual relationships well, but may lack sensitivity to fine-grained facial details. In this work, we propose a hybrid CNN–Vision Transformer architecture, referred to as Autistic Fusion, which combines a ResNet-18 backbone for local feature extraction with a Vision Transformer for global representation learning. A lightweight projection and fusion module is introduced to integrate these complementary features efficiently. The model is trained using class-weighted loss, Mixup augmentation, and cosine annealing optimization on the FER2013 dataset using standard training, validation, and test splits. Experimental results show that the proposed model achieves a test accuracy of 71.48% and a macro F1-score of 69.96%, demonstrating competitive performance on a challenging benchmark. In particular, the model shows improved performance on underrepresented classes such as disgust, indicating effective handling of class imbalance. An ablation study further confirms that the hybrid architecture provides consistent improvements over individual CNN and transformer models. The proposed approach offers a practical and robust solution for facial emotion recognition, highlighting the benefit of combining local and global feature representations in deep learning frameworks.
Full text
13,766 characters
· extracted from
preprint-html
· click to expand
A Dual-Branch CNN–Vision Transformer Model with Feature Fusion for Robust Facial Expression Recognition | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A Dual-Branch CNN–Vision Transformer Model with Feature Fusion for Robust Facial Expression Recognition Bakhita Salman, Jorge Madrigal, Muneeb Yassin This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9192925/v1 This work is licensed under a CC BY 4.0 License Status: Under Revision Version 1 posted 12 You are reading this latest preprint version Abstract Facial emotion recognition (FER) on in-the-wild datasets such as FER2013 remains a challenging task due to label noise, significant class imbalance, and variations in lighting, pose, and occlusion. Convolutional neural networks (CNNs) are effective at extracting local facial features but often struggle to capture long-range dependencies across facial regions. In contrast, Vision Transformers (ViTs) model global contextual relationships well, but may lack sensitivity to fine-grained facial details. In this work, we propose a hybrid CNN–Vision Transformer architecture, referred to as Autistic Fusion, which combines a ResNet-18 backbone for local feature extraction with a Vision Transformer for global representation learning. A lightweight projection and fusion module is introduced to integrate these complementary features efficiently. The model is trained using class-weighted loss, Mixup augmentation, and cosine annealing optimization on the FER2013 dataset using standard training, validation, and test splits. Experimental results show that the proposed model achieves a test accuracy of 71.48% and a macro F1-score of 69.96%, demonstrating competitive performance on a challenging benchmark. In particular, the model shows improved performance on underrepresented classes such as disgust, indicating effective handling of class imbalance. An ablation study further confirms that the hybrid architecture provides consistent improvements over individual CNN and transformer models. The proposed approach offers a practical and robust solution for facial emotion recognition, highlighting the benefit of combining local and global feature representations in deep learning frameworks. Facial emotion recognition FER2013 CNN Vision Transformer hybrid class imbalance deep learning ResNet feature fusion Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Revision Version 1 posted Editorial decision: Revision requested 15 May, 2026 Reviews received at journal 04 May, 2026 Reviews received at journal 29 Apr, 2026 Reviewers agreed at journal 27 Apr, 2026 Reviews received at journal 25 Apr, 2026 Reviewers agreed at journal 25 Apr, 2026 Reviewers agreed at journal 20 Apr, 2026 Reviewers agreed at journal 20 Apr, 2026 Reviewers invited by journal 20 Apr, 2026 Editor assigned by journal 27 Mar, 2026 Submission checks completed at journal 27 Mar, 2026 First submitted to journal 22 Mar, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9192925","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":630008022,"identity":"2f94ac25-0528-46ef-9200-df00a25000a2","order_by":0,"name":"Bakhita Salman","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABA0lEQVRIie3OMUsDMRTA8VcO6tJy6+vQ3lcICNbS0s+ScJD7CoKiB8XnUulW+jF0c6wcXJfDrjmyeItOgmMdFHOhIg5pOzrkDwkvJD8IgM/3D2PQbAK3Y5CabWRWox6CPcQa+1LiYQR+SYbbCzfpH+XyudqMIbp5vL77eFhfhvOMEM6GInWQwVSuGOcxsEJQeVtoRC0MKRInYcuEkPPA/FCQapNGqEmDMjdZv9bkCqJZReUnPWFkydcOomRuSAagBOk2LZFZkrrJYPEikctVi6mKdJfizr0Wk1OeJ8cu0g/lSWczOu9Fszgv32gc9nRSqfeLYddFfmr9PfI9z30+n8+3u2+nhl5NId5z9gAAAABJRU5ErkJggg==","orcid":"","institution":"Texas A\u0026M International University","correspondingAuthor":true,"prefix":"","firstName":"Bakhita","middleName":"","lastName":"Salman","suffix":""},{"id":630008023,"identity":"b2c56d98-9973-414f-8155-c5e4412145da","order_by":1,"name":"Jorge Madrigal","email":"","orcid":"","institution":"Texas A\u0026M International University","correspondingAuthor":false,"prefix":"","firstName":"Jorge","middleName":"","lastName":"Madrigal","suffix":""},{"id":630008024,"identity":"25a0c216-db89-43b5-b742-3c55ec5a728d","order_by":2,"name":"Muneeb Yassin","email":"","orcid":"","institution":"Texas A\u0026M International University","correspondingAuthor":false,"prefix":"","firstName":"Muneeb","middleName":"","lastName":"Yassin","suffix":""}],"badges":[],"createdAt":"2026-03-22 17:38:11","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9192925/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9192925/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":108007183,"identity":"9af6967d-5ea4-4588-9a3f-1db84d2b5c17","added_by":"auto","created_at":"2026-04-28 12:58:48","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1040471,"visible":true,"origin":"","legend":"","description":"","filename":"TransformerModelFacialExpressionRecognition.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9192925/v1_covered_4f1d0ac0-4ce7-451b-a8f2-404e2fa75118.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"A Dual-Branch CNN–Vision Transformer Model with Feature Fusion for Robust Facial Expression Recognition","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"discover-applied-sciences","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [Discover Applied Sciences](https://link.springer.com/journal/42452)","snPcode":"42452","submissionUrl":"https://submission.springernature.com/new-submission/42452/3","title":"Discover Applied Sciences","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Discover Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Facial emotion recognition, FER2013, CNN, Vision Transformer hybrid, class imbalance, deep learning, ResNet, feature fusion","lastPublishedDoi":"10.21203/rs.3.rs-9192925/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9192925/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eFacial emotion recognition (FER) on in-the-wild datasets such as FER2013 remains a challenging task due to label noise, significant class imbalance, and variations in lighting, pose, and occlusion. Convolutional neural networks (CNNs) are effective at extracting local facial features but often struggle to capture long-range dependencies across facial regions. In contrast, Vision Transformers (ViTs) model global contextual relationships well, but may lack sensitivity to fine-grained facial details. In this work, we propose a hybrid CNN\u0026ndash;Vision Transformer architecture, referred to as Autistic Fusion, which combines a ResNet-18 backbone for local feature extraction with a Vision Transformer for global representation learning. A lightweight projection and fusion module is introduced to integrate these complementary features efficiently. The model is trained using class-weighted loss, Mixup augmentation, and cosine annealing optimization on the FER2013 dataset using standard training, validation, and test splits. Experimental results show that the proposed model achieves a test accuracy of 71.48% and a macro F1-score of 69.96%, demonstrating competitive performance on a challenging benchmark. In particular, the model shows improved performance on underrepresented classes such as disgust, indicating effective handling of class imbalance. An ablation study further confirms that the hybrid architecture provides consistent improvements over individual CNN and transformer models. The proposed approach offers a practical and robust solution for facial emotion recognition, highlighting the benefit of combining local and global feature representations in deep learning frameworks.\u003c/p\u003e","manuscriptTitle":"A Dual-Branch CNN–Vision Transformer Model with Feature Fusion for Robust Facial Expression Recognition","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-28 04:57:53","doi":"10.21203/rs.3.rs-9192925/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-05-15T10:49:53+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-04T06:19:14+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-29T09:15:07+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"301961086514893454226236136777246321603","date":"2026-04-27T05:10:37+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-25T10:35:24+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"214585606149700466613698625423212382450","date":"2026-04-25T09:48:53+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"255734411512437965810424929200347905260","date":"2026-04-20T09:23:39+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"320537238345427385706658623102236220928","date":"2026-04-20T08:49:42+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-04-20T08:26:49+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-03-27T12:35:47+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-03-27T12:35:40+00:00","index":"","fulltext":""},{"type":"submitted","content":"Discover Applied Sciences","date":"2026-03-22T17:25:58+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"discover-applied-sciences","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [Discover Applied Sciences](https://link.springer.com/journal/42452)","snPcode":"42452","submissionUrl":"https://submission.springernature.com/new-submission/42452/3","title":"Discover Applied Sciences","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Discover Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"e0e7d479-b7d9-4e67-bab6-4183b51aa311","owner":[],"postedDate":"April 28th, 2026","published":true,"recentEditorialEvents":[{"type":"decision","content":"Revision requested","date":"2026-05-15T10:49:53+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-04T06:19:14+00:00","index":56,"fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"in-revision","subjectAreas":[],"tags":[],"updatedAt":"2026-05-15T10:55:09+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-28 04:57:53","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9192925","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9192925","identity":"rs-9192925","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}
Text is read by the "Ask this paper" AI Q&A widget below. Extraction quality varies by source — PMC NXML preserves structure cleanly, OA-HTML may include some navigation residue, and OA-PDF can have broken hyphenation. The publisher copy (via DOI) is the canonical version.
My notes (saved in your browser only)
Ask this paper
Answers must be backed by verbatim quotes from this paper's full text. Hallucinated quotes are dropped automatically; if no verbatim passage answers the question, we say so. How this works
Citation neighborhood (no data yet)
We don't have any in-corpus citations linked to this paper yet. This is a recent paper (2026) — citers typically take a year or two to land, and the OpenAlex reference graph may still be filling in.
Source provenance
- europepmc
- last seen: 2026-05-20T01:45:00.602351+00:00