The Impact of Class Imbalance on Hybrid CNN‑ViT‑SVM for Facial Expression Recognition: A Controlled Study on RAISE‑FER

The Impact of Class Imbalance on Hybrid CNN‑ViT‑SVM for Facial Expression Recognition: A Controlled Study on RAISE‑FER | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article The Impact of Class Imbalance on Hybrid CNN‑ViT‑SVM for Facial Expression Recognition: A Controlled Study on RAISE‑FER Ramanankandrasana Tiana Julianno Fidele This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9624055/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Even though deep learning has greatly improved facial expression recognition (FER), it is still difficult to tell apart similar expressions and to deal with class imbalance. Hybrid CNN‑ViT architectures are commonly used to capture local and global features, but how robust they are to imbalance is poorly understood. Using the RAISE‑FER dataset with a controlled experiment, we had a balanced baseline (224k images to learn from) and an augmented imbalanced version (933k images). We tested three different configurations: CNN‑only (MobileNetV3), ViT‑only (ViT‑B/16), and a hybrid CNN-ViT with an SVM using RBF kernel. The hybrid model showed the highest accuracy of 86.9%, on the balanced dataset, over ViT‑only (85.9%) and CNN‑only (80.9%). ViT‑only outperformed the hybrid on the imbalanced dataset (86.7% vs. 85.8% accuracy). F1 scores for the minority class, Disgust, suffered greatly from class imbalance (-13.9 points hybrid, -11.5 points ViT) while performance on the majority class, Happy, increased (+ 2.7 and + 4.0 points respectively). These findings suggest that feature fusion works well on balanced data but can yield less satisfactory results when very imbalanced. Our proposed study provides the first controlled examination of hybrid CNN‑ViT‑SVM robustness to imbalance on RAISE‑FER dataset and practical guidelines for model selection in real FER applications. Artificial Intelligence and Machine Learning facial expression recognition class imbalance hybrid CNN-ViT vision transformer SVM RAISE-FER Full Text Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9624055","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":635140388,"identity":"7d347f12-ca22-4109-ba12-6f5f1e1973a4","order_by":0,"name":"Ramanankandrasana Tiana Julianno Fidele","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA5UlEQVRIiWNgGAWjYBAC9gbmhgMMDAd4INwKIGZmbsCrhbGBEVnLGZAWRsJagNQBKK8NIoZfy4zExgMfau7ImLMfPvbg47zaaP52oJYfFdvwaWk4OOPYMx7LnrR0w5nbjufOOAy0uefMbbxaDvM2HOYxOJBjJs277VhuA1ALM2MbMVrOvzGT/jvnWO58QloE4VpuAG1hbKjJ3UBIizTPQ5BfQFqepUn2HDuQuxGo5SA+v/CxJx/+8KHmsL3B+eRjEj9q6nLnnT988MGPCtxa0MFhMHmAaPVAUEeK4lEwCkbBKBghAACgnWQnITlPbAAAAABJRU5ErkJggg==","orcid":"https://orcid.org/0009-0001-1638-8666","institution":"Wuhan Textile University","correspondingAuthor":true,"prefix":"","firstName":"Ramanankandrasana","middleName":"Tiana Julianno","lastName":"Fidele","suffix":""}],"badges":[],"createdAt":"2026-05-06 02:57:30","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-9624055/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9624055/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":108805406,"identity":"081b304a-e5c4-4a71-a675-7c28a0bd597f","added_by":"auto","created_at":"2026-05-08 15:25:52","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1066911,"visible":true,"origin":"","legend":"","description":"","filename":"RamanankandrasanapaperRAISEFER.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9624055/v1_covered_3efc8923-e9fe-4289-b9a4-9f6c84b11d78.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003eThe Impact of Class Imbalance on Hybrid CNN‑ViT‑SVM for Facial Expression Recognition: A Controlled Study on RAISE‑FER\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"Wuhan Textile University","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"facial expression recognition, class imbalance, hybrid CNN-ViT, vision transformer, SVM, RAISE-FER","lastPublishedDoi":"10.21203/rs.3.rs-9624055/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9624055/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eEven though deep learning has greatly improved facial expression recognition (FER), it is still difficult to tell apart similar expressions and to deal with class imbalance. Hybrid CNN‑ViT architectures are commonly used to capture local and global features, but how robust they are to imbalance is poorly understood. Using the RAISE‑FER dataset with a controlled experiment, we had a balanced baseline (224k images to learn from) and an augmented imbalanced version (933k images). We tested three different configurations: CNN‑only (MobileNetV3), ViT‑only (ViT‑B/16), and a hybrid CNN-ViT with an SVM using RBF kernel. The hybrid model showed the highest accuracy of 86.9%, on the balanced dataset, over ViT‑only (85.9%) and CNN‑only (80.9%). ViT‑only outperformed the hybrid on the imbalanced dataset (86.7% vs. 85.8% accuracy). F1 scores for the minority class, Disgust, suffered greatly from class imbalance (-13.9 points hybrid, -11.5 points ViT) while performance on the majority class, Happy, increased (+\u0026thinsp;2.7 and +\u0026thinsp;4.0 points respectively). These findings suggest that feature fusion works well on balanced data but can yield less satisfactory results when very imbalanced. Our proposed study provides the first controlled examination of hybrid CNN‑ViT‑SVM robustness to imbalance on RAISE‑FER dataset and practical guidelines for model selection in real FER applications.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e","manuscriptTitle":"The Impact of Class Imbalance on Hybrid CNN‑ViT‑SVM for Facial Expression Recognition: A Controlled Study on RAISE‑FER","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-05-07 05:29:46","doi":"10.21203/rs.3.rs-9624055/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"2faeee9c-302a-4bbe-ad62-6e9e2c8bdb68","owner":[],"postedDate":"May 7th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":67596623,"name":"Artificial Intelligence and Machine Learning"}],"tags":[],"updatedAt":"2026-05-07T05:29:46+00:00","versionOfRecord":[],"versionCreatedAt":"2026-05-07 05:29:46","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9624055","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9624055","identity":"rs-9624055","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}