EGY-MER: Establishing The First Egyptian Arabic Multimodal Emotion Recognition Dataset for Affective Computing

EGY-MER: Establishing The First Egyptian Arabic Multimodal Emotion Recognition Dataset for Affective Computing | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article EGY-MER: Establishing The First Egyptian Arabic Multimodal Emotion Recognition Dataset for Affective Computing Amany Hussein, May Hussien, Ammar Hassona, Wolfgang Minker, Mohammed A.-M. Salem, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8006895/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 15 You are reading this latest preprint version Abstract This research paper introduces EGY-MER, a new multimodal dataset for emotion recognition in the Egyptian Arabic language. The aim is to fill an important research gap in affective computing regarding this dialect. The data samples were collected and organized using the MODALINK pipeline, which provides synchronized multimodal alignment and high-quality annotations. Each sample comprises transcribed speech, corresponding audio, and facial frames with an associated emotion category. Three pretrained encoders were used for establishing baseline results: text data was processed with AraBERTv2, speech with Wav2Vec2-ER, and vision data with a Swin Transformer. A late-fusion strategy was used to combine high-level representations from each encoder. Baseline experiments revealed that combining the different modalities results in improved emotion recognition performance, in contradiction to the unimodal configurations. Weighted F1 and macro-F1 scores suggest the potential of cross-modal features for capturing affective cues in Egyptian Arabic. In addition, the results demonstrate the dataset’s consistency and applicability in multimodal learning research. This research presents the first dataset for multimodal emotion recognition in Egyptian Arabic, along with reproducible baselines. The main aim is that the dataset and provided benchmark models will facilitate further research in emotion recognition for low-resource languages, multimodal fusion, and affective computing in Arabic. Multimodal Emotion Recognition Egyptian Arabic Affective Computing Low-Resource Language Fusion Techniques Deep Learning Arabic NLP Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 21 Dec, 2025 Reviews received at journal 10 Dec, 2025 Reviews received at journal 04 Dec, 2025 Reviews received at journal 19 Nov, 2025 Reviewers agreed at journal 11 Nov, 2025 Reviews received at journal 11 Nov, 2025 Reviewers agreed at journal 10 Nov, 2025 Reviewers agreed at journal 10 Nov, 2025 Reviewers agreed at journal 10 Nov, 2025 Reviewers agreed at journal 10 Nov, 2025 Reviewers agreed at journal 10 Nov, 2025 Reviewers invited by journal 10 Nov, 2025 Editor assigned by journal 10 Nov, 2025 Submission checks completed at journal 07 Nov, 2025 First submitted to journal 01 Nov, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8006895","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":547638850,"identity":"bb88075e-c626-4c8d-9714-146618c947d1","order_by":0,"name":"Amany Hussein","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABAklEQVRIiWNgGAWjYPCCA1CygoEHzEpgYCZWyxlStTAwtsFFcGvhn3b42WMehjuJ8/sPHzxcOK9WxuAAj9mDBwzWcri0SNxOMzfmYXiWuOFGWsLhmduO8wC1mBskMKQb47TmdoKZNA/D4cQNEjwGh3m3HeMxuP/GTCIBKNKAQ4f87fRvYC3z+89/OMw75xjIFrCWelxaDG7nQGxpOJDDcJi3oQauJQGXuwxv55RJzjE4bAz0i8FhnmMHeCQPsJVJJBikG+KyRe52+jaJNxWHZYEh9vgzT02dPd8B5m2SPyqs5XF6HwiYeAzg7MMwB+PTAIzAHwh2HX6lo2AUjIJRMCIBAIlzWMmI4rxpAAAAAElFTkSuQmCC","orcid":"","institution":"German International University","correspondingAuthor":true,"prefix":"","firstName":"Amany","middleName":"","lastName":"Hussein","suffix":""},{"id":547638851,"identity":"ab2760d2-d544-4bf5-9da5-7b363a5b7e53","order_by":1,"name":"May Hussien","email":"","orcid":"","institution":"German International University","correspondingAuthor":false,"prefix":"","firstName":"May","middleName":"","lastName":"Hussien","suffix":""},{"id":547638852,"identity":"488cd069-2303-440d-89b8-2947317ff165","order_by":2,"name":"Ammar Hassona","email":"","orcid":"","institution":"German International University","correspondingAuthor":false,"prefix":"","firstName":"Ammar","middleName":"","lastName":"Hassona","suffix":""},{"id":547638853,"identity":"d781abe1-bb5b-4389-ad34-eb8322fc3d1a","order_by":3,"name":"Wolfgang Minker","email":"","orcid":"","institution":"University of Ulm","correspondingAuthor":false,"prefix":"","firstName":"Wolfgang","middleName":"","lastName":"Minker","suffix":""},{"id":547638854,"identity":"3248f15b-d60e-4852-8663-aa909aaf1815","order_by":4,"name":"Mohammed A.-M. Salem","email":"","orcid":"","institution":"German University in Cairo","correspondingAuthor":false,"prefix":"","firstName":"Mohammed","middleName":"A.-M.","lastName":"Salem","suffix":""},{"id":547638855,"identity":"efb9d10a-e009-44bd-8cb7-b0c9c4071446","order_by":5,"name":"Nada Sharaf","email":"","orcid":"","institution":"German International University","correspondingAuthor":false,"prefix":"","firstName":"Nada","middleName":"","lastName":"Sharaf","suffix":""}],"badges":[],"createdAt":"2025-11-01 16:23:11","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8006895/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8006895/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":96392972,"identity":"6435371f-4023-4292-a605-d6340520c01e","added_by":"auto","created_at":"2025-11-20 14:43:25","extension":"json","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":7246,"visible":true,"origin":"","legend":"","description":"","filename":"f42e092f7783407b9c986352a31de6e3.json","url":"https://assets-eu.researchsquare.com/files/rs-8006895/v1/8bc8bfb77a1007080eaeedb4.json"},{"id":96453109,"identity":"ccdc4987-e50d-48e7-8868-ebad696ac2d8","added_by":"auto","created_at":"2025-11-21 09:58:10","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1010818,"visible":true,"origin":"","legend":"","description":"","filename":"EGYMEREstablishingTheFirstEgyptianArabicMultimodalEmotionRecognitionDatasetforAffectiveComputing6.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8006895/v1_covered_4e48caf6-197c-4d80-9edb-43c63dd6d3c0.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"EGY-MER: Establishing The First Egyptian Arabic Multimodal Emotion Recognition Dataset for Affective Computing ","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"language-resources-and-evaluation","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"lrev","sideBox":"Learn more about [Language Resources and Evaluation](http://link.springer.com/journal/10579)","snPcode":"10579","submissionUrl":"https://submission.nature.com/new-submission/10579/3","title":"Language Resources and Evaluation","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Multimodal Emotion Recognition, Egyptian Arabic, Affective Computing, Low-Resource Language, Fusion Techniques, Deep Learning, Arabic NLP","lastPublishedDoi":"10.21203/rs.3.rs-8006895/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8006895/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\nThis research paper introduces EGY-MER, a new multimodal dataset for emotion recognition in the Egyptian Arabic language. The aim is to fill an important research gap in affective computing regarding this dialect. \nThe data samples were collected and organized using the MODALINK pipeline, which provides synchronized multimodal alignment and high-quality annotations. Each sample comprises transcribed speech, corresponding audio, and facial frames with an associated emotion category. Three pretrained encoders were used for establishing baseline results: text data was processed with AraBERTv2, speech with Wav2Vec2-ER, and vision data with a Swin Transformer. A late-fusion strategy was used to combine high-level representations from each encoder.\nBaseline experiments revealed that combining the different modalities results in improved emotion recognition performance, in contradiction to the unimodal configurations. Weighted F1 and macro-F1 scores suggest the potential of cross-modal features for capturing affective cues in Egyptian Arabic. In addition, the results demonstrate the dataset’s consistency and applicability in multimodal learning research.\nThis research presents the first dataset for multimodal emotion recognition in Egyptian Arabic, along with reproducible baselines. The main aim is that the dataset and provided benchmark models will facilitate further research in emotion recognition for low-resource languages, multimodal fusion, and affective computing in Arabic.","manuscriptTitle":"EGY-MER: Establishing The First Egyptian Arabic Multimodal Emotion Recognition Dataset for Affective Computing","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-11-20 14:43:20","doi":"10.21203/rs.3.rs-8006895/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-12-21T17:09:17+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-12-11T04:29:40+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-12-05T02:44:26+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-11-19T15:31:17+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"129876890168428568979572410981179138648","date":"2025-11-12T03:22:30+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-11-11T05:27:20+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"74301634166898275767090079637292617696","date":"2025-11-11T04:42:26+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"172706855194335385521327772649532531063","date":"2025-11-11T04:38:31+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"322910930992505167831349605938836574353","date":"2025-11-11T04:11:46+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"249389222901004788069014236485534220985","date":"2025-11-11T03:11:33+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"89226038469969049690883898014215983633","date":"2025-11-10T20:17:47+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-11-10T20:15:14+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-11-10T13:43:11+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-11-07T11:25:26+00:00","index":"","fulltext":""},{"type":"submitted","content":"Language Resources and Evaluation","date":"2025-11-01T16:07:10+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"language-resources-and-evaluation","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"lrev","sideBox":"Learn more about [Language Resources and Evaluation](http://link.springer.com/journal/10579)","snPcode":"10579","submissionUrl":"https://submission.nature.com/new-submission/10579/3","title":"Language Resources and Evaluation","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"499f5e75-4884-4229-812a-634849817478","owner":[],"postedDate":"November 20th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-01-23T17:38:24+00:00","versionOfRecord":[],"versionCreatedAt":"2025-11-20 14:43:20","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8006895","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8006895","identity":"rs-8006895","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}