Explaining Facial Expression Recognition for Human Understanding and Model Personalisation

Explaining Facial Expression Recognition for Human Understanding and Model Personalisation | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Explaining Facial Expression Recognition for Human Understanding and Model Personalisation Sanjeev Nahulanthran, Mor Vered, Dana Kulić, Leimin Tian This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9582396/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 7 You are reading this latest preprint version Abstract Facial expression recognition (FER) systems utilize machine learning algorithms to perceive human facial expressions from image or video data. While FER systems have broad applications in human-computer/robot interaction, significant challenges remain in deploying FER systems to real-world scenarios. One such challenge is the limited transparency of current FER systems, which fail to provide users with relevant information when the system is operating incorrectly. Another challenge stems from the diversity of facial expressions, which are shaped by individual identity, cultural background, and situational factors, thereby introducing significant variability that impacts the reliability of these algorithms. We investigate the use of eXplainable AI (XAI) as a way to mitigate some of these issues by not only providing greater transparency but also by personalizing the model. We developed a Facial Action Unit (FAUs) based explanation method (DEFAULTS) and compare the usability of this XAI method in terms of user understanding and appropriate trust between other well-known XAI methods. We found that our novel DEFAULTS explanation method significantly improved user understanding while engendering higher appropriate trust as compared to other state-of-the-art XAI methods. We further investigated the effect that DEFAULTS explanations can have in the personalisation of the FER model by introducing FACT-MATCH , a new algorithm that uses information from the DEFAULTS explanations to identify additional samples with which to retrain the model, without any user intervention. We compared FACT-MATCH against state-of-the-art Active Learning (AL) methods and showed that FACT-MATCH is able to better improve baseline model median accuracy when compared to existing AL methods. Our work is the first to show that FAU-based explanations are useful both for increasing FER transparency to end-users and improving FER personalisation, addressing key challenges in the adoption of FER systems and paving the path for future work in this new sub-field of eXplainable Active Learning for Facial Expression Recognition. Facial Expression Recognition eXplainable Artificial Intelligence Affect Recognition Trust Personalization Active Learning eXplainable Active Learning Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Reviews received at journal 18 May, 2026 Reviewers agreed at journal 15 May, 2026 Reviewers agreed at journal 13 May, 2026 Reviewers invited by journal 07 May, 2026 Editor assigned by journal 07 May, 2026 Submission checks completed at journal 05 May, 2026 First submitted to journal 30 Apr, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9582396","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":641580874,"identity":"35a2d4ea-33d1-4a04-bd29-11b2f07d447a","order_by":0,"name":"Sanjeev Nahulanthran","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABNUlEQVRIie2RMWvCQBTHXwjEJeJ6Eqhf4YVAUmn6XZIlXc5QKEgHKU52sbo6tV/BLplTDpxOs97QIVBwcsgkdlB6Up0u1LXD/ZZ378Hv3h8egEbzT8HfYgOU5646T8C6oEQA5NgZswsKKIpp/6F0npf5ffXIIGgsFyQafD4FDlt/3dIwbU2XOVR9poTiaeTOOIPuOE1ItFiT7iQJvF6WPBCRRsZspSpA0WuOEsCc+iSyGEEOvtPLWDwUNprNkaJ0phv09gepFBupHI5KY+t0pfJWcDT3qgJCbjGGIaCQW+LRUbF9x5DKPKdoGqqCYoPueBHa8uFdxxPWnnO7337Jkvhd/vYxXt2pwaiPuwG5woK6otqylgyWke8sjF8L7pa7/k3NPSw8naCGvH5slvVzjUaj0Zz4AbEac0azOcr8AAAAAElFTkSuQmCC","orcid":"","institution":"Monash University","correspondingAuthor":true,"prefix":"","firstName":"Sanjeev","middleName":"","lastName":"Nahulanthran","suffix":""},{"id":641580875,"identity":"b9e92d2d-969b-4c74-9596-872a57994a05","order_by":1,"name":"Mor Vered","email":"","orcid":"","institution":"Monash University","correspondingAuthor":false,"prefix":"","firstName":"Mor","middleName":"","lastName":"Vered","suffix":""},{"id":641580876,"identity":"05a1c72e-c8df-44fe-bdcc-f9bff3fdf632","order_by":2,"name":"Dana Kulić","email":"","orcid":"","institution":"Monash University","correspondingAuthor":false,"prefix":"","firstName":"Dana","middleName":"","lastName":"Kulić","suffix":""},{"id":641580877,"identity":"b537d5e9-227a-41aa-9e12-770dcc0b6352","order_by":3,"name":"Leimin Tian","email":"","orcid":"","institution":"CSIRO Robotics","correspondingAuthor":false,"prefix":"","firstName":"Leimin","middleName":"","lastName":"Tian","suffix":""}],"badges":[],"createdAt":"2026-05-01 03:24:57","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9582396/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9582396/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":109760789,"identity":"1365a06f-de75-449e-a879-a75370a5686d","added_by":"auto","created_at":"2026-05-22 07:29:09","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":773512,"visible":true,"origin":"","legend":"","description":"","filename":"JAAMASPaper.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9582396/v1_covered_a9f93679-a8ef-45d9-8f8c-374e283c9827.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Explaining Facial Expression Recognition for Human Understanding and Model Personalisation","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"autonomous-agents-and-multi-agent-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"agnt","sideBox":"Learn more about [Autonomous Agents and Multi-Agent Systems](http://link.springer.com/journal/10458)","snPcode":"10458","submissionUrl":"https://submission.nature.com/new-submission/10458/3","title":"Autonomous Agents and Multi-Agent Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Facial Expression Recognition, eXplainable Artificial Intelligence, Affect Recognition, Trust, Personalization, Active Learning, eXplainable Active Learning","lastPublishedDoi":"10.21203/rs.3.rs-9582396/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9582396/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eFacial expression recognition (FER) systems utilize machine learning algorithms to perceive human facial expressions from image or video data. While FER systems have broad applications in human-computer/robot interaction, significant challenges remain in deploying FER systems to real-world scenarios. One such challenge is the limited transparency of current FER systems, which fail to provide users with relevant information when the system is operating incorrectly. Another challenge stems from the diversity of facial expressions, which are shaped by individual identity, cultural background, and situational factors, thereby introducing significant variability that impacts the reliability of these algorithms. We investigate the use of eXplainable AI (XAI) as a way to mitigate some of these issues by not only providing greater transparency but also by personalizing the model. We developed a Facial Action Unit (FAUs) based explanation method (DEFAULTS) and compare the usability of this XAI method in terms of user understanding and appropriate trust between other well-known XAI methods. We found that our novel \u003cem\u003eDEFAULTS\u003c/em\u003e explanation method significantly improved user understanding while engendering higher appropriate trust as compared to other state-of-the-art XAI methods. We further investigated the effect that DEFAULTS explanations can have in the personalisation of the FER model by introducing \u003cem\u003eFACT-MATCH\u003c/em\u003e, a new algorithm that uses information from the DEFAULTS explanations to identify additional samples with which to retrain the model, without any user intervention. We compared FACT-MATCH against state-of-the-art Active Learning (AL) methods and showed that FACT-MATCH is able to better improve baseline model median accuracy when compared to existing AL methods. Our work is the first to show that FAU-based explanations are useful both for increasing FER transparency to end-users and improving FER personalisation, addressing key challenges in the adoption of FER systems and paving the path for future work in this new sub-field of eXplainable Active Learning for Facial Expression Recognition.\u003c/p\u003e","manuscriptTitle":"Explaining Facial Expression Recognition for Human Understanding and Model Personalisation","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-05-18 19:21:19","doi":"10.21203/rs.3.rs-9582396/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"editorInvitedReview","content":"","date":"2026-05-18T08:25:50+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"79817573699977605180471312752700016420","date":"2026-05-15T06:04:36+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"7960414334540314603206752207107390940","date":"2026-05-13T15:40:18+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-05-07T16:16:28+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-05-07T16:11:08+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-05-05T04:47:21+00:00","index":"","fulltext":""},{"type":"submitted","content":"Autonomous Agents and Multi-Agent Systems","date":"2026-05-01T03:21:16+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"autonomous-agents-and-multi-agent-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"agnt","sideBox":"Learn more about [Autonomous Agents and Multi-Agent Systems](http://link.springer.com/journal/10458)","snPcode":"10458","submissionUrl":"https://submission.nature.com/new-submission/10458/3","title":"Autonomous Agents and Multi-Agent Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"26fc4ca6-0f0b-48ea-9c92-52fbac32aaae","owner":[],"postedDate":"May 18th, 2026","published":true,"recentEditorialEvents":[{"type":"editorInvitedReview","content":"","date":"2026-05-18T08:25:50+00:00","index":14,"fulltext":""},{"type":"reviewerAgreed","content":"79817573699977605180471312752700016420","date":"2026-05-15T06:04:36+00:00","index":13,"fulltext":""},{"type":"reviewerAgreed","content":"7960414334540314603206752207107390940","date":"2026-05-13T15:40:18+00:00","index":11,"fulltext":""},{"type":"reviewersInvited","content":"7","date":"2026-05-07T16:16:28+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-05-07T16:11:08+00:00","index":"","fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-05-18T19:21:19+00:00","versionOfRecord":[],"versionCreatedAt":"2026-05-18 19:21:19","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9582396","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9582396","identity":"rs-9582396","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}