FedSER-XAI: PSO-Optimized Multi-StreamCross-Attention Transformer with Graph Featuresfor Explainable Federated Speech EmotionRecognitione

FedSER-XAI: PSO-Optimized Multi-StreamCross-Attention Transformer with Graph Featuresfor Explainable Federated Speech EmotionRecognitione | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article FedSER-XAI: PSO-Optimized Multi-StreamCross-Attention Transformer with Graph Featuresfor Explainable Federated Speech EmotionRecognitione Eman Abdulrahman Alkhamali, Arwa Allinjawi, Rehab Bahaaddin Ashari, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7530459/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 24 Nov, 2025 Read the published version in Scientific Reports → Version 1 posted 19 You are reading this latest preprint version Abstract Federated learning for speech emotion recognition faces fundamental challengesin simultaneously achieving high performance, privacy preservation, and modelinterpretability. This paper introduces FedSER-XAI, a novel framework thatintegrates Particle Swarm Optimization (PSO)-based feature selection, multi-stream cross-attention mechanisms, and graph-based feature extraction withinan explainable federated learning architecture. Our approach combines VisionTransformer processing of mel-spectrograms with temporal-spatial graph convo-lutional networks to capture both contextual and structural speech relationships.The PSO algorithm achieves 78.1% dimensionality reduction (228→50 features)while improving discriminative power. The multi-stream architecture processestraditional acoustic features alongside novel graph-based representations derivedfrom visibility and correlation graphs, fused through Transformer-based cross-attention mechanisms. Extensive evaluation on EMODB and SAVEE datasetsdemonstrates exceptional performance: 99.9% and 97.2% accuracy in central-ized settings, with remarkable federated performance of 99.7% and 97.2% using8 emotion-specialized clients. The framework achieves rapid convergence within110 communication rounds, representing minimal performance degradation (0.2%for EMODB) while preserving privacy. Cross-dataset evaluation on CREMA-Dyields 68% accuracy, demonstrating reasonable generalization. The compre-hensive explainability framework using SHAP and LIME provides global andlocal interpretations, validating that graph-based features contribute significantlyto emotion discrimination. FedSER-XAI represents the first explainable feder-ated speech emotion recognition system, advancing trustworthy AI for sensitivehealthcare and human-computer interaction applications Biological sciences/Computational biology and bioinformatics Physical sciences/Engineering Physical sciences/Mathematics and computing Speech emotion recognition Federated learning Explainable AI Cross-attention mechanisms Graph neural networks Particle swarm optimization Privacy preservation Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 24 Nov, 2025 Read the published version in Scientific Reports → Version 1 posted Editorial decision: Revision requested 01 Oct, 2025 Reviews received at journal 30 Sep, 2025 Reviewers agreed at journal 30 Sep, 2025 Reviewers agreed at journal 29 Sep, 2025 Reviews received at journal 28 Sep, 2025 Reviewers agreed at journal 27 Sep, 2025 Reviewers agreed at journal 27 Sep, 2025 Reviews received at journal 26 Sep, 2025 Reviewers agreed at journal 18 Sep, 2025 Reviewers agreed at journal 17 Sep, 2025 Reviews received at journal 17 Sep, 2025 Reviewers agreed at journal 16 Sep, 2025 Reviewers agreed at journal 16 Sep, 2025 Reviewers agreed at journal 16 Sep, 2025 Reviewers invited by journal 16 Sep, 2025 Editor invited by journal 08 Sep, 2025 Editor assigned by journal 05 Sep, 2025 Submission checks completed at journal 04 Sep, 2025 First submitted to journal 03 Sep, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7530459","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":517109989,"identity":"fda68e56-4ca3-4495-b65b-0a7295afc132","order_by":0,"name":"Eman Abdulrahman Alkhamali","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABFklEQVRIie3OP0vDQBjH8ecIxOU064VI8hYeEARJ0beSEEiXIBWh6FgClyWQtZuvwXeQcKBLqGukg+niVNAuEkFpLxUnD/wzOdx3CHeQD78D0On+aQxGACaQSftx6zO/I7glKULwG9L/xn5ErJ20ajocnO5Zgl++dgLsrEZ4HgvwslJJ7PwmOsoxPjeh4nMaCHBogmQ6E4B1oCTYJIeMogg5mfA5SOJCgsYulwTU5OTh6cV+w3XIDcLPup5YSzTeJfGKVr3CqOlQLENuEg7bhzG5QiSBRr3C6vjA38co5JSkDo2H1J4+jqp8NqTYqFesTCzulxfH4VVxu1h1A99ld9F124191yvUK1+i/af8POh0Op3ub20AGhxY21lkjdAAAAAASUVORK5CYII=","orcid":"","institution":"King Abdul-Aziz University","correspondingAuthor":true,"prefix":"","firstName":"Eman","middleName":"Abdulrahman","lastName":"Alkhamali","suffix":""},{"id":517109990,"identity":"9781f215-ea85-4449-923f-8e3a8622cc0d","order_by":1,"name":"Arwa Allinjawi","email":"","orcid":"","institution":"King Abdul-Aziz University","correspondingAuthor":false,"prefix":"","firstName":"Arwa","middleName":"","lastName":"Allinjawi","suffix":""},{"id":517109991,"identity":"79978897-18f9-48d9-a7b0-ab71c895e105","order_by":2,"name":"Rehab Bahaaddin Ashari","email":"","orcid":"","institution":"King Abdul-Aziz University","correspondingAuthor":false,"prefix":"","firstName":"Rehab","middleName":"Bahaaddin","lastName":"Ashari","suffix":""},{"id":517109992,"identity":"3a52f0ff-e094-4c47-b563-54542bb69808","order_by":3,"name":"Mohammed Tawfik","email":"","orcid":"","institution":"Ajloun National University","correspondingAuthor":false,"prefix":"","firstName":"Mohammed","middleName":"","lastName":"Tawfik","suffix":""}],"badges":[],"createdAt":"2025-09-03 22:23:04","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7530459/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7530459/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41598-025-28686-z","type":"published","date":"2025-11-24T15:57:29+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":92041271,"identity":"6a3cab47-c6d1-4d83-bd97-5e92fa96771d","added_by":"auto","created_at":"2025-09-24 03:12:39","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1773754,"visible":true,"origin":"","legend":"","description":"","filename":"Eman2025.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7530459/v1/a3a016dca22af4fc91db2ba2.pdf"},{"id":92041270,"identity":"9252b07e-8a07-4ade-aa87-4ef444cba1a3","added_by":"auto","created_at":"2025-09-24 03:12:39","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":6732,"visible":true,"origin":"","legend":"","description":"","filename":"544da8ce570146a18d4664c606cae59a.json","url":"https://assets-eu.researchsquare.com/files/rs-7530459/v1/93c26d24b6654cc7300e4b39.json"},{"id":97179311,"identity":"5c1ec27b-e963-4e82-8f7d-6d5fda17a49c","added_by":"auto","created_at":"2025-12-01 16:14:34","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1238553,"visible":true,"origin":"","legend":"","description":"","filename":"Eman2025.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7530459/v1_covered_19684305-fe2f-4a07-8efc-10b12dfe2238.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"FedSER-XAI: PSO-Optimized Multi-StreamCross-Attention Transformer with Graph Featuresfor Explainable Federated Speech EmotionRecognitione","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Speech emotion recognition, Federated learning, Explainable AI, Cross-attention mechanisms, Graph neural networks, Particle swarm optimization, Privacy preservation","lastPublishedDoi":"10.21203/rs.3.rs-7530459/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7530459/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Federated learning for speech emotion recognition faces fundamental challengesin simultaneously achieving high performance, privacy preservation, and modelinterpretability. This paper introduces FedSER-XAI, a novel framework thatintegrates Particle Swarm Optimization (PSO)-based feature selection, multi-stream cross-attention mechanisms, and graph-based feature extraction withinan explainable federated learning architecture. Our approach combines VisionTransformer processing of mel-spectrograms with temporal-spatial graph convo-lutional networks to capture both contextual and structural speech relationships.The PSO algorithm achieves 78.1% dimensionality reduction (228→50 features)while improving discriminative power. The multi-stream architecture processestraditional acoustic features alongside novel graph-based representations derivedfrom visibility and correlation graphs, fused through Transformer-based cross-attention mechanisms. Extensive evaluation on EMODB and SAVEE datasetsdemonstrates exceptional performance: 99.9% and 97.2% accuracy in central-ized settings, with remarkable federated performance of 99.7% and 97.2% using8 emotion-specialized clients. The framework achieves rapid convergence within110 communication rounds, representing minimal performance degradation (0.2%for EMODB) while preserving privacy. Cross-dataset evaluation on CREMA-Dyields 68% accuracy, demonstrating reasonable generalization. The compre-hensive explainability framework using SHAP and LIME provides global andlocal interpretations, validating that graph-based features contribute significantlyto emotion discrimination. FedSER-XAI represents the first explainable feder-ated speech emotion recognition system, advancing trustworthy AI for sensitivehealthcare and human-computer interaction applications","manuscriptTitle":"FedSER-XAI: PSO-Optimized Multi-StreamCross-Attention Transformer with Graph Featuresfor Explainable Federated Speech EmotionRecognitione","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-09-24 03:12:35","doi":"10.21203/rs.3.rs-7530459/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-10-01T16:47:53+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-10-01T02:13:57+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"69317956869407821276480236924629693476","date":"2025-09-30T14:43:54+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"84845117828014595208801495786645316515","date":"2025-09-29T08:48:30+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-09-28T16:33:16+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"65712604159738751993953040495367313265","date":"2025-09-28T00:13:48+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"319122245859717385755367087369750857639","date":"2025-09-27T07:52:50+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-09-27T03:27:32+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"165500005158375755889741358015509145157","date":"2025-09-18T12:53:46+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"36557491933083353409343716811400581669","date":"2025-09-18T01:51:07+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-09-17T22:26:25+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"125694254271421402286634815557530801553","date":"2025-09-16T14:18:57+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"41495122789185734873338737267733919283","date":"2025-09-16T13:11:54+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"235338644548812777286984015258296324381","date":"2025-09-16T08:56:17+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-09-16T08:10:23+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2025-09-08T17:24:18+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-09-05T10:30:11+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-09-04T05:07:28+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2025-09-03T22:08:20+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"4323fad7-27fd-4981-9194-ec3cc3d6add3","owner":[],"postedDate":"September 24th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":54949143,"name":"Biological sciences/Computational biology and bioinformatics"},{"id":54949144,"name":"Physical sciences/Engineering"},{"id":54949145,"name":"Physical sciences/Mathematics and computing"}],"tags":[],"updatedAt":"2025-12-01T16:08:54+00:00","versionOfRecord":{"articleIdentity":"rs-7530459","link":"https://doi.org/10.1038/s41598-025-28686-z","journal":{"identity":"scientific-reports","isVorOnly":false,"title":"Scientific Reports"},"publishedOn":"2025-11-24 15:57:29","publishedOnDateReadable":"November 24th, 2025"},"versionCreatedAt":"2025-09-24 03:12:35","video":"","vorDoi":"10.1038/s41598-025-28686-z","vorDoiUrl":"https://doi.org/10.1038/s41598-025-28686-z","workflowStages":[]},"version":"v1","identity":"rs-7530459","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7530459","identity":"rs-7530459","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}