Multi-Modal Data Fusion With Federated Multi-Head Attention for Diabetic Retinopathy Severity Classification
preprint
OA: closed
AI-generated summary
This study developed a multi-modal deep learning approach fusing fundus images and EHR data with federated learning to accurately classify diabetic retinopathy severity while preserving patient privacy.
One-sentence paraphrase of the abstract; not a substitute for reading it. No clinical advice. How this works
AI-generated deep summary
The study develops and evaluates a multi-modal deep learning model for diabetic retinopathy (DR) severity classification that combines fundus images with EHR-derived tabular features (HbA1c, diabetes duration, and age). Using a ResNet-18 backbone for image feature extraction, an MLP for EHR processing, and a multi-head attention fusion mechanism, the authors report improved classification performance (higher accuracy and AUROC) versus image-only and EHR-only baselines, with attention visualizations highlighting retinal regions. They further implement federated learning across four simulated client sites using FedML and find the federated model achieves performance close to centralized training, with only a minor AUROC decrease. The paper is a preprint and uses the RetinaMNIST dataset (augmented with synthetic EHR data), which is an explicit caveat regarding data realism and peer-reviewed validation. The paper does not explicitly discuss endometriosis or adenomyosis; it was included in the corpus via a keyword match in the upstream search index.
Read from the paper's body, not the abstract. Not a substitute for reading the paper. No clinical advice. How this works
Abstract
Abstract Purpose Diabetic retinopathy (DR) is a leading cause of vision loss. Deep learning models have achieved high accuracy in DR detection from fundus images, but often ignore valuable patient information in electronic health records (EHR). This study develops a multi-modal data fusion approach for DR severity classification, integrating fundus images with key EHR features (glycated hemoglobin HbA1c, diabetes duration, and age) to improve performance. We also extend it to a federated learning framework for decentralized training across sites while preserving privacy. Key research questions: Can attention-based fusion of imaging and EHR data enhance DR classification over single-modality models? Does federated learning achieve comparable results to centralized training without sharing sensitive data? Methods The architecture utilizes ResNet-18 for extracting features from fundus images and a multilayer perceptron (MLP) for processing EHR tabular data. A multi-head attention mechanism fuses modalities to learn task-relevant interactions. Federated learning is implemented via FedML, simulating training across four client sites. Experiments use the RetinaMNIST dataset (1,600 fundus images) augmented with synthetic EHR data. Results The multi-modal model outperformed image-only and EHR-only baselines, achieving higher accuracy and area under the ROC curve (AUROC). The federated learning variant yielded performance close to the centrally trained model, with only a minor decrease in AUROC. Visualizations of attention weights revealed clinically relevant retinal regions. Conclusion Incorporating EHR data via attention-based fusion significantly improves DR severity grading, while federated learning facilitates secure multi-center collaboration without data sharing. This approach holds promise for real-world clinical deployment in privacy-sensitive environments.
Full text
17,190 characters
· extracted from
preprint-html
· click to expand
Multi-Modal Data Fusion With Federated Multi-Head Attention for Diabetic Retinopathy Severity Classification | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Multi-Modal Data Fusion With Federated Multi-Head Attention for Diabetic Retinopathy Severity Classification John Ezembu, Rita Orji, Oladapo Oyebode This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8191406/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Purpose Diabetic retinopathy (DR) is a leading cause of vision loss. Deep learning models have achieved high accuracy in DR detection from fundus images, but often ignore valuable patient information in electronic health records (EHR). This study develops a multi-modal data fusion approach for DR severity classification, integrating fundus images with key EHR features (glycated hemoglobin HbA1c, diabetes duration, and age) to improve performance. We also extend it to a federated learning framework for decentralized training across sites while preserving privacy. Key research questions: Can attention-based fusion of imaging and EHR data enhance DR classification over single-modality models? Does federated learning achieve comparable results to centralized training without sharing sensitive data? Methods The architecture utilizes ResNet-18 for extracting features from fundus images and a multilayer perceptron (MLP) for processing EHR tabular data. A multi-head attention mechanism fuses modalities to learn task-relevant interactions. Federated learning is implemented via FedML, simulating training across four client sites. Experiments use the RetinaMNIST dataset (1,600 fundus images) augmented with synthetic EHR data. Results The multi-modal model outperformed image-only and EHR-only baselines, achieving higher accuracy and area under the ROC curve (AUROC). The federated learning variant yielded performance close to the centrally trained model, with only a minor decrease in AUROC. Visualizations of attention weights revealed clinically relevant retinal regions. Conclusion Incorporating EHR data via attention-based fusion significantly improves DR severity grading, while federated learning facilitates secure multi-center collaboration without data sharing. This approach holds promise for real-world clinical deployment in privacy-sensitive environments. Diabetic retinopathy multimodal data fusion attention mechanism federated learning electronic health records medical imaging Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8191406","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":551249350,"identity":"f3d9acfc-0764-434d-9349-e69269b9e82b","order_by":0,"name":"John Ezembu","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA+0lEQVRIiWNgGAWjYJACZjCWYGB8DBORIFYLszFJWkDK2KSJ0sLPf/jo5sI2BnZ+6d5j1QUVh+XMGZgP3uZhuGPXgEOLZMOxtNsz2xiYJeecS7s948xhY8sGtmRrHoZnybi0GBzsMbvNC9RicCMHxLiduOEAj5k0D8PhZFwOsz/M/w2upRiopX7DAf5veLUYsPGwwbUwA7UkGBzgYQNpscOlReIMm9ntGeckmCVn5BhLzzjz33DDYTZjyzkGhxNwaeHvP/zsdkGZTTK/RI7h54KKNHmD480Pb7ypOGyPSwvMMiSXg6PJgCGxgYAeTJcTsmUUjIJRMApGDgAAA7dQWXsyU/4AAAAASUVORK5CYII=","orcid":"","institution":"Bowen University","correspondingAuthor":true,"prefix":"","firstName":"John","middleName":"","lastName":"Ezembu","suffix":""},{"id":551249351,"identity":"0302cd9e-1f62-49d1-8f4a-9ab95c46e664","order_by":1,"name":"Rita Orji","email":"","orcid":"","institution":"Dalhousie University","correspondingAuthor":false,"prefix":"","firstName":"Rita","middleName":"","lastName":"Orji","suffix":""},{"id":551249352,"identity":"1ed7a5ad-2e63-4478-b282-e6c8908c91d1","order_by":2,"name":"Oladapo Oyebode","email":"","orcid":"","institution":"Dalhousie University","correspondingAuthor":false,"prefix":"","firstName":"Oladapo","middleName":"","lastName":"Oyebode","suffix":""}],"badges":[],"createdAt":"2025-11-24 09:26:03","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8191406/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8191406/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":97138618,"identity":"90bcecae-ae06-48d1-b6fb-39786e805b1f","added_by":"auto","created_at":"2025-12-01 09:59:07","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":562905,"visible":true,"origin":"","legend":"","description":"","filename":"MultimodalDataFusionPaperJHIR.docx","url":"https://assets-eu.researchsquare.com/files/rs-8191406/v1/7c03a5f31e205cc8cf245aa9.docx"},{"id":97003230,"identity":"397158f1-ab1d-437a-817c-a5b7e2a9bca4","added_by":"auto","created_at":"2025-11-28 14:02:54","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":6066,"visible":true,"origin":"","legend":"","description":"","filename":"a32303c4072f4cd2b0a14591c5aba588.json","url":"https://assets-eu.researchsquare.com/files/rs-8191406/v1/32f7c4e50c742f5b190674be.json"},{"id":97003240,"identity":"262cb017-910d-44ef-bb07-4842d1b05428","added_by":"auto","created_at":"2025-11-28 14:02:54","extension":"xml","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":98851,"visible":true,"origin":"","legend":"","description":"","filename":"a32303c4072f4cd2b0a14591c5aba5881enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-8191406/v1/556ec670656424523cd8aed4.xml"},{"id":97139276,"identity":"1dca48b0-cbae-480e-9f3b-23d5a595cb4a","added_by":"auto","created_at":"2025-12-01 09:59:56","extension":"png","order_by":3,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":176713,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-8191406/v1/b621c418e26301bf06a6edef.png"},{"id":97003234,"identity":"172dec4c-336c-4ce6-80f2-79ddb94b019e","added_by":"auto","created_at":"2025-11-28 14:02:54","extension":"png","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":123111,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-8191406/v1/12dcb847e2f08d0429873edc.png"},{"id":97003237,"identity":"84744535-7957-43c3-9c70-cd56a59c646c","added_by":"auto","created_at":"2025-11-28 14:02:54","extension":"png","order_by":5,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":71860,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-8191406/v1/c19ad12d5354aefd1a363d8a.png"},{"id":97139367,"identity":"0a111629-ef3e-4d71-9c1c-56d8669f9a4e","added_by":"auto","created_at":"2025-12-01 10:00:09","extension":"jpeg","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":135820,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage4.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-8191406/v1/cbc6cea86adc1f5088f0244a.jpeg"},{"id":97003238,"identity":"b6979ed8-0361-48d2-a727-4f0e89502387","added_by":"auto","created_at":"2025-11-28 14:02:54","extension":"png","order_by":7,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":10587,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-8191406/v1/db4e036f2320564a9287898b.png"},{"id":97138627,"identity":"492c86d4-d0a5-43f8-9e8b-3af7cf17efae","added_by":"auto","created_at":"2025-12-01 09:59:08","extension":"png","order_by":8,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":28402,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-8191406/v1/1854e29235d9d1bfc6d9ab07.png"},{"id":97003232,"identity":"91c215c7-a66a-412c-ba3e-eb43bdf7ac8a","added_by":"auto","created_at":"2025-11-28 14:02:54","extension":"png","order_by":9,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":15912,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-8191406/v1/5087034082293ef3c24e1020.png"},{"id":97137714,"identity":"0e93a6f6-60cc-4562-bc4b-1b73438742e8","added_by":"auto","created_at":"2025-12-01 09:58:05","extension":"png","order_by":10,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":63768,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-8191406/v1/26ab923c85de38ff37ee3fe1.png"},{"id":97003242,"identity":"43555ec1-05a8-4d8f-b42c-43cbd50cece8","added_by":"auto","created_at":"2025-11-28 14:02:54","extension":"xml","order_by":11,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":94047,"visible":true,"origin":"","legend":"","description":"","filename":"a32303c4072f4cd2b0a14591c5aba5881structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-8191406/v1/094fd3cf18f2590b8ae4a755.xml"},{"id":97003243,"identity":"3cd89e26-869d-497a-b699-71afd0b6ba18","added_by":"auto","created_at":"2025-11-28 14:02:54","extension":"html","order_by":12,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":108960,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-8191406/v1/a93eef6750611044dea3386b.html"},{"id":97144737,"identity":"7a3b906d-bfbc-414c-a846-abad076186ef","added_by":"auto","created_at":"2025-12-01 10:11:50","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":679553,"visible":true,"origin":"","legend":"","description":"","filename":"MultimodalDataFusionPaperJHIR.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8191406/v1_covered_0239a223-67f2-407d-bd2a-757fdd847b3c.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Multi-Modal Data Fusion With Federated Multi-Head Attention for Diabetic Retinopathy Severity Classification","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Diabetic retinopathy, multimodal data fusion, attention mechanism, federated learning, electronic health records, medical imaging","lastPublishedDoi":"10.21203/rs.3.rs-8191406/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8191406/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003ePurpose\u003c/h2\u003e\u003cp\u003eDiabetic retinopathy (DR) is a leading cause of vision loss. Deep learning models have achieved high accuracy in DR detection from fundus images, but often ignore valuable patient information in electronic health records (EHR). This study develops a multi-modal data fusion approach for DR severity classification, integrating fundus images with key EHR features (glycated hemoglobin HbA1c, diabetes duration, and age) to improve performance. We also extend it to a federated learning framework for decentralized training across sites while preserving privacy. Key research questions: Can attention-based fusion of imaging and EHR data enhance DR classification over single-modality models? Does federated learning achieve comparable results to centralized training without sharing sensitive data?\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e\u003cp\u003eThe architecture utilizes ResNet-18 for extracting features from fundus images and a multilayer perceptron (MLP) for processing EHR tabular data. A multi-head attention mechanism fuses modalities to learn task-relevant interactions. Federated learning is implemented via FedML, simulating training across four client sites. Experiments use the RetinaMNIST dataset (1,600 fundus images) augmented with synthetic EHR data.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e\u003cp\u003eThe multi-modal model outperformed image-only and EHR-only baselines, achieving higher accuracy and area under the ROC curve (AUROC). The federated learning variant yielded performance close to the centrally trained model, with only a minor decrease in AUROC. Visualizations of attention weights revealed clinically relevant retinal regions.\u003c/p\u003e\u003ch2\u003eConclusion\u003c/h2\u003e\u003cp\u003eIncorporating EHR data via attention-based fusion significantly improves DR severity grading, while federated learning facilitates secure multi-center collaboration without data sharing. This approach holds promise for real-world clinical deployment in privacy-sensitive environments.\u003c/p\u003e","manuscriptTitle":"Multi-Modal Data Fusion With Federated Multi-Head Attention for Diabetic Retinopathy Severity Classification","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-11-28 14:02:49","doi":"10.21203/rs.3.rs-8191406/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"bbc4e3ee-41c7-42f0-b93b-e98a08e62416","owner":[],"postedDate":"November 28th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-11-28T14:02:51+00:00","versionOfRecord":[],"versionCreatedAt":"2025-11-28 14:02:49","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8191406","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8191406","identity":"rs-8191406","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}
Text is read by the "Ask this paper" AI Q&A widget below. Extraction quality varies by source — PMC NXML preserves structure cleanly, OA-HTML may include some navigation residue, and OA-PDF can have broken hyphenation. The publisher copy (via DOI) is the canonical version.
My notes (saved in your browser only)
Ask this paper
Answers must be backed by verbatim quotes from this paper's full text. Hallucinated quotes are dropped automatically; if no verbatim passage answers the question, we say so. How this works
Citation neighborhood (no data yet)
We don't have any in-corpus citations linked to this paper yet. This is a recent paper (2025) — citers typically take a year or two to land, and the OpenAlex reference graph may still be filling in.
Source provenance
- europepmc
- last seen: 2026-05-20T01:45:00.602351+00:00