Multimodal Brain Tumor Classification via Triple Fusion Attention and Transformer-Based Feature Integration

Multimodal Brain Tumor Classification via Triple Fusion Attention and Transformer-Based Feature Integration | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Multimodal Brain Tumor Classification via Triple Fusion Attention and Transformer-Based Feature Integration G. Isha, F. D Astrabel Sherlin This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8858830/v1 This work is licensed under a CC BY 4.0 License Status: Under Revision Version 1 posted 12 You are reading this latest preprint version Abstract Brain tumor classification using multimodal neuroimaging continues to be a difficult undertaking because the heterogeneous kind of tumor appearance across imaging modalities and the complexity of meaningful feature integration. To overcome these restrictions, this research proposes a novel hybrid fusion pipeline that leverages the complimentary qualities of MRI and PET modalities through a synergistic combination of advanced deep learning and radiomics techniques. The pipeline begins with modality-specific denoising to enhance data quality, followed by precise tumor segmentation using a Multimodal Swin Transformer U-Net (MM-SwinUNet), capable of simulating cross-modal interactions and long-range dependency. For feature extraction, Vision Transformer (ViT) embeddings are utilized for MRI, capturing rich semantic representations, while handcrafted and CNN-based Deep Radiomics features are extracted from PET, preserving modality-specific morphological and metabolic information. These distinct features are then unified using a Triple Fusion Attention Module (TFAM), which dynamically attends to relevant features across modalities to form robust fused representations. To combat high dimensionality and enhance class separability, the fused features undergo Supervised UMAP embedding with Local Discriminant Structure Preservation. The final classification is performed using a SimCLR-pretrained ViT model, fine-tuned on the fused feature space to leverage contrastive pretraining and improve generalization. This novel pipeline demonstrates improved classification accuracy, enhanced modality synergy, and clinical interpretability. The proposed method holds significant promise for aiding diagnostic decision-making and advancing the role of AI in neuro-oncology. Brain Tumor Classification Multimodal Fusion Triple Fusion Attention Vision Transformer Deep Radiomics MM-SwinUNet SimCLR Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Revision Version 1 posted Editorial decision: Revision requested 21 Apr, 2026 Reviews received at journal 17 Apr, 2026 Reviewers agreed at journal 08 Apr, 2026 Reviews received at journal 08 Apr, 2026 Reviewers agreed at journal 19 Mar, 2026 Reviewers agreed at journal 17 Mar, 2026 Reviewers agreed at journal 24 Feb, 2026 Reviewers agreed at journal 23 Feb, 2026 Reviewers invited by journal 23 Feb, 2026 Editor assigned by journal 22 Feb, 2026 Submission checks completed at journal 19 Feb, 2026 First submitted to journal 12 Feb, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8858830","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":596754452,"identity":"efd1e1ee-f76c-49de-b7af-f0cabb118c02","order_by":0,"name":"G. Isha","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABPUlEQVRIie2RMUvDQBTHX3iQOFy6uFyImK9woRAtVDv6NSyBZElRcBEULATOJda1Tn4FXTorhfQrFK5DpZDJIaVQghTxWhykScBRML/h8d7jfvy5O4CKir8Igvqjh0NQtfB1mspJ1X6nUKiR2LX7awVLc7aUfRo4u+R7LMI60ZJpBsdnTIRvi/NraqnU87D5MbBqCEo6D3KKPSQHdgRu42kS181+TG1OZvGs0xM2R0DjYZBXQqJSAsjY+NRB2Stc8/x6JxKKVFTUixQtMVZwIxV/ieSTtjgEjtmIRKtMsRAck8BQKoGDOqdtviMVyES7TGFIHHOPjZgxDi5MvUddvn7ku65wOSph0V2s+1FivF9esdrYf16QZfPo8VZ+ZbYSmyadF6S8bOr2WuGb2s2dX6cUbgFWxeuKioqKf8kX++hiOCLEd9gAAAAASUVORK5CYII=","orcid":"","institution":"Anna University, Chennai","correspondingAuthor":true,"prefix":"","firstName":"G.","middleName":"","lastName":"Isha","suffix":""},{"id":596754453,"identity":"09c8fade-08c7-4947-8368-6d7a23129fd3","order_by":1,"name":"F. D Astrabel Sherlin","email":"","orcid":"","institution":"Anna University, Chennai","correspondingAuthor":false,"prefix":"","firstName":"F.","middleName":"D Astrabel","lastName":"Sherlin","suffix":""}],"badges":[],"createdAt":"2026-02-12 07:24:12","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8858830/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8858830/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":103777820,"identity":"f07bbc81-de1b-4d5c-b5eb-5c9848e2b2a0","added_by":"auto","created_at":"2026-03-02 19:24:52","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":790486,"visible":true,"origin":"","legend":"","description":"","filename":"IJCIS.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8858830/v1_covered_6d527669-c9c1-4fe0-be4c-4366ffb6722b.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Multimodal Brain Tumor Classification via Triple Fusion Attention and Transformer-Based Feature Integration","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"international-journal-of-computational-intelligence-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [International Journal of Computational Intelligence Systems](https://link.springer.com/journal/44196)","snPcode":"44196","submissionUrl":"https://submission.springernature.com/new-submission/44196/3","title":"International Journal of Computational Intelligence Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Open","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Brain Tumor Classification, Multimodal Fusion, Triple Fusion Attention, Vision Transformer, Deep Radiomics, MM-SwinUNet, SimCLR","lastPublishedDoi":"10.21203/rs.3.rs-8858830/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8858830/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eBrain tumor classification using multimodal neuroimaging continues to be a difficult undertaking because the heterogeneous kind of tumor appearance across imaging modalities and the complexity of meaningful feature integration. To overcome these restrictions, this research proposes a novel hybrid fusion pipeline that leverages the complimentary qualities of MRI and PET modalities through a synergistic combination of advanced deep learning and radiomics techniques. The pipeline begins with modality-specific denoising to enhance data quality, followed by precise tumor segmentation using a Multimodal Swin Transformer U-Net (MM-SwinUNet), capable of simulating cross-modal interactions and long-range dependency. For feature extraction, Vision Transformer (ViT) embeddings are utilized for MRI, capturing rich semantic representations, while handcrafted and CNN-based Deep Radiomics features are extracted from PET, preserving modality-specific morphological and metabolic information. These distinct features are then unified using a Triple Fusion Attention Module (TFAM), which dynamically attends to relevant features across modalities to form robust fused representations. To combat high dimensionality and enhance class separability, the fused features undergo Supervised UMAP embedding with Local Discriminant Structure Preservation. The final classification is performed using a SimCLR-pretrained ViT model, fine-tuned on the fused feature space to leverage contrastive pretraining and improve generalization. This novel pipeline demonstrates improved classification accuracy, enhanced modality synergy, and clinical interpretability. The proposed method holds significant promise for aiding diagnostic decision-making and advancing the role of AI in neuro-oncology.\u003c/p\u003e","manuscriptTitle":"Multimodal Brain Tumor Classification via Triple Fusion Attention and Transformer-Based Feature Integration","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-03-02 19:23:53","doi":"10.21203/rs.3.rs-8858830/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-04-21T15:31:19+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-17T13:27:11+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"175410810731272557464484101840322479021","date":"2026-04-08T05:52:30+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-08T04:03:51+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"163368784921351031839965222217130865012","date":"2026-03-19T15:05:37+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"152553238696346622642298468110399064530","date":"2026-03-17T10:57:06+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"146799412909647911860695728120510078070","date":"2026-02-25T04:35:09+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"156101366822071359462440952476378075037","date":"2026-02-24T04:30:02+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-02-23T13:15:10+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-02-23T00:51:00+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-02-19T10:28:36+00:00","index":"","fulltext":""},{"type":"submitted","content":"International Journal of Computational Intelligence Systems","date":"2026-02-12T07:14:24+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"international-journal-of-computational-intelligence-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [International Journal of Computational Intelligence Systems](https://link.springer.com/journal/44196)","snPcode":"44196","submissionUrl":"https://submission.springernature.com/new-submission/44196/3","title":"International Journal of Computational Intelligence Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Open","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"0a3f3b66-3a55-4755-82a6-1852f1ff676d","owner":[],"postedDate":"March 2nd, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"in-revision","subjectAreas":[],"tags":[],"updatedAt":"2026-04-21T15:41:05+00:00","versionOfRecord":[],"versionCreatedAt":"2026-03-02 19:23:53","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8858830","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8858830","identity":"rs-8858830","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}