PolicySegNet: A Policy-Based Reinforcement Learning Framework with Pretrained Embeddings and Transformer Decoder for Joint Brain Tumors Segmentation and Classification in MRI

PolicySegNet: A Policy-Based Reinforcement Learning Framework with Pretrained Embeddings and Transformer Decoder for Joint Brain Tumors Segmentation and Classification in MRI | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article PolicySegNet: A Policy-Based Reinforcement Learning Framework with Pretrained Embeddings and Transformer Decoder for Joint Brain Tumors Segmentation and Classification in MRI Vishv Patel, Vandana Patel, Aakash Shinde This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6802822/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 20 Aug, 2025 Read the published version in Egyptian Journal of Radiology and Nuclear Medicine → Version 1 posted You are reading this latest preprint version Abstract PolicySegNet is a novel hybrid deep learning architecture developed for joint brain tumor segmentation and classification using MRI scans. It combines a pretrained SegFormer-B4 encoder (with a MiT backbone, originally trained on the ADE20K dataset) as a fixed feature extractor with a UNet-inspired decoder for segmentation and a lightweight classification head for tumor type identification. Unlike typical fine-tuning approaches, the SegFormer encoder remains frozen, enabling efficient training on limited domain-specific data. PolicySegNet uniquely integrates a policy-based reinforcement learning algorithm—specifically Proximal Policy Optimization (PPO)—to jointly optimize the decoder and classifier based on a reward signal that balances segmentation accuracy with classification performance. The segmentation task involves four distinct binary masks, each representing a tumor class. Experimental results on a multi-class brain tumor MRI dataset demonstrate strong performance: on the training set, the model achieves a segmentation accuracy of 0.9961 and classification accuracy of 0.9133, on the validation set, it achieves a segmentation accuracy of 0.9936 and classification accuracy of 0.9175, and on the test set, it achieves a segmentation accuracy of 0.9924 and classification accuracy of 0.8803. During training, the model attains a reward of 0.7295. These results showcase the potential of combining transformer-based vision features with reinforcement learning strategies for improved medical image analysis, while requiring fewer computational resources due to the fixed encoder and lightweight architectural design. Classification image embeddings MRI images semantic segmentation reinforcement learning vision transformer Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 20 Aug, 2025 Read the published version in Egyptian Journal of Radiology and Nuclear Medicine → Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6802822","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":473240593,"identity":"13cc6ca1-631c-4f29-b96a-4d4b06b76206","order_by":0,"name":"Vishv Patel","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA/UlEQVRIie3PMWrDMBSAYQmBM2bVMV4I1M3kg3SR8BpBRg+pERTUJQfwUNIruEugm4zBXgReA10ccoF009iXjKW2OxaqfxDo8T6QCAmF/mo94QQI1SAyvDGmJ/YZIeJGmN307jqgvyIESSQ+T+Y6mSDxrKt6ae6TmK8Xpdw/PsyfkfjsMEhWu5SBNFy+F+slyEOrippqunMfgwRsGnHhuADn7pA0SiNh1IyQ7jzzSBIksZcvjXqdJMc0IiLjtGyNAKm3qpwiq+K85Ehk2T5ZEI1Vb0iqsb/Ec3m6XCBPADcXfpurfVdXvc9GHvbtXt9OO7j/A8nHlkOhUOif9gXxf2DxmG2y+AAAAABJRU5ErkJggg==","orcid":"","institution":"","correspondingAuthor":true,"prefix":"","firstName":"Vishv","middleName":"","lastName":"Patel","suffix":""},{"id":473240594,"identity":"6f786a8a-cc03-42da-8f74-75ad3abffb03","order_by":1,"name":"Vandana Patel","email":"","orcid":"","institution":"University of California, San Diego","correspondingAuthor":false,"prefix":"","firstName":"Vandana","middleName":"","lastName":"Patel","suffix":""},{"id":473240595,"identity":"193379bd-af48-46fe-9751-be929a074f0f","order_by":2,"name":"Aakash Shinde","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Aakash","middleName":"","lastName":"Shinde","suffix":""}],"badges":[],"createdAt":"2025-06-02 13:53:21","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6802822/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6802822/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1186/s43055-025-01557-3","type":"published","date":"2025-08-20T16:12:52+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":89847015,"identity":"4e84b2bb-7dab-4372-a03e-a27b579faa94","added_by":"auto","created_at":"2025-08-25 16:37:46","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":664160,"visible":true,"origin":"","legend":"","description":"","filename":"manuscriptSNEJRNM2025anonymised.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6802822/v1_covered_68a29805-599b-4ac5-a4d5-8a360e5404cb.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"PolicySegNet: A Policy-Based Reinforcement Learning Framework with Pretrained Embeddings and Transformer Decoder for Joint Brain Tumors Segmentation and Classification in MRI","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Classification, image embeddings, MRI images, semantic segmentation, reinforcement learning, vision transformer","lastPublishedDoi":"10.21203/rs.3.rs-6802822/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6802822/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e \u003cb\u003ePolicySegNet\u003c/b\u003e is a novel hybrid deep learning architecture developed for joint brain tumor segmentation and classification using MRI scans. It combines a pretrained SegFormer-B4 encoder (with a MiT backbone, originally trained on the ADE20K dataset) as a fixed feature extractor with a UNet-inspired decoder for segmentation and a lightweight classification head for tumor type identification. Unlike typical fine-tuning approaches, the SegFormer encoder remains frozen, enabling efficient training on limited domain-specific data. PolicySegNet uniquely integrates a policy-based reinforcement learning algorithm\u0026mdash;specifically Proximal Policy Optimization (PPO)\u0026mdash;to jointly optimize the decoder and classifier based on a reward signal that balances segmentation accuracy with classification performance. The segmentation task involves four distinct binary masks, each representing a tumor class. Experimental results on a multi-class brain tumor MRI dataset demonstrate strong performance: on the training set, the model achieves a segmentation accuracy of 0.9961 and classification accuracy of 0.9133, on the validation set, it achieves a segmentation accuracy of 0.9936 and classification accuracy of 0.9175, and on the test set, it achieves a segmentation accuracy of 0.9924 and classification accuracy of 0.8803. During training, the model attains a reward of 0.7295. These results showcase the potential of combining transformer-based vision features with reinforcement learning strategies for improved medical image analysis, while requiring fewer computational resources due to the fixed encoder and lightweight architectural design.\u003c/p\u003e","manuscriptTitle":"PolicySegNet: A Policy-Based Reinforcement Learning Framework with Pretrained Embeddings and Transformer Decoder for Joint Brain Tumors Segmentation and Classification in MRI","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-06-19 04:54:16","doi":"10.21203/rs.3.rs-6802822/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"e93db1d8-3479-484a-8cbd-0a3af5bfaca5","owner":[],"postedDate":"June 19th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-08-25T16:30:04+00:00","versionOfRecord":{"articleIdentity":"rs-6802822","link":"https://doi.org/10.1186/s43055-025-01557-3","journal":{"identity":"egyptian-journal-of-radiology-and-nuclear-medicine","isVorOnly":false,"title":"Egyptian Journal of Radiology and Nuclear Medicine"},"publishedOn":"2025-08-20 16:12:52","publishedOnDateReadable":"August 20th, 2025"},"versionCreatedAt":"2025-06-19 04:54:16","video":"","vorDoi":"10.1186/s43055-025-01557-3","vorDoiUrl":"https://doi.org/10.1186/s43055-025-01557-3","workflowStages":[]},"version":"v1","identity":"rs-6802822","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6802822","identity":"rs-6802822","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}