Semantic segmentation feature fusion network based on transformer

Semantic segmentation feature fusion network based on transformer | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Semantic segmentation feature fusion network based on transformer Tianping Li, Zhaotong Cui, Hua Zhang This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4543188/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 19 Feb, 2025 Read the published version in Scientific Reports → Version 1 posted 12 You are reading this latest preprint version Abstract Convolutional neural networks have demonstrated efficacy in acquiring local features and spatial details; however, they struggle to obtain global information, which could potentially compromise the segmentation of important regions of an image. Transformer can increase the expressiveness of pixels by establishing global relationships between them. Moreover, some transformer-based self-attentive methods do not combine the advantages of convolution, which makes the model require more computational parameters. This work uses both Transformer and CNN structures to improve the relationship between image-level regions and global information to improve segmentation accuracy and performance in order to address these two issues and improve the semantic segmentation segmentation results at the same time. We first build a Feature Alignment Module (FAM) module to enhance spatial details and improve channel representations. Second, we compute the link between similar pixels using a Transformer structure, which enhances the pixel representation. Finally, we design a Pyramid Convolutional Pooling Module (PCPM) that both compresses and enriches the feature maps, as well as determines the global correlations among the pixels, to reduce the computational burden on the transformer. These three elements come together to form a transformer-based semantic segmentation feature fusion network (FFTNet). Our method yields 82.5% mIoU, according to experimental results based on the Cityscapes test dataset. Furthermore, we conducted various visualization tests using the Pascal VOC 2012 and Cityscapes datasets. The results show that our approach outperforms alternative approaches. Physical sciences/Engineering Physical sciences/Physics/Information theory and computation Smantic segmentation transformer feature fusion Attention Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 19 Feb, 2025 Read the published version in Scientific Reports → Version 1 posted Editorial decision: Revision requested 01 Oct, 2024 Reviews received at journal 24 Sep, 2024 Reviewers agreed at journal 15 Sep, 2024 Reviews received at journal 10 Sep, 2024 Reviewers agreed at journal 30 Aug, 2024 Reviews received at journal 30 Aug, 2024 Reviewers agreed at journal 27 Aug, 2024 Reviewers invited by journal 24 Aug, 2024 Editor assigned by journal 24 Aug, 2024 Editor invited by journal 12 Jun, 2024 Submission checks completed at journal 10 Jun, 2024 First submitted to journal 06 Jun, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4543188","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":318169879,"identity":"d0f47c08-1909-42b6-8bb8-7b0f19185aa7","order_by":0,"name":"Tianping Li","email":"","orcid":"","institution":"Shandong Normal University","correspondingAuthor":false,"prefix":"","firstName":"Tianping","middleName":"","lastName":"Li","suffix":""},{"id":318169880,"identity":"94e87a40-8113-47e3-ad8d-e2cd1ce3d327","order_by":1,"name":"Zhaotong Cui","email":"","orcid":"","institution":"Shandong Normal University","correspondingAuthor":false,"prefix":"","firstName":"Zhaotong","middleName":"","lastName":"Cui","suffix":""},{"id":318169881,"identity":"06fc5e7b-ead1-4de1-b862-6843306f31f6","order_by":2,"name":"Hua Zhang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAv0lEQVRIiWNgGAWjYDACCR4gUcHA2ADi8BCv5QzJWhjbSNEiP7v3mMTPeXWy2yUSGB+8bWOQNyekhXHOuTTJ3m2HjXfOSGA2nNvGYLizgYAWZokcMwnebQcSN9xIYJPmbWNIMDhAQAsbUIvk3zl1IC3sv4nSwgPUIs3bwAy2hZkoLRISOcbWMscOG28487BZcs45CcMNhLTIz8gxvPmmpk52w/Hkgx/elNnIE7QFCYCjRoJ49aNgFIyCUTAKcAMAagU9DOFiVA8AAAAASUVORK5CYII=","orcid":"","institution":"Shandong Normal University","correspondingAuthor":true,"prefix":"","firstName":"Hua","middleName":"","lastName":"Zhang","suffix":""}],"badges":[],"createdAt":"2024-06-07 03:26:22","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4543188/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4543188/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41598-025-90518-x","type":"published","date":"2025-02-19T15:57:33+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":77052623,"identity":"04246d03-e34b-4082-b411-d5de85aff880","added_by":"auto","created_at":"2025-02-24 16:18:41","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":645518,"visible":true,"origin":"","legend":"","description":"","filename":"Manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4543188/v1_covered_852d887e-dd2d-42df-aee0-7a9e122f166e.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Semantic segmentation feature fusion network based on transformer","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Smantic segmentation, transformer, feature fusion, Attention","lastPublishedDoi":"10.21203/rs.3.rs-4543188/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4543188/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eConvolutional neural networks have demonstrated efficacy in acquiring local features and spatial details; however, they struggle to obtain global information, which could potentially compromise the segmentation of important regions of an image. Transformer can increase the expressiveness of pixels by establishing global relationships between them. Moreover, some transformer-based self-attentive methods do not combine the advantages of convolution, which makes the model require more computational parameters. This work uses both Transformer and CNN structures to improve the relationship between image-level regions and global information to improve segmentation accuracy and performance in order to address these two issues and improve the semantic segmentation segmentation results at the same time. We first build a Feature Alignment Module (FAM) module to enhance spatial details and improve channel representations. Second, we compute the link between similar pixels using a Transformer structure, which enhances the pixel representation. Finally, we design a Pyramid Convolutional Pooling Module (PCPM) that both compresses and enriches the feature maps, as well as determines the global correlations among the pixels, to reduce the computational burden on the transformer. These three elements come together to form a transformer-based semantic segmentation feature fusion network (FFTNet). Our method yields 82.5% mIoU, according to experimental results based on the Cityscapes test dataset. Furthermore, we conducted various visualization tests using the Pascal VOC 2012 and Cityscapes datasets. The results show that our approach outperforms alternative approaches.\u003c/p\u003e","manuscriptTitle":"Semantic segmentation feature fusion network based on transformer","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-06-27 21:04:07","doi":"10.21203/rs.3.rs-4543188/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-10-01T06:03:09+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-09-24T07:38:19+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"66674683529124297889766982076858546665","date":"2024-09-15T08:50:08+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-09-10T12:45:43+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"309766125814279442998011513291111208531","date":"2024-08-31T03:46:16+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-08-30T09:08:19+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"317488060712183366280031144325483128917","date":"2024-08-27T06:37:51+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-08-24T11:00:44+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-08-24T10:31:04+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2024-06-12T16:33:12+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-06-10T08:25:28+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2024-06-07T03:25:04+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"e060d46b-76cb-4567-b8b5-ff6788bd1fac","owner":[],"postedDate":"June 27th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":33638195,"name":"Physical sciences/Engineering"},{"id":33638196,"name":"Physical sciences/Physics/Information theory and computation"}],"tags":[],"updatedAt":"2025-02-24T16:02:17+00:00","versionOfRecord":{"articleIdentity":"rs-4543188","link":"https://doi.org/10.1038/s41598-025-90518-x","journal":{"identity":"scientific-reports","isVorOnly":false,"title":"Scientific Reports"},"publishedOn":"2025-02-19 15:57:33","publishedOnDateReadable":"February 19th, 2025"},"versionCreatedAt":"2024-06-27 21:04:07","video":"","vorDoi":"10.1038/s41598-025-90518-x","vorDoiUrl":"https://doi.org/10.1038/s41598-025-90518-x","workflowStages":[]},"version":"v1","identity":"rs-4543188","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4543188","identity":"rs-4543188","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}