RSGM: A Multimodal Image Matching Scheme Guided by Regional Semantics

RSGM: A Multimodal Image Matching Scheme Guided by Regional Semantics | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article RSGM: A Multimodal Image Matching Scheme Guided by Regional Semantics Fangwei Jin, Yun Liao, Junhui Liu, Xu Qian, YunPeng Li, RongRui Teng, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5662582/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Multimodal image feature matching technology plays an important role in computer vision research, and a lot of work has been done to improve the matching performance. However, existing matching methods lack sufficient extraction of descriptor information and clear semantic guidance during the matching process. Most descriptors only contain pixel information, neglecting region-based and inter-image information. While some approaches attempt to incorporate semantic information into descriptors, they do not utilize this information during the actual matching stage, leading to potential incorrect matches when similar object types are present in the images. In order to solve these problems, We put forward a new multimodal local image feature matching method that includes a novel descriptor extraction method and a new matching scheme. During the feature extraction and fusion stage, similarity information, regional semantic information of image pairs, as well as neighboring pixel information of corresponding match points from different images are fused to enrich the feature descriptors. In the matching stage, a method guided by regional semantic information(RSGM) is employed to enhance the overall matching effect. Finally, extensive experiments demonstrate that RSGM performs exceptionally well on multimodal image datasets. The code for RSGM is available at: https://github.com/LiaoYun0x0/RSGM. Multimodal Image Feature Matching Regional Semantic Information Similarity Information Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5662582","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":392959461,"identity":"60b3704d-771b-4314-a3eb-f35906964c83","order_by":0,"name":"Fangwei Jin","email":"","orcid":"","institution":"Yunnan University","correspondingAuthor":false,"prefix":"","firstName":"Fangwei","middleName":"","lastName":"Jin","suffix":""},{"id":392959462,"identity":"35fc04cf-14dd-4887-b5b4-84095f6b3a45","order_by":1,"name":"Yun Liao","email":"","orcid":"","institution":"Yunnan University","correspondingAuthor":false,"prefix":"","firstName":"Yun","middleName":"","lastName":"Liao","suffix":""},{"id":392959463,"identity":"f57e4859-9f9e-4445-afb1-e9852c16b8da","order_by":2,"name":"Junhui Liu","email":"","orcid":"","institution":"Yunnan University","correspondingAuthor":false,"prefix":"","firstName":"Junhui","middleName":"","lastName":"Liu","suffix":""},{"id":392959465,"identity":"12ba6252-3c20-41bb-8c61-0c0529f6e3a1","order_by":3,"name":"Xu Qian","email":"","orcid":"","institution":"Yunnan University","correspondingAuthor":false,"prefix":"","firstName":"Xu","middleName":"","lastName":"Qian","suffix":""},{"id":392959466,"identity":"a931f561-da3e-4954-b925-37fdc09b7610","order_by":4,"name":"YunPeng Li","email":"","orcid":"","institution":"Yunnan University","correspondingAuthor":false,"prefix":"","firstName":"YunPeng","middleName":"","lastName":"Li","suffix":""},{"id":392959467,"identity":"051b8c99-5b8b-4227-8cb8-1deae3c348a6","order_by":5,"name":"RongRui Teng","email":"","orcid":"","institution":"Yunnan University","correspondingAuthor":false,"prefix":"","firstName":"RongRui","middleName":"","lastName":"Teng","suffix":""},{"id":392959468,"identity":"f5dcbadc-132f-4fc0-a14a-785f8c35d793","order_by":6,"name":"ZongXiao Hu","email":"","orcid":"","institution":"Yunnan University","correspondingAuthor":false,"prefix":"","firstName":"ZongXiao","middleName":"","lastName":"Hu","suffix":""},{"id":392959469,"identity":"195b9917-e3ae-4ef4-9258-f0808a1cfbb5","order_by":7,"name":"Qing Duan","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAt0lEQVRIiWNgGAWjYFCCBCCugDAlSNByhmQtjG2kaOE7nmP4uXDe4TyDA8wHb/Mw2OUR1CJ55o2x9Mxth4sNDrAlW/MwJBcT1GJwI8dAmnfb4cQNB3jMpHkYDiQ2EKHF+DfvHJAW/m9EazGT5m0A28JGnBbJM8/KrHmOpSfOPMxmbDnHIJmwFr7jyZtv89RYJ/Ydb354402FHWEtDAfAZDMDAzPYnQTVw7XUEaN0FIyCUTAKRioAAEfzPWyiSIBWAAAAAElFTkSuQmCC","orcid":"","institution":"Yunnan University","correspondingAuthor":true,"prefix":"","firstName":"Qing","middleName":"","lastName":"Duan","suffix":""}],"badges":[],"createdAt":"2024-12-17 14:23:18","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5662582/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5662582/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":77371383,"identity":"c10a13cd-a749-41c5-aee9-3cdc33b746de","added_by":"auto","created_at":"2025-02-28 00:46:36","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":23624382,"visible":true,"origin":"","legend":"","description":"","filename":"RSGM.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5662582/v1_covered_228da026-7d7a-4e6b-aee4-f855d070622c.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"RSGM: A Multimodal Image Matching Scheme Guided by Regional Semantics","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Multimodal Image, Feature Matching, Regional Semantic Information, Similarity Information","lastPublishedDoi":"10.21203/rs.3.rs-5662582/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5662582/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eMultimodal image feature matching technology plays an important role in computer vision research, and a lot of work has been done to improve the matching performance. However, existing matching methods lack sufficient extraction of descriptor information and clear semantic guidance during the matching process. Most descriptors only contain pixel information, neglecting region-based and inter-image information. While some approaches attempt to incorporate semantic information into descriptors, they do not utilize this information during the actual matching stage, leading to potential incorrect matches when similar object types are present in the images. In order to solve these problems, We put forward a new multimodal local image feature matching method that includes a novel descriptor extraction method and a new matching scheme. During the feature extraction and fusion stage, similarity information, regional semantic information of image pairs, as well as neighboring pixel information of corresponding match points from different images are fused to enrich the feature descriptors. In the matching stage, a method guided by regional semantic information(RSGM) is employed to enhance the overall matching effect. Finally, extensive experiments demonstrate that RSGM performs exceptionally well on multimodal image datasets. The code for RSGM is available at: https://github.com/LiaoYun0x0/RSGM.\u003c/p\u003e","manuscriptTitle":"RSGM: A Multimodal Image Matching Scheme Guided by Regional Semantics","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-12-23 18:21:29","doi":"10.21203/rs.3.rs-5662582/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"4ff9a93e-fc00-4c6a-8a5a-2e71cd19a7cb","owner":[],"postedDate":"December 23rd, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-02-28T00:38:19+00:00","versionOfRecord":[],"versionCreatedAt":"2024-12-23 18:21:29","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5662582","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5662582","identity":"rs-5662582","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}