Multimodal Query Enhancement for Image Retrieval using Transformer Networks - MQIRTN

Multimodal Query Enhancement for Image Retrieval using Transformer Networks - MQIRTN | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Multimodal Query Enhancement for Image Retrieval using Transformer Networks - MQIRTN Seyed Mohammad Bagher Hosseini, Soodeh Bakhshandeh This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4213905/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract In today's world, with the dramatic increase of digital data and the variety of information, image recovery plays a very vital role in people's lives. For example, imagine a person in an online store looking for a dress similar to a friend's dress, but with distinct criteria such as blue flowers and red flags. The main challenge in this field is the existence of a semantic gap between the user's intention and the machine's understanding. The goal of image retrieval using multivariate (image-text) queries is to reduce this gap. An effective retrieval searches for images matching specific variations specified in a text expression related to the image. Existing methods use Convolutional Neural Networks (CNNs) and recurrent neural networks (LSTM) to extract image and text features that do not express the semantic relationships between the image and text well. In this regard, in order to improve these profiles, a transformer-based architecture is presented. In this research, Vision Transformer (ViT) is used to extract image features and Bidirectional Encoder (BERT) is used to extract text features. After combining the image and text features and applying the attention mechanism on images and text to enhance visual and text understanding. The proposed model of this research is inspired by deep metric learning and seeks to reduce the distance between the image and the input text with the searched images and the user's intent. In addition, a symmetric constraint has been used to improve the model indices. The evaluation results show that the proposed approach from the base model provides better performance in multimodal query-based image retrieval. To reproduce the results of this research, our source code is available at: https://github.com/smb-h/mqirtn Image retrieval Content-based image retrieval Text-based image retrieval Semantic gaps Cross-modal image retrieval Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4213905","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":287870235,"identity":"d1610c3c-3f14-47c4-9d61-8235e4816c81","order_by":0,"name":"Seyed Mohammad Bagher Hosseini","email":"","orcid":"","institution":"Department of Computer Engineering, South Tehran Branch, Islamic Azad University","correspondingAuthor":false,"prefix":"","firstName":"Seyed","middleName":"Mohammad Bagher","lastName":"Hosseini","suffix":""},{"id":287870236,"identity":"d7f3e589-7940-40fb-8172-4b7dba51e385","order_by":1,"name":"Soodeh Bakhshandeh","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABH0lEQVRIie2QMUvDUBDHX3jwupx0fRDbfoVIICj6YRoKmV6K0LVIQIhLTNcUBL9CpuKm8jBTQtc3uBU6OFkCBQeLlziJyZBN5P22O+7H/e8I0Wj+KNwIyJgAEIqFEXD6VDWPuyhsXDWhiwJW1W1VRjfXz2r3QKb9qJDl5fx1sDCj8k3NT4H05EvaoFh5Njlb5mTGi9gzk2xrL++K1bnIMBh4nmpSuHDMo5C4wRocCky6qfJXtmCocHCalFGCyicq92uwSzhI91GJrS0O7QpRqBiopEVk4TrcwgXd+GG7Ut9yG/LZSZ5jwljaifIc6sccWMst9cc+wovpMBcYbC8Hi2SyKcX+atjvyawx2Df8R8XqkrWP/4a+d5nWaDSaf88XwRNnX1IL2bgAAAAASUVORK5CYII=","orcid":"","institution":"Department of Computer Engineering, East Tehran Branch, Islamic Azad University","correspondingAuthor":true,"prefix":"","firstName":"Soodeh","middleName":"","lastName":"Bakhshandeh","suffix":""}],"badges":[],"createdAt":"2024-04-03 16:03:53","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4213905/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4213905/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":54453730,"identity":"350d6593-c100-4e8d-b5fe-518fb8e513ef","added_by":"auto","created_at":"2024-04-10 18:40:32","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1012270,"visible":true,"origin":"","legend":"","description":"","filename":"MQIRTNSumbitted.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4213905/v1_covered_7d5d4087-3180-4444-9f5c-78771b726b16.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Multimodal Query Enhancement for Image Retrieval using Transformer Networks - MQIRTN","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Image retrieval, Content-based image retrieval, Text-based image retrieval, Semantic gaps, Cross-modal image retrieval","lastPublishedDoi":"10.21203/rs.3.rs-4213905/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4213905/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"In today's world, with the dramatic increase of digital data and the variety of information, image recovery plays a very vital role in people's lives. For example, imagine a person in an online store looking for a dress similar to a friend's dress, but with distinct criteria such as blue flowers and red flags. The main challenge in this field is the existence of a semantic gap between the user's intention and the machine's understanding. The goal of image retrieval using multivariate (image-text) queries is to reduce this gap. An effective retrieval searches for images matching specific variations specified in a text expression related to the image. Existing methods use Convolutional Neural Networks (CNNs) and recurrent neural networks (LSTM) to extract image and text features that do not express the semantic relationships between the image and text well. In this regard, in order to improve these profiles, a transformer-based architecture is presented. In this research, Vision Transformer (ViT) is used to extract image features and Bidirectional Encoder (BERT) is used to extract text features. After combining the image and text features and applying the attention mechanism on images and text to enhance visual and text understanding. The proposed model of this research is inspired by deep metric learning and seeks to reduce the distance between the image and the input text with the searched images and the user's intent. In addition, a symmetric constraint has been used to improve the model indices. The evaluation results show that the proposed approach from the base model provides better performance in multimodal query-based image retrieval. To reproduce the results of this research, our source code is available at: https://github.com/smb-h/mqirtn","manuscriptTitle":"Multimodal Query Enhancement for Image Retrieval using Transformer Networks - MQIRTN","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-04-10 18:24:23","doi":"10.21203/rs.3.rs-4213905/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"fdb209ae-86d9-4ac5-9859-34905566c6ef","owner":[],"postedDate":"April 10th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-04-10T18:24:26+00:00","versionOfRecord":[],"versionCreatedAt":"2024-04-10 18:24:23","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4213905","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4213905","identity":"rs-4213905","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}