MLTU: Mixup Long-Tail Unsupervised Zero-Shot Image Classification on Vision-Language Models

MLTU: Mixup Long-Tail Unsupervised Zero-Shot Image Classification on Vision-Language Models | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article MLTU: Mixup Long-Tail Unsupervised Zero-Shot Image Classification on Vision-Language Models Yunpeng Jia, Xiufen Ye, Xinkui Mei, Yusong Liu, Shuxiang Guo This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4139909/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 01 Jun, 2024 Read the published version in Multimedia Systems → Version 1 posted 9 You are reading this latest preprint version Abstract Vision-language models, such as Contrastive Language-Image Pretraining (CLIP), have demonstrated powerful capabilities in image classification under zero-shot settings. However, current Zero-Shot Learning (ZSL) relies on manually tagged samples of known classes through supervised learning, resulting in a waste of labor costs and limitations on foreseeable classes in real-world applications. To address these challenges, we propose the Mixup Long-Tail Unsupervised (MLTU) approach for open-world ZSL problems. The proposed approach employed a novel long-tail mixup loss that integrated class-based re-weighting assignments with a given mixup factor for each mixed visual embedding. To mitigate the adverse impact over time, we adopted a noisy learning strategy to filter out samples that generated incorrect labels. We reproduced the unsupervised results of existing state-of-the-art long-tail and noisy learning approaches. Experimental results demonstrate that MLTU achieves significant improvements in classification compared to these proven existing approaches on public datasets. Moreover, it serves as a plug-and-play solution for amending previous assignments and enhancing unsupervised performance. MLTU enables the automatic classification and correction of incorrect predictions caused by the projection bias of CLIP. Zero-Shot Learning Unsupervised Learning Long-Tail Learning Mixup Augmentation Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 01 Jun, 2024 Read the published version in Multimedia Systems → Version 1 posted Editorial decision: Revision requested 16 Apr, 2024 Reviews received at journal 15 Apr, 2024 Reviews received at journal 12 Apr, 2024 Reviewers agreed at journal 11 Apr, 2024 Reviewers agreed at journal 04 Apr, 2024 Reviewers invited by journal 04 Apr, 2024 Editor assigned by journal 30 Mar, 2024 Submission checks completed at journal 20 Mar, 2024 First submitted to journal 20 Mar, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4139909","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":282058162,"identity":"dd24baab-d971-404d-a2f5-6642c6c3bd35","order_by":0,"name":"Yunpeng Jia","email":"","orcid":"","institution":"Harbin Engineering University","correspondingAuthor":false,"prefix":"","firstName":"Yunpeng","middleName":"","lastName":"Jia","suffix":""},{"id":282058163,"identity":"c5d81a53-58db-4054-a972-1430bd858b2a","order_by":1,"name":"Xiufen Ye","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA7UlEQVRIiWNgGAWjYJCCAww2DAz8EgxsMC4xWtIYGCRnkKKFAaTF4AaxWgyO9xge5kmwyTO+3fvswcc2Bjm+GwmMnwvwaTlzLAGoJa3Y7M5xc8OZbQzGkjcSmKVn4NNyI/nAYd4fhxO33Uhjk+ZtY0jccCOBjZkHn5b7DxuAtvxP3DwDqOVvG0M9YS03mA8AtRxI3CAB1MLYxpBgQEiL5Jm0hINzEpITZ9xIYzfsOSdhOPPMw2ZpfFr4jp8x/vAmwS6xH+iwBz/KbOT5jicf/IxPi8IBVL4EEDM24NHAwCCPX3oUjIJRMApGARAAAAoiUS+QIxskAAAAAElFTkSuQmCC","orcid":"","institution":"Harbin Engineering University","correspondingAuthor":true,"prefix":"","firstName":"Xiufen","middleName":"","lastName":"Ye","suffix":""},{"id":282058164,"identity":"acaf3c8a-17d3-4b74-a5d3-fbdfb40a288b","order_by":2,"name":"Xinkui Mei","email":"","orcid":"","institution":"Harbin Engineering University","correspondingAuthor":false,"prefix":"","firstName":"Xinkui","middleName":"","lastName":"Mei","suffix":""},{"id":282058165,"identity":"d02d4ba1-19f2-4c0b-b7f6-5892ed180efc","order_by":3,"name":"Yusong Liu","email":"","orcid":"","institution":"Harbin Medical University","correspondingAuthor":false,"prefix":"","firstName":"Yusong","middleName":"","lastName":"Liu","suffix":""},{"id":282058166,"identity":"ebf31fe6-53f1-4103-b85c-906b38d04a56","order_by":4,"name":"Shuxiang Guo","email":"","orcid":"","institution":"Southern University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Shuxiang","middleName":"","lastName":"Guo","suffix":""}],"badges":[],"createdAt":"2024-03-21 00:59:21","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4139909/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4139909/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s00530-024-01373-1","type":"published","date":"2024-06-01T17:25:52+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":59746301,"identity":"850f1016-29e5-4991-b6fa-f3c86cd8dd23","added_by":"auto","created_at":"2024-07-05 17:25:59","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":820034,"visible":true,"origin":"","legend":"","description":"","filename":"MLTUmanuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4139909/v1_covered_a0872c62-0323-4ed0-94b6-0a0b7499aaaa.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"MLTU: Mixup Long-Tail Unsupervised Zero-Shot Image Classification on Vision-Language Models","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"multimedia-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"mmsj","sideBox":"Learn more about [Multimedia Systems](http://link.springer.com/journal/530)","snPcode":"530","submissionUrl":"https://submission.nature.com/new-submission/530/3","title":"Multimedia Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Zero-Shot Learning, Unsupervised Learning, Long-Tail Learning, Mixup Augmentation","lastPublishedDoi":"10.21203/rs.3.rs-4139909/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4139909/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eVision-language models, such as Contrastive Language-Image Pretraining (CLIP), have demonstrated powerful capabilities in image classification under zero-shot settings. However, current Zero-Shot Learning (ZSL) relies on manually tagged samples of known classes through supervised learning, resulting in a waste of labor costs and limitations on foreseeable classes in real-world applications. To address these challenges, we propose the Mixup Long-Tail Unsupervised (MLTU) approach for open-world ZSL problems. The proposed approach employed a novel long-tail mixup loss that integrated class-based re-weighting assignments with a given mixup factor for each mixed visual embedding. To mitigate the adverse impact over time, we adopted a noisy learning strategy to filter out samples that generated incorrect labels. We reproduced the unsupervised results of existing state-of-the-art long-tail and noisy learning approaches. Experimental results demonstrate that MLTU achieves significant improvements in classification compared to these proven existing approaches on public datasets. Moreover, it serves as a plug-and-play solution for amending previous assignments and enhancing unsupervised performance. MLTU enables the automatic classification and correction of incorrect predictions caused by the projection bias of CLIP.\u003c/p\u003e","manuscriptTitle":"MLTU: Mixup Long-Tail Unsupervised Zero-Shot Image Classification on Vision-Language Models","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-03-24 19:49:11","doi":"10.21203/rs.3.rs-4139909/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-04-16T08:21:30+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-04-15T15:23:50+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-04-12T08:32:17+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"5eacb26b-40b6-49dc-9414-5b21e1b406b0","date":"2024-04-12T02:09:10+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"2c4639e7-7097-4dd1-9f81-36e8ace520c4","date":"2024-04-04T23:57:52+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-04-04T23:55:21+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-03-30T13:55:15+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-03-21T03:10:57+00:00","index":"","fulltext":""},{"type":"submitted","content":"Multimedia Systems","date":"2024-03-21T00:57:00+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"multimedia-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"mmsj","sideBox":"Learn more about [Multimedia Systems](http://link.springer.com/journal/530)","snPcode":"530","submissionUrl":"https://submission.nature.com/new-submission/530/3","title":"Multimedia Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"676e1b5c-6cde-4d88-874b-6305b3029081","owner":[],"postedDate":"March 24th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2024-07-05T17:25:52+00:00","versionOfRecord":{"articleIdentity":"rs-4139909","link":"https://doi.org/10.1007/s00530-024-01373-1","journal":{"identity":"multimedia-systems","isVorOnly":false,"title":"Multimedia Systems"},"publishedOn":"2024-06-01 17:25:52","publishedOnDateReadable":"June 1st, 2024"},"versionCreatedAt":"2024-03-24 19:49:11","video":"","vorDoi":"10.1007/s00530-024-01373-1","vorDoiUrl":"https://doi.org/10.1007/s00530-024-01373-1","workflowStages":[]},"version":"v1","identity":"rs-4139909","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4139909","identity":"rs-4139909","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}