Patch-CLIP - Contrastive Health Record-Image Joint Training with Patch Embedding Loss

Patch-CLIP - Contrastive Health Record-Image Joint Training with Patch Embedding Loss | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Patch-CLIP - Contrastive Health Record-Image Joint Training with Patch Embedding Loss Sheethal Bhat, Awais Mansoor, Bogdan Georgescu, Mathias Zinnen, and 5 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5959624/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Vision-Language (VL) models such as Contrastive Language-Image pretraining (CLIP) use multimodal self-supervised learning (SSL) methods to extract maximal information from large-scale datasets. This enables the trained model to learn key image encodings while correlating them to corresponding textual information through a contrastive loss function that maximizes the similarity of VL pairs. Due to weak supervision provided by the text information, these VL models demonstrate strong zero-shot classification performance; however, their performance on downstream tasks such as object detection and localization remains suboptimal [1]. In this work, we introduce a novel contrastive loss function that aligns image patch embeddings with text embeddings. These patch embeddings that are usually discarded output from the image encoder can naturally incorporate location information during unsupervised training. The proposed approach improves both localization and classification performance, thus allowing key findings to be localized without the need for a complex downstream object detection framework. We evaluated the performance of our proposed method on two chest X-ray (CXR) datasets for abnormality detection and localization tasks. The experiments achieved state-of-the-art (SOTA) results for 8 abnormality detection tasks. Moreover, the patch prediction maps introduced in work considerably reduce False Positive (FP) rates at a given sensitivity, compared to saliency maps. Biological sciences/Computational biology and bioinformatics/Image processing Biological sciences/Computational biology and bioinformatics/Machine learning Computer-Assisted Diagnosis CLIP FROC curves Vision-Langauge models Full Text Additional Declarations No competing interests reported. Supplementary Files additionalresultsvideo.zip Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5959624","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":411806288,"identity":"7469401b-2681-4195-b4c1-e3b142487870","order_by":0,"name":"Sheethal Bhat","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA6klEQVRIiWNgGAWjYDACCSBmbGCQYZDgAbIq5EBiBoS1HGxg4AFrOXDGmFQtB9uI0CI/u/ng44877HgYpHsPfv44z0De4ADzxgf4tBjcOZZscPBMMg+DzLlkiYPbDAw3HGArxmuNgUSOmcTBNmagw3IMgFr+MG44wGMmgddhM/K//zjYVg/SYvzj4BwDe6AW8x94PXMjhw3o68MgLUDrGgwSQbbg1WFwI81Y4mzbcR42mTNmFmeOGSTPPMxWTMBhyQ8/VLZVy/FL9xjfqKgxsO073rzxA15rYIANzmImSv0oGAWjYBSMAnwAAPo8TC1z90uCAAAAAElFTkSuQmCC","orcid":"","institution":"Siemens Healthineers","correspondingAuthor":true,"prefix":"","firstName":"Sheethal","middleName":"","lastName":"Bhat","suffix":""},{"id":411806289,"identity":"a5699b87-676b-4143-bbac-ad7f5cbd7cb1","order_by":1,"name":"Awais Mansoor","email":"","orcid":"","institution":"Siemens Medical Solutions","correspondingAuthor":false,"prefix":"","firstName":"Awais","middleName":"","lastName":"Mansoor","suffix":""},{"id":411806290,"identity":"5e78d977-6319-4c78-8d1c-c6a9913db974","order_by":2,"name":"Bogdan Georgescu","email":"","orcid":"","institution":"Siemens Medical Solutions","correspondingAuthor":false,"prefix":"","firstName":"Bogdan","middleName":"","lastName":"Georgescu","suffix":""},{"id":411806292,"identity":"ab1cb8b6-3130-4745-b879-c6b5e4a544b5","order_by":3,"name":"Mathias Zinnen","email":"","orcid":"","institution":"Friedrich-Alexander-Universität","correspondingAuthor":false,"prefix":"","firstName":"Mathias","middleName":"","lastName":"Zinnen","suffix":""},{"id":411806294,"identity":"dc0287b7-cde0-45e7-bd63-b2cea7955d44","order_by":4,"name":"Pranjal Sahu","email":"","orcid":"","institution":"Siemens Medical Solutions","correspondingAuthor":false,"prefix":"","firstName":"Pranjal","middleName":"","lastName":"Sahu","suffix":""},{"id":411806297,"identity":"46ad6149-2e0b-41f0-816d-ccd517f84d6e","order_by":5,"name":"Adarsh B. Panambur","email":"","orcid":"","institution":"Friedrich-Alexander-Universität","correspondingAuthor":false,"prefix":"","firstName":"Adarsh","middleName":"B.","lastName":"Panambur","suffix":""},{"id":411806299,"identity":"533a510e-5590-44b3-837b-8a0e8342d5d1","order_by":6,"name":"Florin C. Ghesu","email":"","orcid":"","institution":"Siemens Healthineers","correspondingAuthor":false,"prefix":"","firstName":"Florin","middleName":"C.","lastName":"Ghesu","suffix":""},{"id":411806304,"identity":"86ecc4e8-cdfd-48d8-b1c5-a2439f551b62","order_by":7,"name":"Sasa Grbic","email":"","orcid":"","institution":"Siemens Medical Solutions","correspondingAuthor":false,"prefix":"","firstName":"Sasa","middleName":"","lastName":"Grbic","suffix":""},{"id":411806307,"identity":"1e5cb127-6bdb-42cb-9a5c-260bc3ec0f0d","order_by":8,"name":"Andreas Maier","email":"","orcid":"","institution":"Friedrich-Alexander-Universität","correspondingAuthor":false,"prefix":"","firstName":"Andreas","middleName":"","lastName":"Maier","suffix":""}],"badges":[],"createdAt":"2025-02-04 16:08:24","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5959624/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5959624/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":75709058,"identity":"ef154648-fd1d-4d25-8ead-9a5a66bc3fd4","added_by":"auto","created_at":"2025-02-07 10:47:19","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":8579073,"visible":true,"origin":"","legend":"","description":"","filename":"NPJPatchCLIPonCXRs.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5959624/v1_covered_81222473-55ae-43b1-8bbe-645e8e146db7.pdf"},{"id":75697686,"identity":"6a2ec89a-d8c2-434d-843f-5cc70006f8cc","added_by":"auto","created_at":"2025-02-07 08:44:56","extension":"zip","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":122952875,"visible":true,"origin":"","legend":"","description":"","filename":"additionalresultsvideo.zip","url":"https://assets-eu.researchsquare.com/files/rs-5959624/v1/63eb7f2c31fbaeeee7ab7c28.zip"}],"financialInterests":"No competing interests reported.","formattedTitle":"Patch-CLIP - Contrastive Health Record-Image Joint Training with Patch Embedding Loss","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Computer-Assisted Diagnosis, CLIP, FROC curves, Vision-Langauge models","lastPublishedDoi":"10.21203/rs.3.rs-5959624/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5959624/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eVision-Language (VL) models such as Contrastive Language-Image pretraining (CLIP) use multimodal self-supervised learning (SSL) methods to extract maximal information from large-scale datasets. This enables the trained model to learn key image encodings while correlating them to corresponding textual information through a contrastive loss function that maximizes the similarity of VL pairs. Due to weak supervision provided by the text information, these VL models demonstrate strong zero-shot classification performance; however, their performance on downstream tasks such as object detection and localization remains suboptimal [1]. In this work, we introduce a novel contrastive loss function that aligns image patch embeddings with text embeddings. These patch embeddings that are usually discarded output from the image encoder can naturally incorporate location information during unsupervised training. The proposed approach improves both localization and classification performance, thus allowing key findings to be localized without the need for a complex downstream object detection framework. We evaluated the performance of our proposed method on two chest X-ray (CXR) datasets for abnormality detection and localization tasks. The experiments achieved state-of-the-art (SOTA) results for 8 abnormality detection tasks. Moreover, the patch prediction maps introduced in work considerably reduce False Positive (FP) rates at a given sensitivity, compared to saliency maps.\u003c/p\u003e","manuscriptTitle":"Patch-CLIP - Contrastive Health Record-Image Joint Training with Patch Embedding Loss","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-02-07 08:44:45","doi":"10.21203/rs.3.rs-5959624/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"95c271f2-41be-4a5d-9904-2a9d135b1a9f","owner":[],"postedDate":"February 7th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":43912091,"name":"Biological sciences/Computational biology and bioinformatics/Image processing"},{"id":43912092,"name":"Biological sciences/Computational biology and bioinformatics/Machine learning"}],"tags":[],"updatedAt":"2025-02-08T05:53:05+00:00","versionOfRecord":[],"versionCreatedAt":"2025-02-07 08:44:45","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5959624","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5959624","identity":"rs-5959624","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}