Towards Visually Interpreting Variational Autoencoders

Towards Visually Interpreting Variational Autoencoders | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Towards Visually Interpreting Variational Autoencoders Runze Li, Wenqian Liu, Meng Zheng, Max Torop, Milind Rajadhyaksha, and 7 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4331222/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Recent advances in convolutional neural network (CNN) interpretability have led to a wide-variety of gradient-based visual attention techniques for generating visual attention maps. However, most of these methods require a classification-type design architecture, and consequently concentrate on classification/categorization-type tasks. Extending these methods to generate visual attention maps for other kinds of computer vision models, e.g., variational autoencoders (VAE) is not trivial. In this paper, we present a method that helps bridge this crucial gap, proposing to compute VAE attention as a means for interpreting the latent space learned by a VAE. We first present methods to generate visual attention maps from the learned latent space, and then show how they can be used in a variety of applications: localizing anomalies in images, including medical imagery, and improved latent space disentanglement. We conduct extensive experiments on a wide-variety of benchmark datasets to demonstrate the efficacy of the proposed VAE attention. Physical sciences/Mathematics and computing/Computer science Physical sciences/Mathematics and computing/Information technology Attention maps anomaly localization disentanglement learning explainable latent space learned by VAE Full Text Additional Declarations There is NO Competing Interest. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4331222","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":300694299,"identity":"9aa5c05d-79e0-4750-8170-e5a8c2c7f1c0","order_by":0,"name":"Runze Li","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAq0lEQVRIiWNgGAWjYDACdsaGA5L/bCAcHqK0MDM2PrBgSyNJCwOzQQXbYRK0mDczt0nc4Dmf2D/7AOODt21EaJE5zNgmOUPiduKMcwnMhnOJ0SLBzNgmLWFwO3EDDwObNC/RWv4knANpYf9NrJZmA4kDB8C2MBOrpfGBZEOy8YwzjM2Sc84Ro4W9/cEByQY72f4e5oMf3pQRoQUJMDaQpn4UjIJRMApGAW4AADfpL+I+mwVDAAAAAElFTkSuQmCC","orcid":"","institution":"UC Riverside","correspondingAuthor":true,"prefix":"","firstName":"Runze","middleName":"","lastName":"Li","suffix":""},{"id":300694300,"identity":"3413d6ef-a74b-4a6c-9905-7d270e5eda76","order_by":1,"name":"Wenqian Liu","email":"","orcid":"","institution":"Inria CHROMA","correspondingAuthor":false,"prefix":"","firstName":"Wenqian","middleName":"","lastName":"Liu","suffix":""},{"id":300694301,"identity":"a6d00a68-80b1-4bfd-9961-12583aff875b","order_by":2,"name":"Meng Zheng","email":"","orcid":"","institution":"United Imaging Intelligence","correspondingAuthor":false,"prefix":"","firstName":"Meng","middleName":"","lastName":"Zheng","suffix":""},{"id":300694302,"identity":"c6ac838c-14e4-4a6c-af00-390e29e9cc1d","order_by":3,"name":"Max Torop","email":"","orcid":"","institution":"Northeastern University","correspondingAuthor":false,"prefix":"","firstName":"Max","middleName":"","lastName":"Torop","suffix":""},{"id":300694303,"identity":"da886d8a-29b8-49d4-a00e-a8d9f0af23dc","order_by":4,"name":"Milind Rajadhyaksha","email":"","orcid":"https://orcid.org/0000-0002-6323-4547","institution":"Memorial Sloan Kettering Cancer Center","correspondingAuthor":false,"prefix":"","firstName":"Milind","middleName":"","lastName":"Rajadhyaksha","suffix":""},{"id":300694305,"identity":"0581bfec-0581-4941-8fad-96231dec7d0c","order_by":5,"name":"Jennifer Dy","email":"","orcid":"","institution":"Northeastern University","correspondingAuthor":false,"prefix":"","firstName":"Jennifer","middleName":"","lastName":"Dy","suffix":""},{"id":300694307,"identity":"71acbb33-26b7-4c14-ac12-facdff6ab2f8","order_by":6,"name":"Kivanc Kose","email":"","orcid":"https://orcid.org/0000-0003-3185-2639","institution":"Memorial Sloan Kettering Cancer Cente","correspondingAuthor":false,"prefix":"","firstName":"Kivanc","middleName":"","lastName":"Kose","suffix":""},{"id":300694309,"identity":"a0d7b7a2-4d67-4ec7-a906-6cb936a76bca","order_by":7,"name":"Srikrishna Karanam","email":"","orcid":"","institution":"Adobe Research","correspondingAuthor":false,"prefix":"","firstName":"Srikrishna","middleName":"","lastName":"Karanam","suffix":""},{"id":300694310,"identity":"7eda684c-bd8e-4211-a071-f649a145bdc9","order_by":8,"name":"Ziyan Wu","email":"","orcid":"","institution":"United Imaging Intelligence","correspondingAuthor":false,"prefix":"","firstName":"Ziyan","middleName":"","lastName":"Wu","suffix":""},{"id":300694311,"identity":"c6c1ce40-7f48-4fc5-a35f-49ad74080da0","order_by":9,"name":"Bir Bhanu","email":"","orcid":"https://orcid.org/0000-0001-8971-6416","institution":"UC Riverside","correspondingAuthor":false,"prefix":"","firstName":"Bir","middleName":"","lastName":"Bhanu","suffix":""},{"id":300694312,"identity":"00e39ce8-7bd7-43d9-8610-b301795346a4","order_by":10,"name":"Richard Radke","email":"","orcid":"","institution":"Rensselaer Polytechnic Institute","correspondingAuthor":false,"prefix":"","firstName":"Richard","middleName":"","lastName":"Radke","suffix":""},{"id":300694313,"identity":"85b91f85-6af9-40a7-aa91-bf5843dee4ff","order_by":11,"name":"Octavia Camps","email":"","orcid":"","institution":"Northeastern University","correspondingAuthor":false,"prefix":"","firstName":"Octavia","middleName":"","lastName":"Camps","suffix":""}],"badges":[],"createdAt":"2024-04-26 18:30:43","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4331222/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4331222/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":60987065,"identity":"0bd895cd-8932-44a9-8cd9-f0aefd098e3b","added_by":"auto","created_at":"2024-07-24 10:18:44","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":5229981,"visible":true,"origin":"","legend":"","description":"","filename":"EXPVAENMISubmission.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4331222/v1_covered_1cf25f1a-6c2d-4a68-8ddb-73114ca75bee.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"Towards Visually Interpreting Variational Autoencoders","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Attention maps, anomaly localization, disentanglement learning, explainable, latent space learned by VAE","lastPublishedDoi":"10.21203/rs.3.rs-4331222/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4331222/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eRecent advances in convolutional neural network (CNN) interpretability have led to a wide-variety of gradient-based visual attention techniques for generating visual attention maps. However, most of these methods require a classification-type design architecture, and consequently concentrate on classification/categorization-type tasks. Extending these methods to generate visual attention maps for other kinds of computer vision models, e.g., variational autoencoders (VAE) is not trivial. In this paper, we present a method that helps bridge this crucial gap, proposing to compute \u003cem\u003eVAE attention\u003c/em\u003e as a means for interpreting the latent space learned by a VAE. We first present methods to generate visual attention maps from the learned latent space, and then show how they can be used in a variety of applications: localizing anomalies in images, including medical imagery, and improved latent space disentanglement. We conduct extensive experiments on a wide-variety of benchmark datasets to demonstrate the efficacy of the proposed VAE attention.\u003c/p\u003e","manuscriptTitle":"Towards Visually Interpreting Variational Autoencoders","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-05-10 05:01:55","doi":"10.21203/rs.3.rs-4331222/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"a4756b95-24eb-4c13-8cfb-8c2da2078d2c","owner":[],"postedDate":"May 10th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":31728291,"name":"Physical sciences/Mathematics and computing/Computer science"},{"id":31728292,"name":"Physical sciences/Mathematics and computing/Information technology"}],"tags":[],"updatedAt":"2024-07-24T10:10:33+00:00","versionOfRecord":[],"versionCreatedAt":"2024-05-10 05:01:55","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4331222","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4331222","identity":"rs-4331222","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}