3D Magic Mirror: Clothing Reconstruction from a Single Image via a Causal Perspective

3D Magic Mirror: Clothing Reconstruction from a Single Image via a Causal Perspective | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article 3D Magic Mirror: Clothing Reconstruction from a Single Image via a Causal Perspective Zhedong Zheng, Jiayin Zhu, Wei Ji, Yi Yang, Tat-seng Chua This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6439497/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 9 You are reading this latest preprint version Abstract This paper aims to study a self-supervised 3D clothing reconstruction method, which recovers the geometry shape and texture of human clothing from a single image. Compared with existing methods, we observe that three primary challenges remain:(1) 3D ground-truth meshes of clothing are usually inaccessible due to annotation difficulties and time costs; (2) Conventional template-based methods are limited to modeling non-rigid objects, e.g., handbags and dresses, which are common in fashion images;(3) The inherent ambiguity compromises the model training, such as the dilemma between a large shape with a remote camera or a small shape with a close camera. In an attempt to address the above limitations, we propose a causality-aware self-supervised learning method to adaptively reconstruct 3D non-rigid objects from 2D images without 3D annotations. In particular, to solve the inherent ambiguity among four implicit variables, i.e., camera position, shape, texture, and illumination, we introduce an explainable structural causal map (SCM) to build our model. The proposed model structure follows the spirit of the causal map, which explicitly considers the prior template in the camera estimation and shape prediction. When optimization, the causality intervention tool, i.e., two expectation-maximization loops, is deeply embedded in our algorithm to (1) disentangle four encoders and (2) facilitate the prior template. Extensive experiments on two 2D fashion benchmarks (ATR and Market-HQ) show that the proposed method could yield high-fidelity 3D reconstruction. Furthermore, we also verify the scalability of the proposed method on a fine-grained bird dataset, i.e., CUB. Physical sciences/Mathematics and computing Physical sciences/Mathematics and computing/Computer science 3D Mesh Reconstruction Fashion Generation Expectation Maximization Causality Self-supervised Learning Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 28 Oct, 2025 Reviews received at journal 27 Oct, 2025 Reviewers agreed at journal 16 Oct, 2025 Reviews received at journal 06 Oct, 2025 Reviewers agreed at journal 01 Oct, 2025 Reviewers invited by journal 19 Jun, 2025 Editor assigned by journal 30 Apr, 2025 Submission checks completed at journal 15 Apr, 2025 First submitted to journal 13 Apr, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6439497","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":474481074,"identity":"78185d48-e508-44cf-aa06-a014dca6760f","order_by":0,"name":"Zhedong Zheng","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA8UlEQVRIiWNgGAWjYDACCTBiYDAAER/AQgkkaGGcQbIWZh5itMjPbj544+OOWntz9t7Dr23btjHws+cYMN2owK2Fcc6xZMuZZ44zW/acS7PObbvNINnzxoA55wxuLcwSOWbSvG3H2Axu5JgZg7QAGQbMuW24tbBJ5H8DaeExuP/GzNgSqMWekBYeiRw2oJYaCYMbPMaPGUG2SBDQIiGRZmw5s+2AgcGZHDPGnnO3eSTOPCs4jM8v8jOSH9742FZnb3D8jPGHH2W35fjbkzc+zsETYlBwGOIvkEtBrAMENTAw1IEI5g9EqBwFo2AUjIIRCABow0+jfg/PFAAAAABJRU5ErkJggg==","orcid":"","institution":"University of Macau","correspondingAuthor":true,"prefix":"","firstName":"Zhedong","middleName":"","lastName":"Zheng","suffix":""},{"id":474481075,"identity":"e5d528d7-9952-4305-bc6d-909022dd1c80","order_by":1,"name":"Jiayin Zhu","email":"","orcid":"","institution":"National University of Singapore","correspondingAuthor":false,"prefix":"","firstName":"Jiayin","middleName":"","lastName":"Zhu","suffix":""},{"id":474481076,"identity":"75059795-273d-4b54-80e7-e35c470b8fdd","order_by":2,"name":"Wei Ji","email":"","orcid":"","institution":"Nanjing University","correspondingAuthor":false,"prefix":"","firstName":"Wei","middleName":"","lastName":"Ji","suffix":""},{"id":474481077,"identity":"977910b0-bc72-43e2-b5dd-a0658a035497","order_by":3,"name":"Yi Yang","email":"","orcid":"","institution":"Zhejiang University","correspondingAuthor":false,"prefix":"","firstName":"Yi","middleName":"","lastName":"Yang","suffix":""},{"id":474481078,"identity":"3e7f5b88-2479-4959-8d89-51ff45398852","order_by":4,"name":"Tat-seng Chua","email":"","orcid":"","institution":"National University of Singapore","correspondingAuthor":false,"prefix":"","firstName":"Tat-seng","middleName":"","lastName":"Chua","suffix":""}],"badges":[],"createdAt":"2025-04-13 13:38:09","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6439497/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6439497/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":85211460,"identity":"c5aa21d8-7962-4b59-b139-da75be694c41","added_by":"auto","created_at":"2025-06-23 12:32:37","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2564213,"visible":true,"origin":"","legend":"","description":"","filename":"latexsource.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6439497/v1_covered_0a314c90-480c-43cb-919f-3b50a2490d17.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"3D Magic Mirror: Clothing Reconstruction from a Single Image via a Causal Perspective","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"npj-artificial-intelligence","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [npj Artificial Intelligence](https://www.nature.com/npjai)","snPcode":"443878","submissionUrl":"https://submission.springernature.com/new-submission/443878/3","title":"npj Artificial Intelligence","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"3D Mesh Reconstruction, Fashion Generation, Expectation Maximization, Causality, Self-supervised Learning","lastPublishedDoi":"10.21203/rs.3.rs-6439497/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6439497/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"This paper aims to study a self-supervised 3D clothing reconstruction method, which recovers the geometry shape and texture of human clothing from a single image. Compared with existing methods, we observe that three primary challenges remain:(1) 3D ground-truth meshes of clothing are usually inaccessible due to annotation difficulties and time costs; (2) Conventional template-based methods are limited to modeling non-rigid objects, e.g., handbags and dresses, which are common in fashion images;(3) The inherent ambiguity compromises the model training, such as the dilemma between a large shape with a remote camera or a small shape with a close camera. In an attempt to address the above limitations, we propose a causality-aware self-supervised learning method to adaptively reconstruct 3D non-rigid objects from 2D images without 3D annotations. In particular, to solve the inherent ambiguity among four implicit variables, i.e., camera position, shape, texture, and illumination, we introduce an explainable structural causal map (SCM) to build our model. The proposed model structure follows the spirit of the causal map, which explicitly considers the prior template in the camera estimation and shape prediction. When optimization, the causality intervention tool, i.e., two expectation-maximization loops, is deeply embedded in our algorithm to (1) disentangle four encoders and (2) facilitate the prior template. Extensive experiments on two 2D fashion benchmarks (ATR and Market-HQ) show that the proposed method could yield high-fidelity 3D reconstruction. Furthermore, we also verify the scalability of the proposed method on a fine-grained bird dataset, i.e., CUB. ","manuscriptTitle":"3D Magic Mirror: Clothing Reconstruction from a Single Image via a Causal Perspective","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-06-23 12:24:29","doi":"10.21203/rs.3.rs-6439497/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-10-28T05:59:07+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-10-27T19:47:00+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"177661992495955132158151974535697915647","date":"2025-10-16T16:59:49+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-10-07T02:20:01+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"62034933219226182223458508528795341863","date":"2025-10-01T14:30:34+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-06-19T13:56:58+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-05-01T02:55:00+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-04-15T04:24:23+00:00","index":"","fulltext":""},{"type":"submitted","content":"npj Artificial Intelligence","date":"2025-04-13T13:25:34+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"npj-artificial-intelligence","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [npj Artificial Intelligence](https://www.nature.com/npjai)","snPcode":"443878","submissionUrl":"https://submission.springernature.com/new-submission/443878/3","title":"npj Artificial Intelligence","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"d97b7522-c6fc-4bef-9767-653995d01c75","owner":[],"postedDate":"June 23rd, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":50389045,"name":"Physical sciences/Mathematics and computing"},{"id":50389046,"name":"Physical sciences/Mathematics and computing/Computer science"}],"tags":[],"updatedAt":"2026-02-12T15:58:35+00:00","versionOfRecord":[],"versionCreatedAt":"2025-06-23 12:24:29","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6439497","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6439497","identity":"rs-6439497","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}