Evaluating the Quality of Synthetic Data in Health Care

Evaluating the Quality of Synthetic Data in Health Care | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Evaluating the Quality of Synthetic Data in Health Care Ivana Nanevski, Maryam Mohebi, Sebastian Jäger, Karen Otte, Matthias Schulte-Althoff, and 3 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6320382/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Machine Learning (ML) research in healthcare remains challenging as large, privacy preserving open data sets are lacking. Synthetic data could offer a solution – but the value of synthetic data depends on diverse and conflicting criteria such as utility, fidelity, and privacy, which are rarely evaluated comprehensively. To close this gap, we explore the trade-off between these metrics in an empirical evaluation across a broad spectrum of generative models, datasets and metrics. Our results indicate that no single generative model excels across all metrics and datasets. In contrast, we find that for every dataset, different generative methods work best -- highlighting the need for automation for synthetic data methods. We investigate the dependency between privacy and utility metrics and demonstrate that the first two main variance directions of all metrics capture the trade-offs between fidelity, utility, and privacy well enough to support design choices of generative models in healthcare. Health sciences/Health care Health sciences/Medical research Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6320382","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":446330877,"identity":"b4bd8ff4-d822-41aa-8130-dc41ad8b0721","order_by":0,"name":"Ivana Nanevski","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABO0lEQVRIie3RMWvCQBTA8RcCyRLq+oJivsILgYSC9LNYBLuI3UImaxHi4gfQpR/D+cqBWa7OFh2EQqYW0kUcpO01UawmdC40/yG8I/ldDg6grOwPph1Hlj4J9JGWrXSlf/LBKaEfxBCanOVKLSZwJIcJOwdydox9F9iK11vY3Vbuebx+D3tOZfI2e0lgVfcGargGv5E7GLY9ewR0OWYzz56E3MVl94YYxE6NK0OCeTtHDOaiAUTAmFs1BGvAsuPi4we/HqtKiErI8yTamDtJLBZtJOk1rGfhIgN+tyefOaKP5OaSEBNyCFSXFkZKmpgRVkD8ao0cspnwzXHAHVt02yggttO/NOetc2IN9Kn5GtSpvoimmFDPfoieZhjAykJ9GGPiXxXcCWTXgrkTfNcsBlmV/m9vy8rKyv5zXwGnamaPy6CPAAAAAElFTkSuQmCC","orcid":"","institution":"Berliner Hochschule für Technik","correspondingAuthor":true,"prefix":"","firstName":"Ivana","middleName":"","lastName":"Nanevski","suffix":""},{"id":446330878,"identity":"5315c714-b8fc-4a18-8ee4-c46b946682c3","order_by":1,"name":"Maryam Mohebi","email":"","orcid":"","institution":"Berliner Hochschule für Technik","correspondingAuthor":false,"prefix":"","firstName":"Maryam","middleName":"","lastName":"Mohebi","suffix":""},{"id":446330879,"identity":"ffa24544-21e2-4f1d-ae46-290b6c368869","order_by":2,"name":"Sebastian Jäger","email":"","orcid":"","institution":"Berliner Hochschule für Technik","correspondingAuthor":false,"prefix":"","firstName":"Sebastian","middleName":"","lastName":"Jäger","suffix":""},{"id":446330880,"identity":"8ce79492-7529-4ce3-8604-159161a669f3","order_by":3,"name":"Karen Otte","email":"","orcid":"","institution":"Berlin Institute of Health at Charité - Universitätsmedizin Berlin","correspondingAuthor":false,"prefix":"","firstName":"Karen","middleName":"","lastName":"Otte","suffix":""},{"id":446330881,"identity":"561a1fd4-af49-4262-aa8a-9f37514dee5d","order_by":4,"name":"Matthias Schulte-Althoff","email":"","orcid":"","institution":"Freie Universität Berlin","correspondingAuthor":false,"prefix":"","firstName":"Matthias","middleName":"","lastName":"Schulte-Althoff","suffix":""},{"id":446330882,"identity":"b4b1f1da-cc8c-4ef4-b8d0-7e2919e240be","order_by":5,"name":"Fabian Prasser","email":"","orcid":"","institution":"Berlin Institute of Health at Charité - Universitätsmedizin Berlin","correspondingAuthor":false,"prefix":"","firstName":"Fabian","middleName":"","lastName":"Prasser","suffix":""},{"id":446330883,"identity":"3cbf691d-2c6c-4183-9af5-af0f76c61fcd","order_by":6,"name":"Daniel Fürstenau","email":"","orcid":"","institution":"Freie Universität Berlin","correspondingAuthor":false,"prefix":"","firstName":"Daniel","middleName":"","lastName":"Fürstenau","suffix":""},{"id":446330884,"identity":"8448a545-f104-4485-9f74-d08a3cbd2414","order_by":7,"name":"Felix Biessmann","email":"","orcid":"","institution":"Berliner Hochschule für Technik","correspondingAuthor":false,"prefix":"","firstName":"Felix","middleName":"","lastName":"Biessmann","suffix":""}],"badges":[],"createdAt":"2025-03-27 12:08:08","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6320382/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6320382/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":83789956,"identity":"7e6d6dd7-1c64-4651-9675-58994d38b0b9","added_by":"auto","created_at":"2025-06-02 18:46:29","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":675193,"visible":true,"origin":"","legend":"","description":"","filename":"EvaluatingtheQualityofSyntheticDatainHealthCare.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6320382/v1_covered_222f1fae-7f6d-4102-99d1-4f6aa3e29e6c.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Evaluating the Quality of Synthetic Data in Health Care","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-6320382/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6320382/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Machine Learning (ML) research in healthcare remains challenging as large, privacy preserving open data sets are lacking. Synthetic data could offer a solution – but the value of synthetic data depends on diverse and conflicting criteria such as utility, fidelity, and privacy, which are rarely evaluated comprehensively. To close this gap, we explore the trade-off between these metrics in an empirical evaluation across a broad spectrum of generative models, datasets and metrics. Our results indicate that no single generative model excels across all metrics and datasets. In contrast, we find that for every dataset, different generative methods work best -- highlighting the need for automation for synthetic data methods. We investigate the dependency between privacy and utility metrics and demonstrate that the first two main variance directions of all metrics capture the trade-offs between fidelity, utility, and privacy well enough to support design choices of generative models in healthcare.","manuscriptTitle":"Evaluating the Quality of Synthetic Data in Health Care","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-04-25 04:40:43","doi":"10.21203/rs.3.rs-6320382/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"439528a5-9c58-4285-9789-d29131f8e855","owner":[],"postedDate":"April 25th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":47507980,"name":"Health sciences/Health care"},{"id":47507981,"name":"Health sciences/Medical research"}],"tags":[],"updatedAt":"2025-06-02T18:38:21+00:00","versionOfRecord":[],"versionCreatedAt":"2025-04-25 04:40:43","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6320382","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6320382","identity":"rs-6320382","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}