Cross-validation under data dependence: a review-derived taxonomy and trustcv, a leakage-aware Python toolkit

Cross-validation under data dependence: a review-derived taxonomy and trustcv, a leakage-aware Python toolkit | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Cross-validation under data dependence: a review-derived taxonomy and trustcv, a leakage-aware Python toolkit Abdolamir Karbalaie, Farhad Abtahi This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9357577/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Inappropriate cross-validation can inflate performance estimates in medical machine learning , yet validation methods for grouped, temporal, and spatial data remain fragmented across literatures and incompletely supported in existing tools. We addressed this gap through a methodological study combining evidence synthesis, controlled synthetic illustration, and open-source implementation. Using purposive bidirectional snowballing from canonical seed papers, we identified and screened 29 cross-validation methods relevant to medical machine learning and organized them into a four-category taxonomy: independent and identically distributed (i.i.d., n = 9), grouped (n = 8), temporal (n = 8), and spatial (n = 4). Inclusion required documented medical or health-related application, or clear transferability to medical data structures. All 29 methods were implemented in trustcv ( https://ki-smile.github.io/trustcv ), a framework-agnostic Python toolkit that provides automated detection of six leakage types and structure-aware validation workflows. Controlled synthetic grouped benchmarks showed that, in the medium-leakage scenario, ignoring patient-level structure inflated area under the receiver operating characteristic curve (AUC) relative to Group K-Fold by 18.2 and 11.7 percentage points at the observation and patient levels, respectively. Implementation reliability was confirmed through automated verification of key correctness properties across 141 unit tests in 11 test modules. Together, the taxonomy and toolkit provide a practical foundation for more reliable , structure-aware model evaluation in medical machine learning. Biological sciences/Computational biology and bioinformatics Health sciences/Health care Physical sciences/Mathematics and computing Health sciences/Medical research Data leakage Grouped data Temporal validation Spatial validation Machine learning Full Text Additional Declarations No competing interests reported. Supplementary Files SuppleP4Sub1.pdf Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9357577","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":620419592,"identity":"e67d5810-c335-4001-8255-27471a987a3a","order_by":0,"name":"Abdolamir Karbalaie","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABFklEQVRIie3Qv0vDQBTA8RcOrsuDWy8Q0n/hQqEiiPev5BA6OShORaEXCnER54j/RF2kY+EgLhVXQxzS/0ARCoJILx38AV7r6HDfKQf58N4dgM/3H+MgoAFAAAoQaIiAQLDQYgtJvxG0hCR/IfBF7FeoNwh2Pb5p0rPbaEdT9Tqc7iM7J+WwOALJXEOeyhORljVGM2rC+/kBckMH1USAunKNejzs85TWyKGjwywnCAb7VWO3FbPfRXdNPtZk/JblI+watjy2REoHES1ReUtoaacYFAZpYBcLJo69kvYu6tISQge72fwOE0N7YSG4KhxT4tq+2POylryT96psehrHD2bxcvG+J5nr+p+Rn0e+7X+fz+fzbWgFRRZXvdmfnlUAAAAASUVORK5CYII=","orcid":"","institution":"Karolinska Institutet","correspondingAuthor":true,"prefix":"","firstName":"Abdolamir","middleName":"","lastName":"Karbalaie","suffix":""},{"id":620419593,"identity":"d3644c67-6bfc-474a-95e0-9e87eac67fbb","order_by":1,"name":"Farhad Abtahi","email":"","orcid":"","institution":"Karolinska Institutet","correspondingAuthor":false,"prefix":"","firstName":"Farhad","middleName":"","lastName":"Abtahi","suffix":""}],"badges":[],"createdAt":"2026-04-08 13:24:45","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9357577/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9357577/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":107051391,"identity":"87f5e7fe-f997-427d-bc7f-a65d702ee763","added_by":"auto","created_at":"2026-04-16 08:27:26","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1347176,"visible":true,"origin":"","legend":"","description":"","filename":"P4Sub2.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9357577/v1_covered_a0c760bc-9e56-47ca-8dc8-e95afdb97205.pdf"},{"id":106939204,"identity":"0bfbe1f6-4503-4a58-862a-4c99784b2a1e","added_by":"auto","created_at":"2026-04-15 04:36:15","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":7640462,"visible":true,"origin":"","legend":"","description":"","filename":"SuppleP4Sub1.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9357577/v1/fd4a4a97dddb6bde99e2e920.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Cross-validation under data dependence: a review-derived taxonomy and trustcv, a leakage-aware Python toolkit","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Data leakage, Grouped data, Temporal validation, Spatial validation, Machine learning","lastPublishedDoi":"10.21203/rs.3.rs-9357577/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9357577/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Inappropriate cross-validation can inflate performance estimates in medical machine learning , yet validation methods for grouped, temporal, and spatial data remain fragmented across literatures and incompletely supported in existing tools. We addressed this gap through a methodological study combining evidence synthesis, controlled synthetic illustration, and open-source implementation. Using purposive bidirectional snowballing from canonical seed papers, we identified and screened 29 cross-validation methods relevant to medical machine learning and organized them into a four-category taxonomy: independent and identically distributed (i.i.d., n = 9), grouped (n = 8), temporal (n = 8), and spatial (n = 4). Inclusion required documented medical or health-related application, or clear transferability to medical data structures. All 29 methods were implemented in trustcv (https://ki-smile.github.io/trustcv), a framework-agnostic Python toolkit that provides automated detection of six leakage types and structure-aware validation workflows. Controlled synthetic grouped benchmarks showed that, in the medium-leakage scenario, ignoring patient-level structure inflated area under the receiver operating characteristic curve (AUC) relative to Group K-Fold by 18.2 and 11.7 percentage points at the observation and patient levels, respectively. Implementation reliability was confirmed through automated verification of key correctness properties across 141 unit tests in 11 test modules. Together, the taxonomy and toolkit provide a practical foundation for more reliable , structure-aware model evaluation in medical machine learning.","manuscriptTitle":"Cross-validation under data dependence: a review-derived taxonomy and trustcv, a leakage-aware Python toolkit","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-15 04:36:11","doi":"10.21203/rs.3.rs-9357577/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"573e7fa0-c380-45ad-85b9-d40fe69640c2","owner":[],"postedDate":"April 15th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":66027947,"name":"Biological sciences/Computational biology and bioinformatics"},{"id":66027948,"name":"Health sciences/Health care"},{"id":66027949,"name":"Physical sciences/Mathematics and computing"},{"id":66027950,"name":"Health sciences/Medical research"}],"tags":[],"updatedAt":"2026-04-16T08:26:26+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-15 04:36:11","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9357577","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9357577","identity":"rs-9357577","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}