Case-Control Matching Erodes Feature Discriminability for Machine Learning-Based Sepsis Prediction in ICUs: A Retrospective Cohort Study

Case-Control Matching Erodes Feature Discriminability for Machine Learning-Based Sepsis Prediction in ICUs: A Retrospective Cohort Study | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Case-Control Matching Erodes Feature Discriminability for Machine Learning-Based Sepsis Prediction in ICUs: A Retrospective Cohort Study Sophia Ehlers, Youssef Farag, Fanny Tranchellini, Tim Hahn, Catherine Jutzeler, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9066923/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 8 You are reading this latest preprint version Abstract Background: Sepsis is a leading cause of mortality in the intensive care unit (ICU), and early detection using machine learning (ML) models is critical for timely intervention. To address methodological challenges such as class imbalance and differences in patient trajectories, researchers increasingly adapt case-control matching from epidemiology. However, its impact on predictive modeling performance in ICU sepsis prediction remains insufficiently understood. Methods: We conducted a retrospective multi-cohort analysis using three large harmonized ICU datasets: HiRID, MIMIC-IV, and eICU. We evaluated the effects of case-control matching on both feature discriminability and predictive performance. Matching strategies incorporated temporal alignment and demographic criteria, and were compared against original imbalanced cohorts and undersampled cohorts at equivalent case-to-control ratios. To quantify changes in feature significance, we applied linear mixed-effects models across clinical variables. We then trained multiple ML models, including random forests, balanced random forests, LightGBM, XGBoost, logistic regression, and convolutional neural networks, and evaluated performance on the original test sets using AUROC and normalized AUPRC. Results: Case-control matching consistently reduced the number of significant predictive features across all three cohorts. In the original datasets, 35 to 43 features showed significant differences between septic and non-septic patients, whereas this number declined to 24 to 29 in the most strongly matched settings. In contrast, undersampling largely preserved feature discriminability. Models trained on the original imbalanced cohorts showed robust performance, while models trained on undersampled cohorts often achieved very strong discrimination. However, models trained on matched cohorts exhibited high degradation, with AUROC values frequently to around 0.50 and normalized AUPRC dropping to baseline or below. These patterns were consistent across datasets, matching ratios, and model classes. Conclusion: Case-control matching creates a critical trade-off in ML-based sepsis prediction: although it satisfies the epidemiological objective of balancing cohorts, it removes clinically informative differences that are essential for prediction. Our findings caution against the uncritical transfer of methods designed for causal inference into predictive modeling tasks in the ICU and highlight the need for strategies that preserve predictive signal while addressing dataset imbalance. Full Text Additional Declarations No competing interests reported. Supplementary Files Supplementv1.pdf Cite Share Download PDF Status: Under Review Version 1 posted Reviews received at journal 23 Apr, 2026 Reviewers agreed at journal 13 Apr, 2026 Reviewers agreed at journal 12 Apr, 2026 Reviewers invited by journal 03 Apr, 2026 Editor invited by journal 11 Mar, 2026 Editor assigned by journal 10 Mar, 2026 Submission checks completed at journal 10 Mar, 2026 First submitted to journal 08 Mar, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9066923","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":619050951,"identity":"d594e62e-441a-4241-ac78-1ad8352dadc5","order_by":0,"name":"Sophia Ehlers","email":"","orcid":"","institution":"University of Münster","correspondingAuthor":false,"prefix":"","firstName":"Sophia","middleName":"","lastName":"Ehlers","suffix":""},{"id":619050956,"identity":"ea379aa8-c446-4b87-bd40-3e5d31310da8","order_by":1,"name":"Youssef Farag","email":"","orcid":"","institution":"ETH Zurich","correspondingAuthor":false,"prefix":"","firstName":"Youssef","middleName":"","lastName":"Farag","suffix":""},{"id":619050958,"identity":"a29a3050-62c0-40b4-b7eb-c1901e6c9a16","order_by":2,"name":"Fanny Tranchellini","email":"","orcid":"","institution":"ETH Zurich","correspondingAuthor":false,"prefix":"","firstName":"Fanny","middleName":"","lastName":"Tranchellini","suffix":""},{"id":619050960,"identity":"82c93f8a-7e60-4ce9-8ad7-9d0363082710","order_by":3,"name":"Tim Hahn","email":"","orcid":"","institution":"University of Münster","correspondingAuthor":false,"prefix":"","firstName":"Tim","middleName":"","lastName":"Hahn","suffix":""},{"id":619050962,"identity":"0951eeb0-ed3d-4298-85ea-f859313d2b74","order_by":4,"name":"Catherine Jutzeler","email":"","orcid":"","institution":"ETH Zurich","correspondingAuthor":false,"prefix":"","firstName":"Catherine","middleName":"","lastName":"Jutzeler","suffix":""},{"id":619050963,"identity":"d482ff01-9225-4517-bb8f-1fbdf215f3e6","order_by":5,"name":"Lakmal Meegahapola","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABH0lEQVRIie3QsWqDQBjA8QPhXI7a8UJt8gpfEKSQl/GQnssJHTOULoKTODf0JZw6W6S69AGUZCmChZKhECgZQuiZJpsGx0LvPxx3nD/uToRUqj/YBcJyvEsRSlkgJwhRpMkVomYfwQcCJwItwU5LyADiaEdCoN3pJ3p+m2xhxeLCDWAL2WT0KDaf5f0NQXr2mnQRwvMqgoYt3t4DFkE2faL+80zk8mKE87LzYl5YEshYUrIglRNnLIklsCSU2J3E+Air3ZG87A5ENJbYnyGU58vTKW57yhUVWu2H50jDlyY0VvsWywRvuogaW/NjSnDPWwyD29V6vrqOC68ereezCS3ceiO+H8aGnuVdpKNLB9PfPzM4I9W+hn+tUqlU/6AfihxsQASG7yQAAAAASUVORK5CYII=","orcid":"","institution":"ETH Zurich","correspondingAuthor":true,"prefix":"","firstName":"Lakmal","middleName":"","lastName":"Meegahapola","suffix":""}],"badges":[],"createdAt":"2026-03-08 23:38:33","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9066923/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9066923/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":106725035,"identity":"aa2184b9-ae9d-4856-af56-ef69468896de","added_by":"auto","created_at":"2026-04-12 18:31:07","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":13114969,"visible":true,"origin":"","legend":"","description":"","filename":"Paperv4.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9066923/v1_covered_d8b36e88-b4b9-4e9a-b17e-62896fa6b161.pdf"},{"id":106503670,"identity":"5c27f440-a36c-44df-9d41-7a7f20e7c7a1","added_by":"auto","created_at":"2026-04-09 09:36:07","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":4819377,"visible":true,"origin":"","legend":"","description":"","filename":"Supplementv1.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9066923/v1/abc45eae8ee01d754d24faf5.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Case-Control Matching Erodes Feature Discriminability for Machine Learning-Based Sepsis Prediction in ICUs: A Retrospective Cohort Study","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-informatics-and-decision-making","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"midm","sideBox":"Learn more about [BMC Medical Informatics and Decision Making](http://bmcmedinformdecismak.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/midm/default.aspx","title":"BMC Medical Informatics and Decision Making","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-9066923/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9066923/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eBackground: Sepsis is a leading cause of mortality in the intensive care unit (ICU), and early detection using machine learning (ML) models is critical for timely intervention. To address methodological challenges such as class imbalance and differences in patient trajectories, researchers increasingly adapt case-control matching from epidemiology. However, its impact on predictive modeling performance in ICU sepsis prediction remains insufficiently understood.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eMethods: We conducted a retrospective multi-cohort analysis using three large harmonized ICU datasets: HiRID, MIMIC-IV, and eICU. We evaluated the effects of case-control matching on both feature discriminability and predictive performance. Matching strategies incorporated temporal alignment and demographic criteria, and were compared against original imbalanced cohorts and undersampled cohorts at equivalent case-to-control ratios. To quantify changes in feature significance, we applied linear mixed-effects models across clinical variables. We then trained multiple ML models, including random forests, balanced random forests, LightGBM, XGBoost, logistic regression, and convolutional neural networks, and evaluated performance on the original test sets using AUROC and normalized AUPRC.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eResults: Case-control matching consistently reduced the number of significant predictive features across all three cohorts. In the original datasets, 35 to 43 features showed significant differences between septic and non-septic patients, whereas this number declined to 24 to 29 in the most strongly matched settings. In contrast, undersampling largely preserved feature discriminability. Models trained on the original imbalanced cohorts showed robust performance, while models trained on undersampled cohorts often achieved very strong discrimination. However, models trained on matched cohorts exhibited high degradation, with AUROC values frequently to around 0.50 and normalized AUPRC dropping to baseline or below. These patterns were consistent across datasets, matching ratios, and model classes.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eConclusion: Case-control matching creates a critical trade-off in ML-based sepsis prediction: although it satisfies the epidemiological objective of balancing cohorts, it removes clinically informative differences that are essential for prediction. Our findings caution against the uncritical transfer of methods designed for causal inference into predictive modeling tasks in the ICU and highlight the need for strategies that preserve predictive signal while addressing dataset imbalance.\u003c/p\u003e","manuscriptTitle":"Case-Control Matching Erodes Feature Discriminability for Machine Learning-Based Sepsis Prediction in ICUs: A Retrospective Cohort Study","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-09 09:35:57","doi":"10.21203/rs.3.rs-9066923/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"editorInvitedReview","content":"","date":"2026-04-23T23:26:37+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"47812611230479473523367841038254118107","date":"2026-04-13T17:13:02+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"146621233470245107501124478188474823217","date":"2026-04-12T11:25:03+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-04-03T15:03:14+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2026-03-11T12:04:32+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-03-10T11:08:13+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-03-10T11:07:49+00:00","index":"","fulltext":""},{"type":"submitted","content":"BMC Medical Informatics and Decision Making","date":"2026-03-08T23:28:56+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-informatics-and-decision-making","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"midm","sideBox":"Learn more about [BMC Medical Informatics and Decision Making](http://bmcmedinformdecismak.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/midm/default.aspx","title":"BMC Medical Informatics and Decision Making","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"be4e4b4d-4b0e-41ae-bb50-4aa0d236aff8","owner":[],"postedDate":"April 9th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-04-09T09:35:57+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-09 09:35:57","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9066923","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9066923","identity":"rs-9066923","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}