Sparse Foundation Models for Continous-Time EHRs

Sparse Foundation Models for Continous-Time EHRs | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Method Article Sparse Foundation Models for Continous-Time EHRs William Ferrel, Timothy Chang, Samuel Lawrence This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8819405/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Electronic health records (EHRs) are characterized by extreme sparsity, irregular sampling, and event-driven observation processes, yet most existing foundation models treat them as dense sequences through discretization and tokenization. This mismatch forces models to infer temporal dynamics implicitly through depth and attention, conflating missingness with semantic content and limiting robustness under real-world sparsity. We propose a Sparse Foundation Model (SFM) that represents patient trajectories as continuous-time latent states governed by neural ordinary differential equations, with event-conditioned residual updates applied only at observed times. This formulation unifies continuous-time dynamics with ResNet-style depth, yielding a sparsity-adaptive architecture whose effective depth is determined by the data rather than a fixed temporal grid. We pretrain SFM using self-supervised objectives defined over irregular event streams, enabling the model to reason explicitly about missing observations and variable temporal gaps. Across MEDS-DEV and FoMoH benchmarks, SFM consistently outperforms state-of-the-art EHR foundation models, including CEHR-BERT, CEHR-GPT, CLMBR, and MOTOR, with particularly large gains under extreme sparsity, limited labeled supervision, and unseen prediction horizons. Beyond accuracy, SFM exhibits improved calibration, temporal stability, and robustness to observation dropout. These results suggest that aligning foundation model architecture with the sparse, continuous-time structure of clinical data is critical for scalable and reliable healthcare representation learning. Sparsity EHR Foudation Models Full Text Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8819405","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Method Article","associatedPublications":[],"authors":[{"id":587622616,"identity":"495a54e6-0b32-440b-82b7-c0f6b303968a","order_by":0,"name":"William Ferrel","email":"","orcid":"","institution":"Carnegie Mellon University","correspondingAuthor":false,"prefix":"","firstName":"William","middleName":"","lastName":"Ferrel","suffix":""},{"id":587622681,"identity":"8137281c-5164-4c3c-8aef-867dc57b3e65","order_by":1,"name":"Timothy Chang","email":"","orcid":"","institution":"Carnegie Mellon University","correspondingAuthor":false,"prefix":"","firstName":"Timothy","middleName":"","lastName":"Chang","suffix":""},{"id":587622682,"identity":"5fcb60a7-5fc2-4ecf-b207-e5909b9f8712","order_by":2,"name":"Samuel Lawrence","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA2klEQVRIiWNgGAWjYDACdv7PDyT/2AAZDAzMDAYgoQQCWpgZzAwsG9JADOK1GEhUNhyGamEgQot8M0OCwc0d5+X5mXlMNxcU2DHws+cY4NVicJjhwMOZZ24bzmzmMbs9wyCZQbLnDQEtzIwNxhJstxMMDgO18BgA3XmDgC3yzcwM0n/YzsG01DPYE9LCcJiNQUKy7QBMy2FgaBD0Cw+bgcSZZKBf2MqAfjnOI3HmWQF+h7X3MD+QqLCT52dv3na74E+1HH978gb8DkMHPKQpHwWjYBSMglGAFQAAf+o/IGEkLv0AAAAASUVORK5CYII=","orcid":"","institution":"Columbia University","correspondingAuthor":true,"prefix":"","firstName":"Samuel","middleName":"","lastName":"Lawrence","suffix":""}],"badges":[],"createdAt":"2026-02-08 05:57:04","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-8819405/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8819405/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":102297987,"identity":"a68db5da-9e3e-46a6-b5d9-e089a803e3df","added_by":"auto","created_at":"2026-02-10 10:30:00","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":243371,"visible":true,"origin":"","legend":"","description":"","filename":"HealthFM6.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8819405/v1_covered_e7ddeb7e-1339-4073-81c5-d77f596043b3.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003eSparse Foundation Models for Continous-Time EHRs\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"Columbia University","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Sparsity, EHR, Foudation Models","lastPublishedDoi":"10.21203/rs.3.rs-8819405/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8819405/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eElectronic health records (EHRs) are characterized by extreme sparsity, irregular sampling, and event-driven observation processes, yet most existing foundation models treat them as dense sequences through discretization and tokenization. This mismatch forces models to infer temporal dynamics implicitly through depth and attention, conflating missingness with semantic content and limiting robustness under real-world sparsity. We propose a Sparse Foundation Model (SFM) that represents patient trajectories as continuous-time latent states governed by neural ordinary differential equations, with event-conditioned residual updates applied only at observed times. This formulation unifies continuous-time dynamics with ResNet-style depth, yielding a sparsity-adaptive architecture whose effective depth is determined by the data rather than a fixed temporal grid. We pretrain SFM using self-supervised objectives defined over irregular event streams, enabling the model to reason explicitly about missing observations and variable temporal gaps. Across MEDS-DEV and FoMoH benchmarks, SFM consistently outperforms state-of-the-art EHR foundation models, including CEHR-BERT, CEHR-GPT, CLMBR, and MOTOR, with particularly large gains under extreme sparsity, limited labeled supervision, and unseen prediction horizons. Beyond accuracy, SFM exhibits improved calibration, temporal stability, and robustness to observation dropout. These results suggest that aligning foundation model architecture with the sparse, continuous-time structure of clinical data is critical for scalable and reliable healthcare representation learning.\u003c/p\u003e","manuscriptTitle":"Sparse Foundation Models for Continous-Time EHRs","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-02-10 09:28:13","doi":"10.21203/rs.3.rs-8819405/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"6deba287-e087-4518-8bdc-a6a22962ec24","owner":[],"postedDate":"February 10th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-02-10T09:28:13+00:00","versionOfRecord":[],"versionCreatedAt":"2026-02-10 09:28:13","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8819405","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8819405","identity":"rs-8819405","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}