TOO-BERT: A Trajectory Order Objective BERT for self-supervised representation learning of temporal healthcare data

TOO-BERT: A Trajectory Order Objective BERT for self-supervised representation learning of temporal healthcare data | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article TOO-BERT: A Trajectory Order Objective BERT for self-supervised representation learning of temporal healthcare data Ali Amirahmadi, Farzaneh Etminani, Jonas Bjork, Olle Melander, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3959125/v3 This work is licensed under a CC BY 4.0 License Status: Posted Version 3 posted You are reading this latest preprint version Show more versions Abstract Background: The growing availability of Electronic Health Records (EHRs) presents an opportunity to enhance patient care by uncovering hidden health risks and improving informed decisions through advanced deep learning methods. However, modeling EHR sequential data, denoted patient trajectories, is complex due to the evolving relationships between diagnoses and treatments over time, where medical conditions and interventions alter the likelihood of future health outcomes over time. While BERT-inspired models have shown promise in modeling EHR sequences by pretraining on the masked language modeling (MLM) objective, they struggle to fully capture the intricate, temporal dynamics of disease progression and medical interventions. Methods: In this study, we introduce TOO-BERT, a novel adaptation that enhances MLM pretrained transformers by explicitly incorporating temporal information from patient trajectories. TOO-BERT encourages the model to learn complex temporal dependencies between diagnoses and treatments using a new self-supervised learning task, the Temporal Order Objective (TOO). This is achieved through two proposed methods: Conditional Code Swapping (CCS) and Conditional Visit Swapping (CVS). Results: We evaluate TOO-BERT on two datasets, MIMIC-IV hospitalization records and the Malmo Diet cohort—comprising approximately 10 and 8 million medical codes, respectively. TOO-BERT demonstrates superior performance in predicting Heart Failure (HF), Alzheimer’s Disease (AD), and Prolonged Length of Stay (PLS) compared to standard MLM-pretrained transformers, and notably excels in HF prediction even with limited fine-tuning data. Conclusions: Our results underscore the effectiveness of integrating temporal ordering objectives into MLM-pretrained models, enabling deeper insights into the complex relationships in EHR data. Attention analysis further reveals TOO-BERT’s ability to capture and represent sophisticated structural patterns within patient trajectories. Biological sciences/Biological techniques/Bioinformatics Physical sciences/Engineering/Electrical and electronic engineering Physical sciences/Mathematics and computing/Information technology Physical sciences/Mathematics and computing/Computer science representation learning patient trajectories Masked language model electronic health records deep learning disease prediction Full Text Additional Declarations The authors declare no competing interests. Supplementary Files SupplementalMaterial2.pdf Supplemental Material Cite Share Download PDF Status: Posted Version 3 posted You are reading this latest preprint version Show more versions Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3959125","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":370495133,"identity":"9cd3b223-c60e-45d2-af87-c5b0d62c3e22","order_by":0,"name":"Ali Amirahmadi","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA5ElEQVRIiWNgGAWjYDACdsYGICkBRgw8BjYJBgS1MKNqSSNGC4wB1sJwmLAW/mbmBoafORb5/LO7Ez+8KTifZ87A/vABPi0ShxkbGHu3SVjOuHN2s+Qcg9vFlg08xvhtAmph4N0mYcBwI3eDNI/B7cQNB3jYJPDpkAfZ8heoRf5G7ubfPAbngFrYn//Ap8UAqIUZZIvBjdxtQFsOALUwmOF1lyFQy2FZoBZDoBbLOQbJiRsO8xjjdZjc8faHD99uqzOQAzrsxps/dokbgCIf8FoDBAdQucxYFY2CUTAKRsEoIAUAACovS6ZNr1uFAAAAAElFTkSuQmCC","orcid":"","institution":"Halmstad University","correspondingAuthor":true,"prefix":"","firstName":"Ali","middleName":"","lastName":"Amirahmadi","suffix":""},{"id":370495134,"identity":"f2b47cab-95a0-4407-8c93-7199d2d96ba2","order_by":1,"name":"Farzaneh Etminani","email":"","orcid":"","institution":"Halmstad University","correspondingAuthor":false,"prefix":"","firstName":"Farzaneh","middleName":"","lastName":"Etminani","suffix":""},{"id":370495135,"identity":"105a1633-bdff-4ed1-a318-69f9a7d58757","order_by":2,"name":"Jonas Bjork","email":"","orcid":"","institution":"Lund University","correspondingAuthor":false,"prefix":"","firstName":"Jonas","middleName":"","lastName":"Bjork","suffix":""},{"id":370495136,"identity":"94d05aa5-84b7-4b3f-abfb-0ecac13c0a6e","order_by":3,"name":"Olle Melander","email":"","orcid":"","institution":"Lund University","correspondingAuthor":false,"prefix":"","firstName":"Olle","middleName":"","lastName":"Melander","suffix":""},{"id":370495137,"identity":"dbec31e2-fdd6-409c-aa94-b84583055ac6","order_by":4,"name":"Mattias Ohlsson","email":"","orcid":"","institution":"Halmstad University","correspondingAuthor":false,"prefix":"","firstName":"Mattias","middleName":"","lastName":"Ohlsson","suffix":""}],"badges":[],"createdAt":"2024-02-15 16:07:12","currentVersionCode":3,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-3959125/v3","doiUrl":"https://doi.org/10.21203/rs.3.rs-3959125/v3","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":79622582,"identity":"a1f06023-0a1f-471c-bcf8-7f09741c8e02","added_by":"auto","created_at":"2025-03-31 23:37:41","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1594072,"visible":true,"origin":"","legend":"","description":"","filename":"JamiralTOOBert5.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3959125/v3_covered_095cc211-5808-499b-804f-88ced6102530.pdf"},{"id":79622299,"identity":"df0c7c68-e403-4f0c-8d49-1acbf162492d","added_by":"auto","created_at":"2025-03-31 23:21:39","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":7619383,"visible":true,"origin":"","legend":"\u003cp\u003eSupplemental Material\u003c/p\u003e","description":"","filename":"SupplementalMaterial2.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3959125/v3/b23beb15b42ba107923e8980.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"TOO-BERT: A Trajectory Order Objective BERT for self-supervised representation learning of temporal healthcare data","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":true,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"representation learning, patient trajectories, Masked language model, electronic health records, deep learning, disease prediction","lastPublishedDoi":"10.21203/rs.3.rs-3959125/v3","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3959125/v3","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eBackground: The growing availability of Electronic Health Records (EHRs) presents an opportunity to enhance patient care by uncovering hidden health risks and improving informed decisions through advanced deep learning methods. However, modeling EHR sequential data, denoted patient trajectories, is complex due to the evolving relationships between diagnoses and treatments over time, where medical conditions and interventions alter the likelihood of future health outcomes over time. While BERT-inspired models have shown promise in modeling EHR sequences by pretraining on the masked language modeling (MLM) objective, they struggle to fully capture the intricate, temporal dynamics of disease progression and medical interventions.\u003c/p\u003e\n\u003cp\u003eMethods: In this study, we introduce TOO-BERT, a novel adaptation that enhances MLM pretrained transformers by explicitly incorporating temporal information from patient trajectories. TOO-BERT encourages the model to learn complex temporal dependencies between diagnoses and treatments using a new self-supervised learning task, the Temporal Order Objective (TOO). This is achieved through two proposed methods: Conditional Code Swapping (CCS) and Conditional Visit Swapping (CVS).\u003c/p\u003e\n\u003cp\u003eResults: We evaluate TOO-BERT on two datasets, MIMIC-IV hospitalization records and the Malmo Diet cohort—comprising approximately 10 and 8 million medical codes, respectively. TOO-BERT demonstrates superior performance in predicting Heart Failure (HF), Alzheimer’s Disease (AD), and Prolonged Length of Stay (PLS) compared to standard MLM-pretrained transformers, and notably excels in HF prediction even with limited fine-tuning data.\u003c/p\u003e\n\u003cp\u003eConclusions: Our results underscore the effectiveness of integrating temporal ordering objectives into MLM-pretrained models, enabling deeper insights into the complex relationships in EHR data. Attention analysis further reveals TOO-BERT’s ability to capture and represent sophisticated structural patterns within patient trajectories.\u003c/p\u003e","manuscriptTitle":"TOO-BERT: A Trajectory Order Objective BERT for self-supervised representation learning of temporal healthcare data","msid":"","msnumber":"","nonDraftVersions":[{"code":3,"date":"2025-03-31 23:21:35","doi":"10.21203/rs.3.rs-3959125/v3","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}},{"code":2,"date":"2024-10-25 16:40:33","doi":"10.21203/rs.3.rs-3959125/v2","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}},{"code":1,"date":"2024-03-05 02:11:05","doi":"10.21203/rs.3.rs-3959125/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"84a970e6-4380-4ab2-8890-de5c5dbc85a0","owner":[],"postedDate":"March 31st, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":39436713,"name":"Biological sciences/Biological techniques/Bioinformatics"},{"id":39436714,"name":"Physical sciences/Engineering/Electrical and electronic engineering"},{"id":39436715,"name":"Physical sciences/Mathematics and computing/Information technology"},{"id":39436716,"name":"Physical sciences/Mathematics and computing/Computer science"}],"tags":[],"updatedAt":"2024-10-18T05:38:51+00:00","versionOfRecord":[],"versionCreatedAt":"2025-03-31 23:21:35","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v3","identity":"rs-3959125","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3959125","identity":"rs-3959125","version":["v3"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}