Exploring the Potential and Limitations of Deep Learning and Explainable AI for Longitudinal Life Course Analysis

Exploring the Potential and Limitations of Deep Learning and Explainable AI for Longitudinal Life Course Analysis | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Exploring the Potential and Limitations of Deep Learning and Explainable AI for Longitudinal Life Course Analysis Helen Coupland, Neil Scheidwasser, Alexandros Katsiferis, Megan Davies, and 5 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5582136/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 24 Apr, 2025 Read the published version in BMC Public Health → Version 1 posted 8 You are reading this latest preprint version Abstract Background: Understanding the complex interplay between life course exposures, such as adverse childhood experiences and environmental factors, and disease risk is essential for developing effective public health interventions. Traditional epidemiological methods, such as regression models and risk scoring, are limited in their ability to capture the non-linear and temporally dynamic nature of these relationships. Deep learning (DL) and explainable artificial intelligence (XAI) are increasingly applied within healthcare settings to identify influential risk factors and enable personalised interventions. However, significant gaps remain in understanding their utility and limitations, especially for sparse longitudinal life course data and how the influential patterns identified using explainability are linked to underlying causal mechanisms. Methods: We conducted a controlled simulation study to assess the performance of various state-of-the-art DL architectures including CNNs and (attention-based) RNNs against XGBoost and logistic regression. Input data was simulated to reflect a generic and generalisable scenario with different rules used to generate multiple realistic outcomes based upon epidemiological concepts. Multiple metrics were used to assess model performance in the presence of class imbalance and SHAP values were calculated. Results: We find that DL methods can accurately detect dynamic relationships that baseline linear models and tree-based methods cannot. However, there is no one model that consistently outperforms the others across all scenarios. We further identify the superior performance of DL models in handling sparse feature availability over time compared to traditional machine learning approaches. Additionally, we examine the interpretability provided by SHAP values, demonstrating that these explanations often misalign with causal relationships, despite excellent predictive and calibrative performance. Conclusions: These insights provide a foundation for future research applying DL and XAI to life course data, highlighting the challenges associated with sparse healthcare data, and the critical need for advancing interpretability frameworks in personalised public health. Deep Learning Life Course Epidemiology Explainable Artificial Intelligence Full Text Additional Declarations No competing interests reported. Supplementary Files LifecoursesimulationpaperepiSUPP10.pdf Cite Share Download PDF Status: Published Journal Publication published 24 Apr, 2025 Read the published version in BMC Public Health → Version 1 posted Editorial decision: Revision requested 28 Mar, 2025 Reviews received at journal 27 Mar, 2025 Reviewers agreed at journal 26 Mar, 2025 Reviews received at journal 26 Mar, 2025 Reviewers agreed at journal 26 Mar, 2025 Reviewers invited by journal 26 Mar, 2025 Submission checks completed at journal 26 Mar, 2025 First submitted to journal 25 Mar, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5582136","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":434506586,"identity":"80b60d22-939c-464e-9fdb-93f62ac43d09","order_by":0,"name":"Helen Coupland","email":"","orcid":"","institution":"University of Copenhagen","correspondingAuthor":false,"prefix":"","firstName":"Helen","middleName":"","lastName":"Coupland","suffix":""},{"id":434506587,"identity":"9de558f2-dce0-4201-b779-48bf2e3092eb","order_by":1,"name":"Neil Scheidwasser","email":"","orcid":"","institution":"University of Copenhagen","correspondingAuthor":false,"prefix":"","firstName":"Neil","middleName":"","lastName":"Scheidwasser","suffix":""},{"id":434506588,"identity":"28ed5761-5e6b-4c23-a8b7-ccc5d67f0f08","order_by":2,"name":"Alexandros Katsiferis","email":"","orcid":"","institution":"University of Copenhagen","correspondingAuthor":false,"prefix":"","firstName":"Alexandros","middleName":"","lastName":"Katsiferis","suffix":""},{"id":434506589,"identity":"f803a0ae-1cc1-4fab-913d-2b83f2995a86","order_by":3,"name":"Megan Davies","email":"","orcid":"","institution":"University of Copenhagen","correspondingAuthor":false,"prefix":"","firstName":"Megan","middleName":"","lastName":"Davies","suffix":""},{"id":434506590,"identity":"f3311312-9ae0-469d-97dd-ab508e1774fc","order_by":4,"name":"Seth Flaxman","email":"","orcid":"","institution":"University of Oxford","correspondingAuthor":false,"prefix":"","firstName":"Seth","middleName":"","lastName":"Flaxman","suffix":""},{"id":434506591,"identity":"65fa8535-b384-49e8-93ed-2e215c14ea26","order_by":5,"name":"Naja Hulvej Rod","email":"","orcid":"","institution":"University of Copenhagen","correspondingAuthor":false,"prefix":"","firstName":"Naja","middleName":"Hulvej","lastName":"Rod","suffix":""},{"id":434506592,"identity":"8059c2cf-3c41-4472-b209-319201985f68","order_by":6,"name":"Swapnil Mishra","email":"","orcid":"","institution":"National University of Singapore","correspondingAuthor":false,"prefix":"","firstName":"Swapnil","middleName":"","lastName":"Mishra","suffix":""},{"id":434506593,"identity":"2a5f25f7-918b-4ea1-bbb3-f5d07bc6d983","order_by":7,"name":"Samir Bhatt","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA5ElEQVRIiWNgGAWjYJCCA0CUAKQZH4CIBh4gyXOAGC1szMwGRGthgGphkyBKi3z76cTDFQx38vjn9x+r/Nl2R7aB54wBw5szuLUYnMndcPAMw7NiiWPMbLd5254ZN/D2GDDOuYFHCwNQSwPD4cQGkBbGNiCDn8eAmecDHof1v4VomQ/UUviTGC0MN6C2bABqYeAFaQE6jJkHn8NugGwxeJa48ViysTTPucPGbTzHCg7OweN9+f7czR8bKu4kzjt88OHHH2WHZft5kjc+eHMMj8OggYAAbAzgyB0Fo2AUjIJRQAkAAHA4XabCaKN4AAAAAElFTkSuQmCC","orcid":"","institution":"University of Copenhagen","correspondingAuthor":true,"prefix":"","firstName":"Samir","middleName":"","lastName":"Bhatt","suffix":""},{"id":434506594,"identity":"cfc06881-f5d9-442a-b68b-d190a5d79b21","order_by":8,"name":"H. Juliette T. Unwin","email":"","orcid":"","institution":"University of Bristol","correspondingAuthor":false,"prefix":"","firstName":"H.","middleName":"Juliette T.","lastName":"Unwin","suffix":""}],"badges":[],"createdAt":"2024-12-04 19:23:17","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5582136/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5582136/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1186/s12889-025-22705-4","type":"published","date":"2025-04-24T15:57:43+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":81569648,"identity":"8f1bb465-0c81-450b-9cf7-f3ee76993bf8","added_by":"auto","created_at":"2025-04-28 16:09:29","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2916417,"visible":true,"origin":"","legend":"","description":"","filename":"ExploringthePotentialandLimitationsofDeepLearningandExplainableAIforLongitudinalLifeCourseAnalysis11.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5582136/v1_covered_c13ddb36-2103-4b14-9237-37c9d231429a.pdf"},{"id":79317934,"identity":"34d3100a-8ddc-481d-a0f3-a9ed8563a819","added_by":"auto","created_at":"2025-03-27 04:01:28","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":2558152,"visible":true,"origin":"","legend":"","description":"","filename":"LifecoursesimulationpaperepiSUPP10.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5582136/v1/a4173937e25f17ca79c05fd5.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Exploring the Potential and Limitations of Deep Learning and Explainable AI for Longitudinal Life Course Analysis","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-public-health","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"pubh","sideBox":"Learn more about [BMC Public Health](http://bmcpublichealth.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/pubh/default.aspx","title":"BMC Public Health","twitterHandle":"@BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Deep Learning, Life Course Epidemiology, Explainable Artificial Intelligence","lastPublishedDoi":"10.21203/rs.3.rs-5582136/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5582136/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Background: Understanding the complex interplay between life course exposures, such as adverse childhood experiences and environmental factors, and disease risk is essential for developing effective public health interventions. Traditional epidemiological methods, such as regression models and risk scoring, are limited in their ability to capture the non-linear and temporally dynamic nature of these relationships. Deep learning (DL) and explainable artificial intelligence (XAI) are increasingly applied within healthcare settings to identify influential risk factors and enable personalised interventions. However, significant gaps remain in understanding their utility and limitations, especially for sparse longitudinal life course data and how the influential patterns identified using explainability are linked to underlying causal mechanisms.\n\nMethods: We conducted a controlled simulation study to assess the performance of various state-of-the-art DL architectures including CNNs and (attention-based) RNNs against XGBoost and logistic regression. Input data was simulated to reflect a generic and generalisable scenario with different rules used to generate multiple realistic outcomes based upon epidemiological concepts. Multiple metrics were used to assess model performance in the presence of class imbalance and SHAP values were calculated. \n\nResults: We find that DL methods can accurately detect dynamic relationships that baseline linear models and tree-based methods cannot. However, there is no one model that consistently outperforms the others across all scenarios. We further identify the superior performance of DL models in handling sparse feature availability over time compared to traditional machine learning approaches. Additionally, we examine the interpretability provided by SHAP values, demonstrating that these explanations often misalign with causal relationships, despite excellent predictive and calibrative performance.\n\nConclusions: These insights provide a foundation for future research applying DL and XAI to life course data, highlighting the challenges associated with sparse healthcare data, and the critical need for advancing interpretability frameworks in personalised public health. ","manuscriptTitle":"Exploring the Potential and Limitations of Deep Learning and Explainable AI for Longitudinal Life Course Analysis","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-03-27 04:01:23","doi":"10.21203/rs.3.rs-5582136/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-03-28T10:40:21+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-03-27T09:15:19+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"150756367104384753308491895037900915952","date":"2025-03-26T21:37:24+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-03-26T20:22:14+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"306163483346314667513521595196497219598","date":"2025-03-26T20:12:58+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-03-26T16:28:46+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-03-26T05:36:11+00:00","index":"","fulltext":""},{"type":"submitted","content":"BMC Public Health","date":"2025-03-25T11:50:17+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-public-health","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"pubh","sideBox":"Learn more about [BMC Public Health](http://bmcpublichealth.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/pubh/default.aspx","title":"BMC Public Health","twitterHandle":"@BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"4dc02f95-f7da-4651-bfa8-3034d1d193cb","owner":[],"postedDate":"March 27th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2025-04-28T16:02:15+00:00","versionOfRecord":{"articleIdentity":"rs-5582136","link":"https://doi.org/10.1186/s12889-025-22705-4","journal":{"identity":"bmc-public-health","isVorOnly":false,"title":"BMC Public Health"},"publishedOn":"2025-04-24 15:57:43","publishedOnDateReadable":"April 24th, 2025"},"versionCreatedAt":"2025-03-27 04:01:23","video":"","vorDoi":"10.1186/s12889-025-22705-4","vorDoiUrl":"https://doi.org/10.1186/s12889-025-22705-4","workflowStages":[]},"version":"v1","identity":"rs-5582136","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5582136","identity":"rs-5582136","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}