StrokeClassifier: Ischemic Stroke Etiology Classification by Ensemble Consensus Modeling Using Electronic Health Records

StrokeClassifier: Ischemic Stroke Etiology Classification by Ensemble Consensus Modeling Using Electronic Health Records | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article StrokeClassifier: Ischemic Stroke Etiology Classification by Ensemble Consensus Modeling Using Electronic Health Records Ho-Joon Lee, Lee H. Schwamm, Lauren Sansing, Hooman Kamel, Adam de Havenon, and 7 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3367169/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 17 May, 2024 Read the published version in npj Digital Medicine → Version 1 posted You are reading this latest preprint version Abstract Determining the etiology of an acute ischemic stroke (AIS) is fundamental to secondary stroke prevention efforts but can be diagnostically challenging. We trained and validated an automated classification machine intelligence tool, StrokeClassifier , using electronic health record (EHR) text data from 2,039 non-cryptogenic AIS patients at 2 academic hospitals to predict the 4-level outcome of stroke etiology determined by agreement of at least 2 board-certified vascular neurologists’ review of the stroke hospitalization EHR. StrokeClassifier is an ensemble consensus meta-model of 9 machine learning classifiers applied to features extracted from discharge summary texts by natural language processing. StrokeClassifier was externally validated in 406 discharge summaries from the MIMIC-III dataset reviewed by a vascular neurologist to ascertain stroke etiology. Compared with stroke etiologies adjudicated by vascular neurologists, nine base classifiers performed well with a mean cross-validated area under the receiver operating curve (AUCROC) of 0.90. Their ensemble meta-model, StrokeClassifier , achieved a mean cross-validated accuracy of 0.74 and weighted F1 of 0.74. In the MIMIC-III cohort, the accuracy and weighted F1 of StrokeClassifier were 0.70, and 0.71, respectively. SHapley Additive exPlanation analysis revealed that the top 5 features contributing to stroke etiology prediction were atrial fibrillation, age, middle cerebral artery occlusion, internal carotid artery occlusion, and frontal stroke location. We then designed a certainty heuristic to deem a StrokeClassifier diagnosis as confidently non-cryptogenic by the degree of consensus among the 9 classifiers, and applied it to 788 cryptogenic patients. This reduced the percentage of the cryptogenic strokes from 25.2–7.2% of all ischemic strokes. StrokeClassifier is a validated artificial intelligence tool that rivals the performance of vascular neurologists in classifying ischemic stroke etiology for individual patients. With further training, StrokeClassifier may have downstream applications including its use as a clinical decision support system. Health sciences/Neurology/Neurological disorders/Stroke Health sciences/Risk factors Full Text Additional Declarations Yes there is potential Competing Interest. H.L., L.H.S., and R.S. are co-inventors of U.S. Provisional Patent Application No. 63/505,006, “Methods of Training an Algorithm To Predict Ischemic Stroke Etiology”. H.M.K. works under contract with the Centers for Medicare & Medicaid Services to support quality measurement programs, was a recipient of a research grant from Johnson & Johnson, through Yale University, to support clinical trial data sharing; was a recipient of a research agreement, through Yale University, from the Shenzhen Center for Health Information for work to advance intelligent disease prevention and health promotion; collaborates with the National Center for Cardiovascular Diseases in Beijing; receives payment from the Arnold & Porter Law Firm for work related to the Sanofi clopidogrel litigation, from the Martin Baughman Law Firm for work related to the Cook Celect IVC filter litigation, and from the Siegfried and Jensen Law Firm for work related to Vioxx litigation; chairs a Cardiac Scientific Advisory Board for UnitedHealth; was a member of the IBM Watson Health Life Sciences Board; is a member of the Advisory Board for Element Science, the Advisory Board for Facebook, and the Physician Advisory Board for Aetna; and is the co-founder of Hugo Health, a personal health information platform, and co-founder of Refactor Health, a healthcare AI-augmented data management company. A.D.H. has received consultant fees from Integra and Novo Nordisk, has equity in TitinKM and Certus, and receives author fees from UpToDate. K.N.S. reports investigator‐initiated clinical research funding to Yale from Hyperfine, Inc, Biogen, and Bard; reports from Sense and Zoll, for data and safety monitoring services; compensation from Cerevasc for consultant services; compensation from Rhaeos for consultant services, compensation from Certus for consultant services; and a patent pending for Stroke wearables licensed to Alva Health. S.K. is on the scientific advisory board of KovaDx and AI Therapeutics. H.K. reports compensation from Novo Nordisk for end point review committee services; compensation from Medtronic for other services; compensation from Janssen Biotech for other services; compensation from Boehringer Ingelheim for end point review committee services; and employment by Weill Cornell Medical College. L.H.S. reports compensation as a scientific consultant regarding trial design and conduct on late window thrombolysis and member of steering committee for Genentech (TIMELESS NCT03785678); user interface design and usability to LifeImage (privately held teleradiology company); member of a Data Safety Monitoring Board (DSMB) for Penumbra (MIND NCT03342664; PI, multicenter trial of stroke prevention in atrial fibrillation for Medtronic (Stroke AF NCT02700945). There were no competing interests reported by the remaining authors. Supplementary Files SupplementaryInformation.pdf Cite Share Download PDF Status: Published Journal Publication published 17 May, 2024 Read the published version in npj Digital Medicine → Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3367169","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":236394651,"identity":"9f235b51-07e5-4ce5-9c99-843171a69e25","order_by":0,"name":"Ho-Joon Lee","email":"","orcid":"https://orcid.org/0000-0003-3616-5387","institution":"Department of Genetics and Yale Center for Genome Analysis, Yale School of Medicine, New Haven, CT","correspondingAuthor":false,"prefix":"","firstName":"Ho-Joon","middleName":"","lastName":"Lee","suffix":""},{"id":236394653,"identity":"e1439e39-6443-484f-a4f1-283eba0eec0a","order_by":1,"name":"Lee H. Schwamm","email":"","orcid":"https://orcid.org/0000-0003-0592-9145","institution":"Department of Neurology and Comprehensive Stroke Center, Massachusetts General Hospital and Harvard Medical School Boston, MA; Department of Neurology, Yale School of Medicine, New Haven, CT","correspondingAuthor":false,"prefix":"","firstName":"Lee","middleName":"H.","lastName":"Schwamm","suffix":""},{"id":236394655,"identity":"710fbe80-46f7-400b-bb06-dab32fe9e787","order_by":2,"name":"Lauren Sansing","email":"","orcid":"","institution":"Department of Neurology, Yale School of Medicine, New Haven, CT","correspondingAuthor":false,"prefix":"","firstName":"Lauren","middleName":"","lastName":"Sansing","suffix":""},{"id":236394657,"identity":"aae821f5-e683-422a-8a0e-ce657ba3f0ed","order_by":3,"name":"Hooman Kamel","email":"","orcid":"","institution":"Department of Neurology, Weill Cornell Medicine, New York City, NY","correspondingAuthor":false,"prefix":"","firstName":"Hooman","middleName":"","lastName":"Kamel","suffix":""},{"id":236394659,"identity":"6e33e47f-1ba3-4fd2-8675-c2d8e656d461","order_by":4,"name":"Adam de Havenon","email":"","orcid":"","institution":"Department of Neurology, Yale School of Medicine, New Haven, CT","correspondingAuthor":false,"prefix":"","firstName":"Adam","middleName":"","lastName":"de Havenon","suffix":""},{"id":236394661,"identity":"d710439b-938d-44c0-b138-2b407c35c277","order_by":5,"name":"Ashby C. Turner","email":"","orcid":"","institution":"Department of Neurology and Comprehensive Stroke Center, Massachusetts General Hospital and Harvard Medical School Boston, MA","correspondingAuthor":false,"prefix":"","firstName":"Ashby","middleName":"C.","lastName":"Turner","suffix":""},{"id":236394663,"identity":"37fd0347-8e08-4134-8b89-1be6927ec0f9","order_by":6,"name":"Kevin N. Sheth","email":"","orcid":"https://orcid.org/0000-0003-2003-5473","institution":"Department of Neurology, Yale School of Medicine, New Haven, CT","correspondingAuthor":false,"prefix":"","firstName":"Kevin","middleName":"N.","lastName":"Sheth","suffix":""},{"id":236394665,"identity":"fbe974c4-a5d8-46ca-b3ad-e2f09a67b18e","order_by":7,"name":"Smita Krishnaswamy","email":"","orcid":"https://orcid.org/0000-0001-5823-1985","institution":"Departments of Genetics and Computer Science, Yale School of Medicine, New Haven, CT","correspondingAuthor":false,"prefix":"","firstName":"Smita","middleName":"","lastName":"Krishnaswamy","suffix":""},{"id":236394666,"identity":"c0d0e38d-a89f-4516-a41c-5f7a36bb319d","order_by":8,"name":"Cynthia Brandt","email":"","orcid":"","institution":"Department of Biomedical Informatics and Data Science, Yale School of Medicine, New Haven, CT","correspondingAuthor":false,"prefix":"","firstName":"Cynthia","middleName":"","lastName":"Brandt","suffix":""},{"id":236394667,"identity":"2ae123ed-0f48-41a1-b44a-dea79be6700f","order_by":9,"name":"Hongyu Zhao","email":"","orcid":"https://orcid.org/0000-0003-1195-9607","institution":"Departments of Biostatistics, Yale School of Public Health, New Haven, CT","correspondingAuthor":false,"prefix":"","firstName":"Hongyu","middleName":"","lastName":"Zhao","suffix":""},{"id":236394668,"identity":"171a0169-a03b-45d1-aec3-d8ab77531078","order_by":10,"name":"Harlan Krumholz","email":"","orcid":"https://orcid.org/0000-0003-2046-127X","institution":"Department of Internal Medicine, Yale School of Medicine, New Haven, CT","correspondingAuthor":false,"prefix":"","firstName":"Harlan","middleName":"","lastName":"Krumholz","suffix":""},{"id":236394650,"identity":"13ed56d4-62ce-45d5-8a8a-07c359662680","order_by":11,"name":"Richa Sharma","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA70lEQVRIiWNgGAWjYDACZgY2EJUA5nxgYJAB0RJEa2GcwcDAQ1gLA5IWZh5itJi3Mz978IOhLo9/9unEz7Ztdjz8DMwHb/Pg0SJzmM3csIfhcLHEudzN0rltyTySDWzJ1vi0SDDzsEnwMBxIbDjDuwGohZnH4ACPmTQhLZJ/GOoS55/h3fzbsq2ex/4A/zeCWoAKmBM3nOHdJs3YdpjHgAEsgk8Lm5m0jMHhxI1ALZY9547zSBxmM7acg08L/+Fnkm8q6hLnAR1240dZtRx/e/PDG2/waIEAAyjNCIojZoLKUcAf0pSPglEwCkbByAAArP0/O+/OqhgAAAAASUVORK5CYII=","orcid":"","institution":"Department of Neurology, Yale School of Medicine, New Haven, CT","correspondingAuthor":true,"prefix":"","firstName":"Richa","middleName":"","lastName":"Sharma","suffix":""}],"badges":[],"createdAt":"2023-09-18 19:30:27","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3367169/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3367169/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41746-024-01120-w","type":"published","date":"2024-05-17T04:00:00+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":56697432,"identity":"a22cb85f-5fe9-4bd0-b9b0-e2d5fa359598","added_by":"auto","created_at":"2024-05-18 07:09:31","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2409294,"visible":true,"origin":"","legend":"","description":"","filename":"Manuscriptwithfigures.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3367169/v1_covered_abe589b1-02c9-4fa0-9747-5168ef9eda19.pdf"},{"id":45570032,"identity":"77273b00-0907-425f-838b-68e4ceb84aa6","added_by":"auto","created_at":"2023-10-31 16:25:34","extension":"pdf","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":494014,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryInformation.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3367169/v1/bad7a5fdba121d2ba19baafc.pdf"}],"financialInterests":"\u003cb\u003eYes\u003c/b\u003e there is potential Competing Interest.\nH.L., L.H.S., and R.S. are co-inventors of U.S. Provisional Patent Application No. 63/505,006, “Methods of Training an Algorithm To Predict Ischemic Stroke Etiology”. H.M.K. works under contract with the Centers for Medicare \u0026 Medicaid Services to support quality measurement programs, was a recipient of a research grant from Johnson \u0026 Johnson, through Yale University, to support clinical trial data sharing; was a recipient of a research agreement, through Yale University, from the Shenzhen Center for Health Information for work to advance intelligent disease prevention and health promotion; collaborates with the National Center for Cardiovascular Diseases in Beijing; receives payment from the Arnold \u0026 Porter Law Firm for work related to the Sanofi clopidogrel litigation, from the Martin Baughman Law Firm for work related to the Cook Celect IVC filter litigation, and from the Siegfried and Jensen Law Firm for work related to Vioxx litigation; chairs a Cardiac Scientific Advisory Board for UnitedHealth; was a member of the IBM Watson Health Life Sciences Board; is a member of the Advisory Board for Element Science, the Advisory Board for Facebook, and the Physician Advisory Board for Aetna; and is the co-founder of Hugo Health, a personal health information platform, and co-founder of Refactor Health, a healthcare AI-augmented data management company. A.D.H. has received consultant fees from Integra and Novo Nordisk, has equity in TitinKM and Certus, and receives author fees from UpToDate. K.N.S. reports investigator‐initiated clinical research funding to Yale from Hyperfine, Inc, Biogen, and Bard; reports from Sense and Zoll, for data and safety monitoring services; compensation from Cerevasc for consultant services; compensation from Rhaeos for consultant services, compensation from Certus for consultant services; and a patent pending for Stroke wearables licensed to Alva Health. S.K. is on the scientific advisory board of KovaDx and AI Therapeutics. H.K. reports compensation from Novo Nordisk for end point review committee services; compensation from Medtronic for other services; compensation from Janssen Biotech for other services; compensation from Boehringer Ingelheim for end point review committee services; and employment by Weill Cornell Medical College. L.H.S. reports compensation as a scientific consultant regarding trial design and conduct on late window thrombolysis and member of steering committee for Genentech (TIMELESS NCT03785678); user interface design and usability to LifeImage (privately held teleradiology company); member of a Data Safety Monitoring Board (DSMB) for Penumbra (MIND NCT03342664; PI, multicenter trial of stroke prevention in atrial fibrillation for Medtronic (Stroke AF NCT02700945). There were no competing interests reported by the remaining authors.","formattedTitle":"StrokeClassifier: Ischemic Stroke Etiology Classification by Ensemble Consensus Modeling Using Electronic Health Records","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"npj-digital-medicine","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"npjdigitalmed","sideBox":"Learn more about [npj Digital Medicine](http://www.nature.com/npjdigitalmed/)","snPcode":"41746","submissionUrl":"https://submission.springernature.com/new-submission/41746/3","title":"npj Digital Medicine","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-3367169/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3367169/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eDetermining the etiology of an acute ischemic stroke (AIS) is fundamental to secondary stroke prevention efforts but can be diagnostically challenging. We trained and validated an automated classification machine intelligence tool, \u003cem\u003eStrokeClassifier\u003c/em\u003e, using electronic health record (EHR) text data from 2,039 non-cryptogenic AIS patients at 2 academic hospitals to predict the 4-level outcome of stroke etiology determined by agreement of at least 2 board-certified vascular neurologists\u0026rsquo; review of the stroke hospitalization EHR. \u003cem\u003eStrokeClassifier\u003c/em\u003e is an ensemble consensus meta-model of 9 machine learning classifiers applied to features extracted from discharge summary texts by natural language processing. \u003cem\u003eStrokeClassifier\u003c/em\u003e was externally validated in 406 discharge summaries from the MIMIC-III dataset reviewed by a vascular neurologist to ascertain stroke etiology.\u003c/p\u003e \u003cp\u003eCompared with stroke etiologies adjudicated by vascular neurologists, nine base classifiers performed well with a mean cross-validated area under the receiver operating curve (AUCROC) of 0.90. Their ensemble meta-model, \u003cem\u003eStrokeClassifier\u003c/em\u003e, achieved a mean cross-validated accuracy of 0.74 and weighted F1 of 0.74. In the MIMIC-III cohort, the accuracy and weighted F1 of \u003cem\u003eStrokeClassifier\u003c/em\u003e were 0.70, and 0.71, respectively. SHapley Additive exPlanation analysis revealed that the top 5 features contributing to stroke etiology prediction were atrial fibrillation, age, middle cerebral artery occlusion, internal carotid artery occlusion, and frontal stroke location. We then designed a certainty heuristic to deem a \u003cem\u003eStrokeClassifier\u003c/em\u003e diagnosis as confidently non-cryptogenic by the degree of consensus among the 9 classifiers, and applied it to 788 cryptogenic patients. This reduced the percentage of the cryptogenic strokes from 25.2\u0026ndash;7.2% of all ischemic strokes.\u003c/p\u003e \u003cp\u003e \u003cem\u003eStrokeClassifier\u003c/em\u003e is a validated artificial intelligence tool that rivals the performance of vascular neurologists in classifying ischemic stroke etiology for individual patients. With further training, \u003cem\u003eStrokeClassifier\u003c/em\u003e may have downstream applications including its use as a clinical decision support system.\u003c/p\u003e","manuscriptTitle":"StrokeClassifier: Ischemic Stroke Etiology Classification by Ensemble Consensus Modeling Using Electronic Health Records","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2023-10-31 16:25:29","doi":"10.21203/rs.3.rs-3367169/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"npj-digital-medicine","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"npjdigitalmed","sideBox":"Learn more about [npj Digital Medicine](http://www.nature.com/npjdigitalmed/)","snPcode":"41746","submissionUrl":"https://submission.springernature.com/new-submission/41746/3","title":"npj Digital Medicine","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"b2a42ab7-7504-4e9c-904b-28443b123a9e","owner":[],"postedDate":"October 31st, 2023","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":25031915,"name":"Health sciences/Neurology/Neurological disorders/Stroke"},{"id":25031916,"name":"Health sciences/Risk factors"}],"tags":[],"updatedAt":"2024-05-18T07:09:23+00:00","versionOfRecord":{"articleIdentity":"rs-3367169","link":"https://doi.org/10.1038/s41746-024-01120-w","journal":{"identity":"npj-digital-medicine","isVorOnly":false,"title":"npj Digital Medicine"},"publishedOn":"2024-05-17 04:00:00","publishedOnDateReadable":"May 17th, 2024"},"versionCreatedAt":"2023-10-31 16:25:29","video":"","vorDoi":"10.1038/s41746-024-01120-w","vorDoiUrl":"https://doi.org/10.1038/s41746-024-01120-w","workflowStages":[]},"version":"v1","identity":"rs-3367169","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3367169","identity":"rs-3367169","version":["v1"]},"buildId":"_2-kVJe1T_tPrBINL-cwx","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}