An ensemble method for identifying consistent models in interpretable machine learning

An ensemble method for identifying consistent models in interpretable machine learning | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article An ensemble method for identifying consistent models in interpretable machine learning Kristin Persson, Solomon Oyakhire This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5784379/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Machine learning (ML) is becoming indispensable for accelerating understanding and design in science. However, limited availability of data in the physical sciences necessarily directs data-driven methods towards feature-based and interpretable ML models. In this work, we demonstrate that today’s common interpretable ML tools lack consistency in feature and model selection when applied to different train-test splits of the same dataset. We trace these inconsistencies to variations in dataset distributions, insufficient regressor (feature) complexity, and strong pairwise correlations between regressors. To address these issues, we introduce an ensemble method that pairs the least absolute shrinkage and selection operator (LASSO) with Bayesian criterion-informed forward stepwise selection. Using this ensemble method, we provide new insights across five thematically different small data sets (with sample size between 23 – 1046 data points and number of features between 8 -18), focused on predicting the oxygen evolution reaction activity of metal oxides, the adsorption energy of various adsorbates on catalysts, the work function of oxides, the yield strength of steel, and the efficiency of lithium metal batteries. Physical sciences/Mathematics and computing/Computer science Physical sciences/Materials science/Materials for energy and catalysis/Electrocatalysis Physical sciences/Materials science/Materials for energy and catalysis/Batteries Full Text Additional Declarations There is NO Competing Interest. Supplementary Files supportinginformationensemblemethod.docx Supporting information Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5784379","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":427927044,"identity":"0f943cf5-de4f-48b6-86ac-221fcb5329b5","order_by":0,"name":"Kristin Persson","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAy0lEQVRIiWNgGAWjYBACCTiLvbmN4QFpWngOtjEkkKZFIpFILZLtZw8+rvhTJ2cu+bDtQQJDrZzBAQJapHnykg3Pth02tpyd2G6QwHDcmKAWOYYcM8nGhgOJG24ntkkkMBxLnNlASAv/G/OfDX/qEjfcPEikFmmJHDPGBjbmxA03GEFaahL7CehgkJzxLlmyEegXgzMgvxgcMOYnpEXifO7Bj0CHyRkcP3zswYeKOjk2QlqAUYjMMThMWAOaFoY6YrSMglEwCkbBCAMAMmpCaalnXFEAAAAASUVORK5CYII=","orcid":"https://orcid.org/0000-0003-2495-5509","institution":"Lawrence Berkeley National Laboratory","correspondingAuthor":true,"prefix":"","firstName":"Kristin","middleName":"","lastName":"Persson","suffix":""},{"id":427927045,"identity":"39b00be7-ba64-4f18-93dd-5de76937dba8","order_by":1,"name":"Solomon Oyakhire","email":"","orcid":"","institution":"University of California, Berkeley","correspondingAuthor":false,"prefix":"","firstName":"Solomon","middleName":"","lastName":"Oyakhire","suffix":""}],"badges":[],"createdAt":"2025-01-07 22:30:13","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5784379/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5784379/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":80749450,"identity":"9a5793df-e5aa-4e05-bdc7-57776d2d7cbc","added_by":"auto","created_at":"2025-04-16 16:04:53","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1127277,"visible":true,"origin":"","legend":"Article File","description":"","filename":"maintextensemblemethodforintepretableMLfinal.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5784379/v1_covered_cc9cf805-108b-4a72-8514-296975fefb46.pdf"},{"id":78492953,"identity":"fb4ca69e-d8db-44be-b8d5-89c83127bcfc","added_by":"auto","created_at":"2025-03-14 03:10:50","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":5382493,"visible":true,"origin":"","legend":"Supporting information","description":"","filename":"supportinginformationensemblemethod.docx","url":"https://assets-eu.researchsquare.com/files/rs-5784379/v1/a18e44011cf48a50c6c60aaa.docx"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"An ensemble method for identifying consistent models in interpretable machine learning","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-5784379/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5784379/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Machine learning (ML) is becoming indispensable for accelerating understanding and design in science. However, limited availability of data in the physical sciences necessarily directs data-driven methods towards feature-based and interpretable ML models. In this work, we demonstrate that today’s common interpretable ML tools lack consistency in feature and model selection when applied to different train-test splits of the same dataset. We trace these inconsistencies to variations in dataset distributions, insufficient regressor (feature) complexity, and strong pairwise correlations between regressors. To address these issues, we introduce an ensemble method that pairs the least absolute shrinkage and selection operator (LASSO) with Bayesian criterion-informed forward stepwise selection. Using this ensemble method, we provide new insights across five thematically different small data sets (with sample size between 23 – 1046 data points and number of features between 8 -18), focused on predicting the oxygen evolution reaction activity of metal oxides, the adsorption energy of various adsorbates on catalysts, the work function of oxides, the yield strength of steel, and the efficiency of lithium metal batteries.","manuscriptTitle":"An ensemble method for identifying consistent models in interpretable machine learning","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-03-14 03:10:45","doi":"10.21203/rs.3.rs-5784379/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"30453138-2fa6-49c3-8e35-f22c2e6cb42b","owner":[],"postedDate":"March 14th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":45593133,"name":"Physical sciences/Mathematics and computing/Computer science"},{"id":45593134,"name":"Physical sciences/Materials science/Materials for energy and catalysis/Electrocatalysis"},{"id":45593135,"name":"Physical sciences/Materials science/Materials for energy and catalysis/Batteries"}],"tags":[],"updatedAt":"2025-04-16T15:56:43+00:00","versionOfRecord":[],"versionCreatedAt":"2025-03-14 03:10:45","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5784379","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5784379","identity":"rs-5784379","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}