Subset Mixture Model: Interpretable Bayesian Regression for Categorical Features with Principled Uncertainty

Subset Mixture Model: Interpretable Bayesian Regression for Categorical Features with Principled Uncertainty | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Subset Mixture Model: Interpretable Bayesian Regression for Categorical Features with Principled Uncertainty Aaron Danielson This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9272926/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Regression on tabular data with categorical features presents a persistent tension: interpretable models (GAMs, rule lists) are transparent but cannot capture higher-order feature interactions, while flexible black-box models (gradient-boosted trees) capture interactions but provide neither interpretability nor calibrated uncertainty. We introduce the \textbf{Subset Mixture Model (SMM)}, a supervised learning method that resolves this tension via empirical-Bayes aggregation of partition estimators. Each non-empty feature subset $s \subseteq {1,\ldots,D}$ induces a natural estimator $f_s(\mathbf{x})$, the empirical cell mean of training examples sharing the same values of $s$ as $\mathbf{x}$. SMM learns a convex combination of all $2^D - 1$ such estimators under a Dirichlet prior, with weights $\hat{\bm{\pi}}$ forming an interpretable, data-driven distribution over feature interaction orders. Weight uncertainty is propagated to the predictive distribution via the Laplace approximation, yielding a closed-form aleatoric/epistemic decomposition without post-hoc calibration. We prove an exact finite-sample oracle inequality: SMM's empirical NLL is within $(1/n)\log|\mathcal{S}|$ of the best single subset estimator, connecting the method to classical aggregation theory~\citep{cesabianchi2006prediction,tsybakov2003optimal}. On six benchmark datasets against eight baselines, SMM achieves the best NLL on three of six datasets and competitive RMSE throughout, producing well-calibrated predictive intervals without requiring post-hoc recalibration. Code and package: \url{ https://github.com/aaronjdanielson/subset-mixture-model} ; \url{ https://pypi.org/project/subset-mixture-model/} . interpretable machine learning empirical Bayes mixture model uncertainty quantification categorical features tabular regression Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9272926","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":617095853,"identity":"ff88ff4f-4987-40dd-b139-862ce0b7b32c","order_by":0,"name":"Aaron Danielson","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABO0lEQVRIie2PMUvDQBTHXwi0y0nXQGnyFe64wSFVv0qOQG5pwTFjoJAuRdf4LeKmkykP6hLpGsGhoVAXB7ulU00aBRsJXR3uxw13//d+vHsACsX/Ra+OlpQ38/B2ftW0oF2BSuGHpkohJxWoFRH8NLUpvelLvr4G2zrv6sm88N/kbSTfV/kDiKfujG0LGA7i5EgxUsl5BJI9TjoOknQzjrIRC0QKYkYWbp+AxxsKJF6nzFGLkVDUQhwH2UgLRFgqhpv0AVA0FGu5OShXMfY+57sQpZXJvFasfLIrYN9UaFZPEeUUSM5CdGjmsO8p+sIow6bCso3OCZVujB1a7oLsPv1gkQgNPks9zybU5XfHirn0tDXx7Yt4iXle+GiZz3K13YXDwXSa8tfCvxzcNNavv/c3MtpLCoVCoTjJF2eYfr13JB70AAAAAElFTkSuQmCC","orcid":"","institution":"The University of Texas at Austin","correspondingAuthor":true,"prefix":"","firstName":"Aaron","middleName":"","lastName":"Danielson","suffix":""}],"badges":[],"createdAt":"2026-03-31 01:39:27","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9272926/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9272926/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":107485861,"identity":"a623c73e-a4ed-4b29-833e-082fa6601bc4","added_by":"auto","created_at":"2026-04-22 02:36:38","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":512229,"visible":true,"origin":"","legend":"","description":"","filename":"smmsubmission.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9272926/v1_covered_b9390d18-101e-4dfc-b926-c5ef7e1b0ded.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Subset Mixture Model: Interpretable Bayesian Regression for Categorical Features with Principled Uncertainty","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":true,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"interpretable machine learning, empirical Bayes, mixture model, uncertainty quantification, categorical features, tabular regression","lastPublishedDoi":"10.21203/rs.3.rs-9272926/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9272926/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Regression on tabular data with categorical features presents a persistent\ntension: interpretable models (GAMs, rule lists) are transparent but cannot\ncapture higher-order feature interactions, while flexible black-box models\n(gradient-boosted trees) capture interactions but provide neither\ninterpretability nor calibrated uncertainty.\nWe introduce the \\textbf{Subset Mixture Model (SMM)}, a supervised learning\nmethod that resolves this tension via empirical-Bayes aggregation of\npartition estimators.\nEach non-empty feature subset $s \\subseteq \\{1,\\ldots,D\\}$ induces a\nnatural estimator $f_s(\\mathbf{x})$, the empirical cell mean of training\nexamples sharing the same values of $s$ as $\\mathbf{x}$.\nSMM learns a convex combination of all $2^D - 1$ such estimators under\na Dirichlet prior, with weights $\\hat{\\bm{\\pi}}$ forming an interpretable,\ndata-driven distribution over feature interaction orders.\nWeight uncertainty is propagated to the predictive distribution via the\nLaplace approximation, yielding a closed-form aleatoric/epistemic\ndecomposition without post-hoc calibration.\nWe prove an exact finite-sample oracle inequality: SMM's empirical NLL\nis within $(1/n)\\log|\\mathcal{S}|$ of the best single subset estimator,\nconnecting the method to classical aggregation\ntheory~\\citep{cesabianchi2006prediction,tsybakov2003optimal}.\nOn six benchmark datasets against eight baselines, SMM achieves the best\nNLL on three of six datasets and competitive RMSE throughout, producing\nwell-calibrated predictive intervals without requiring post-hoc recalibration.\nCode and package: \\url{https://github.com/aaronjdanielson/subset-mixture-model};\n\\url{https://pypi.org/project/subset-mixture-model/}.","manuscriptTitle":"Subset Mixture Model: Interpretable Bayesian Regression for Categorical Features with Principled Uncertainty","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-20 07:36:45","doi":"10.21203/rs.3.rs-9272926/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"f93b63c7-f05c-454d-8be0-d659ec137dd8","owner":[],"postedDate":"April 20th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-04-21T17:08:15+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-20 07:36:45","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9272926","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9272926","identity":"rs-9272926","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}