Probabilistic Machine Learning for Healthcare.

In this section, we show how probabilistic models can aid in different parts of the model development and model maintenance pipeline in healthcare. Here, we assume that the problem has already been reduced to a collection of features and a (potentially real-valued) response. The goal is to produce a model that predicts the response from the features. The section highlights probabilistic methods—referring to both models with probabilistic outputs or models with probabilistic computation in the development pipeline-to address key challenges in building predictive models for medicine. Missing values are a prevalent problem in clinical data that can impede predictive models, and probabilistic models allow for the modeling of the underlying data mechanism for missingness . Healthcare datasets are generally retrospective, observational, and frequently incomplete as a result. Consider a longitudinal clinical dataset of patient visits following a diagnosis of diabetes. For each patient visit, a clinician may choose not to measure all possible biomarker values, resulting in missing values. Patients may also vary in their number of clinical visits, resulting in missing visits for some patients compared to the maximum number of patient visits. Traditionally, machine learning models require completely observed datasets. Because removal of missing values may result in a dataset that is too small, or the removal may induce statistical bias, the search for other methods to accommodate missingness is an active area of research. In cases where predictive performance is of greatest importance, the model can directly incorporate missingness. One example might be passing indicators of observation as features, which provide the most information about the response ( 10 ). Using these indicators of observation in a time series, recurrent neural networks have been used to predict patient outcomes in the intensive care unit ( 11 ). Additionally, deep probabilistic models can marginalize missing values to predict time to coronary heart disease ( 12 ). The predictive performance may not be the only quantity of interest for a model. Researchers may also be interested in parameter estimation, e.g., using coefficients to model features importance. A main method to address missing data for parameter estimation is imputation , meaning the replacement of missing values based on information from the observed values. After imputation, the transformed data is used for the resulting predictive model. Imputation methods range from using the mean or median of the observed values to prediction of the missing value for each observation. One popular imputation method is multiple imputation using chained equations where missing features are imputed using the posterior predictive distribution of the missing data conditional on the observed data ( 13 ). Implicit in imputation methods is an assumption about the underlying data generating process ( 14 ), namely that the data is either missing completely at random (MCAR) or missing at random (MAR). MCAR is the assumption that the missingness of a data is completely random and uncorrelated whereas MAR refers to missingness of data that depends on the observed data. Notably, imputation methods cannot support data that is missing not at random (MNAR), meaning the missingness correlates with an unobserved characteristic. Using either the assumption of MCAR or MAR, imputation methods can leverage probabilistic methods such as Gaussian methods ( 15 ), causal diagrams ( 16 ), or models using auxiliary information ( 17 ). Identification of MNAR requires additional assumptions, for example semi-parametric estimation using an instrumental variable ( 18 ). Additionally, modeling the data missingness process allows for the model stability when the mechanism for data missingness changes, e.g. across hospitals or across time. Similar to how probabilistic models can address missing features as described in Section 3.1 , probabilistic models can capture the probability distribution of the possible outcome events when the patient labels are not observed. Labels in healthcare often depend on a patient’s state at some point in the future. This gap in time between observed features and observed label means that the labels may be unobserved or censored for some patients ( 19 ). We may be interested in the time to event, e.g., death, for a patient given observed features and may observe previous patients, only some of whom have observed times to event. A simple machine learning method might regress on the time to event only for patients with observed labels, but this simplification generally underestimates the time to event because patients with a longer gap between observed features and time to event are less likely to be observed. In contrast, probabilistic models to characterize the survival function are trained with the label likelihood and can directly address censored observations by computing the probability that the observed label falls in the censoring interval through integration when observations are censored at random. The general assumption that makes survival problems tractable is censoring at random where the censoring and event time are independent given the observed features. Under the censoring at random assumption, consider a patient that has no event until a censoring time c with features x , the likelihood under a distribution p can be computed as ∫ c ∞   p ( a ∣ x ) da . This approach has been used in combination with deep neural networks in recent work on deep survival analysis ( 10 , 20 , 12 , 21 ). The evaluation of a survival function requires consideration about the probability distribution of outcomes. The Brier score ( 22 ) is the metric traditionally used for estimating the survival function. Evaluation with the Brier score requires appropriate adjustment for censoring by estimating the (inverse) probability of censoring. Non-probabilistic methods like survival forests ( 23 ) have also been used to incorporate right censoring. Survival forests side-step explicit parametrization of censoring mechanism but build on ensembles of random trees to non-parametrically estimate a cumulative hazard function (a probabilistic quantity) using the Nelson—Aalen estimator ( 24 ). In the presence of more complex forms of censoring, like interval censoring which is common in epidemiological studies, uncertainty over the interval of censoring should be modeled to fix bias in survival estimates. This statistical bias is particularly problematic if the interval periods themselves are long. Modeling the uncertainties over the censoring mechanism have been demonstrated to improve estimation over imputation techniques ( 25 ) and has been only recently explored in machine learning literature ( 26 ) in a non-probabilistic framework using random forests. Probabilistic machine learning models can ensure that risk scores used in clinical settings are calibrated , meaning that the risk estimates accurately characterize the actual risk. Risk scores like the Framingham risk score ( 27 ) for cardiovascular disease predictions are routinely used for clinical decision making, diagnostic tools, or determining subsequent treatment pathways. There is a general expectation that in addition to predicting the correct binary label y (whether a patient will develop coronary heart disease in 10 years), the actual risk estimate of the event is available as well. These risk estimates can be obtained only from a machine learning model that frames the supervised learning problem as that of estimating the probability p θ ( y ∣ x ) . Therefore, support vector machines (SVMs) will not directly provide such risk estimates without further processing. How well do machine learning models trained to estimate p θ ( y ∣ x ) characterize this risk? This can be understood by quantifying how well-calibrated a model is. A machine learning method is well-calibrated if for all examples, x , for which the model provides the same estimate p ˆ of p θ ( y ∣ x ) , the proportion of these examples that are associated with the prediction ( y = 1 ) is equal to p ˆ . Calibration is therefore an inherently probabilistic concept. While traditional machine learning methods like logistic-regression and shallow neural networks typically produce well-calibrated risk scores, modern neural networks notoriously may not ( 28 ). Further, even if a model is calibrated at the population level, subpopulation level miscalibration can further amplify inequities in clinical decision making ( 29 ). Since calibration significantly affects optimality of downstream clinical decision making ( 30 ), diagnostic, and other decision support models should consider probabilistic frameworks for supervised model design. Calibration of machine learning models can be quantified using reliability diagrams which evaluate the estimated risk of a group of example p ˆ compared to the expected sample accuracy for all p ˆ . A well-calibrated model will be close to the identity function in the diagram (see Figure 1 ). Assessing the expected (or maximum) difference between the estimated confidence/risk of a model and empirical estimates, a quantity known as the Expected (or Maximum) Calibration Error ( 31 ) can summarize how well-calibrated a model is (lower is better). Reliability diagrams can also be assessed at subpopulation levels to determine miscalibration challenges due to lack of samples within groups. Calibration of machine learning methods can be improved post-hoc using techniques such as Platt’s scaling ( 32 ), which corrects for calibration errors after model training by learning a mapping from learned risk scores to calibrated risk scores by estimating a sigmoid mapping on a validation set. These methods can also be used for non-probabilistic models like SVMs that do not produce risk scores as a by-product of supervised learning. Platt’s scaling is essentially refitting a logistic regression (a probabilistic model) to obtain risk scores from deterministic models like SVMs. When data is scarce, flexible non-parametric methods like isotonic regression can be used for calibration ( 33 ). Sigmoid function refitting as done in Platt’s scaling can also be extended to multiclass settings by modeling multiclass problems as a one-vs-all classification ( 34 , 35 ). Calibration can also be improved with other post-hoc methods. Binning methods to obtain calibrated neural networks have also been proposed recently ( 36 , 34 ). This class of methods suitably re-estimate risk scores to directly improve/optimize calibration metrics. Several types of uncertainties arise when modeling clinical outcomes of interest from finite amount of data. Machine learning model predictions are based on a finite random samples, making the model predictions themselves random. That is, any machine learning model (deterministic or probabilistic) is a function of the random samples from the data generating distribution and hence, has an associated uncertainty. The overall uncertainty of a machine learning model prediction, captured by p ( y ∣ x ) is known as the predictive uncertainty of the model. To see how the uncertainty can affect downstream decision making, consider a breast cancer staging model that predicts risk of an adverse event. If the model-estimated disease stage has different variances for different features values, the resulting decisions can be suboptimal if a single decision threshold is used. Predictive uncertainty can be further decomposed into different sources. The first source, called aleatoric uncertainty, measures the noise in the labels in the true data generating process. For instance, diagnostic labels can have some uncertainty when different clinicians annotate the same sample ( 37 ). The second source stems from the uncertainty about an estimated model’s match to the true data generating distribution. This uncertainty is called model uncertainty, also known as epistemic uncertainty. Model uncertainty is a combination of the choice of the model class used to approximate a property of the true data generating distribution, as well as the fact that multiple parameters θ within the same model class can approximate the data well. When a large enough sample size is available and assuming correctness of the model class, uncertainty can be quantified using asymptotic analyses ( 38 ). Large scale datasets may not always be available. In this case, uncertainty is either captured via bootstrapping samples ( 39 ) without explicit probabilistic modeling or using a fully Bayesian framework . Bootstrapping is a sampling with replacement procedure, a proxy to quantify uncertainty over the data sample, so that multiple model estimates can be obtained. Predictions from different candidate models then give an estimate of variability in predictions, i.e., capturing uncertainty. Neural networks can also be retrained on bootstrapped samples to quantify this uncertainty although performance may degrade with fewer data samples ( 40 ). These methods can be used to assess overall uncertainty as well as calibration of a model, but still only provide expectations over outcomes rather than a full characterization of the predictive uncertainty. Thus, the onus is on the end user to calculate probabilistic quantities of interest like variance that will be useful for downstream decision making. In a fully Bayesian framework, the outcome of interest is naturally modeled as a distribution i.e. p ( y ∣ x ) . Additionally, distributional assumptions are made over the model parameter θ itself allowing to fully characterize as well as decompose the predictive distribution. The relationship between overall uncertainty and model uncertainty p ( θ ∣ 𝒟 ) (where 𝒟 is the dataset sample from the generating distribution) is captured by the following relationship: p ( y ∣ x ) = ∫ θ p ( y ∣ x , θ ) p ( θ ∣ 𝒟 ) d θ Notably, such uncertainty estimation via Bayesian modeling is more challenging to incorporate for non-probabilistic machine learning models like support vector machines. In deep neural networks, Bayesian deep learning ( 41 ) is commonly employed for capturing this uncertainty over predictions, i.e. p ( y ∣ x ) . These methods usually capture intermediate uncertainties i.e. over parameters p ( θ ∣ x ) during model training. For instance, one method reinterprets traditional regularization methods like dropout regularization in a probabilistic framework ( 42 ). However, eliciting good prior distributions over parameters, required for Bayesian modeling, can be a challenge for modern deep neural networks. Even in simple models, misspecification of such priors can lead to unreliable uncertainty estimates ( 43 ). Recently, extensive evaluation of Bayesian neural networks and neural network ensemble methods for uncertainty characterization has been demonstrated on critical care datasets eICU and MIMIC-III ( 44 ). Similar to bootstrapping, neural network ensembles do not explicitly model the predictive distribution to obtain uncertainty estimates. In one example, model predictions and hence downstream decisions can vary widely for individuals due to these uncertainties and quantifying this uncertainty can reduce the possibility of sub-optimal clinical decisions ( 44 ). After clinical model development, the data setting in which the model is used may differ from the setting in which the model was trained, a shift that can be characterized and accommodated by probabilistic models. For example, model performance may degrade when the data distribution for development and for deployment differ across locations ( 45 ) or across time ( 46 ). This shift of data can threaten the performance of machine learning models in deployment. Consider a setting where a model is trained to predict an outcome y based on patient features x . When a model is trained on a source domain with data distribution p ( y ∣ x ) , the concern is that the setting of deployment may have a different distribution in p ( x ) ( covariate shift ) or p ( y ∣ x ) (label shift), as seen in Figure 2 . Probabilistic models can describe this process and improve robustness of models over different data distributions. Probabilistic models can address covariate shift through identification and correction. Covariate shift is the condition where the feature distribution changes. For example, urban and rural hospitals have differences in clinical care patient populations. Rural hospitals have higher prevalence of chronic obstructive pulmonary disease including higher rates of Medicare hospitalizations and death ( 47 ). In model settings with long time-series sequences, for example continuous monitoring in medical settings, change point detection identifies abrupt changes in observed data where p ( x ) shifts dramatically ( 48 , 49 ). In settings with clear and known demarcations between source and target domains, for example across hospitals, probabilistic models can adapt model development on the source domain to a target domain for deployment. One technique uses importance sampling, based on estimates of the probability of the domain given the features, to reweight observed datapoints ( 50 , 51 ). Modeling and correcting for data shifts allows for more accurate and robust clinical models. Label shift, meaning p ( y ∣ x ) changes, is the mechanism of labels conditional on features changes and may be due to changes in treatment policy or induced by the clinical model itself ( 52 ). For example, the clinical protocol for disease management may change as additional scientific discoveries shape medical knowledge, resulting in multiple myeloma survival rates improving dramatically with one-year relative survival rates increasing from 69% in 1973 to to 82% in 2013 due to better treatment policies ( 53 ). Without reasoning about this shift, models may yield erroneous and outdated results. In one famous example, a model predicting pneumonia risk stratification did not account for patients with asthma being given more severe interventions; the predictions for pneumonia patients with asthma yielded lower risk when clinical literature reveals the opposite ( 54 ). In that case, models can be trained to be anticipate shifts in policy based on the treatment policy distribution ( 55 ) and therefore more readily update for changes in clinical treatment protocol. Alternatively, the model itself may induce change in the data after deployment. For example, a diagnostic model may yield predictions that guide treatment policy where high risk patients receive more aggressive treatments, meaning the model may not be calibrated to the outcomes that occur after model deployment. One method to address this shift is to produce probabilistic models that yield stable and calibrated predictions for not only pre-deployment outcomes but also against future outcomes that manifest from the data shifts ( 56 ). Strategic behavior from the individuals may also affect the data distribution, for example, with otherwise healthy patients adapting to the deployed clinical model for treatment and displaying behaviors similar to sick patients in order to receive additional treatment. Causal inference methods can model complex systems to address changes in strategic behavior ( 57 , 58 ). The fairness of a clinical model is a key concern to ensure unbiased predictions for medical tasks, and probabilistic models help enable the defining, auditing, and ameliorating this algorithmic bias. Because of the high-stakes setting of healthcare, researchers have raised numerous concerns about fairness and algorithmic bias in clinical contexts ( 59 , 60 , 61 ). There exist many definitions of algorithmic fairness ( 62 , 63 , 64 , 65 ), and probabilistic models are crucial to representing and addressing many notions of ethics and disparity. One branch of fairness focuses on the notion of individual fairness ( 63 ). This fairness definition implies that a patient with similar characteristics to another patient should not receive worse clinical care than the other. A key requirement is that similar patients receive similar probabilistic predictions, using some notion of similarity ( 65 ). Another branch of concerns analyzes fairness across groups . In this family of fairness definitions, algorithmic bias is assessed based on known and pre-defined sensitive attribute groups, e.g. race, gender, socioeconomic status. Because some definitions are impossible to satisfy simultaneously ( 66 , 67 ), definition choice is crucial in different health settings. The fundamental idea across group fairness definitions is that a definition of performance should not differ across groups and statistically significant violations of this assumption could prove algorithmic bias. Although not all definitions of group fairness require probabilistic models, we outline a few examples of group fairness definitions that leverage the data distributions. For example, the two groups may not adjust to changes in data shift in the same way. Probabilistic models can learn models on the source data distribution that will adapt to a target data distribution while satisfying group fairness definitions ( 68 ). In another example, examining the calibration of an algorithm across each group may reveal larger disparities ( 69 , 70 ), and health settings may require calibration across multiple subgroups ( 71 ). Lastly, probabilistic models can reason about regression on continuous outcome variables, for example in the case of health care costs. When comparing predictions for healthcare costs between patients with mental health and substance abuse disorders compared to patients without them, the sensitive attribute group and the residual error from a learned model should be independent. Researchers can then measure the covariance between the sensitive attribute group and the residual error as a proxy for independence to detect the level of algorithmic bias in the model ( 72 ). The quest to propose solutions to algorithmic bias is still in its infancy. Notably, any balance between algorithmic fairness and accuracy may be ethically challenging for health settings. Although non-probabilistic models solutions to algorithmic bias certainly exist ( 60 ), probabilistic models consider how to fix discrimination within the constraints of the model. One method seeks to induce independence between model predictions and the sensitive attribute through a learned latent representation that maximizes performance and minimizes dependence on the sensitive attribute ( 64 ). Another approach alters the objective function of the predictive model to regularize for algorithmic fairness based on a specified definitions ( 73 , 74 ). Lastly, probabilistic models can reason about the effectiveness of additional data collection or other actions using the existing data ( 75 ).

Machine learning for healthcare holds promise to reshape healthcare. In this review, we give a broad overview of the role that probabilistic modeling plays in medicine. This review does not touch on all of the areas of healthcare that benefit from probabilistic modeling. For example, causal inference ( 138 ) is a central question in medicine. Probabilistic methods have been used to estimate causal effects in HIV treatment ( 139 ), and in general, probabilistic techniques have been shown to give some of the most accurate inferences at a well-known causal inference challenge ( 140 ). As another example, healthcare often involves time-series data, which benefits from a probabilistic approach for providing uncertainty estimates over forecasts for the entire sequence ( 141 ). We believe that a probabilistic perspective can yield significant benefits when building machine learning models for healthcare; this review covers specific examples in building predictive models, phenotyping, sample generation, and learning policies. We encourage further research into both methodologies and applications with probabilistic machine learning models in healthcare.

Generative models refer to the class of probabilistic models trained to produce samples that match samples from the distribution from which observed data is collected. For example, given data vector x , we can sample from p ( x ) using a generative model to produce synthetic but realistic data, as shown in Figure 4 . Recent advances in deep learning have led to promising generative models. Generative adversarial networks (GANs) ( 104 ) use two neural networks to first create artificial imitations of the training data and then separately decide whether a given sample was genuine or counterfeit. Other models include deep likelihood models like normalizing flows ( 105 ) where a simple initial density is transformed into a more complex one by applying a sequence of invertible transformations. In this review, we focus on three promising cases for generative models in medicine. First, we describe the the use of generative models to overcome data deficiency or accessibility challenges in clinical data. Next, we describe how generative models can be used for clinical tasks like abnormality detection and modality translation in medical imaging. Lastly, we address the generation of viable candidates in the drug discovery process. Clinical predictive models often suffer from poor performance because of insufficient data stemming from patients with rare conditions, labels that are costly to obtain, or other logistical challenges. Data augmentation using generative models can create synthetic data that is similar to the true dataset of interest to mitigate the effects of insufficient data. Particularly in medical imaging ( 106 ), generative models can then accelerate model development that would be otherwise impeded by small dataset sizes. Data augmentation can address small or imbalanced datasets through the generation of synthetic datapoints. For example, GANs have been used to augment all three classes of liver lesions ( 107 ), to increase the number of positive examples of bone lesions ( 108 ) or brain metastases ( 109 ), and to provide additional red blood cell images for downstream segmentation tasks ( 110 ). Note that when data augmentation is applied to a dataset, the augmented data does add any additional information, but the augmented data can regularize the model or help mitigate class imbalance. Data augmentation can address privacy and security concerns that impede model development by creating synthetic data that can be shared across institutions. Researchers created synthetic patients using a GAN based on actual patients in a systolic blood pressure trial using GANs with differential privacy constraints. The synthetic patients were similar enough to the original trial patients that models trained on each dataset yielded effectively the same results ( 111 ). Machine learning models trained using the synthetic data generalized well to the original training data. Because sharing individual-level data across clinical institutions often requires close collaboration and extensive data use agreements, data augmentation and distribution of privacy-preserving synthetic data can circumvent these restrictions. Generative models can assist with clinical tasks including abnormality detection and modality translation in medical imaging ( 112 ). With regards to abnormality detection, images from healthy patients are used to learn a latent space. Then, a reference medical image potentially containing an abnormality is encoded into the healthy latent space. Samples are drawn from the latent space near the encoded image and compared against the reference medical image. Any differences between the generated samples and the reference medical image are highlighted and analyzed for medical abnormalities ( 113 , 114 ). The task of translation of medical imaging between modalities — e.g., positron emission tomography (PET) scan to computed tomography (CT) scan — is necessary for tasks like attenuation correction of PET data using CT scans. Two different encoders are learned for two medical imaging modalities with the same latent space. Then, one image can be translated into another modality by mapping first to the latent space and then decoding to the second modality ( 115 ). Currently the development of de novo drug-like compounds relies on the identification of new molecular entities. While prior methods have relied on manual selection of candidates or molecular predictors to estimate viability, sampling from generative models can create large virtual chemical libraries, which can then be screened more efficiently for in silico drug discovery purposes. Methods for candidate generation focus on use cases that are goal-based or distribution-based ( 116 ). Goal-based describes methods that generate molecular structures that perform well according to a scoped goal like structural or physiochemical features, and they do not necessarily rely on probabilistic models. Distribution-based methods observe acceptable molecules and generate similar molecular structures, a task well-suited for generative models. Using GANs, researchers have produced novel small-molecule organicestructures ( 117 ). However, molecules generated by GANs may lack diversity ( 118 ), a problem that other generative models such as VAEs do not suffer ( 5 ). One model combines distribution-based and goal-based approaches by learning a latent representation of molecules using a VAE and then using reinforcement learning using a reward function, similar to the goal-based scores, to explore the space ( 119 ).

We now pivot to three major healthcare application areas where probabilistic modeling has been extensively used, starting with phenotyping. Phenotyping ( 76 ) is the process of producing concise representations of medical concepts or diagnoses composed of observable clinical traits that could be used to facilitate cohort selection or trait definition ( 77 ). From a latent variable perspective, we can hypothesize that simple latent “summaries” explain the variation in observed patient records, and serve as a governing factor in determining how different patients will progress ( 78 ) or react to different interventions ( 79 ). Prior work has identified many forms of electronic phenotyping, including rule-based methods, text processing, noisy data learning, and unsupervised discovery of latent phenotypes ( 80 ). We categorize these efforts into three settings, based primarily on the amount of the level of label supervision used. In some settings the goal is phenotypic matching , where phenotypes are explicitly defined, and the goal is to map noisy data sources into these labels. In others the goal is uncovering latent phenotypes , where there is uncertainty about what phenotypic definitions should be, and the goal is to identify useful characterizations that could impact patient care ( 81 ). Semi-supervised phenotyping is a hybrid approach that straddles matching and discovery, where we assume that some label information is available, e.g., with the use of specific “anchoring” clinical terms ( 82 , 83 ). Both phenotypic matching and uncovering latent phenotypes are visualized in Figure 3 . In this review, we review each of these three settings—phenotype matching, uncovering latent phenotypes, and semi-supervised phenotyping—outlining key areas where probabilistic models have made, or could make, an impact. Phenotypic matching is a computational approach that involves matching or summarizing patient records x into validated phenotypes z . This style of phenotyping is useful when a meaningful classification or alert could be automated for patients that need to be treated early, or more efficiently. For example, structured electronic health record (EHR) data in the form of International Classification of Disease (ICD) billing codes and medication prescriptions have been used to detect coronary heart disease or rheumatoid arthritis ( 84 ). More recent work has also targeted use of natural language processing ( 85 ), with additional unsupervised learning of important features from the structured EHR ( 86 ). Importantly, phenotypic matching assumes that the existing phenotype definitions z are appropriate, robust, and identifiable from the given data x . Known challenges in label leakage, soft labels, confounding and missingness must therefore be considered very carefully ( 87 ). Probabilistic methods could be particularly useful in phenotyping due to the inherent uncertainty in many phenotype definitions, which can change over time as in autism ( 88 ). Some work has moved to integrate notions of uncertainty into matching, either by performing large-scale phenotype estimation from observational data ( 89 ), or by visualizing probabilistic estimates over phenotypes for interactive verification ( 90 ). More generally, models that attempt to discover phenotypes based on underlying data regularities, or use prior labels in a semi-supervised fashion, are well-suited for probabilities modeling. Instead of matching to a known phenotype z , many problems in healthcare require identifying potential phenotypes from data. When phenotypes are unknown, they are hidden. This makes finding unknown phenotypes well-suited to latent variable models. This style of model takes the phenotype of a person to be a latent variable z with a hypothesized prior distribution p ( z ) . The latent variable controls the distribution of the observed data x through the likelihood p ( x ∣ z ) . As a whole, the model with parameters θ is p θ ( z , x ) . The goal during learning is to find parameters of the latent variable model that make the observed data likely. Typically, this is done through an approximation to the posterior p ( z ∣ x ) , which also helps to compute phenotypes given an observation from a single patient. To illustrate, imagine that x is a vector of measured traits like hemoglobin A1C—a test that evaluates the average glucose in blood over the past few months, blood pressure—a measure of the pressure that blood exerts within arteries, and cholesterol—a waxy substance that can develop into fatty deposits in blood vessels. Let the phenotype z be a binary variable marking a patient as belonging into one of two possible phenotypes. The likelihood is set up such that the phenotype is encouraged to explain relationships between the observed traits, e.g., blood pressure and cholesterol have a positive correlation. One salient example of a latent variable model are variational autoencoders (VAEs) ( 91 ), where neural networks are both used to parametrize the likelihood and the posterior approximation. Recent work has used VAEs in a health setting to create low-dimensional latent representations of a phenotype feature space that additionally capture individual rates of change along each dimension during aging ( 92 ). We note that non-probabilistic methods have also identified meaningful subtypes including asthma subtypes through hierarchical clustering ( 81 ) and type 2 diabetes through tensor factorization ( 93 ). Uncovering latent phenotypes (and subcategories) often aims to discover similar patient subgroups that may share the same underlying disease mechanism, or treatment response patterns. In conditions such as asthma, where there is heterogeneity in symptom expression and response to therapy, phenotyping subcategories of z into z 1 … z k can be particularly useful. Data-driven methods for data-driven characterization of wheeze patterns, and the further discovery of biomarkers to identify phenotypes are useful, particularly in childhood ( 94 , 95 ). Treatments themselves can also be heterogenous, and clustering them can reveal treatment profiles at a large scale that would not have been clear in a smaller sample ( 96 ). For example, recent work in endometriosis subtyping provides evidence of different treatments in each of four subtypes learned in an unsupervised mixed-membership model ( 97 ). Importantly, while the subtypes present new knowledge, they also align well with previous clinical knowledge about endometriosis, and reflect direct patient experiences with endometriosis. Probabilistic clustering of time series specifically can be of high-value in clinical settings, because sequential clinical markers from different patients could be heterogeonous. For instance, autoimmune diseases are known to be heterogenous, and hierarchical probabilistic models can be used to infer disease subtypes in such patients and explain away correlations that are not relevant to the question of interest ( 98 ). In more acute settings, neural networks and switching state autoregressive models have been used in the intensive care unit to predict upcoming interventions ( 99 , 100 ). These estimations can be used to group patients that are “maximally activiating” for intervention onsets ( 101 ). Phenotypic matching can initially be very labor intensive, as it requires many manual gold standard annotations of z . In some cases, there is some strong information that can be used for partial phenotype matching, but other facets of the phenotype must be discovered. One potential solution to this is to assume that phenotype labels themselves are weak or “semi-supervised”, with only a few known features from data being clear conditional markers. In this setting, probabilistic modeling approaches can be used to identify additional features that correlate with the anchoring clinical terms ( 82 , 83 ). Anchoring can be thought of as a form of noisy labelling, where observing the positive anchor a i unambiguously reveals the state of latent phenotype z i to be positive, but a negative anchor a i reveals nothing about z i , so that p z i = 1 ∣ a i = 1 = 1 . Other work has similarly used anchor words to provide a form of supervision in topic modeling for characterizing pancreatitis outcomes ( 102 ). Further work in phenotype inference with semi-supervised approaches have used mixed membership models ( 103 ) to inferring binary labels for clinical condition targets when trained on limited samples.

Recent advances in healthcare—from tracking patient health with mobile phones ( 1 ), to predicting peripheral vascular disease from retinal images ( 2 ), to detecting sepsis early in intensive care units ( 3 ), to summarizing medical records to reduce the digital burden on clinicians ( 4 )—have made use of artificial intelligence and machine learning. The power of machine learning is rooted in both the assumptions encoded in a model and the troves of healthcare data used for training. Abstractly, healthcare data can be thought of as coming from a probability distribution called the data generating distribution. It is through this data generating distribution that learning and evaluating a machine learning model can be formalized. For example, the error of a machine learning model on a test set, i.e., data that is not used during training, measures how well this model performs on new samples from the data generating distribution. In many uses of machine learning models in healthcare, the data generating distribution does not need to be made explicit. For instance, making predictions or finding regression coefficients may not require consideration of the data generating distribution. We illustrate why a probabilistic machine learning model is helpful in healthcare. Imagine modeling the survival time for melanoma patients based on stage and demographic information. If two observed stage, demographic pairs have the same average survival time, but the variance in survival differs, the average survival time can mislead planning. A probabilistic model of survival would instead provide a more holistic view by returning a distribution over survival times given the stage and demographic information. Such a probabilistic model would enable both the patient and the provider to better plan for the future by incorporating uncertainty into the decision making process. For example, a melanoma patient with high certainty about a short survival time might choose to make specific life adjustments. The probabilistic perspective can aid the entire machine learning model development and maintenance pipeline. Missing values can be mollified by probabilistic models. Censoring of labels, such as survival time, can be addressed through probabilistic models. The probabilistic view can also aid in maintaining deployed models by detecting shifts in a data generating distribution over time. Probabilistic machine learning models have a myriad of practical uses in healthcare. A type of probabilistic model called a latent variable model can be used for phenotpying. Probabilistic models can also be used for simulation, which has seen success in scientific discovery like in drug development ( 5 ). The probabilistic approach has shown promise in learning policies for diabetes and sepsis management, among other applications that use reinforcement learning ( 6 , 7 ). However, estimating probabilistic models comes with a cost. Probabilistic models can require more mental and computational labor than non-probabilistic approaches, though this labor is being reduced through the introduction of probabilistic building blocks in common machine learning toolkits ( 8 , 9 ). This review focuses on the uses of probabilistic models in healthcare. At the end of this review, the goal is for the reader to understand how probability plays a role in building models and how it can help address challenges that occur in machine learning models in healthcare. This review is organized as follows. The first section sets up mathematical notation for the basic concepts around probabilistic models. The subsequent sections are organized by use cases in healthcare that make use of the probabilistic perspective.

To make the distinction between a probabilistic model and a deterministic model clear, we present an example. Consider features x and a count-valued response y like lymphocyte count. Now take a model with parameter θ denoted g θ ( x ) ; this model is a function that predicts lymphocyte count based on the features and can be learned by finding the parameters that minimize the squared error between the model predictions and the observed response. For simplicity we do not consider a fixed data set, but rather assume access to the data generating distribution F . See the margin notes for common notation used in this review, including response, probability, and expectation. With this setup, the model g θ ( x ) can be trained by finding θ to minimize E x , y ~ F g θ ( x ) − y 2 . If trained well, the model g θ will be close to the expected value of y given the observed features x . This type of model is deterministic and can be used to make predictions. However, without assumptions, this model says nothing about the distribution of the response. A probabilistic model represents a probability distribution. A probabilistic model of a response y given features x , rather than being a function, would be a probability distribution p θ ( y ∣ x ) . One way to train such a model from data is to maximize the likelihood of the observations, E x , y ~ F log p θ ( y ∣ x ) . If trained well, the model p θ would be close to the conditional distribution of y given the features x in the data generating distribution. The probabilistic model p θ can not only be used to compute the average value of y given a particular observed feature (by computing E p θ ( y ∣ x ) [ y ] , but also the conditional variance and other statistics as well. Note that binary classifiers are special in that their probability distribution is completely characterized by its expectation. 1 Probabilistic models go beyond regression. We list different flavors of probabilistic models— predictive models, generative models, and latent variable models —in the margin notes. Probabilistic models can refer to both models with probabilistic outputs or probabilistic modeling components in a broader estimation pipeline. In the subsequent sections, we will go into more detail on the role of probabilistic models in healthcare for building predictive models ( Section 3 ), for phentopying ( Section 4 ), for simulation ( Section 5 ), and for sequential decision making ( Section 6 ).

Sequential decision making is a core part of healthcare ( 120 , 121 , 122 , 123 , 7 , 124 , 125 , 126 ), and probabilistic models enable a variety of approaches including characterizing randomness in patient disease progression and stochasticity of clinician intervention practices ( 6 , 127 , 120 ). Consider a patient admitted to the intensive care unit (ICU) as they develop respiratory failure. Clinicians will attempt to manage the patient’s condition with a series of advanced respiratory interventions over the coming hours/days with the goal of restoring their function for a successful discharge and survival. During the ICU stay, a sequence of interventions are done. For example, starting with a mechanical ventilation for urgent and initial resuscitation of the patient, to prevent further deterioration. After this is stabilized, further interventions like more therapeutic treatments and additional differential diagnosis follow. This is an example of clinical care that is a series of interventions to improve the chances of patient survival. Similarly, treating progression of chronic conditions is a sequential set of interventions to manage disease severity, just over months/years of disease progression. In machine learning, the closest analogue to modeling this problem is known as Reinforcement learning (RL). The primary goal of an RL algorithm can be to either learn a function or a distribution over possible interventions given patient state (policy learning) or evaluating the potential reward of an existing policy (policy evaluation). RL can be seen as a generalization of supervised machine learning learning where the goal is to make a sequence of optimal decisions to maximize long term rewards ( 126 ) (patient survival in the ICU example). In this setup, a clinician (learning agent) interacts within an environment (patients) via actions (ventilation and other interventions) it performs, observes the changes in the environment (patient state) in order to successfully discharge the patient (maximize a longer term numerical reward) ( 128 ). Canonical RL assumes the ability to conduct experiments in the target environment, which allows the learning algorithm to collect samples to learn a reasonable policy, called “model-free” learning, explored for diabetes management ( 129 ) and sepsis management ( 130 , 7 ). In model-free learning, an RL agent does not model the underlying transition dynamics of the data, which are how a patient state changes based on their current state and the current intervention performed by the clinician. In healthcare, it is not feasible to conduct online experiments, due to ethical concerns around patient safety, neither is it easy to collect vast amounts of data. Embedding prior knowledge about transition dynamics is therefore key in healthcare and can reduce the need to collect many samples. This is usually done via “model-based” RL, which involves explicitly modeling the transition dynamics to learn a policy. For example, consider the problem of managing type-1 diabetes, where the goal would be to determine amount of insulin to administer based on continuous glucose monitoring ( 6 ). While the underlying physiological glucagon kinetics and secretion are well characterized ( 131 ), carbohydrate intake of a patient is stochastic ( 6 ). This makes the transition dynamics of the patient’s glucose model partially stochastic ( 132 ) and should be modeled accordingly for healthcare RL tasks. Clinical data is rife with different forms of uncertainty. The nature of uncertainty can be due to noisy or incomplete observations of patient state. In Section 3.4 , we focused on capturing uncertainty for predictive and diagnostics tasks. Here, we focus on how uncertainty should be accounted for treatment planning in sequential decision making. First, there is inherent randomness in how patients are treated as well as in healthcare data-collection. For example, dosages can differ across hospital practices; so can reporting standards and practices ( 133 , 134 ). There may also be some uncertainty between when a drug was administered versus when the intervention was recorded. Modeling policies deterministically can then lead to misspecification in the RL problem formulation. In the extreme case where data-collection and recording artefacts are modeled without any wiggle room to model the randomness, one runs the risk of blind imitation of such artefacts rather than learning clinically meaningful treatment policies. Most RL methods (policy learning or evaluation) in healthcare fall in the framework of off-policy RL. Off-policy learning refers to policy learning when observational data on patient progression and interventions are available from a clinician’s treatment path, called the behavior policy. The goal is to either learn a better policy or estimate the reward of a different policy on these patients. Off-policy means the algorithm is unable to interact with the environment (patients) to collect new samples. Here again one of the main statistical challenge is sample efficiency to learn a better policy. Along with modeling the transition dynamics, the inability to collect additional data (by actively intervening on patients) can be mitigated by framing off-policy learning as a causal (and therefore a probabilistic) estimation problem: “would the patient have survived had they not been treated?” This framing allows the RL algorithm to explore how outcomes could have changed under a slightly different set of interventions without actual experimentation. Another way to address this is if a similar patient happened to be in the dataset who was not treated and one could similarly estimate a good treatment policy based on the estimated efficacy of the treatment. Causal modeling make certain assumptions on the underlying probabilistic data-generating mechanisms, one could sample such counterfactual trajectories from the original observed data itself ( 135 ). Thus, with fewer effective number of samples, reliable stochastic policies can be learned with a probabilistic approach to RL. In healthcare, the benefits of causal probabilistic modeling have become a complementary toolbox that can be leveraged for training reliable policies. In many cases, clinicians’ interventions are done with more implicit information than available to an RL algorithm. This problem is known as partial observability and is yet another source of uncertainty in reliable policy learning. The conventional Markov Decision Process (MDP) framework can be augmented to handle partial observability, known as Partially Observable MDPs. The added modeling complexity now involves learning an observation function p ( o ∣ s ) , which characterizes likely observations o an RL algorithm perceives based on the potential states s of the patient. Confounding in observational health data is one such example source of partial observability ( 133 ). Consider a case where socio-economic status is unavailable to an RL agent to learn from but was used by clinicians, who may have used costlier treatments for wealthy patients. The goal is then to learn a policy when a behavior policy operated on more (and complete state information) compared to what is available to the RL algorithm. In this case, learning with partial observations involves estimating the posterior over unobserved states p ( s ∣ ⋅ ) ( 136 , 137 ). The benefits of such probabilistic inference have been demonstrated for off-policy evaluation in the presence of such confounding for treatment planning ( 137 ).