Mitigation of outcome conflation in predicting patient outcomes using electronic health records.

Objectives Artificial intelligence (AI) models utilizing electronic health record data for disease prediction can enhance risk stratification but may lack specificity, which is crucial for reducing the economic and psychological burdens associated with false positives. This study aims to evaluate the impact of confounders on the specificity of single-outcome prediction models and assess the effectiveness of a multi-class architecture in mitigating outcome conflation.

We evaluated a state-of-the-art model predicting pancreatic cancer from disease code sequences in an independent cohort of 2.3 million patients and compared this single-outcome model with a multi-class model designed to predict multiple cancer types simultaneously. Additionally, we conducted a clinical simulation experiment to investigate the impact of confounders on the specificity of single-outcome prediction models.

While we were able to independently validate the pancreatic cancer prediction model, we found that its prediction scores were also correlated with ovarian cancer, suggesting conflation of outcomes due to underlying confounders. Building on this observation, we demonstrate that the specificity of single-outcome prediction models is impaired by confounders using a clinical simulation experiment. Introducing a multi-class architecture improves specificity in predicting cancer types compared to the single-outcome model while preserving performance, mitigating the conflation of outcomes in both the real-world and simulated contexts.

Our results highlight the risk of outcome conflation in single-outcome AI prediction models and demonstrate the effectiveness of a multi-class approach in mitigating this issue.

The number of predicted outcomes needs to be carefully considered when employing AI disease risk prediction models.

electronic health records, artificial intelligence, disease prediction, confounding factors (epidemiology), pancreatic neoplasms

Electronic health records (EHR) are a valuable source of clinical data that can be leveraged for many applications, such as disease risk prediction,1,2 hospital readmission prediction,3,4 clinical decision support system,5 and drug safety surveillance.6 Artificial intelligence (AI) models have successfully utilized EHR data to forecast the risk of various diseases, including sepsis,5 neonatal complications,7,8 and cancer.1 The utilization of AI for disease risk prediction paves the way for more personalized healthcare by identifying individuals at high risk of developing specific diseases and facilitating more efficient allocation of healthcare resources.9 Despite these successes, few AI models have found their way into clinical practice,10 and the practical implications of false-positive predictions of potential population-wide deployment of AI-enabled disease screening programs have not been widely studied.11,12 Algorithms using readily available EHR data without the need for additional blood testing could be particularly useful for diseases with a low incidence and high mortality, such as certain types of cancer, as they would enable early detection or implementation of effective surveillance programs. However, many of these algorithms only quantify a patient’s risk for one specific cancer at a time, while in reality patients are simultaneously at risk for multiple cancers which may be co-located to the same body area or organs and may share similar non-specific symptoms like weight loss.13,14 Focusing on single-outcome predictions without accounting for confounders therefore may lead to misdiagnosis by reducing specificity. Additionally, the inherent challenge of not being able to identify all observed and unobserved confounders means that the real-world utility of these predictions remains uncertain. Clinical implementation of AI-enabled disease risk stratification algorithms requires careful analysis of the specificity of predictions due to the high potential costs of downstream interventions. If two diseases have similar features, a single-outcome model might incorrectly predict an individual to have disease 1 when in reality disease 2 is present. When translated into clinical practice, this leads to inappropriate subsequent workup for disease 1 which may be unnecessary and miss detection of disease 2 altogether. We hypothesized that AI models might be especially prone to such errors if the diseases share common risk factors, symptoms, or underlying biological processes. In the EHR, these diverse phenomena will be represented as a correlation of shared high-dimensional features with the outcomes. Such risk factors for both disease 1 and 2 can therefore act as confounders and might affect model specificity, in particular when they are not controlled for. We therefore speculated that conflation of outcomes might be present in AI models that use EHRs to make single-outcome predictions. Here, we evaluate a state-of-the-art model1 predicting pancreatic cancer from disease code sequences in clinical histories, show that its predictions are correlated with ovarian cancer leading to outcome conflation and confirm that this phenomenon is not limited to this particular model by replicating it in a simulated clinical dataset. Finally, we demonstrate that a simple modification in the form of multi-class prediction has the potential to address this issue, although further theoretical and empirical analysis is required to develop a generalizable solution.

Real world dataset: MarketScan database and cohort generation For the development of cancer risk prediction models, we used the MarketScan Research Databases 3.0 with de-identified healthcare claims records from over 250 million individuals between 2007 and 2021, encompassing medical, drug, and dental data, laboratory tests, outpatient encounters, and hospital admissions.15 These data are contributed from a diverse network of contributors, including large employers, managed care organizations, hospitals, and Medicare, ensuring a rich and varied dataset that reflects a broad spectrum of the patient population and healthcare services in the United States. Diagnoses within the MarketScan databases are coded according to the International Classification of Diseases (ICD), transitioning from ICD-9 to ICD-10 coding during the study period. This study was conducted using fully de-identified claims data and thus does not qualify as human subjects research and does not require IRB review. Due to the claims-based nature of the dataset, with patient identifiers tied to insurance coverage, these identifiers may undergo changes in response to alterations in employment status or insurance providers. Our cohort was therefore restricted to patients continuously enrolled for at least 132 months in the main MarketScan Databases or the Medicare Supplement to ensure sufficiently long disease trajectories (n = 2 298 013). After filtering out all patients without any diagnosis or born in 1990 or later, n = 2 290 730 patients were included. Additionally, patients which were continuously enrolled for at least 36 months until their death were added (n = 111 927). In MarketScan, death is only recorded if it occurs in hospital and before 2015 (after 2015, this feature was removed due to privacy concerns). For all resulting n = 2 402 657 patients [mean age at baseline 45.6 ± 12.8 (SD) years, 52.35% female], ICD9 and ICD10 diagnostic codes pertaining to inpatient and outpatient encounters were extracted and used for subsequent preprocessing. A flow chart depicting the cohort generation is shown in Figure S3. Real world dataset: use of ICD codes and SNOMED clinical terms As MarketScan utilizes ICD9 codes for all entries until the end of 2014, and almost exclusively ICD10 codes after 2015, we decided to map all ICD codes to the SNOMED CT vocabulary. For this, we first truncated all ICD codes (except ICD9 codes starting with E and V) after three characters (eg, ICD-10 code F31.9 to F31). ICD9 codes starting with E and V were truncated after four characters (eg, ICD-9 code V814.9 to V814). This preprocessing yielded 2853 distinct ICD codes which were subsequently mapped to 2184 unique standard SNOMED CT concepts. Real world dataset: identification of cancer patients in MarketScan, dataset cleanup and preprocessing In the MarketScan cohort of 2 402 657 patients, we first eliminated rare duplicate diagnostic codes recorded on identical dates. For labeling, the following definitions were applied: Pancreatic cancer patients were identified by at least one instance of SNOMED code 199754 (“Primary malignant neoplasm of pancreas,” equivalent to ICD10 C25 or ICD9 157). Ovarian cancer patients were identified by at least one instance of SNOMED code 200051 or 200052, where 200051 stands for “Primary malignant neoplasm of ovary” matching ICD10 C57 and ICD9 187 and 200052 (“Primary malignant neoplasm of uterine adnexa”) was added as it is also subsumed under ICD9 187. Subsequently, the following filters were applied: Non-cancer patients required a complete 5-year follow-up prior to the last entry (due to death or MarketScan data cutoff in 2021) to exclude potential undiagnosed cancer cases. Patients with fewer than 6 diagnostic entries prior to their endpoint (cancer diagnosis or, for non-cancer patients, death or data cutoff) were excluded. After filtering, the following datasets were used for model development and validation: | Preprocessed Dataset | Total number of patients | Patients with cancer diagnosis | Used for | |---|---|---|---| | PC | 2 271 037 | 7785 | CancerRiskNet | | OC | 2 269 659 | 9016 | CancerRiskNet (Flip Labels evaluation) | | PC and OC | 2 271 103 | 7752 (PC) 9005 (OC) | MultiRiskNet | Minor variations in the number of total patients are due to eg, pancreatic cancer patients being labeled as having no cancer in the ovarian cancer dataset and consequently potentially leading to exclusions under filters (i) or (ii). Small differences in the number of patients with pancreatic or ovarian cancer between the three preprocessed datasets are due to the extremely rare number of patients with both pancreatic and ovarian cancer, where patients are labeled based on the first listed cancer diagnosis in the dataset used for MultiRiskNet. The final dataset was randomly split into training (80%), development (10%), and test (10%) data, with each patient only belonging to one category. Real world dataset: model development and training For CancerRiskNet and MultiRiskNet model development, we used the training procedures described in the original publication,1 including data augmentation, as well as prediction endpoints function annotation indicating cancer occurrence at different timepoints (3, 6, 12, 36, and 60 months), and balanced sampling of patient trajectories in the training set such that each batch has an approximately equal number of positive (cancer) and negative (non-cancer) trajectories. We conducted an extensive random hyperparameter search. The top performing approach was selected on the basis of the mean AUPRC for the five prediction time points on the test set. Relative risk curves The RR of an AI model used for screening or diagnosis is the relative increase in risk of the outcome in the group with positive predictions from the model compared to the risk of the outcome in the group with positive predictions based on a random classifier. By this definition, the RR can be related to threshold-dependent metrics derived from a confusion matrix in the following way: (1) the probability of the outcome in the positive group from the AI model is equal to the positive predictive value (PPV) which is also known as precision; (2) the probability of the outcome in the positive group from the random classifier is equal to the prevalence of the outcome in the dataset because the probability of making a positive prediction for a random classifier is equal to prevalence. It then follows that the RR is a prevalence-normalized PPV, or RR = PPV/prevalence. To comprehensively analyze the predictive capabilities of an AI model, we plot the RR over a range of decision thresholds, yielding an RR curve. When evaluating the RR at a specific decision threshold to compare different models, we refer to this number as the RR score, eg, RR0.1% = PPV0.1%/prevalence with PPV0.1% denoting the PPV at decision threshold 0.1% (classifying 0.1% samples as positive). Clinical simulation experiment: dataset generation We generated a synthetic dataset comprising 250 000 entries to simulate a clinical scenario with longitudinal data (Figure S8). For each entry, three independent random variables A, B, and C (representing core risk factors) were drawn from a uniform distribution in [0,1]. For each entry, the number of timepoints was randomly chosen between 1 and Tmax. For each time point, nfeatures features associated with each core risk factor were synthesized by linearly combining the original risk factor value, weighted by a factor of 1-noise, with noise drawn from a uniform distribution in [0, 1] and weighted by noise. Additionally, 50—3 * nfeatures were randomly generated features with uniform distribution in [0,1] to mimic the complexity EHRs. In total, each entry therefore contained 50 features per time point. This method ensured that each set of features retained a link to its corresponding risk factor while introducing variability to simulate the complex and noisy nature of real-world clinical data. Risk factors A and B each contributed independently to the respective disease risks, while factor C served as a common risk factor influencing both outcomes. This was modeled as follows: For each entry, two composite risks Risk1 and Risk2 were defined by where α modulates the contribution of the common risk factor C. The final outcome labels were determined by applying thresholds to the calculated risks, introducing variability in the outcome distribution and allowing for the simulation of different disease prevalence rates. To define labels of the entries based on their risks, we defined two thresholds θ1 and θ2 and labeled entries with Risk1 > θ1 as disease 1, indicating the first hypothetical condition, and entries with Risk2 > θ2 as disease 2, indicating the second condition. Entries with both Risk1 > θ1 and Risk2 > θ2 were randomly assigned to either 1 or 2. This simplifying assumption was chosen in the context of our eventual comparison with CancerRiskNet where outcome prevalence of cancer is low and it is extremely uncommon for multiple primary cancers to simultaneously occur. All other entries were labeled with 0. θ1 and θ2 were set such that the prevalence of the outcomes 1 and 2 were 3% each. In the dataset reported in Figure 1, the following parameters were used: Tmax = 15, nfeatures = 3, noise = 0.4. The experiments were conducted 25 times each for α = 0.0, 0.1, 0.2, …, 0.9. Clinical simulation experiment: model architecture and training The transformer encoder was implemented using two layers of the TransformerEncoderLayer module of pytorch-2.0.0, each with 4 attention heads, a feedforward dimension of 32, and dropout of 0.2 for regularization. The final layer either consisted of a binary (SingleRN) or multi-class (MultiRN) classification layer. The training process involved splitting the dataset into training, validation, and test sets, with a 70:10:20 ratio. We utilized the Adam optimizer with cross-entropy loss, learning rate of 0.001, and weight decay of 0.001 for regularization. The models were trained with batch size of 512 for 20 epochs, and the loss was determined on the validation set after each epoch with early stopping to prevent overfitting. Model performance was assessed using AUROC and AUPRC. Additionally, we plotted Relative Risk (RR, see below for a detailed definition) curves to assess the prevalence-normalized precision of the machine learning model. This metric provided insight into the specificity of the predictions and the model’s ability to distinguish between outcomes that share similar risk factors. To further investigate the model’s ability to differentiate between correlated outcomes, we performed a secondary analysis by flipping the labels of the outcomes. This analysis assessed whether the features learned by the model for one outcome could inadvertently predict another, indicating a conflation of features between the two outcomes. Feature importance To identify the features utilized by the models in predicting their respective outcomes (PC and OC), we employed integrated gradients as the attribution method, following the approach described.1 As a minor modification, we applied L2 normalization per sample to ensure the feature importance scores were quantitatively comparable. Code availability The code used to create the datasets, train the models, and generate the figures of the clinical simulation experiment is available under https://github.com/momsenr/ClinicalSimulationExperiment. MultiRiskNet is available under https://github.com/momsenr/MultiRiskNet. Robustness of methods and reporting We adhered to the MINIMAR16 and MI-CLAIM17 standards to enhance the robustness and transparency of our methods. These standards ensure that our reporting includes the necessary information to understand our intended predictions, target populations, potential hidden biases, and how we developed these models.

CancerRiskNet is a state-of-the-art model that predicts pancreatic cancer (PC) risk at 3, 6, 12, 36, and 60 months into the future from a time series of ICD codes.1 We first replicated the original findings by utilizing the Merative Marketscan Research Databases,15 analyzing clinical histories from 2.3M patients between 2007 and 2021 (Table 1, Figure S1), which included 7786 pancreatic cancer cases (Figure 1A). As shorter prediction frames may introduce information leakage, for example, through codes recorded before a pancreatic cancer diagnosis that suggest it—such as neoplasm of the digestive tract—we primarily report results at 36 months, where such effects should be minimal. Our CancerRiskNet model, independently trained on the MarketScan dataset, achieved an AUROC of 0.879 at 36 months (Figure S2A). This result precisely matches the performance reported by the original study using the Danish National Patient Registry dataset,1 with performance at the other prediction intervals similarly aligning with the initial findings. To better compare the two models, we calculated RR curves, which quantify the improvement of a model over a random classifier over a range of decision thresholds and illustrate the prediction accuracy for different scenarios (eg, classifying the top 0.1% vs top 1% of model predictions as high risk), and the relative risk (RR) score, which evaluates the RR at a specific decision threshold. For the most at-risk 0.1% as defined by the model probabilities, our model trained on the MarketScan dataset achieved an RR score of 201.4 for 36 months, outperforming the original1 CancerRiskNet’s 104.7 (Figure S2A). While the original model was only used to predict PC, we hypothesized that the same model architecture could be used for prediction of other conditions and would therefore provide the ideal setting to analyze specificity of predictions. We therefore trained a model to predict ovarian cancer (OC) and were able to predict OC occurrence at 36 months with an AUROC = 0.809 and RR score of 102.6 (Figure S2B), yielding a modest reduction at this and the other prediction time points compared to the PC model. In summary, both independently trained single-outcome prediction models exhibited high performance in short- and long-term risk prediction for PC and OC. Table 1. | All (n = 2402431) | Pancreatic Cancer (n = 8055) | Ovarian Cancer (n = 10017) | | |---|---|---|---| | Demographics | ||| | Age, mean ± SD | 42.34 ± 9.86 | 48.68 ± 8.36 | 46.74 ± 8.78 | | Sex (% female) | 52.35 | 48.68 | 98.64 | | Comorbidities | ||| | Diabetes mellitus (Type 2) | 25.54% | 51.10% | 34.00% | | Obesity | 34.82% | 33.43% | 42.88% | | Hypertension | 59.03% | 80.78% | 68.76% | | Endometriosis | 3.17% | 2.15% | 13.60% | | Chronic kidney disease | 9.82% | 22.58% | 14.14% | | Non-malignant diseases of pancreas | 2.63% | 66.01% | 6.32% | Age is reported at the beginning of the observation period in 2007. Because PC and OC share risk factors18,19 and initial symptom presentation,14 we then considered whether the single-outcome prediction models appropriately distinguished these two cancers. Occurrence of PC and OC in the same patient is extremely rare, and AI models should ideally be able to accurately distinguish between PC and OC risk. In an optimal scenario, the model predictions for PC and OC risks should not show any correlation. However, when examining the 12-month predictions for PC and OC risk of the individually trained models, we observed a moderate correlation, with a Spearman coefficient of r = 0.61, indicating conflation of these diseases. This illustrates the danger of using single-outcome models for prediction in real-world scenarios where patients are always susceptible to many diseases and not just the single disease targeted by the model (Figure 1B). Conflation of these risks is particularly concerning due to its implications for subsequent diagnostic work-up and cancer surveillance, and could lead to false security or inappropriate allocation of healthcare resources. We hypothesized that adding a multiclass prediction head to the CancerRiskNet architecture would allow the model to learn the difference between the two diseases. This new model, called MultiRiskNet, simultaneously predicts PC and OC risk by employing a multi-class classifier as the last layer of the neural network (Figure 1A, right). Performance of MultiRiskNet to predict pancreatic cancer at 36 months was comparable to CancerRiskNet, with an AUROC = 0.880 and a RR score of 192.5 (Figure 1C, Figures S2A and S3A). The Spearman’s correlation coefficient of MultiRiskNet PC and OC predictions was r = −0.07, indicating that MultiRiskNet showed strong differentiation between cancer types (Figure 1D). To evaluate the model’s ability to predict OC when trained to predict PC, we also conducted a secondary analysis by flipping the labels (Figure S4). If the model’s performance on predicting OC in this situation exceeds that of a random classifier, this suggests conflation of PC and OC. To further quantify outcome conflation, we calculated relative risk (RR) ratio curves for both the model and the flip-label analysis (Figure 1E, Figure S4). MultiRiskNet showed a reduced rate of false-positive OC predictions, as indicated by the improved flip-label RR curves (Figure 1E). For example, for a 36 month prediction timeframe and decision threshold of 0.1% (corresponding to identifying 1000 patients as high-risk out of 1M), this translates to a 60.1% reduction of patients who develop OC but were falsely predicted to develop PC (Figure 1E). Similar improvements were obtained for the other prediction timeframes (Figure S5). Accordingly, MultiRiskNet showed a small improvement in True Positive Rate (TPR) and Positive Predictive Value (PPV) compared to CancerRiskNet, while yielding a similar True Negative Rate (TNR) and Negative Predictive Value (NPV) (Figure S6). While the performance advantage of MultiRiskNet over CancerRiskNet decreases with longer prediction timeframes (Figure S5), MultiRiskNet demonstrates a markedly lower rate of false positives even for longer timeframes. This robustness makes MultiRiskNet a preferable approach for early detection and screening applications, where reducing false positives is critical. As a control experiment, we trained a model to predict trigeminal neuralgia (TN), a disease unrelated to pancreatic cancer (PC) but with a similar incidence and age distribution (Figure S7A and B). The predictions of the PC and TN model showed only a very weak correlation (r = 0.16), suggesting that the observed correlation between PC and OC predictions in the single-outcome model is unlikely to be driven solely by general data patterns, such as age or incidence distributions (Figure S7C). To investigate the basis of the correlation between PC and OC predictions, we conducted a feature importance analysis and found that the strongest predictive features differed between the single-outcome PC and OC models (Figure S7D). This indicates that outcome conflation is not driven by a few dominant predictors but rather by the cumulative effect of many small contributing factors (Figure S7D). These findings highlight the importance of explicitly modeling multiple outcomes to mitigate outcome conflation and enhance the interpretability of predictions. To further analyze outcome conflation of single-outcome models, we devised a simulated clinical scenario where we could vary the confounding strength of a shared risk factor: We generated a synthetic dataset comprising 250 000 patients, each with a longitudinal timeline with consisting of 50 features per time point and two distinct outcome labels, representative of the mutually-exclusive presence of diseases 1 and 2 with a prevalence of 3% each (Figure 2A and Figure S8). The confounding strength of a shared risk factor α was varied to quantify the impact on the model’s predictive performance for disease 1 and disease 2. Similar to our cancer risk prediction model, we trained a transformer-based model to either predict each disease individually (single-outcome RiskNet, SingleRN) or both diseases jointly using a multi-class classifier (multiclass RiskNet, MultiRN, Figure 2A). Both SingleRN and MultiRN were able to predict the disease they were trained on with comparable area under the receiver operating characteristic (AUROC) curve for all correlation factors tested (Figure 2B). With increasing α, both models showed an inverted U-shape in area under the precision recall curve (AUPRC) and PPV (Figure 2C and Figure S9A-C), indicating that incorporating the additional information provided by the common risk factor C improves model performance for low and medium correlation of C with disease 1 and 2, but hurts model performance for high values of α due to conflation of disease 1 and 2. With increasing α, MultiRN was less susceptible than SingleRN of conflating the learned disease with the confounded disease as evidenced by the evaluation with flipped labels (Figure 2D). Additionally, MultiRN exhibited a marked enhancement in prediction specificity at higher decision thresholds, as evidenced by its reduced susceptibility to the confounders (Figure 2C and D, and Figure S9A), while yielding comparable TPR, TNR, and NPV (Figure S9A-C). Furthermore, the underlying representations learned by the MultiRN models were more consistent than the SingleRN models, notably at higher decision thresholds, reflected in narrower confidence intervals on the RR curves (Figure 2D). In conclusion, explicitly allowing a multi-class model to distinguish between diseases with shared risk factors improves the specificity of predictions, leads to more consistent models, and avoids conflation of outcomes.

Our external validation of a state-of-the-art model predicting PC achieved comparable performance to the original publication,1 but unexpectedly also was able to predict OC as evidenced by an undesirable correlation between PC and OC predictions. To mitigate the resulting conflation of PC and OC predictions, we introduced a multiclass architecture MultiRiskNet which showed improved prediction specificity without a drop in model performance. In particular, MultiRiskNet showed a lower rate of false positive predictions for OC when assessing PC risk. This improved differentiation is especially important for distinguishing between conditions with overlapping symptoms or risk factors and is crucial in clinical settings where the cost of false positives can be high, both in terms of patient experience and healthcare resource allocation. While multi-task learning has been widely employed to improve disease prediction,20,21 prior research has also shown that it can also inadvertently encode non-causal associations, particularly when tasks share overlapping patterns in the training data.22 Moreover, confounding due to target shift has been recognized as a challenge of conventional multi-task models, requiring adjustments to disentangle true disease relationships from spurious correlations introduced by the training data.23 Our findings reinforce these concerns, demonstrating that outcome conflation is not driven by a single dominant predictor but rather by the cumulative effect of multiple weakly contributing factors. Despite the improvements achieved with MultiRiskNet, it remains an open question whether a single multi-class prediction model is the optimal approach for cancer screening. Different types of cancer, while sharing certain similarities, also have distinct clinical features. An alternative strategy could involve using a general model to screen for “any cancer” and then applying type-specific models to determine the particular cancer type, balancing the trade-off between broader screening and disease-specific diagnostics. While we chose PC and OC as examples to illustrate the challenges associated with specificity and how they can be mitigated, we also observed a similar conflation of outcomes in a simulated clinical dataset with highly correlated features. Thus, these findings may extend to any single-outcome prediction task. Our findings emphasize the importance of considering false positive predictions in these scenarios and provide insights for the implementation of AI models for population-level disease screening strategies. Supplementary Material Acknowledgments Data for this project were accessed using the Stanford Center for Population Health Sciences (PHS) Data Core. The PHS Data Core is supported by a National Institutes of Health National Center for Advancing Translational Science Clinical and Translational Science Award (UL1TR003142) and from Internal Stanford funding. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. Certain data were supplied by Merative as part of one or more MarketScan Research Databases. Any analysis, interpretation, or conclusion based on these data is solely that of the authors and not Merative. Contributor Information S Momsen Reincke, Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University, Stanford, CA 94305, United States; Department of Pediatrics, Stanford University School of Medicine, Stanford, CA 94305, United States; Department of Biomedical Data Science, Stanford University, Stanford, CA 94305, United States. Camilo Espinosa, Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University, Stanford, CA 94305, United States; Department of Pediatrics, Stanford University School of Medicine, Stanford, CA 94305, United States; Department of Biomedical Data Science, Stanford University, Stanford, CA 94305, United States. Philip Chung, Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University, Stanford, CA 94305, United States. Tomin James, Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University, Stanford, CA 94305, United States; Department of Pediatrics, Stanford University School of Medicine, Stanford, CA 94305, United States; Department of Biomedical Data Science, Stanford University, Stanford, CA 94305, United States. Eloïse Berson, Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University, Stanford, CA 94305, United States; Department of Biomedical Data Science, Stanford University, Stanford, CA 94305, United States; Department of Pathology, Stanford University, Stanford, CA 94305, United States. Nima Aghaeepour, Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University, Stanford, CA 94305, United States; Department of Pediatrics, Stanford University School of Medicine, Stanford, CA 94305, United States; Department of Biomedical Data Science, Stanford University, Stanford, CA 94305, United States. Author contributions S. Momsen Reincke (Conceptualization, Investigation, Formal analysis, Software, Writing—original draft), Camilo Espinosa (Investigation, Writing—review & editing), Philip Chung (Investigation, Writing—review & editing), Tomin James (Investigation, Software, Writing—review & editing), Eloïse Berson (Investigation, Writing—review & editing), and Nima Aghaeepour (Conceptualization, Investigation, Formal analysis, Project administration, Supervision, Writing—review & editing) Supplementary material Supplementary material is available at Journal of the American Medical Informatics Association online. Funding This work was supported by the NIH [R35GM138353, RF1AG07744], Burroughs Wellcome Fund [1019816], the March of Dimes, the Robertson Foundation, the Alfred E. Mann Foundation, and the Bill and Melinda Gates Foundation [INV-037517]. Conflicts of interest None declared. Data availability The EHR data utilized in this study were obtained from Merative, a third-party provider of healthcare data. Due to the proprietary and sensitive nature of this data, it cannot be made publicly available. Access to the data is subject to institutional subscription agreements and compliance with data use agreements. The data used in the clinical simulation experiment are provided in the associated code repository https://github.com/momsenr/ClinicalSimulationExperiment.

- 1. Placido D, Yuan B, Hjaltelin JX, et al. A deep learning algorithm to predict risk of pancreatic cancer from disease trajectories. Nat Med. 2023;29:1113-1122. [DOI] [PMC free article] [PubMed] [Google Scholar] - 2. Tomašev N, Glorot X, Rae JW, et al. A clinically applicable approach to continuous prediction of future acute kidney injury. Nature. 2019;572:116-119. [DOI] [PMC free article] [PubMed] [Google Scholar] - 3. Jiang LY, Liu XC, Nejatian NP, et al. Health system-scale language models are all-purpose prediction engines. Nature. 2023;619:357-362. [DOI] [PMC free article] [PubMed] [Google Scholar] - 4. Rajkomar A, Oren E, Chen K, et al. Scalable and accurate deep learning with electronic health records. NPJ Digital Med. 2018;1:18. [DOI] [PMC free article] [PubMed] [Google Scholar] - 5. Adams R, Henry KE, Sridharan A, et al. Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis. Nat Med. 2022;28:1455-1460. [DOI] [PubMed] [Google Scholar] - 6. Davis SE, Zabotka L, Desai RJ, et al. Use of electronic health record data for drug safety signal identification: a scoping review. Drug Saf. 2023;46:725-742. [DOI] [PMC free article] [PubMed] [Google Scholar] - 7. De Francesco D, Reiss JD, Roger J, et al. Data-driven longitudinal characterization of neonatal health and morbidity. Sci Transl Med. 2023;15:eadc9854. [DOI] [PMC free article] [PubMed] [Google Scholar] - 8. Espinosa CA, Khan W, Khanam R, et al. Multiomic signals associated with maternal epidemiological factors contributing to preterm birth in low- and middle-income countries. Sci. Adv. 2023;9:eade7692. [DOI] [PMC free article] [PubMed] [Google Scholar] - 9. Rajpurkar P, Chen E, Banerjee O, Topol EJ. AI in health and medicine. Nat Med. 2022;28:31-38. [DOI] [PubMed] [Google Scholar] - 10. Kelly CJ, Karthikesalingam A, Suleyman M, Corrado G, King D. Key challenges for delivering clinical impact with artificial intelligence. BMC Med. 2019;17:195. [DOI] [PMC free article] [PubMed] [Google Scholar] - 11. Wong A, Otles E, Donnelly JP, et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern Med. 2021;181:1065-1070. [DOI] [PMC free article] [PubMed] [Google Scholar] - 12. Turnbull C, Wald N, Sullivan R, et al. GRAIL-Galleri: why the special treatment? Lancet. 2024;403:431-432. [DOI] [PubMed] [Google Scholar] - 13. Jung AW, Holm PC, Gaurav K, et al. Multi-cancer risk stratification based on national health data: a retrospective modelling and validation study. Lancet Digit Health. 2024;6:e396-e406. [DOI] [PubMed] [Google Scholar] - 14. Koo MM, Hamilton W, Walter FM, Rubin GP, Lyratzopoulos G. Symptom signatures and diagnostic timeliness in cancer patients: a review of current evidence. Neoplasia. 2018;20:165-174. [DOI] [PMC free article] [PubMed] [Google Scholar] - 15. Stanford Center for Population Health Sciences. MarketScan databases. Redivis. 2023. 10.57761/kg3j-nh50 [DOI] [Google Scholar] - 16. Hernandez-Boussard T, Bozkurt S, Ioannidis JPA, Shah NH. MINIMAR (MINimum Information for Medical AI Reporting): developing reporting standards for artificial intelligence in health care. J Am Med Inform Assoc. 2020;27:2011-2015. [DOI] [PMC free article] [PubMed] [Google Scholar] - 17. Norgeot B, Quer G, Beaulieu-Jones BK, et al. Minimum information about clinical artificial intelligence modeling: the MI-CLAIM checklist. Nat Med. 2020;26:1320-1324. [DOI] [PMC free article] [PubMed] [Google Scholar] - 18. Klein AP. Pancreatic cancer epidemiology: understanding the role of lifestyle and inherited risk factors. Nat Rev Gastroenterol Hepatol. 2021;18:493-502. [DOI] [PMC free article] [PubMed] [Google Scholar] - 19. Matulonis UA, Sood AK, Fallowfield L, et al. Ovarian cancer. Nat Rev Dis Primers. 2016;2:16061. [DOI] [PMC free article] [PubMed] [Google Scholar] - 20. Zou B, Lampos V, Cox I. Multi-task learning improves disease models from web search. In: Proceedings of the 2018 World Wide Web Conference on World Wide Web—WWW ’18. ACM Press; 2018, 87-96. 10.1145/3178876.3186050 [DOI] - 21. Wang X, Wang F, Hu J. A multi-task learning framework for joint disease risk prediction and comorbidity discovery. In: 2014 22nd International Conference on Pattern Recognition. IEEE, 2014, 220-225. 10.1109/ICPR.2014.47 [DOI] - 22. Hu Z, Zhao Z, Yi X, et al. Improving multi-task generalization via regularizing spurious correlation. In: Advances in Neural Information Processing Systems. Vol. 35. Curran Associates, Inc.; 2022:11450-11466. 10.48550/arxiv.2205.09797 [DOI] - 23. Makino T, Geras KJ, Cho K. Generative multitask learning mitigates target-causing confounding. In: Advances in Neural Information Processing Systems. Vol. 35. Curran Associates, Inc.; 2022:36546-36558. 10.48550/arxiv.2202.04136 [DOI] Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Supplementary Materials Data Availability Statement The EHR data utilized in this study were obtained from Merative, a third-party provider of healthcare data. Due to the proprietary and sensitive nature of this data, it cannot be made publicly available. Access to the data is subject to institutional subscription agreements and compliance with data use agreements. The data used in the clinical simulation experiment are provided in the associated code repository https://github.com/momsenr/ClinicalSimulationExperiment.