Comparing Massively-Multitask Regression Algorithms for Drug Discovery

Comparing Massively-Multitask Regression Algorithms for Drug Discovery | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Comparing Massively-Multitask Regression Algorithms for Drug Discovery Eric J Martin, Xiang-Wei Zhu, Patrick Riley, Steven Kearnes, Ekaterina A Sosnina, and 6 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7482715/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 05 Feb, 2026 Read the published version in Journal of Computer-Aided Molecular Design → Version 1 posted 11 You are reading this latest preprint version Abstract Massively-multitask regression models (MMRMs) have revolutionized activity prediction for drug discovery. MMRMs trained on millions of compounds and many thousands of assays can predict bioactivity with accuracy comparable to 4-concentration IC 50 experiments. This report compares six MMRMs: pQSAR, Alchemite, MT-DNN, MetaNN, Macau and IMC. Models were trained by experts in each method, on identical sets of 159 kinase and 4276 diverse ChEMBL assays, employing the same, realistically novel, training/test set splits. MMRMs performed much better than single-task random forest regression (ST-RFR) models for our use-case of imputing full bioactivity profiles for the very sparse compound collection on which the models were trained. Five MMRMs train all models simultaneously, so must leave out test-set measurements for all assays to avoid leakage (i.e. 25% of data). One method trains models one-at-a-time, and trains on all but the test data for that assay (< 1% of data). All algorithms were compared both using 75/25 splits, and when possible, 99+/<1 splits. Many evaluations achieved similar accuracy when tested on the same split. When evaluated on 75/25 splits, all MMRMs performed much worse than when evaluated on 99+/<1% splits. Thus, while many produce comparable high-accuracy final production models (trained on all the data), models that require 75/25 splits cannot evaluate the accuracy of those final models. While outstanding for imputations, MMRMs proved little better than ST-RFR for compounds very unlike the training collection. Thus, MMRMs are best for hit-finding, off-target, promiscuity, MoA, polypharmacology or drug-repurposing within the training collection. Besides accuracy, other pros and cons of each method are discussed. multitask regression imputation QSAR virtual screening algorithm comparison drug discovery Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 Figure 9 Introduction Quantitative structure-activity relationships (QSAR), invented by Hansch and Fujita in 1963, 2 , 3 was based on the earlier linear free energy relationships (LFER) of Hammet, first published in 1937. 4 Despite reporting very high correlations with experimental activity, the impact of Hansch analysis on drug discovery was minimal for several reasons: reported correlations were fits, not predictions; models were only useful for congeneric series; descriptors were substituent constants that required experimental determination; model applicability domains were too narrow for virtual screening, mechanism-of-action (MoA) determination, hit-list triaging or off-target prediction; models were simple linear, quadratic or bilinear least-squares regression; and too few observations for the number of descriptors, along with the lack of multiple-hypothesis adjustment when many descriptors were tested, led to overfitting. Even when held-out test sets were eventually employed, they were leave-one-out or random test sets that only tested a much narrower applicability domain than typical real-life use-cases. Over time, these problems were solved. Despite the many decades-long history of LFER and QSAR, the last obstacle to practical use, applicability domain, fell only recently. Evaluating model accuracy across a relevant applicability domain was addressed by designing held-out test sets wherein the test compounds are realistically novel with respect to the training set. Expanding the applicability domain was first addressed by proteochemometric modeling 5 , which included both protein and compound descriptors and trained a single large model across many protein targets. Because it trains on and predicts target-compound pairs, the number of compounds and measurements informing the models is much larger, greatly enhancing the applicability domain. It is limited, however, by not differentiating between different assays for the same protein target, which can vary widely for legitimate biological reasons. 6 It also cannot effectively include models for phenotypic screens, physicochemical properties, pharmacokinetic properties, or other assay types. Massively-multitask regression models (MMRMs) were subsequently developed. These algorithms train models collectively on up to many thousands of endpoints of any type. In drug discovery, they can combine models for biochemical, cellular, ADME (absorption, distribution, metabolism, excretion), physicochemical, binding or in vivo assays, greatly expanding the chemical matter and assay data informing all the models. They do not combine assays for the same target, but produce separate models for each assay, including separating activation vs. inhibition, yet all other assays for that target still inform the models with optimal scaling (along with all other assays in the multitask assay collection). In multitask models, each measurement and each prediction are for a compound-assay pair. The training matrix for multitask models covering thousands to millions of compounds across hundreds to thousands of assays is generally very sparse, sometimes as much as 99.9% missing values. An important application of MMRMs is “imputing” those missing values, i.e. predicting activities on the full set of assays for compounds already measured for a few of the assays and used in training the MMRM. This distinguishes them from single-task models, where predictions are always on a single assay for new compounds not used to train the model. Thus, in evaluating single-task models a compound in the training set is never in the test set. In evaluating multi-task models, where a compound is associated with many assays, compounds in the test set for one assay are frequently in the training sets for other assays. It was previously reported that the accuracy of compound-target pair predictions in multitask models is heavily influenced by whether the compounds were part of the training data in other assays (i.e. imputations). 7 The accuracy of predictions for entirely new compounds, known as “cold-start” predictions, can be significantly worse. One key objective of this study was to compare the accuracy of cold-start predictions to imputations, including the number of supporting assays, and how that affects the uses for which MMRMs are suited. The much higher accuracy and wider applicability domain of MMRMs than corresponding single-assay models greatly improves virtual screening (including virtual counter-screens for artifacts or selectivity). The near-quantitative predictions, rather than mere classification, allow sophisticated hit triaging based on predicted potency, ligand efficiency, lipophilic efficiency, selectivity, etc. Most importantly, having 1000s of semiquantitative models for each compound enables previously unavailable predictions, such as discovering mechanisms of action (MoA) for phenotypic screens, promiscuity, off-target activity to identify potential toxicities, identification of polypharmacology or drug repurposing. A recent study included comparison of several multitask kinase models. 8 It differs from this work in that it compares classification models, not regression models. Unlike this study, it uses random tests sets, assumes all assays for the same kinase are equivalent, does not distinguish cold-starts from imputations and does no significance testing. Another recent MMRM study trained on a large ChEMBL dataset and showed good performance on 6 datasets. 9 This article compares the MMRMs for 6 algorithms: Profile-QSAR (pQSAR), Alchemite, a meta learner (MetaNN), a multitask feed-forward neural network (MT-DNN), Bayesian matrix factorization (Macau) and inductive matrix completion (IMC). Each model uses the exact same data, including the same “realistic” training/test set splits, and each is trained by an expert, in many cases an author, of the algorithm. Brief description of MMRM algorithms: pQSAR: Profile-QSAR (pQSAR), first presented in 2006, was the first general MMRM developed for drug discovery. 10 At that time, pQSAR included models for only 115 Novartis biochemical and cellular kinase assays. 11 Furthermore, it was only tested on random held-out test sets, evaluating an unrealistically small applicability domain. By 2019 pQSAR had expanded to 11,805 Novartis biochemical, cellular and ADME assays covering all protein families, of which 8558 (72%) achieved average accuracy comparable to 4-concentration experimental IC 50 s using the “realistic” test set that evaluated a realistically broad applicability domain. Y-scrambling showed a chance-correlation rate of only 0.2% on the 11,805 Novartis assays, and 1.5% on the 4276 assays in this study—the higher rate for ChEMBL likely due to smaller assay sizes. 1 , 6 , 12 pQSAR also enables federated models, where innocuous partial models are shared among collaborating companies without exposing compounds, targets or bioactivities. With these, each partner can build pQSAR models for their internal assays virtually identical to the models they would build if all the data were actually shared. 13 Very briefly, the pQSAR algorithm is a multi-task, 2-level, stacked model as illustrated in Fig. 1 A. For level-1, conventional single-task random-forest regression (ST-RFR) models are built on the available experimental data for each individual assay, describing the compounds by Morgan 2 substructural fingerprints. For level-2, a separate PLS model is then built for each assay, one-at-a-time, now describing each compound not by its chemical structure, but by the profile of predicted (or experimental) bioactivities from all the other ST-RFR models besides the one being currently trained—hence the name “Profile-QSAR.” Alchemite: Alchemite is a commercial machine learning software tool originally developed for use in materials design and discovery 14 , 15 and more recently applied in catalyst 16 and drug 17 – 20 discovery among other areas. The general structure of Alchemite is described elsewhere, 21 but in brief, it is a multiple imputation approach, where machine learning models of each assay are trained using all other assay and chemical descriptor information; the values missing from each assay are imputed using the appropriate model, all simultaneously (as opposed to sequentially as in some other multiple imputation approaches); and the process is repeated, now utilizing imputed information for the input assays as well as experimental data where available (Fig. 1 B). Automated variable selection is carried out to ensure only those assays with high enough correlation and overlap of available data with the target assay are used as input variables at each stage. Typically, two or three cycles of imputation are carried out, as selected through cross-validation within the training set; in the models below two cycles were used. Beyond making predictions for individual compound-assay pairs, Alchemite also uses bootstrap samples of the training data to generate uncertainty estimates for its predictions. In industrial applications uncertainty quantification is vital to enable confident prioritization of compounds predicted to be performant against desired criteria: it is often a better use of limited resources to experimentally validate a suggestion that is predicted with high confidence to marginally achieve the target criteria rather than a suggestion that has a slightly improved expected performance but much increased uncertainty, as the former suggestion will have a higher overall probability of achieving success. Beyond making predictions with their corresponding uncertainties, the full Alchemite suite also includes packages for adaptive experimental design, interpretability, 22 and cold-start prediction, all within a no-code web-based graphical user interface. MetaNN: MetaNN was developed in 2020 and is inspired by the success of gradient-based meta-learning, which uses meta-knowledge learned from previous tasks to facilitate learning for new tasks. MetaNN is designed as a meta-learner that trains a deep neural network for each individual assay, initializing it from a well-generalized consensus DNN optimized across all assays. This algorithm creates task-specific neural networks while requiring minimal learning for each individual task, thus minimizing the risk of overfitting due to limited training data. Figure 1 C shows an overview of the workflow of gradient-based meta-learning algorithms for predicting bioactivity: we first train a neural network with weight initialization shared across all source assays; then for a source assay, the initialized neural network based on the bioactivity values of its training set compounds is established; then the adapted neural networks on the test set compounds of each source assay are evaluated, and the error rates of all source assays are averaged and backpropagated to improve the generalization ability of the initialization; finally the learned initialization is transferred to facilitate training for each target assay dataset. Meta-learning has been proven to be a powerful paradigm for transferring knowledge from previous tasks to facilitate learning for new tasks while reducing the risk of overfitting. The key is to optimize the generalization ability of the initialization, which is measured by the performance of the adapted model on each task's query set. Unfortunately, this generalization metric may cause the initialization to overfit on meta-training tasks, severely impairing the ability to generalize and adapt to new tasks. To develop more flexible and powerful algorithm, MetaNN uses task augmentation to increase the dependence of target predictions on the support set and provide additional knowledge to optimize model initialization. In the MetaNN algorithm, specifically, we use MetaMix, which linearly combines features and labels of samples from both the support and query sets and thus can actively use "more data" to augment meta-training tasks when evaluating generalization. MT-DNN: A Multitask Deep Neural Network (MT-DNN) is an extension of the traditional Deep Neural Network (DNN) architecture, designed to handle multiple learning tasks concurrently. While the conceptual foundation for neural networks was laid in the mid-20th century, significant advancements in computational power, algorithmic development, and data availability in recent decades have enabled the creation of more sophisticated systems like MT-DNNs. MT-DNNs stand out for their ability to process multiple learning tasks concurrently, utilizing shared representations to both generalize and specialize where necessary. This multitasking capability is especially beneficial in scenarios where individual tasks may have limited data, as the shared structure allows for knowledge transfer and regularization across tasks. Architecturally, MT-DNNs share the layered structure of DNNs, comprising input, hidden, and output layers. However, the feature of MT-DNN lies in the organization of these hidden layers: hidden layers can be divided into shared layers for extracting common features across tasks and task-specific layers for improving each task individually. Initially, the network leverages shared layers to learn general features relevant to all tasks. Then, task-specific layers focus on refining this knowledge to address the nuances of each distinct task. This layered organization is adaptable to align with the specific tasks, the nature of the data, and the targeted goals, allowing for a customizable architecture that may, for instance, vary the incorporation or extent of task-specific layers to optimize learning and outcomes. In this research, only shared layers were utilized, emphasizing the extraction and use of common features across multiple tasks without incorporating task-specific layers (Fig. 1 D). This nuanced architecture allows MT-DNNs to effectively process and learn from diverse datasets across multiple tasks, representing a significant evolution in the capabilities of neural network models. Macau: The Macau approach, introduced in 2015, 23 represents a significant advancement in matrix factorization techniques, specifically tailored for handling heterogeneous and sparse datasets. Demonstrating its efficacy in recommender systems, particularly with the MovieLens dataset, Macau has also been applied to more complex domains. Its application in biochemistry for tasks such as drug-protein activity prediction underscores its versatility beyond simple recommender system approaches. This versatility is further evidenced by its ability to handle heterogeneous data and incorporate side information across various fields, which is especially valuable in scientific and research-oriented applications. In 2017, 24 a substantial improvement to the Macau method was introduced, addressing computational limitations. At its core, Macau, illustrated in Fig. 1 E, utilizes an advanced Bayesian matrix factorization technique combined with integrating high-dimensional side information. This Bayesian probabilistic approach treats matrix factorization as a probabilistic model, where priors are assigned to the parameters. During factorization, the posterior distributions of these parameters are updated based on observed data. Macau utilizes Gibbs sampling, a Markov Chain Monte Carlo method, for sampling from these posterior distributions. It allows Macau to navigate the probable parameter space effectively, providing robust predictions while managing the uncertainties inherent in the data and model. A notable advancement in Macau's methodology, highlighted in the 2017 update, is the adoption of Krylov methods and a new prior for the link matrix, designed to scale with the magnitude of latent variables. These enhancements enabled Macau to boost its capability to analyze large and complex datasets significantly. Inductive Matrix Completion (IMC): Inductive Matrix Completion (IMC) is a paradigm often used as a benchmark or baseline when comparing different matrix factorization models. IMC revolutionized traditional matrix completion methods by incorporating auxiliary data (like compound and target features) into the process. Pioneered by Prateek Jain and Inderjit S. Dhillon in the work "Provable Inductive Matrix Completion", 25 IMC was developed to overcome the shortcomings of conventional matrix completion techniques. Since its introduction, IMC has inspired a wealth of subsequent studies across various domains, establishing it as a valuable methodology for leveraging side information in predictive modeling tasks where matrix completion is applicable. Beyond its initial use in recommender systems, IMC has found widespread application in various domains, including genomics for predicting gene-disease associations, text mining, and social network analysis. IMC algorithm involves two main steps: the factorization of the observed matrix into latent factors and the utilization of side information through matrix multiplication. Specifically, an observed matrix W can be approximated as the product of two low-rank matrices of latent features, U and V , further multiplied by additional matrices X and Y , which contain the side information. This process can be formalized as W ≈ X T UVY Figure 1. The optimization focuses on minimizing the discrepancy between the observed and predicted entries by iteratively refining the latent factors U and V , while keeping the side information constant. Methods ChEMBL assay collections: The models were compared on 2 previously published ChEMBL bioactivity data sets: a relatively homogeneous set of 159 kinase dose-response assays from ChEMBL version 20 downloaded on July 6, 2015, and a heterogeneous set of 4276 diverse dose-response assays from ChEMBL version 24.1 downloaded on June 22, 2018. The exact compounds, assays, experimental and pQSAR predicted bioactivities and training/test set splits were in the Supporting Information from 2 papers. 1 , 6 Training/test set splits: We use the “realistic” training/test-set split. It is compared to a random split in Fig. 2 Briefly, the compounds are first clustered. The models are trained on 75% of the data from the largest clusters and tested on the remaining singletons and small clusters. Training-set collections: Training-set collections are only a concern for multitask models. After the individual assays were each split into training and test sets, three training-set (and corresponding test-set) collections were assembled. These collections group together training and test sets for some of the individual assays into training collections and reserve the remaining test sets for the test-set collections. For models trained together, all test-set measurements for those assays must be held out to avoid test-set leakage. Note that the final production models for actual use are trained on 100% of the data, with no held-out measurements. Differences in training - set collections thus do not affect final model quality, only the evaluation of final model quality. The training-set collection affects how similar the evaluation models are to the final production models for an intended use case, here virtual screening of the compound archive used to train the overall multitask models (see below). The three training-set (and corresponding test-set) collections were named by how many assays’ test-sets must be held out when assembling the corresponding training-set collections for the multitask models. These are named “all-out”, “subset-out” and “one-out” as described below. Models and training set collections are similarly named by the test-set collections held out. Tables of colored measurements for each compound (rows) and assay (columns) in Fig. 3 illustrate the three training-set collections for assay A1, and any additional assays co-trained simultaneously with A1, together named “co-A1” assays (indicated with red column headers). Red markers (beneath red headers) are test-set measurements held-out during co-A1 training. Green markers (both light and dark) are co-A1 training measurements. Dark green distinguishes the 75% of “All-train” measurements, assembled from the combined realistic training sets for all individual assays, which are included in all training-set collections. Light green indicates “Some-train” measurements, from the test sets of “co-A1 support assays” (black column headers) not trained with A1, which thus can be included in the training of co-A1 assays. For each collection, red and light-green markers combined comprise the 25% of compounds from the combined realistic test-sets of all assays. Thus, looking left to right, red markers are replaced with light-green markers as fewer models are trained together. As the co-A1 set is thus reduced, ever fewer test-set measurements need be held out. All-out: Methods that train models for all assays simultaneously must leave out the 25% test-set bioactivities for all assays during training as shown in Fig. 3 A. In this case all assays are in the co-A1 set, and only the 75% of all-train measurements from combining all individual training-set assays (dark green) can be used for training, with no additional (light green) co-A1 support test-set measurements. The advantage of all-out models is that they include all assays while only training one MMRM regardless of algorithm, so all algorithms could train models for all assays, even in the large diverse assay collection. The limitation is that they are least like the “none-out” production models trained on 100% the data. Trained just on the large clusters for each assay, these all-train training-set collections contain the most redundant information and lack any of the more informative measurements from singletons and small clusters. They also contain the fewest imputations. Note that in Fig. 3 A, A1 predictions on C4 and C7 represent cold-start predictions that have no supporting green measurements in assays A2 … An. C5 predictions are imputations, supported by a (necessarily dark green) measurement from A3. One-out: Illustrated in Fig. 3 C, methods that train separate models for each assay one-at-a-time can leave out just the test-set data for that assay’s model, while training on both test- and training-set measurements from all other assays. I.e. the co-A1 assay-set consists only of A1 itself, so test-set measurements from assays A2 … An can all be included in training, as they will not leak into the A1 model. Since 99+% of the measurements are included in each one-out model’s training set, these models best reflect the final, production models trained on 100% of the data. Any method can train one-out models, but algorithms that train all the assays simultaneously must build 159 or 4276 separate MMRMs, leaving out just the test set for each assay in turn, and only testing predictions for that assay—a computationally onerous task. Thus, not all algorithms participated in one or both one-out comparisons. Note that for assay A1, test set compound C7 is still a “cold-start” prediction, but unlike the all-out case, C4 is now an imputation, because it is supported by a (light green) some-train measurement on A3. Having fewer cold-starts than the all-out models is presumably an important reason for the better performance of one-out models (see below). If there are 10 assays in the multitask model, one-out models are trained on ~ 90% of the data, 100 assays use 99%, 1000 assays 99.9%, the difference between final production and one-out decreasing as the number of assays in the multitask model increases. Subset-out: All methods could participate in a compromise comparison, the subset-out models, shown in Fig. 3 B. These used full-factorial design, based on assay properties, to sample an assay subset from each of the 2 assay collections. 48 assays were selected from the 4276 diverse assays using 4 properties: compound count ( 100), standard deviation of pIC 50 s ( = 1), pQSAR signed-r 2 (< 0.4, 0.4 < signed-r 2 0.6) and family (kinase, GPCR, phenotypic and other). 12 assays were similarly selected from the 159 kinase assays based on 3 properties: compound count ( 680), standard deviation of pIC 50 s ( = 1) and pQSAR signed-r 2 (< 0.5, 0.5 < signed-r 2 0.65). The assay nearest the centroid of each cell was selected for modeling. MMRM evaluation models were trained for just these subsets of 12 or 48 assays. Model training thus included only realistic training-set data (all-train dark-green markers) for these assays, but both realistic training-set data (dark-green markers) and realistic test-set data (some-train, light-green markers) for the remaining support assays, leaving out just the test-set bioactivities for these 12 or 48 assays (held-out, red markers). Because this was a single training, all methods could participate fully. Model quality, however, could only be evaluated for these 12 or 48 assay subsets. Subset-out models generally have more cold-start predictions than one-out models, but fewer than all-out models, although it depends on the particular subset. One study objective was to compare cold-start predictions to imputations. To guarantee a minimum of 10 cold-start predictions per assay, we selectively removed some bioactivity data from the training set collection of the 12 kinase subset assays. The following was applied for each of those 12 assays: Count the cold-start test-set compounds (n). If n<10, count the IC 50 s for each test-set compound in the subset-out kinase training-set collection. Select the 10-n compounds with the fewest subset-out training-set measurements. Remove those compound’s measurements from the subset-out training sets to ensure at least 10 cold-start predictions for each of the 12 assays. Table 1 and Table 2 show the distributions of compounds and measurements described in Fig. 3 . Note that the All-out and Subset-out columns represent single training-set collections, but One-out includes 159 or 4276 separate models and therefore separate training-set collections. The “number of training measurements” is the total count of light and dark green markers. “Unique training compounds” is the number of rows with any green markers. “Compounds exclusive to training set” is the number of rows with no red markers. These are the compounds that are not in the test set for any model. “Unique test compounds” counts rows with at least one red marker, i.e. for which predictions will be made. “Compounds exclusive to test” set have only red markers, i.e. cold-start compounds. “Number of predictions” is the total number of red points. Cold-start predictions is the total number of red markers in cold-start rows. For one-out, cold-start rows equal cold-start predictions, i.e. test-set compounds measured in only 1 assay. Table 1 Distribution of compounds and predictions in Kinase training-set collections (13,190 unique compounds; 159 assays) All-out Subset-out (12 assays) One-out Number of training measurements 85,675 108,675 114,146 (median) Unique training compounds 10,134 11,205 13,190 Compounds exclusive to training set 9,515 (94%) 10,705 (96%) 9,515 (72%) Unique test compounds 3,675 2,485 3,675 Compounds exclusive to test set (cold-start rows) 3,056 (83%) 1,985 (80%) 2,780 (76%) Number of predictions 28,642 3,427 28,642 Cold-start predictions 13,074 (46%) 2,046 (60%) 2,780 (10%) Table 2 Distribution of compounds and predictions in Diverse training-set collections (496,946 unique compounds; 4276 assays) All-out Subset-out (48 assays) One-out Number of training points 1,024,751 1,364,594 1,368,482 (median) Unique training compounds 409,253 494,425 496,946 Compounds exclusive to training set 319,262 (78%) 493,041 (99%) 319,262 (64%) Unique test compounds 177,684 3,905 177,684 Compounds exclusive to test set (cold-start) 87,693 (49%) 2,521 (65%) 58,514 (33%) Number of predictions 343,749 3,906 343,749 Cold-start predictions 146,405 (43%) 2,521 (65%) 58,514 (17%) Statistical analyses: We treat each test-set collection (all-out, subset-out, one-out, by diverse or kinase) as 6 separate statistical analyses. We follow the general recommendations of Demšar 26 and Benavoli et al. 27 For the analysis of signed-r 2 , we first use a Friedman test to assess whether there is any difference between the models’ performances. Friedman’s test is a non-parametric test that uses the pairing (i.e. that the models are tested on the same datasets). If the Friedman test suggests there is a difference (p < 0.05), we then do post-hoc tests of every model pair with a Wilcoxen signed-rank test. Details are in supplemental file tab_analysis1_full. We chose the Wilcoxen test because it again uses the pairing of the models (as opposed to just looking for a difference in mean signed r 2 ), and only assumes that differences in signed r 2 are ordinal. That is, the test assumes a difference of 0.2 is better than a difference of 0.1, but not exactly twice as good. Because there are many pair predictions, we do a multiple hypothesis testing correction on the p-values using the Simes-Hochberg 28 step up procedure with a target family-wise error rate (FWER) of 5%. This correction aims for at most 5% false positive rate (detecting a difference in models when in fact there is no difference). The analysis of the rate of “successful” models (signed-r 2 > 0.3) is similar. Each assay-model pair is assigned either a 1 or 0 for meeting the criterion. A Friedman test is followed by post-hoc pairwise tests, but here McNemar’s test is used on the 2x2 contingency tables. Detailed results are in supplemental files tab_analysis2_pvalues and tab_analysis2_tables. The same p-value correction is done with the Simes-Hochberg step up procedure. Regression algorithms: Single-task random forest regression: As a benchmark, ST-RFR models were trained for each assay using scikit-learn RandomForestRegressor (v0.20.2). Parameters were defaults except that the number of trees was set to 200. Compound descriptors are Rdkit (v2018.09.1.0) Morgan radius 2 substructure fingerprints of 1024 bits. Profile QSAR: The Profile-QSAR algorithm outlined above was used as previously described in detail, 1 except that experimental measurements were used where available, rather than always using random-forest regression predictions, as the input to level-2 PLS models (5% of the kinase IC 50 s and 0.01% of the diverse-assays IC 50 s). The original “Max2” variant employed a simple variable reduction where three level-2 PLS models were built: one using all the RFR models, one using only those whose predictions correlate with experiment at a threshold of r 2 > 0.05 and a third using only those with a correlation of r 2 > 0.2. The model that performed best on the test set was kept. Here, a consensus model is instead created from averaging the 3 predictions, rather than selecting just one. This avoids any bias due to selecting a model based on the test set, and it produced slightly fewer chance correlations. Alchemite: Alchemite was run using the 20211214 version, which was the production version when the analysis was carried out. Cross-validation to optimize the model hyperparameters was carried out using random 5-fold splits of the training data for the kinase subset-out dataset: the same hyperparameters were then used for all the models for consistency, although we note better results might be obtained by optimizing the hyperparameters separately for each assay collection and training set collection. Performance against all assays was weighted equally in the hyperparameter optimization, no weight was given to the quality of the uncertainty predictions, and Tree-structured Parzen Estimators 29 were used as the optimizer, as part of the overall Alchemite platform. MetaNN: The MetaNN model was previously described 3031 . A more specific description of the algorithm as applied in this work is in the supplemental materials. In this work, compound features were Morgan radius 2 substructure fingerprints of 1024 bits from RDkit v2018.03. The neural network for predicting pIC50 activity was a two-layer Multi-layer Perceptron (MLP) with 500 hidden neurons in each layer. Also, each fully connected layer was followed by a batch normalization layer and a non-linear activation of leakyReLU (negative slope is 0.01). We updated the meta-initialization in a batch-wise manner, with each batch consisting of 8 randomly selected source assays, i.e., N_s = 8. The learning rates to update the meta-initialization (i.e., α) and to learn assay-specific weights (i.e., µ) were 0.001 and 0.01, respectively. We set the Beta distribution to sample values of λ as Beta(0.5,0.5), i.e., α = β = 0.5. We iteratively ran 50 epochs to update the meta-initialization, with each epoch including 500 iterations, while we took only 5 gradient steps to quickly learn weights for the assay-specific neural networks. MT-DNN: We utilized a feed-forward deep neural network (DNN) using the PyTorch Lightning framework. 32 We employed only shared structure for the hidden layers, eliminating task-specific layers. The architecture concludes with distinct output layers for each task. To optimize prediction performance, we conducted a random search of hyperparameters 33 varying: i) the number and size of layers, ii) activation functions (ELU, PReLU, and LeakyReLU), and iii) optimizers (Adam, RAdam, and Yogi). Dropout regularization was applied across all hidden layers with dropout values ranging from 0.1 to 0.5. Model performance was assessed by averaging the results from three separate runs using identical hyperparameters. The Python code for the prediction algorithm is openly accessible on GitHub. 34 Compound features were 2048-bit Morgan 2 fingerprints generated with RDKit v.2021.03.1 for all models. The RAdam optimizer was used with PReLU for activation. Each model operated at a learning rate of 0.0001. The kinase models used an input layer of 2048, three hidden layers (768, 512, and 384 neurons respectively), and an output layer with 149 neurons. Dropout regularization rates for the hidden layers were 0.1, 0.4, and 0.2. The model was trained for 400 epochs. The diverse assays used an input layer with 2048 neurons, four hidden layers (768, 512, 384, and 256 neurons respectively), and an output layer with 4276 neurons. Dropout regularization rates for the hidden layers were 0.1, 0.4, 0.3, and 0.2. The training duration was 240 epochs. Macau: Macau was implemented within the open-source Bayesian Matrix and Tensor Factorization framework, SMURFF. 35 Our use of Macau followed the protocol and examples provided in its official documentation. 36 A pivotal part of employing Macau involved fine-tuning the hyperparameters, specifically adjusting the number of ‘latent dimensions’ and ‘samples to keep’. We determined the best models based on the consensus results from three runs, each using the same set of hyperparameters. This approach was applied to each assay collection and training-set collection, ensuring the identification of the most effective model configuration for each case. Although Macau facilitates the incorporation of features from both compounds and assays, this study exclusively utilized compound features. Compound features were 2048-bit Morgan 2 fingerprints generated with RDKit v.2021.03.1. The optimal hyperparameters for the kinase assays, the optimal number of latent dimensions and samples were [5, 720] for the all-out, [13, 600] for the subset-out, and [14, 680] for the one-out collection. For 4276 diverse assays, the best-performing number of latent dimensions and samples were [50, 150] for the all-out and [40, 730] for the subset-out collection. (Prediction for the diverse one-out collection was not performed due to high computation requirements). IMC: The IMC algorithm was implemented in Python through two primary steps: factorizing the observed matrix into latent factor matrices and utilizing side information via matrix multiplication. Given that only the characteristics of compounds were available, only compound side information, i.e. substructure fingerprints, was used. Therefore, the factorization of the original matrix W can be represented as W ≈ X T UV , where U and V are latent factor matrices, and X is the matrix containing compounds' side information. The optimization process focused on minimizing the discrepancy between the observed and predicted entries by iteratively refining the latent factors while keeping the side information constant. Compound features were 2048-bit Morgan 2 fingerprints generated with RDKit v.2021.03.1. The ranks of the latent matrices U and V were 100 and 200 for the kinase and diverse assay datasets, respectively. Compound descriptors: All methods except Alchemite described the compounds by Morgan 2 fingerprints. Alchemite used a collection of 330 calculated physicochemical properties from the StarDrop software. 37 Results and Discussion This study was designed to examine several key questions about MMRMs: for what use cases do MMRMs have an advantage over much simpler single-task models, how large is that advantage, how to evaluate the accuracy of MMRMs, are some MMRM algorithms more accurate than others, and what are other pros and cons of the different MMRM algorithms? Use-cases for MMRMs: Assessing the utility of any modeling method is inextricably tied to how the models will be used. MMRMs have 2 important characteristics that both highlight and limit their unique applications in drug design: the ability to make near-quantitative IC 50 predictions for thousands of assays for each of millions of compounds, and the limitation that this impressive accuracy on compounds unlike a given assays training set only applies for imputations, not for cold-start predictions on very novel compounds (see below). This points MMRMs toward on- or off-target virtual screens of the compound collection on which the MMRMs were trained, where many or most of the predictions are imputations, or at least near-neighbors of compounds that are imputations. Despite this limitation, having high-quality predictions for 100s to 1000s of assays for a large compound collection still addresses many important drug-discovery problems. Highly accurate on-target virtual screens allow experimental screening and model training on a modest fraction of the collection, followed by a virtual screen of the much larger remainder. More importantly, off-target virtual screens of the entire bioactivity profile allow prediction of potential off-target toxicities, mechanisms-of-action, artifactual hits, changes in mechanism, polypharmacologies or drug repurposing for a compound, or better, a medchem series of related compounds. Such information is difficult to obtain by other means, computational or experimental. MMRM’s semi-quantitative IC 50 predictions, rather than mere binary categories, means they also excel at triaging virtual or experimental screening hits for advancement to hit exploration or lead optimization using predictions of e.g. lipid efficiency, promiscuity and isoform selectivity. However, while the applicability domains are far larger than single-task models, covering the thousands to millions of compounds used to train the entire MMRM, predictions are still most effective for compounds in, or similar to, those in the data sources on which the MMRMs were trained. Thus, in lead optimization, while they excel for MoA or off-target activity prediction of close analogs, for on-target activity prediction they are recommended mainly for project-focused virtual libraries that stick close to the chemical matter used to train the MMRM. For chemical matter outside that chemical space, any small improvement over single-task models does not justify the much higher computational expense. Training/test splits: There are many ways to split training and test sets, and the choice must reflect the use-case. For the use-case of on- or off-target virtual screening of compounds in a corporate archive, the predictions will be for diverse historical compounds from the data source on which the MMRM was trained. A compound the project team selects for experimental testing from a virtual screen of the archive will typically be unlike the already known actives. However, it, or a near analog, will likely have some other historical activity data, i.e. an imputation (or near imputation). For hit-list triaging, off-target, polypharmacology, MoA or drug repurposing prediction, the models will likewise have been trained on compounds with historical data, but structurally unlike those tested in the corresponding project’s on-target assay. We therefore use the “realistic test-set”, which was designed specifically to test performance on compounds from the overall MMRM training set, but very unlike those in the assay being evaluated (see Methods). Additionally, cold-start predictions from an assay’s realistic test set will be on held-out compounds unlike anything in the assay being studied, and also not in the rest of the database, and thus will be our best facsimile of the performance on compounds from other external sources like vendor collections or exploratory generative chemistry. Time-gated splits are a common approach to mimic cold-start, on-target, activity prediction on synthesis candidates in lead optimization, a typical use-case for single-task models. This is not one of the important applications of MMRMs as noted above, and time-gating would be unrelated to our use-cases. Random test sets are much too similar to the training set for any drug design use-case. Scaffold splits are closer to our case, but also generally more similar than the real use cases, since small changes in the core can define a new scaffold. Accuracy comparison between algorithms and training-set collections: The signed square of Pearson’s correlation between prediction and experiment (signed-r 2 = r * |r|) was chosen as the primary model-accuracy figure of merit for several reasons. For many assays, most compounds are inactive, so merely guessing low activity for every compound gives a good MAE or RMSD, whereas signed-r 2 is sensitive to pulling the handful of precious needles out of the haystack of weak activity. Correlation coefficient was chosen over coefficient of determination, which combines correlation with correct absolute value, because assays vary in scale and coefficient of determination can be very sensitive to the absolute range in the test set. The range of activities for different assays can vary greatly even for the same target. For example, the sensitivity of ATP competitive kinase inhibitors will change if assays differ in protein construct, partner proteins in a larger complex, peptide substrate, degree of phosphorylation, cofactor concentrations, etc. Furthermore, besides EC 50 or Kd, some assays report other quantitative endpoints such as IC 90 , MIC, half-life, reaction rate or physicochemical or ADME endpoints. For comparisons across assays, such as identifying PK risks, off-target activities, promiscuity or MoA, it is thus most important to identify the predicted activities unusually high for that particular assay. I.e. correlation is most relevant, rather than getting the correct absolute value. In fact, for off-target, polypharmacology, ADME or MoA prediction, we generally Z-scale the predictions from each model, thus normalizing the predictions for each individual assay. Here, we compare algorithms by 2 metrics: signed-r 2 , and the fraction of models that achieve a heuristic criterion for virtual-screening success of signed-r 2 > 0.3. Mean absolute error (MAE) was used in one analysis, comparing cold-start predictions to imputations, where point-prediction error was aggregated across many assays, bearing in mind this source of incommensurability. The results are summarized in 2 ways: Table 3 gives the median signed-r 2 , and Table 4 the count of assays with signed-r 2 > 0.3 for each algorithm, for each training-set collection for the kinase and diverse assay collections. More precise values are in supplemental files tab_median_signed_r2.csv and tab_signed_r2_good_enough.csv. The corresponding full rank-order distribution plots in Figure S1 are similar in shape, indicating these 2 metrics summarize the results well. Figure 4 and Fig. 5 show the results of statistical significance analyses comparing pairs of models for signed-r 2 and count (signed-r 2 > 0.3) as described in the Methods. The 5 “advanced” methods generally perform better than the baseline single-task ST-RFR and baseline IMC method across the full range of tests. MT-DNN, pQSAR and Alchemite were roughly similar. MetaNN was slightly below those 3 on most tests. Macau’s performance was mixed, roughly comparable to the other 4 on the kinase assays, but substantially worse on the diverse assays. Note that the number of assays varies from 12 to 4276 between test-set collections. With a large number of models, even very small differences can be statistically significant, such as those seen in the diverse all-out case. The does not mean the differences are large enough to matter in practice. Four of the 5 advanced MMRM algorithms contributed kinase one-out models: pQSAR, MT-DNN, metaNN, Macau. One-out models are most like the final none-out models that will be used in production. As Table 3 shows, the IMC MMRM baseline model at median signed-r 2 = 0.26, was still substantially better than ST-RFR single-task baseline at 0.09. The 4 advanced MMRMs were highly successful. The order of performance was MT-DNN = pQSAR > metaNN > Macau > > IMC > > ST-RFR. MT-DNN and pQSAR had median signed-r 2 = 0.59 and 0.58 respectively. MetaNN was close behind at 0.56 and then Macau at 0.53, although Fig. 4 shows that with 159 assays those differences were statistically significant. As Table 4 shows, the top 4 algorithms gave successful models (signed-r 2 > 0.3) on at least 90% of the 159 assays, and as Fig. 5 shows, were statistically equivalent by that measure. By contrast, ST-RFR achieved only 9% successful models with median signed-r 2 = 0.09 and IMC achieved 38% successful models with median signed-r 2 = 0.26. All methods contributed kinase subset-out models. The results were similar to one-out: all 5 advanced MMRMs achieved 9 of 12 successful models, with median signed-r 2 between 0.51 and 0.55. The order of statistical significance was pQSAR = MT-DNN = Alchemite = metaNN = Macau > > IMC > ST-RFR. For the kinase all-out models, the signed-r 2 order of significance was pQSAR = MT-DNN = Alchemite > Macau = metaNN > > ST-RFR > IMC. However, median signed-r 2 was less than 0.2 for all algorithms. and no method achieved more than 20% successful models. All-out models were, thus, not generally able to identify the assays for which the final 100% virtual screening models would be useful on these test-sets. This casts suspicion on interpreting any performance differences. Only pQSAR contributed a one-out MMRM for the diverse assay collection. Median signed-r 2 was 0.41 and 2500 of 4276 assays exceeded signed-r 2 > 0.3, compared to 0.16 and 1300 respectively for ST-RFR. While less dramatic than the 159 kinases, this is a substantial improvement. The decrease likely reflects better transfer of learning among homogeneous kinase assays. The subset of 48 diverse models showed substantial variation between MMRM algorithms. Median signed-r 2 varied between 0.19 and 0.44 for the 5 advanced models, with significance order pQSAR = DNN = ST-RFR = Alchemite ≥ metaNN = Macau > > IMC. Similarly, the proportion of successful models varied from 15% for IMC to 73% for pQSAR. The significance order was pQSAR = DNN = ST-RFR = Alchemite ≥ metaNN = Macau ≥ IMC. The surprising strong performance of ST-RFR on the diverse subset-out models deserves more scrutiny. The 48 assays were selected by full-factorial design to sample a wide but realistic range of 4 assay properties: compound count, dynamic range, pQSAR signed-r 2 and target family (see methods). Unfortunately, the experimental design appears to have selected a very biased sampling of the 4276 assays. Both pQSAR and ST-RFR did better on subset-out than higher-fidelity diverse one-out, but that difference was much greater for the ST-RFR. As Table 3 shows, the median ST-RFR signed-r 2 is 0.36 for the 48 subset models vs. 0.15 for the full set of 4276 one-out models, and 58% of the 48 ST-RFR models achieve signed-r 2 > 0.3 vs. only 30% for the full set of one-out models. For pQSAR, subset-out was much closer to one-out: 0.44 vs. 0.38 and 73% vs. 58%. This suggests that the 48 selected assay were not a representative sampling of the 4276. Thus, the strong ST-RFR subset-out performance appears to be an artifact of the specific selected subset, and it might likely not outperform the final none-out metaNN models, though might still outperform Macau and IMC. The diverse all-out signed-r 2 order was MT-DNN > pQSAR > Alchemite > ST-RFR > MetaNN > Macau > IMC. The order for count of successful models was only slightly different, switching ST-RFR and MetaNN, with DNN > pQSAR > Alchemite > MetaNN = ST-RFR > Macau > > IMC. With so many assays, virtually all differences are statistically significant. Again, since the performance of all-out MMRMs for each algorithm is much worse than subset-out or one-out, with median signed-r 2 < 0.25 and success count < 41%, it is hard to interpret the all-out order. Despite those caveats, the poor performance of Macau on the all-out and subset-out assay collections does stand out. Table 3 Median Signed-r 2 Test Set \ Method PQSAR DNN Alchemite MetaNN Macau IMC ST-RFR Kinase One-Out 0.58 0.59 0.56 0.53 0.26 0.09 Kinase Subset-Out 0.51 0.55 0.55 0.51 0.51 0.29 0.08 Kinase All-Out 0.17 0.17 0.15 0.12 0.12 0.05 0.09 Diverse One-Out 0.38 0.15 Diverse Subset-Out 0.44 0.39 0.36 0.26 0.19 0.02 0.36 Diverse All-Out 0.20 0.23 0.18 0.16 0.10 0.02 0.15 Table 4 Percentage of Models with Signed-r 2 > 0.3 Test Set \ Method PQSAR DNN Alchemite MetaNN Macau IMC ST-RFR Kinase One-Out 91% 94% 92% 90% 38% 9% Kinase Subset-Out 75% 75% 75% 75% 75% 33% 17% Kinase All-Out 20% 13% 16% 7% 11% 0% 9% Diverse One-Out 58% 30% Diverse Subset-Out 73% 65% 60% 48% 31% 15% 58% Diverse All-Out 36% 41% 34% 30% 21% 9% 30% Although the one-out multitask models for each algorithm greatly outperform the all-out models, one might hope that the quality of all-out models might correlate with one-out models, so they could at least be used to predict the relative performance of the final none-out models and to compare the performance of the algorithms. Figure 6 plots the signed-r 2 for models trained with all-out vs. one-out collections for the 159 kinase assays for the 5 MMRMs that contributed kinase one-out models. Figure 7 is similar for pQSAR models on the 4276 diverse assays. Unfortunately, the low correlation coefficients and triangular scatter plots indicate that the order of all-out models is not even a relative indicator of one-out, and by extension of none-out, model quality for our use case of measurements on novel members of the compounds used to train the multitask models. This confirms the suspicion that differences among the all-out MMRMs does not reflect the relative performance for the final none-out models. Subset-out MMRM compromise for model evaluation: Presumably due to the limited, redundant training data and excess of cold-start predictions, all-out models do not reflect well the quality of the final none-out production models for our use case of measurements of novel compounds used in training the multitask model. However, for the assays tested, especially the kinases, the subset-out models are close to the one-out models. This suggests that while training a single all-out MMRM will not indicate the expected performance of the final production models—and training 1000s of separate multitask models is impracticable—a compromise of training multiple, separate multitask subset models could be a reasonable compromise for feasible computation with useful evaluation of final model accuracy for these algorithms. By-assay comparisons: Table 5 summarizes the signed-r 2 correlations across assays between each pair of algorithms for the six training-set collections. Not only were the overall average results similar between several methods, but except for the 2 all-out collections, which did not produce many useful models, which assays produced good or poor models was also very similar between the 5 more-advanced MMRM algorithms. For visual comparison, corresponding trellis plots are presented in Figures S2 -S7. This further supports the hypothesis that these very different algorithms are fundamentally comparable, capturing essentially the same information, and suggests that the quality of the predictions from the MMRMs is limited by the assay data more than by differences between the algorithms. Table 5. Correlations of r 2 across assays between each pair of algorithms for each training set. Imputation vs. cold-start prediction: Previous work has shown that MT-DNN MMRM performance on these assay collections is much better for imputations than for cold-start predictions. 7 We studied this further to understand if this is true for all algorithms, how improvement depends on the number of supporting measurements from other assays, how this impacts performance between training-set collections, and how this affects the cases for which multitask models are helpful in drug discovery practice. Table 6 shows, for each algorithm and training-set collection, the median signed-r 2 and percentage of models with Signed-r 2 > 0.3 for cold-start predictions and imputations. The full rank-order distribution plots in Figures S8 a-f are similar, indicating these 2 metrics summarize the results well. Only a fraction of models in each training-set collection could be analyzed, because for inclusion of a model we required a minimum number of both cold-start and imputation predictions to compute a meaningful signed-r 2 . For kinase all-out, diverse all-out and diverse one-out, the minimum was 8, which still allowed at least 150 assays each. Subset-out training-set collections were small to start with, and with so many supporting assays in the one-out models, their number of cold-start predictions was small. Thus, for diverse subset-out, kinase subset-out and kinase one-out the minimum was reduced to 5 cold-start and 5 imputation predictions. Even so, these training-set collections could only compare from 5 to 9 assays. Table 6 Table of median signed-r 2 and percentage of models with Signed-r 2 > 0.3 of cold-start predictions and imputations for each method for kinase and diverse assay sets also including number of assays for each case. Median signed-r2 test-set collection cold/imp. PQSAR MT-DNN Alchemite MetaNN Macau IMC ST-RFR Assay # Kinase Subset-Out cold 0.06 0.04 0.14 0.09 0.12 0.05 0.02 10 Kinase One-Out cold 0.01 0.00 0.01 0.03 0.00 0.01 6 Kinase All-Out cold 0.07 0.01 0.07 0.03 0.01 0.02 0.07 155 Diverse Subset-Out cold 0.26 0.35 0.06 0.15 0.16 0.10 0.37 6 Diverse One-Out cold 0.08 0.06 245 Diverse All-Out cold 0.05 0.04 0.05 0.03 0.02 0.00 0.06 986 Kinase Subset-Out imp. 0.57 0.60 0.58 0.55 0.56 0.32 0.11 10 Kinase One-Out imp. 0.33 0.18 0.18 0.23 0.00 0.18 6 Kinase All-Out imp. 0.24 0.33 0.24 0.27 0.28 0.07 0.13 155 Diverse Subset-Out imp. 0.77 0.78 0.75 0.45 0.57 0.00 0.28 6 Diverse One-Out imp. 0.22 0.11 245 Diverse All-Out imp. 0.28 0.33 0.26 0.20 0.21 0.00 0.15 986 % Signed-r2 > 0.3 test-set collection cold/imp. PQSAR MT-DNN Alchemite MetaNN Macau IMC ST-RFR Assay # Kinase Subset-Out cold 20% 10% 10% 30% 20% 30% 0% 10 Kinase One-Out cold 0% 0% 17% 17% 0% 17% 6 Kinase All-Out cold 7% 1% 5% 1% 1% 1% 6% 155 Diverse Subset-Out cold 33% 50% 33% 33% 17% 17% 50% 6 Diverse One-Out cold 20% 23% 245 Diverse All-Out cold 18% 17% 20% 12% 9% 5% 18% 986 Kinase Subset-Out imp. 80% 80% 80% 80% 80% 50% 10% 10 Kinase One-Out imp. 50% 50% 50% 50% 17% 33% 6 Kinase All-Out imp. 39% 57% 40% 44% 41% 2% 15% 155 Diverse Subset-Out imp. 100% 100% 83% 67% 83% 17% 50% 6 Diverse One-Out imp. 45% 27% 245 Diverse All-Out imp. 47% 53% 45% 39% 40% 7% 28% 986 Table 6 shows that cold-start MMRM predictions are no better than the corresponding ST-RFRs, although the small number of assays for Diverse Subset-Out, Kinase Subset-Out, and Kinase One-Out makes these comparisons uncertain. Imputations, on the other hand, have substantial improvements. Even the all-out imputations show improvement over ST-RFR. These results suggest that one reason for the improvement seen for one-out over all-out models might be that one-out models have mostly imputations rather than cold-starts. Note that while the cold-start/imputation distinction does not apply to ST-RFR models, for a proper benchmark, the results must still be compared to the corresponding ST-RFR models for the same subset of assays. Note that this analysis was tested on the realistic test set of structures maximally unlike the training set, corresponding to the extremes of each model’s applicability domain. Although activity prediction for compounds similar to those in a model’s training set was not tested in this study, cold-start predictions for compounds with near-neighbors that are imputations, such as analogs in lead optimization, might still be better than single-task. Since the cold-start test compounds were unlike the training set of the model under evaluation, and also are not in the training sets for any other assay, they are our closest simulation of virtual screens against external compound sources like vendor databases or exploratory generative chemistry. In these use cases we thus conclude that multitask models will have little advantage over single-task models, while costing much more. This result also suggests that models should be updated frequently, e.g. monthly or weekly rather than yearly, so off-target predictions on current compounds and series will at least include supporting measurements from assays near the top of the project testing funnel, and thus will be imputations or at least near neighbors of imputations. Beyond the binary distinction of cold-start vs. imputation, Fig. 8 shows the impact of only testing compounds with ever more supporting measurements. Two plots are paired for each test-set/assay collection. The line plots show the average per-prediction difference in absolute errors between each MMRM and the corresponding baseline ST-RFR, aggregated across all assays, binned by the minimum number of supporting experimental measurements required for inclusion. At the bottom is a histogram of the count of predictions contributing to that bin. Error bands are +/- 1.96 * SE. The first bin is unique, representing cold-start predictions (exactly zero supporting measurements). The subsequent bins indicate imputations with at least the listed number of supporting measurements. I.e. the ≥ 1 bin includes all imputations, ≥ 2 is all except those with just 1 supporting assay, etc. Note that the previous Table 6 analysis of signed-r 2 was based on commensurate predictions from individual models, and each model contributes equal weight. Figure 9 plots are aggregated across diverse models for all predictions with the same range of supporting measurements, so assays with more measurements contribute more. Also note that bin ranges on the X-axis get progressively larger going right, so the downward curve is influenced by the increasingly wider bins. The first 5 points increment by a single additional assay, then by 5, 15, 30 and 50. If all bins increased by just 1 assay, the plots would be more linear, but would go way off the page to the right. The multi-task predictions get better when requiring all imputations have successively more supporting measurements. Conversely, looking right to left, performance gets worse as predictions with fewer supporting measurements are included. Aggregating prediction error across all assays is admittedly a crude analysis, and using mean-absolute prediction error rather than signed-r 2 can be dominated by minimally helpful correct predictions for large numbers of inactive compounds. Still, the trend shows that a single supporting measurement helps, and that accuracy continues to improve when more supporting measurements are required for inclusion. The upward curve at the right of the kinase one-out and subset-out plots might indicate artifactual frequent hitters that are followed up in many assays, but by non-specific mechanisms that thus are not well predicted. In summary, for compounds unlike those already tested in a given assay, MMRM cold-start predictions are little better than ST-RFR. For imputations, predictions were better than ST-RFR for compounds tested with just a few supporting assays. The predictions improve as the number of supporting measurements increases until the improvements are substantial. Since MMRMs are expensive, this further supports the high value of employing MMRMs for virtual on- and off-target screens of the compound collection on which the MMRMs were trained, but discourages their use for screens on other compound collections where predictions will be cold starts on novel compounds and cost/benefit will be poor. It also indicates that the improved one-out and subset-out predictions result partly from capturing more supporting training measurements, many of which are missed by the all-out model training sets. Model quality and decisions: Understanding the quality of a given model is important for making decisions about how it should best be used. For example, if undertaking a virtual screen using a fairly low-quality model with signed-r 2 = 0.35, where the false positive rate will be high, one might first order and test 10,000 + compounds at a single concentration, then order dose-response plates to determine IC 50 s just for the single-point hits. A high-quality model with signed-r 2 = 0.6 should be at least as good as a single-concentration percent inhibition. One might, therefore, just order several hundred compounds at 10 concentrations for immediate IC 50 determination, saving maybe 3 months. Similarly, if an off-target or potential MoA activity is predicted for an assay that is currently running in the company, there is very little cost to add your compound to the queue even if the model only has signed-r 2 = 0.3. If the assay is only running at a CRO, the threshold is somewhat higher. If the assay must be restarted internally, the threshold is higher yet. Consensus affects decisions. If most members of a compound series are predicted to hit a given assay, that consensus strengthens the prediction even if signed-r 2 for the model is low. Similarly, if a compound is predicted to hit multiple assays for the same target, which were run at different times on different compound sets, that strengthens the prediction of even marginal models. If a compound from a luciferase assay is predicted to also hit dozens of other unrelated luciferase assays, odds are it is an artifact even if models are good. If a compound is predicted to hit several cytotoxicity assays, that antimicrobial phenotypic virtual screening result is probably not specific. Similar considerations apply to hit-list triaging based on ligand- or lipid-efficiency, isoform selectivity, virtual counter-screens etc. Compound cost also greatly affects the model quality required. A compound that must be repurchased, or worse resynthesized, requires a higher prediction confidence than one pulled from the company archive. Predictions for project-focused library synthesis require a lower threshold than many one-off syntheses. If a novel compound from generative chemistry requires a new and difficult synthesis, the model should be very good indeed to justify the effort. Because it affects important decisions, it is important to have good estimates of final model quality. Statistical significance vs. meaningful difference: These were large datasets. Given enough data, almost any minor systematic accuracy difference between algorithms will be statistically higher than chance. A more important question is whether a difference has practical consequences for your use-cases. What constitutes a meaningful accuracy difference depends on context. A slightly more accurate model for a virtual screen of your current projects primary assay might be important. By contrast, if the number of potential off-target assays that reach signed-r 2 > 0.3 is 5% higher with one algorithm, is that enough reason to select that method over another? That minor distinction becomes just one practical consideration among many in choosing an algorithm. Additional practical pros and cons of each method are therefore discussed below. Practical pros and cons of the algorithms: Minor differences in prediction accuracy is only one reason to choose one prediction algorithm over another. Each algorithm has strengths and weaknesses. PQSAR pros: The accuracy of pQSAR is comparable to the best MMRMs tested, so that is neither a reason to prefer or discount it. Profile-QSAR does have several practical advantages. pQSAR uses one-out training sets. One-out training sets most-closely reflect the final production none-out model performance for the use-case of screening the compound collection on which the models were trained. Since understanding a model’s quality affects decisions for how the model can be used, the use of one-out training sets constitutes pQSAR’s most important advantage. A second strength of pQSAR is facile transfer learning and adaptive learning, i.e., what is required to add a new model to the multitask ensemble, or to update a model by adding or subtracting measurements to an assay with an existing model. Sometimes a new assay becomes available, or one needs an additional endpoint, such as A max , rather than just AC 50 . Retraining a large number of hyperparameters is not needed. Referring back to Fig. 1 A, using the stored matrix of predictions from the RFR models, it simply requires training one additional PLS model. Similarly, if new data are available for an existing model, or if some measurements should be removed because they hit in a counter screen or fail QC, it simply requires retraining that single PLS model. Since the data for a new or updated assay would not have been used to train its own level 2 PLS model, the new model is supported by all the data in the existing multitask model exactly as if the new model had been in the initial training. The new model will not have contributed its support to the other models in the multitask collection, but a single new model’s contribution among 1000s is generally very incremental. Note that if any entirely new compounds are added that had not been tested in any assay (cold-starts), their RFR profiles are missing from the stored matrix, so RFR predictions for those compounds must also be computed before training the new PLS model, which does add a little more time. A third advantage is federated models, i.e. where several companies collaborate by training a super-multitask model covering the union of all their compounds, assays and bioactivity data, without sharing the actual structures or experimental data. Federated models can be built for multitask deep neural networks, but it is difficult. The IMI MELLODDY consortium did it in a 3-year, €18 million collaboration of 16 public and private partners. 38 Collaborative modeling with pQSAR is much simpler. 13 Each company, behind its own firewall, trains innocuous single-task level-1 models on its own data. It assembles the collection of level-1 models into a compiled “black box” program that takes chemical structures in and outputs a vector of numbers which are the bioactivity predictions for the unspecified assays. It can anonymize the predictions further by applying a similarity transformation that mixes the activity predictions but preserves the combined information. Each partner exports its black-box level-1 models into a shared location, from which each company imports its collaborator’s models back inside its firewall. It runs the predictions from the collaborator’s level-1 models on its compounds and adds them to its own level-1 predictions. It then trains the level-2 PLS models for its assays on the expanded level-1 profile. The calculation is virtually the same as if the collaborators had actually shared all the data, except that the work is distributed among the collaborators and each only gets pQSAR models for its own assays. pQSAR models are interpretable in terms of biology. The PLS coefficients from pQSAR models have been shown to capture unknown mechanisms of action and polypharmacologies. 39 This indicates that pQSAR is not employing some kind of meaningless curve fitting, but is learning underlying biology from the bioactivity data. Profile-QSAR is a “turn-key” calculation. It was built with default or standard parameters for the RFR and PLS models. When we apply it to completely different data sets, we don’t tune any parameters—we only change the data. Profile-QSAR is embarrassingly parallel. It runs on conventional CPUs—GPUs or TPUs are not needed. Training Novartis’s very large pQSAR of 20 million measurements from 2 million compounds by 15,000 assays, and making almost 100 billion IC 50 predictions for 6.5 million compounds, takes about 3 days on a modest cluster of 600 CPUs and is repeated every month. This could be an advantage for some users. The pQSAR source code is available at GitHub–- Novartis/pQSAR. PQSAR cons: pQSAR is the slowest algorithm in this study for making cold-start activity predictions on new compounds for just a few assays. This is because the activity of new compounds must first be computed for all the level-1 RFR models before running the few level-2 PLS models. This is only a disadvantage for predictions on a small number of specific assays, because for full-profile off-target, polypharmacology or MOA predictions the cost of level-1 models is amortized over all the level-2 PLS predictions. Thus, this limits its application in cold-start optimization problems like generative chemistry or interactive design, which as mentioned above is only worthwhile for analogs. A work-around is to train fast, single-task models that approximate pQSAR by augmenting the single-task training sets with synthetic data from pQSAR predictions for that assay using all 10,000s to millions of compounds in the whole MMRM training set. For generative chemistry, the optimization procedures exploit the artifacts in any model, which leads to false positives. We therefore also use pQSAR as a valuable orthogonal prediction method to confirm the predicted activity of the best compounds from generative chemistry optimized on other models. Another limitation of pQSAR is that each assay is independent. Unlike some methods, pQSAR has no simple way to explicitly incorporate side information on the outputs, i.e. assay properties such as incubation time or cofactor concentration that might inform the model that some assays are more closely related. This is implicitly incorporated, because assays run at e.g. different incubation times or ATP concentrations are separate assays, but explicit assay properties might improve models in some situations or in other domains besides bioactivity prediction. Alchemite pros: Beyond its performance as generally one of the more accurate multi-task algorithms considered here, Alchemite provides other benefits. Alchemite is available via easy-to-use no-code web graphical user interfaces, including in the drug discovery platform Cerella, 40 which means that it is much more straightforward to deploy to large teams of non-data scientists than research-level tools that require the user to run programs from the command line, use python or install software on their own machines. Beyond simply making imputations of missing assay data or predictions for cold-start compounds (both of which tasks have specific web pages associated with them), Alchemite also has specific components for other tasks: Uncertainty quantification associated with each prediction. This can enable a user to focus on the most confident predictions, which increases the accuracy of the remaining predictions used to take decisions, 41 or, conversely, to focus on compounds that are predicted to be relatively highly performing but have high uncertainty and so may be risky but potentially lucrative options. Uncertainty estimates also enable the identification of outlier data points, which could be false positive or false negative points. The uncertainty quantification in Alchemite is non-parametric, enabling it to capture more complex predictive distributions than a standard Gaussian. An “Importance Matrix” that highlights the inter-assay and descriptor-assay relationships the model is leveraging, which is helpful for identifying mechanisms of action or adverse outcome pathways Bayesian experimental design directly integrated with the predictive modelling component, which proposes experiments based on the Importance Matrix, model uncertainty, and predicted performance against a target profile Alchemite also models categorical data on an equal footing to the continuous/regression data used in this study. This can be useful for example where data is qualified (e.g. >10 uM) and not useful for regression modelling. As Alchemite is used daily in a variety of applications, including in fields beyond drug discovery (e.g. materials science, industrial chemistry, manufacturing), it is flexible enough to apply to many different use-cases beyond assay IC 50 prediction, including pharmacokinetic curve prediction, 42 toxicology modeling, 43 and sensory property prediction. 41 These diverse applications generate varied improvements in the Alchemite approach, all contributing to the same tool. Alchemite cons: Alchemite is commerical software, and therefore not available for free, although an academic program is available. Alchemite does not provide a database of existing data, as generally new problems require specific data to generate good models, but this means that existing data on the problem of interest is required to begin using Alchemite (although the software includes a Design of Experiments component to assist with initial data generation). Due to its applicability to a variety of different domains, Alchemite contains seven hyperparameters that need to be optimized for each new application (as different applications generally are best served by slightly different model parameters): the Alchemite software contains Bayesian hyperparameter optimization tools, but this takes time and computational resource relative to hyperparameter-free machine learning methods. MetaNN pros: MetaNN has been proven strong in bioactivity predictions; particularly, MetaNN performs better in smaller datasets, which have already been well predicted by the consensus DNN. We have applied MetaNN models to several drug discovery projects, both in-house and in the collaborations. In one project, we have observed that MetaNN bioactivity predictions provide much better differentiation even among the most active compounds, which can significantly help prioritizing compound designs; and this is the advantage that single task model cannot achieve. In another project, we have demonstrated that using metaNN models can help to select the sub-series for the next phase of compound optimization. MetaNN has high storage efficiency, storing only one consensus DNN optimized across all tests. This feature enables much faster and more convenient calculations in applications. MetaNN cons: One con of MetaNN is that the consensus DNN must go through a full re-training, which requires certain computational power, and this is needed even when a small fraction of the whole database is updated. MT-DNN pros: MT-DNNs, a sophisticated branch of deep learning techniques, have become widely recognized in machine learning. Besides their excellent predictive capabilities, shown in this study, MT-DNNs possess several advantages that can make them a preferred choice in various applications: Availability of Ready-to-Use Frameworks: MT-DNNs can be implemented using a range of general-purpose deep learning frameworks, including Theano, 44 Tensorflow, 45 Chainer, 46 Torch, 47 and PyTorch. 48 While these tools are not exclusive to MT-DNNs, they facilitate the modeling of such networks by offering robust functionalities, making advanced neural networks accessible to researchers with even basic programming skills. Computational Efficiency: Although deep neural networks generally benefit from GPU acceleration, the extent of efficiency gains can vary. MT-DNNs are particularly well-suited for GPUs, which can execute thousands of parallel operations to significantly speed up computations and minimize memory access latency, especially beneficial when training with large datasets. Knowledge Transfer: MT-DNNs leverage extensive datasets across multiple tasks, enhancing the learning process for each by transferring insights from data-rich tasks to those with less data. This capability helps reduce overfitting and improves the robustness and generalization of the model to new, unseen data. Flexibility in Data Handling: MT-DNNs demonstrate versatility in managing different data types. They can process information related to either the properties of compounds or tasks independently or in combination. Complex Data Management: MT-DNNs efficiently handle data of varying sizes and complexities. They are adept at working with data in complex tensor forms, which enables them to tackle more sophisticated predictive challenges. In this work, we utilized an MT-DNN capable of processing information presented in a matrix form. MT-DNN cons: Despite these advantages, MT-DNNs also present several challenges: Lack of Interpretability: MT-DNNs, like many deep learning models, operate as "black boxes," meaning that the model operation is not easily understandable. It can be a critical drawback in fields requiring transparent decision-making processes. Demand for Substantial Computational Resources: MT-DNNs require significant computational power, which includes the need for high-performance GPUs and substantial memory. It can escalate costs and limit the feasibility of using MT-DNNs in real time. Risk of Negative Transfer: While knowledge transfer is a significant strength of MT-DNNs, there is a potential risk of negative transfer. This occurs when the tasks are not sufficiently related, or the shared features do not benefit all tasks, potentially leading to worse performance than if the tasks were trained independently. Dependency on Task Relatedness: The effectiveness of MT-DNNs heavily depends on the relatedness of the tasks involved. When tasks are too dissimilar, the benefits of shared representations and joint training can decrease, leading to suboptimal outcomes. Macau pros: Macau did very well on the kinase assays, although less well on the diverse assay collection. Macau is a sophisticated machine learning algorithm that can deal with sparse data and integrate a wide range of features, which makes it a powerful tool for multi-task learning across various domains. Unlike many traditional machine learning models that provide point estimates, Macau, being Bayesian, inherently provides probabilistic predictions. This means that for every prediction it makes, it also estimates the uncertainty associated with that prediction. It leads to the main advantages distinguishing Macau from many other multi-task machine learning methods: Improved Risk Assessment: The uncertainty estimation intrinsic to Macau is invaluable for risk assessment. In drug discovery, the confidence level of a prediction can significantly influence the decision-making process, such as selecting candidates in early drug discovery. This capability allows researchers to make more informed and cautious decisions, considering not only the model's predictions but also their reliability. Incorporation of Prior Knowledge: Macau's Bayesian nature facilitates the incorporation of prior knowledge or expert insights into its modeling process. This ability to integrate existing knowledge effectively reduces prediction uncertainties and refines the learning process, making Macau particularly adept in fields where historical data or expert understanding is pivotal. Guidance for Data Collection: Macau’s approach to uncertainty estimation is particularly useful in pinpointing areas where additional data collection could enhance model accuracy. In multi-task learning, this means accurately identifying specific targets or compounds that, if expanded upon, could significantly reduce uncertainty and improve model performance. Utility in Exploratory Analysis: The algorithm’s capability to quantify uncertainty can also be leveraged for exploratory analysis. This feature enables researchers to either concentrate on predictions with low uncertainty for reliable predictions or explore those with high uncertainty to uncover potential anomalies. Enhanced Descriptor Integration: Macau's algorithm excels in simultaneously processing descriptors of both compounds and assays, a critical aspect in drug discovery and chemical informatics. This dual capability allows for a more comprehensive analysis, as it can consider the intricate interplay between chemical properties of compounds and the biological context of assays. Advanced Tensor Data Handling: One of Macau's standout features is its ability to operate not just with matrices but also with tensors of data. This capability is particularly advantageous in scenarios where data is multi-dimensional and complex, such as in multi-factorial drug response studies or when dealing with time-series data. Tensors allow for the incorporation of additional dimensions, like time, dose-response curves, or multi-layered genetic information, providing a more nuanced and detailed analysis. In summary, Macau's approach to uncertainty calculation, rooted in its Bayesian framework, can provide a more nuanced and comprehensive understanding of predictions, enhance the reliability of decision-making processes, and add a layer of transparency and trust to the model's outputs. Macau cons: Like many advanced predictive algorithms, Macau, while offering considerable advantages, also presents notable challenges related to its computational demands, the complexity of its implementation, the dependence on feature quality, and interpretability issues. These challenges are based on Macau's reliance on Bayesian methods and the high level of programming expertise required for its effective application. The implementation of Macau, with its intricate Bayesian foundation, is inherently more complex than more straightforward machine learning algorithms. Consequently, users must possess advanced programming skills and a deep understanding of the algorithm to utilize Macau's capabilities fully. This necessity potentially limits its accessibility to a broader user base. While average researchers might employ Macau for basic predictive tasks, more nuanced applications — such as model tuning that accounts for uncertainty predictions and incorporates expert knowledge — demand a significantly higher level of technical proficiency. Furthermore, effective operation with Macau extends beyond programming knowledge; it requires a comprehensive understanding of Bayesian statistics. A solid foundation in probabilistic modeling and Bayesian inference and familiarity with techniques such as Markov Chain Monte Carlo (MCMC) are essential. Therefore, while Macau offers a wide array of opportunities for enhancing predictive analytics, its sophisticated nature can hinder its widespread and straightforward use. Mastery of both the theoretical underpinnings and practical implementation aspects of Macau is essential for leveraging its full potential in various practical applications. Conclusions 6 MMRM algorithms were compared using realistically novel test sets, trained on large compound clusters and tested on singletons and small clusters, on 2 published assay collections from ChEMBL: a homogenous set of 159 kinase assays and a heterogeneous collection of 4276 diverse assays. The comparison was complicated, because methods that train all models simultaneously must leave out all test-set measurements for all assays to avoid test-set leakage, 25% of the data in our cases. Because realistic test sets are trained on large compound clusters and leave out the most unique compounds, all-out model training-set collections are missing particularly informative measurements. MMRMs which train models one-at-a-time only leave out data for each assay as it is trained, less than 1% and 0.1% of the data in our kinase and diverse assay collections respectively. These one-out models should be very close to the final production models which don’t leave out any data, and thus should give the best estimate of the final none-out models’ predictive performance on the compound collections on which the models were trained. To enable better comparisons across all algorithms, subset-out models were also trained. These only built models for an experimentally designed roughly 10% of kinase or 1% of diverse subsets of the assays, thus leaving out only ~ 10% or 1% of the test data, and including the realistic test-set compound measurements for the remaining ~ 90% or 99% of assays. The analysis found that for the realistic test-set predictions, all-out model predictions were much worse than the one-out models, and by extension the final none-out production models for our use case of predictions on the training archive. The subset-out model accuracy was close to the one-out models, suggesting that for methods for which training thousands of one-out models are impracticable, a compromise of building multiple separate subset-out models could be used to evaluate the likely performance of the final models. Again, this only affects the evaluation of final model quality, not the performance of the final models, which are always none-out models. Generally, the 5 “advanced” MMRMs did substantially better than the benchmark ST-RFR models on the realistic test sets. Comparing algorithms showed that, in most cases, the performance of the better methods, while sometimes statistically distinguishable, were still relatively close, at least on the more homogeneous kinase assay collection. Macau had trouble with the larger diverse assay collection and the MMRM benchmark IMC performed less well, as expected. The 5 very different methods also agreed on which individual assays could be modeled well, and which could not. This supports the hypothesis that many diverse MMRM algorithms are comparable in performance at a fundamental level and suggests that they have reached a limit of what signal can be extracted from the data. The advantage of MMRMs over ST-RFR was marginal for cold-start predictions on these realistic test-set compounds very unlike the individual assay’s training sets. MMRM imputations were much more accurate than ST-RFR on these challenging predictions. The larger proportion of imputations in one-out vs. all-out models likely contributes to their better performance. Even a few supporting measurements improved performance, and the improvement was especially notable for compounds with large numbers of supporting measurements. This implies that MMRMs are best for predictions on the compound-collection used for MMRM training: for on-target virtual screens to find additional novel chemical matter and for hit-list triaging, off-target, promiscuity, MoA, polypharmacology or drug repurposing predictions of existing compounds or their close analogs. It also implies that models should be updated frequently—monthly to weekly—so predictions on recent compounds will include initial measurements and thus be imputations. We expect little advantage for cold-start predictions on compounds outside the multitask training space, such as vendor collections or exploratory generative chemistry, and at significant extra cost. The algorithms differ with respect to other practical considerations. We therefore included a detailed discussion of the pros and cons of each method. While this comparison dealt specifically with drug discovery, similar conclusions should apply to modeling other domains. In fact, only pQSAR was originally developed for drug discovery. Alchemite was originally developed for material science, and matrix factorization, meta-learners and DNNs were used in many diverse domains before drug discovery. Declarations Author Contribution E.M. arranged the collaboration, wrote the main manuscript text, developed pQSAR;X.Z. performed the pQSAR calculations and analysis;P.R. performed analysis, especially significance testing;S.K. performed analysis;E.S. built MT-DNN, Macau and IMC models, performed calculations and analysis;L.T. and Y.W. developed metaNN;Z.W. performed metaNN calculations;T.W., G.C. and MS performed Alchemite calculations and analysis. Acknowledgments: Dr. Li Tian and Mr. Zijian Wang acknowledge "Laboratory for Synthetic Chemistry and Chemical Biology" under the Health@InnoHK Program launched by Innovation and Technology Commission, The Government of Hong Kong Special Administrative Region of the People's Republic of China, for the funding support of this research, as well as for providing the studentship for Mr. Wang from year 2022 to year 2025. Data and Software Availability: The compounds, assays, experimental and pQSAR predicted bioactivities and training/test set splits are available in the Supporting Information from 2 papers. 1,6 The pQSAR source code is openly accessible at GitHub–- Novartis/pQSAR. Macau was implemented within the open-source Bayesian Matrix and Tensor Factorization framework, SMURFF. 35 The Python code for the MT-DNN prediction algorithm is openly accessible on GitHub. 34 Data Availability The compounds, assays, experimental and pQSAR predicted bioactivities and training/test set splits are available in the Supporting Information from 2 papers: https://doi.org/10.1021/ACS.JCIM.9B00375 and https://doi.org/10.1021/acs.jcim.7b00166.The pQSAR source code is openly accessible at [GitHub–- Novartis/pQSAR](https:/github.com/Novartis/pQSAR) .Macau was implemented within the open-source Bayesian Matrix and Tensor Factorization framework, SMURFF.The Python code for the MT-DNN prediction algorithm is openly accessible on GitHub. References Martin, E. J.; Polyakov, V. R.; Zhu, X. W.; Tian, L.; Mukherjee, P.; Liu, X. All-Assay-Max2 PQSAR: Activity Predictions as Accurate as Four-Concentration IC50s for 8558 Novartis Assays. J Chem Inf Model 2019 . https://doi.org/10.1021/ACS.JCIM.9B00375. Hansch, C.; Fujita Vol, T.; Roberts, R. B.; Cowie, D. B.; Britten, R.; Abelson, P. H.; Moses, V.; Holm-Hansen, O.; Bassham, J. A.; Calvin, M.; Mol Biol, J.; Corwin Hansch, B.; Fujita, T. Analysis. A Method for the Correlation of Biological Activity and Chemical Structure. J. Biol. Chem 1963 , 39 (61), 284. Hansch, C.; Fujita Vol, T.; Roberts, R. B.; Cowie, D. B.; Britten, R.; Abelson, P. H.; Moses, V.; Holm-Hansen, O.; Bassham, J. A.; Calvin, M.; Mol Biol, J.; Corwin Hansch, B.; Fujita, T. ρ-σ-π Analysis. A Method for the Correlation of Biological Activity and Chemical Structure. J Am Chem Soc 1964 , 86 (8), 1616–1626. https://doi.org/10.1021/JA01062A035. Hammett, L. P. The Effect of Structure upon the Reactions of Organic Compounds. Benzene Derivatives. J Am Chem Soc 1937 , 59 (1), 96–103. https://doi.org/10.1021/JA01280A022/ASSET/JA01280A022.FP.PNG_V03. Lapinsh, M.; Prusis, P.; Gutcaits, A.; Lundstedt, T.; Wikberg, J. E. S. Development of Proteo-Chemometrics: A Novel Technology for the Analysis of Drug-Receptor Interactions. Biochim Biophys Acta Gen Subj 2001 , 1525 (1–2), 180–190. https://doi.org/10.1016/S0304-4165(00)00187-2. Martin, E. J.; Polyakov, V. R.; Tian, L.; Perez, R. C. Profile-QSAR 2.0: Kinase Virtual Screening Accuracy Comparable to Four-Concentration IC50s for Realistically Novel Compounds. J Chem Inf Model 2017 , 57 (8), 2077–2088. https://doi.org/10.1021/acs.jcim.7b00166. Sosnina, E. A.; Sosnin, S.; Fedorov, M. V. Improvement of Multi-Task Learning by Data Enrichment: Application for Drug Discovery. J Comput Aided Mol Des 2023 , 37 (4), 183–200. https://doi.org/10.1007/s10822-023-00500-w. Wu, J.; Chen, Y.; Wu, J.; Zhao, D.; Huang, J.; Lin, M.; Wang, L. Large-Scale Comparison of Machine Learning Methods for Profiling Prediction of Kinase Inhibitors. J Cheminform 2024 , 16 (1), 13. https://doi.org/10.1186/s13321-023-00799-5. Feng, B.; Liu, Z.; Huang, N.; Xiao, Z.; Zhang, H.; Mirzoyan, S.; Xu, H.; Hao, J.; Xu, Y.; Zhang, M.; Wang, S. A Bioactivity Foundation Model Using Pairwise Meta-Learning. Nat Mach Intell 2024 , 6 (8), 962–974. https://doi.org/10.1038/s42256-024-00876-w. Martin, E.; Sullivan, D. Iterative Kinase Medium-Throughput Screening with Profile-QSAR and Ensemble Surrogate AutoShim for Fast Predictive Kinase Docking without a Crystal Structure. In Keystone symposium: Structure Based Drug Discovery (D6) ; Burley, S. K., Shoichet, B. K., Bartlett, P. A., Eds.; Whistler, BC, Canada, 2006. Martin, E.; Mukherjee, P.; Sullivan, D.; Jansen, J. Profile-QSAR: A Novel Meta-QSAR Method That Combines Activities across the Kinase Family To Accurately Predict Affinity, Selectivity, and Cellular Activity. J Chem Inf Model 2011 , 51 (8), 1942–1956. https://doi.org/10.1021/ci1005004. Martin, E.; Mukherjee, P.; Sullivan, D.; Jansen, J. Profile-QSAR: A Novel Meta-QSAR Method That Combines Activities across the Kinase Family To Accurately Predict Affinity, Selectivity, and Cellular Activity. J. Chem. Inf. Model 2011 , 51 , 1942–1956. https://doi.org/10.1021/ci1005004. Martin, E. J.; Zhu, X. W. Collaborative Profile-QSAR: A Natural Platform for Building Collaborative Models among Competing Companies. J Chem Inf Model 2021 , 61 (4), 1603–1616. https://doi.org/10.1021/ACS.JCIM.0C01342/ASSET/IMAGES/LARGE/CI0C01342_0016.JPEG. Conduit, B. D.; Jones, N. G.; Stone, H. J.; Conduit, G. J. Probabilistic Design of a Molybdenum-Base Alloy Using a Neural Network. Scr Mater 2018 , 146 , 82–86. https://doi.org/10.1016/j.scriptamat.2017.11.008. Conduit, B. D.; Jones, N. G.; Stone, H. J.; Conduit, G. J. Design of a Nickel-Base Superalloy Using a Neural Network. Mater Des 2017 , 131 , 358–365. https://doi.org/10.1016/j.matdes.2017.06.007. Whitehead, T. M.; Chen, F.; Daly, C.; Conduit, G. J. Accelerating the Design of Automotive Catalyst Products Using Machine Learning. Johnson Matthey Technology Review 2022 , 66 (2), 130–136. https://doi.org/10.1595/205651322x16270488736796. Obrezanova, O.; Martinsson, A.; Whitehead, T.; Mahmoud, S.; Bender, A.; Miljković, F.; Grabowski, P.; Irwin, B.; Oprisiu, I.; Conduit, G.; Segall, M.; Smith, G. F.; Williamson, B.; Winiwarter, S.; Greene, N. Prediction of in Vivo Pharmacokinetic Parameters and Time-Exposure Curves in Rats Using Machine Learning from the Chemical Structure. Mol Pharm 2022 , 19 (5), 1488–1504. https://doi.org/10.1021/acs.molpharmaceut.2c00027. Tse, E. G.; Aithani, L.; Anderson, M.; Cardoso-Silva, J.; Cincilla, G.; Conduit, G. J.; Galushka, M.; Guan, D.; Hallyburton, I.; Irwin, B. W. J.; Kirk, K.; Lehane, A. M.; Lindblom, J. C. R.; Lui, R.; Matthews, S.; McCulloch, J.; Motion, A.; Ng, H. L.; Öeren, M.; Robertson, M. N.; Spadavecchio, V.; Tatsis, V. A.; van Hoorn, W. P.; Wade, A. D.; Whitehead, T. M.; Willis, P.; Todd, M. H. An Open Drug Discovery Competition: Experimental Validation of Predictive Models in a Series of Novel Antimalarials. J Med Chem 2021 , 64 (22), 16450–16463. https://doi.org/10.1021/acs.jmedchem.1c00313. Irwin, B. W. J.; Whitehead, T. M.; Rowland, S.; Mahmoud, S. Y.; Conduit, G. J.; Segall, M. D. Deep Imputation on Large‐scale Drug Discovery Data. Applied AI Letters 2021 , 2 (3). https://doi.org/10.1002/ail2.31. Irwin, B. W. J.; Levell, J. R.; Whitehead, T. M.; Segall, M. D.; Conduit, G. J. Practical Applications of Deep Learning to Impute Heterogeneous Drug Discovery Data. J Chem Inf Model 2020 , 60 (6), 2848–2857. https://doi.org/10.1021/acs.jcim.0c00443. Whitehead, T.; Irwin, B.; Hunt, P.; Segall, M.; Conduit, G. Imputation of Assay Bioactivity Data Using Deep Learning. J Chem Inf Model 2019 , 59 (3), 1197 – 1204. https://doi.org/10.1021/acs.jcim.8b00768. Stuckner, J.; Whitehead, T. M.; Parini, R. C.; Conduit, G. J.; Benafan, O.; Arnold, S. M. Design of Materials With Alchemite. NASA Tech Memo 2022 , No. July. Simm, J.; Arany, A.; Zakeri, P.; Haber, T.; Wegner, J. K.; Chupakhin, V.; Ceulemans, H.; Moreau, Y. Macau: Scalable Bayesian Multi-Relational Factorization with Side Information Using MCMC. 2015. Simm, J.; Arany, A.; Zakeri, P.; Haber, T.; Wegner, J. K.; Chupakhin, V.; Ceulemans, H.; Moreau, Y. Macau: Scalable Bayesian Factorization with High-Dimensional Side Information Using MCMC. In 2017 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP) ; 2017; pp 1–6. https://doi.org/10.1109/MLSP.2017.8168143. Jain, P.; Dhillon, I. S. Provable Inductive Matrix Completion. 2013 . Demšar, J. Statistical Comparisons of Classifiers over Multiple Data Sets. Journal of Machine Learning Research 2006 , 7 (1), 1–30. Benavoli, A.; Corani, G.; Mangili, F. Should We Really Use Post-Hoc Tests Based on Mean-Ranks? Journal of Machine Learning Research 2016 , 17 (5), 1–10. HOCHBERG, Y. A Sharper Bonferroni Procedure for Multiple Tests of Significance. Biometrika 1988 , 75 (4), 800–802. https://doi.org/10.1093/biomet/75.4.800. Bergstra, J.; Bardenet, R.; Bengio, Y.; Kégl, B. Algorithms for Hyper-Parameter Optimization. In Advances in Neural Information Processing Systems ; Shawe-Taylor, J., Zemel, R., Bartlett, P., Pereira, F., Weinberger, K. Q., Eds.; Curran Associates, Inc., 2011; Vol. 24. Yao, H.; Huang, L.-K.; Zhang, L.; Wei, Y.; Tian, L.; Zou, J.; Huang, J.; Li, Z. (). Improving Generalization in Meta-Learning via Task Augmentation. In Proceedings of the 38th International Conference on Machine Learning ; Meila, M., Zhang, T., Eds.; Proceedings of Machine Learning Research; PMLR, 2021; Vol. 139, pp 11887–11897. Tian, L.; Huang, H. H.; Martin, E.; Wei, J. Y.; Xu, S.; Huang, J. Transfer Learning Applied in Predicting Small Molecule Bioactivity. bioRxiv 2025 . https://doi.org/10.1101/2025.01.09.632140. Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; Desmaison, A.; Kopf, A.; Yang, E.; DeVito, Z.; Raison, M.; Tejani, A.; Chilamkurthy, S.; Steiner, B.; Fang, L.; Bai, J.; Chintala, S. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32 ; Curran Associates, Inc., 2019; pp 8024–8035. Salakhutdinov, R. M. A. Bayesian Probabilistic Matrix Factorization Using Markov Chain Monte Carlo. In Proceedings of the 25th international conference on Machine learning (ICML ’08) ; ACM: New York, 2008; pp 880–887. Sosnina, E. A.; Sosnin, S.; Fedorov, M. V. ImprovingMTT. GitHub repository . GitHub 2023. https://github.com/ekaterina-sea/ImprovingMTT. Simm, J. SMURFF - Scalable Matrix Factorization Framework . https://github.com/ExaScience/smurff. Simm Jaak. SMURFF - Scalable Matrix Factorization Framework . https://smurff.readthedocs.io/en/release-0.16/index.html. Optibrium StarDrop . https://optibrium.com/stardrop/. Heyndrickx, W.; Mervin, L.; Morawietz, T.; Sturm, N.; Friedrich, L.; Zalewski, A.; Pentina, A.; Humbeck, L.; Oldenhof, M.; Niwayama, R.; Schmidtke, P.; Fechner, N.; Simm, J.; Arany, A.; Drizard, N.; Jabal, R.; Afanasyeva, A.; Loeb, R.; Verma, S.; Harnqvist, S.; Holmes, M.; Pejo, B.; Telenczuk, M.; Holway, N.; Dieckmann, A.; Rieke, N.; Zumsande, F.; Clevert, D.-A.; Krug, M.; Luscombe, C.; Green, D.; Ertl, P.; Antal, P.; Marcus, D.; Do Huu, N.; Fuji, H.; Pickett, S.; Acs, G.; Boniface, E.; Beck, B.; Sun, Y.; Gohier, A.; Rippmann, F.; Engkvist, O.; Göller, A. H.; Moreau, Y.; Galtier, M. N.; Schuffenhauer, A.; Ceulemans, H. MELLODDY: Cross-Pharma Federated Learning at Unprecedented Scale Unlocks Benefits in QSAR without Compromising Proprietary Information. J Chem Inf Model 0 (0), null. https://doi.org/10.1021/acs.jcim.3c00799. Martin, E. J.; Polyakov, V. R.; Zhu, X.-W.; Tian, L.; Mukherjee, P.; Liu, X. All-Assay-Max2 PQSAR: Activity Predictions as Accurate as Four-Concentration IC50s for 8558 Novartis Assays. J Chem Inf Model 2019 . https://doi.org/10.1021/acs.jcim.9b00375. Cerella home page . https://optibriu m.com/cerella/. Mahmoud, S.; Irwin, B.; Chekmarev, D.; Vyas, S.; Kattas, J.; Whitehead, T.; Mansley, T.; Bikker, J.; Conduit, G.; Segall, M. Imputation of Sensory Properties Using Deep Learning. J Comput Aided Mol Des 2021 , 35 (11), 1125–1140. https://doi.org/10.1007/s10822-021-00424-3. Obrezanova, O.; Martinsson, A.; Whitehead, T.; Mahmoud, S.; Bender, A.; Miljković, F.; Grabowski, P.; Irwin, B.; Oprisiu, I.; Conduit, G.; Segall, M.; Smith, G. F.; Williamson, B.; Winiwarter, S.; Greene, N. Prediction of In Vivo Pharmacokinetic Parameters and Time–Exposure Curves in Rats Using Machine Learning from the Chemical Structure. Mol Pharm 2022 , 19 (5), 1488–1504. https://doi.org/10.1021/acs.molpharmaceut.2c00027. Whitehead, T. M.; Strickland, J.; Conduit, G. J.; Borrel, A.; Mucs, D.; Baskerville-Abraham, I. Quantifying the Benefits of Imputation over QSAR Methods in Toxicology Data Modeling. J Chem Inf Model 2024 , 64 (7), 2624–2636. https://doi.org/10.1021/acs.jcim.3c01695. The Theano Development Team; Al-Rfou, R.; Alain, G.; Almahairi, A.; Angermueller, C.; Bahdanau, D.; Ballas, N.; Bastien, F.; Bayer, J.; Belikov, A.; Belopolsky, A.; Bengio, Y.; Bergeron, A.; Bergstra, J.; Bisson, V.; Snyder, J. B.; Bouchard, N.; Boulanger-Lewandowski, N.; Bouthillier, X.; de Brebisson, A.; Breuleux, O.; Carrier, P.-L.; Cho, K.; Chorowski, J.; Christiano, P.; Cooijmans, T.; Cote, M.-A.; Cote, M.; Courville, A.; Dauphin, Y. N.; Delalleau, O.; Demouth, J.; Desjardins, G.; Dieleman, S.; Dinh, L.; Ducoffe, M.; Dumoulin, V.; Kahou, S. E.; Erhan, D.; Fan, Z.; Firat, O.; Germain, M.; Glorot, X.; Goodfellow, I.; Graham, M.; Gulcehre, C.; Hamel, P.; Harlouchet, I.; Heng, J.-P.; Hidasi, B.; Honari, S.; Jain, A.; Jean, S.; Jia, K.; Korobov, M.; Kulkarni, V.; Lamb, A.; Lamblin, P.; Larsen, E.; Laurent, C.; Lee, S.; Lefrancois, S.; Lemieux, S.; Leonard, N.; Lin, Z.; Livezey, J. A.; Lorenz, C.; Lowin, J.; Ma, Q.; Manzagol, P.-A.; Mastropietro, O.; McGibbon, R. T.; Memisevic, R.; van Merrienboer, B.; Michalski, V.; Mirza, M.; Orlandi, A.; Pal, C.; Pascanu, R.; Pezeshki, M.; Raffel, C.; Renshaw, D.; Rocklin, M.; Romero, A.; Roth, M.; Sadowski, P.; Salvatier, J.; Savard, F.; Schlüter, J.; Schulman, J.; Schwartz, G.; Serban, I. V.; Serdyuk, D.; Shabanian, S.; Simon, Étienne; Spieckermann, S.; Subramanyam, S. R.; Sygnowski, J.; Tanguay, J.; van Tulder, G.; Turian, J.; Urban, S.; Vincent, P.; Visin, F.; de Vries, H.; Warde-Farley, D.; Webb, D. J.; Willson, M.; Xu, K.; Xue, L.; Yao, L.; Zhang, S.; Zhang, Y. Theano: A Python Framework for Fast Computation of Mathematical Expressions. arXiv May 2016. https://doi.org/10.48550/arXiv.1605.02688. Abadi, M.; Barham, P.; Chen, J.; Chen, Z.; Davis, A.; Dean, J.; Devin, M.; Ghemawat, S.; Irving, G.; Isard, M.; others. Tensorflow: A System for Large-Scale Machine Learning. In 12th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16) ; 2016; pp 265–283. Tokui, S.; Okuta, R.; Akiba, T.; Niitani, Y.; Ogawa, T.; Saito, S.; Suzuki, S.; Uenishi, K.; Vogel, B.; Vincent, H. Y. Chainer: A Deep Learning Framework for Accelerating the Research Cycle. arXiv August 2019. https://doi.org/10.48550/arXiv.1908.00213. Collobert, R.; Kavukcuoglu, K.; Farabet, C. Torch7: A Matlab-like Environment for Machine Learning. In BigLearn, NIPS Workshop ; 2011. Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; Desmaison, A.; Kopf, A.; Yang, E.; DeVito, Z.; Raison, M.; Tejani, A.; Chilamkurthy, S.; Steiner, B.; Fang, L.; Bai, J.; Chintala, S. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32 ; Wallach, H., Larochelle, H., Beygelzimer, A., d Alché-Buc, F., Fox, E., Garnett, R., Eds.; Curran Associates, Inc., 2019; pp 8024–8035. https://doi.org/10.48550/arXiv.1912.01703. Additional Declarations No competing interests reported. Supplementary Files tabsignedr2goodenough.csv.txt tabmediansignedr2.csv.txt tabanalysis2tables.csv.txt tabanalysis2pvalues.csv.txt tabanalysis1full.csv.txt cvstables.txt Listofsupplementalmaterials.docx Cite Share Download PDF Status: Published Journal Publication published 05 Feb, 2026 Read the published version in Journal of Computer-Aided Molecular Design → Version 1 posted Editorial decision: Revision requested 31 Oct, 2025 Reviews received at journal 31 Oct, 2025 Reviews received at journal 28 Oct, 2025 Reviewers agreed at journal 20 Oct, 2025 Reviewers agreed at journal 17 Oct, 2025 Reviewers agreed at journal 16 Oct, 2025 Reviewers agreed at journal 03 Oct, 2025 Reviewers invited by journal 11 Sep, 2025 Editor assigned by journal 11 Sep, 2025 Submission checks completed at journal 02 Sep, 2025 First submitted to journal 28 Aug, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7482715","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":516172747,"identity":"9adba08e-0f28-4ce3-a3de-4e34b2c58dd9","order_by":0,"name":"Eric J Martin","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA8UlEQVRIiWNgGAWjYFACxgYIzczA+ABVhJAWHmYGZgMwi42gFijgAaqVgGghoJJf+nDjB8Ycu3x7dt5j1QUVh+XN5zc3f2CosYnGpUWyL7FZgnFbsmUPM1/a7RlnDhvOOcbYJsFwLC0Xl/MMzjC2MTBuYzbgYeYxu83blsY4gw0k0nAYpxZ7iJZ6sJZioBZ7oJbmD/i0GPCAtRwGa2HmbbNJBGppkMCnReIMY7NE4rbjBjyHeYylec7YJM9gS2yTSMDjF/4e9ocfPm6rNmDvP2P4madCwnYG8/HHHz7U2ODUAgYJRIiMglEwCkbBKCAFAADhS0rHzk0dcQAAAABJRU5ErkJggg==","orcid":"","institution":"Novartis (United States)","correspondingAuthor":true,"prefix":"","firstName":"Eric","middleName":"J","lastName":"Martin","suffix":""},{"id":516172748,"identity":"91d665b8-a7ac-4060-b388-64ee72b5c101","order_by":1,"name":"Xiang-Wei Zhu","email":"","orcid":"","institution":"Novartis (United States)","correspondingAuthor":false,"prefix":"","firstName":"Xiang-Wei","middleName":"","lastName":"Zhu","suffix":""},{"id":516172749,"identity":"7c0e4e0c-773f-4bc6-9896-f8fbb464cf26","order_by":2,"name":"Patrick Riley","email":"","orcid":"","institution":"DaVinci","correspondingAuthor":false,"prefix":"","firstName":"Patrick","middleName":"","lastName":"Riley","suffix":""},{"id":516172750,"identity":"75dc213a-1a6f-40d9-a002-deae9fa5df87","order_by":3,"name":"Steven Kearnes","email":"","orcid":"","institution":"Genesis Therapeutics","correspondingAuthor":false,"prefix":"","firstName":"Steven","middleName":"","lastName":"Kearnes","suffix":""},{"id":516172751,"identity":"d4888c3c-e277-497b-991f-e6bff648fd41","order_by":4,"name":"Ekaterina A Sosnina","email":"","orcid":"","institution":"Independent Researcher","correspondingAuthor":false,"prefix":"","firstName":"Ekaterina","middleName":"A","lastName":"Sosnina","suffix":""},{"id":516172752,"identity":"6392970b-69aa-40c1-954f-dfcb4912980c","order_by":5,"name":"Li Tian","email":"","orcid":"","institution":"University of Hong Kong","correspondingAuthor":false,"prefix":"","firstName":"Li","middleName":"","lastName":"Tian","suffix":""},{"id":516172753,"identity":"67293fe4-b985-4e5e-bbcf-b101e8c389d2","order_by":6,"name":"Zijian Wang","email":"","orcid":"","institution":"University of Hong Kong","correspondingAuthor":false,"prefix":"","firstName":"Zijian","middleName":"","lastName":"Wang","suffix":""},{"id":516172754,"identity":"2b50f225-8f22-42b0-8a66-90ee156f35d9","order_by":7,"name":"Ying Wei","email":"","orcid":"","institution":"Zhejiang University","correspondingAuthor":false,"prefix":"","firstName":"Ying","middleName":"","lastName":"Wei","suffix":""},{"id":516172755,"identity":"559e8326-1fa5-4f11-b99e-62a94f405470","order_by":8,"name":"Thomas M Whitehead","email":"","orcid":"","institution":"Intellegens Limited","correspondingAuthor":false,"prefix":"","firstName":"Thomas","middleName":"M","lastName":"Whitehead","suffix":""},{"id":516172756,"identity":"30fc58de-7fe2-4bc0-bef9-7b9de1dc2d11","order_by":9,"name":"Gareth J Conduit","email":"","orcid":"","institution":"Intellegens Limited","correspondingAuthor":false,"prefix":"","firstName":"Gareth","middleName":"J","lastName":"Conduit","suffix":""},{"id":516172757,"identity":"50c6f80f-f28e-4b23-be70-bf3f4005e64b","order_by":10,"name":"Matthew D Segall","email":"","orcid":"","institution":"Optibrium Limited","correspondingAuthor":false,"prefix":"","firstName":"Matthew","middleName":"D","lastName":"Segall","suffix":""}],"badges":[],"createdAt":"2025-08-28 18:23:11","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7482715/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7482715/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s10822-026-00761-1","type":"published","date":"2026-02-05T15:59:19+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":91846000,"identity":"9726b7db-cb0b-4dad-805a-fb4b3eda6297","added_by":"auto","created_at":"2025-09-22 10:15:25","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":2294434,"visible":true,"origin":"","legend":"","description":"","filename":"cfMMRMsjcamd.docx","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/35699e3f73e844eed1f375cb.docx"},{"id":91845989,"identity":"24636cec-8a6d-46a1-a895-5206cd9cbbc3","added_by":"auto","created_at":"2025-09-22 10:15:25","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":12053,"visible":true,"origin":"","legend":"","description":"","filename":"57a8b1af71144033804bb4c9933ae1f0.json","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/4e009ede542d72e620043316.json"},{"id":91843759,"identity":"79cf9441-5e55-403c-a0d9-c1d3627915cb","added_by":"auto","created_at":"2025-09-22 10:07:27","extension":"txt","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":36994,"visible":true,"origin":"","legend":"","description":"","filename":"cvstables.txt","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/30c68f5ac77534a85f83f65e.txt"},{"id":91843714,"identity":"78f16140-b07c-4bf1-9b6b-cb93f983776c","added_by":"auto","created_at":"2025-09-22 10:07:25","extension":"txt","order_by":3,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":15579,"visible":true,"origin":"","legend":"","description":"","filename":"tabanalysis1full.csv.txt","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/c3db73343b76e1ccae2ac18e.txt"},{"id":91847993,"identity":"ed3cb292-41b4-40ff-9d09-15fc829f1da8","added_by":"auto","created_at":"2025-09-22 10:31:25","extension":"txt","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":10428,"visible":true,"origin":"","legend":"","description":"","filename":"tabanalysis2pvalues.csv.txt","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/0ea9e05e6599251d09737767.txt"},{"id":91843721,"identity":"151d3f16-42c9-42f5-a550-ce054ebc3fde","added_by":"auto","created_at":"2025-09-22 10:07:25","extension":"txt","order_by":5,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":7963,"visible":true,"origin":"","legend":"","description":"","filename":"tabanalysis2tables.csv.txt","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/44b297a8caf8dd5f72446f22.txt"},{"id":91845999,"identity":"f2033df7-dab5-43bc-a86d-3a04cb54b432","added_by":"auto","created_at":"2025-09-22 10:15:25","extension":"txt","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1695,"visible":true,"origin":"","legend":"","description":"","filename":"tabmediansignedr2.csv.txt","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/c89031505bae7a3ddac2761c.txt"},{"id":91843729,"identity":"b36857c4-a6e0-4651-9010-47a683792d80","added_by":"auto","created_at":"2025-09-22 10:07:25","extension":"txt","order_by":7,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1088,"visible":true,"origin":"","legend":"","description":"","filename":"tabsignedr2goodenough.csv.txt","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/46c9c85f4513d2c6a55334ba.txt"},{"id":91843735,"identity":"216d9c27-a35f-4112-8a65-a9335c45e63f","added_by":"auto","created_at":"2025-09-22 10:07:25","extension":"xml","order_by":8,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":297009,"visible":true,"origin":"","legend":"","description":"","filename":"57a8b1af71144033804bb4c9933ae1f01enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/b7ad7aa578d7f3892420c9d7.xml"},{"id":91843754,"identity":"250b194d-129a-49f5-91fc-1ed99af0ef5d","added_by":"auto","created_at":"2025-09-22 10:07:26","extension":"png","order_by":9,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":286066,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/f8b6b1af0e9f1e01c0d35409.png"},{"id":91843745,"identity":"2fb01a61-0507-4caf-b733-873168862cc2","added_by":"auto","created_at":"2025-09-22 10:07:26","extension":"png","order_by":10,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":491621,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/3de4c12e3e8ce96f47fea625.png"},{"id":91843734,"identity":"15e9df7a-bcb3-4909-a714-414b448f7efe","added_by":"auto","created_at":"2025-09-22 10:07:25","extension":"png","order_by":11,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":491621,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/924e1d9ef10325d9172754bb.png"},{"id":91846010,"identity":"14108d57-bb03-413a-afd0-6c03f7de7fe6","added_by":"auto","created_at":"2025-09-22 10:15:26","extension":"jpeg","order_by":12,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":337381,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage4.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/2e6c82f6c30cdc24771303b1.jpeg"},{"id":91843737,"identity":"4fdec49c-f196-4e8f-9884-2b330a5ebb21","added_by":"auto","created_at":"2025-09-22 10:07:25","extension":"png","order_by":13,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":127903,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/fac4c10cfbdf34421752693a.png"},{"id":91843732,"identity":"da740381-a149-4795-9c5a-3a176dd0e40c","added_by":"auto","created_at":"2025-09-22 10:07:25","extension":"png","order_by":14,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":151457,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/7d8bacfa22af17143ef698d6.png"},{"id":91843758,"identity":"7096f2d4-8e2d-4633-a2af-b4a9033e7160","added_by":"auto","created_at":"2025-09-22 10:07:27","extension":"png","order_by":15,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":295848,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage7.png","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/6d23ebf8d9b5da26159b72d7.png"},{"id":91843743,"identity":"1eada6ec-3111-4bc4-8cbc-9925ffd88a61","added_by":"auto","created_at":"2025-09-22 10:07:26","extension":"jpeg","order_by":16,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":398432,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage8.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/499e3787844ff7bbbc9bc041.jpeg"},{"id":91847566,"identity":"09d3d68b-74d9-4918-a4db-e374ced3357d","added_by":"auto","created_at":"2025-09-22 10:23:26","extension":"jpeg","order_by":17,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":834793,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage9.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/31312ac38dd2923e9f3aafa2.jpeg"},{"id":91843744,"identity":"7dbbd217-9c02-4e91-b685-740036f351f6","added_by":"auto","created_at":"2025-09-22 10:07:26","extension":"png","order_by":18,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":32734,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/ee480e5e8d0a513873a78c39.png"},{"id":91847565,"identity":"c724acd2-b16c-43f3-a7ba-a57d86324885","added_by":"auto","created_at":"2025-09-22 10:23:26","extension":"png","order_by":19,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":142373,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/8b52abbacd7affad9b07935d.png"},{"id":91846005,"identity":"17513323-17dc-42fb-86fb-b2833e61b75a","added_by":"auto","created_at":"2025-09-22 10:15:26","extension":"png","order_by":20,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":142373,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/7653188e7865162a1f0626af.png"},{"id":91846003,"identity":"d57f833a-090c-4b49-964c-afcdd16b38a2","added_by":"auto","created_at":"2025-09-22 10:15:25","extension":"png","order_by":21,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":76354,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/c344d40f6b20a2c84a92f6c2.png"},{"id":91847567,"identity":"674ddae8-98f6-4777-a85a-5061ae147233","added_by":"auto","created_at":"2025-09-22 10:23:26","extension":"png","order_by":22,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":36899,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/05687b5f439e5e267e2b2e7b.png"},{"id":91846008,"identity":"d3fc740e-47a2-4bb4-a397-53b1146e8c31","added_by":"auto","created_at":"2025-09-22 10:15:26","extension":"png","order_by":23,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":43433,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/d8323e88e8d336d28e0be661.png"},{"id":91843752,"identity":"c2c2823f-0095-4d93-b490-63aaf61e7ab9","added_by":"auto","created_at":"2025-09-22 10:07:26","extension":"png","order_by":24,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":92719,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage7.png","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/bdc4aa365db5b8ba69ff9b31.png"},{"id":91843738,"identity":"c01994a8-fc37-4de2-8ba8-38419b0dafaf","added_by":"auto","created_at":"2025-09-22 10:07:26","extension":"png","order_by":25,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":67114,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage8.png","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/3d4abb3c03090582add2882c.png"},{"id":91843740,"identity":"40c92a61-3353-4c9d-bf32-1700fc44524a","added_by":"auto","created_at":"2025-09-22 10:07:26","extension":"png","order_by":26,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":136971,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage9.png","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/f81bba36cc5b7bc58c913fe9.png"},{"id":91846012,"identity":"df110243-9322-4b68-ad7c-f9ed896f42e5","added_by":"auto","created_at":"2025-09-22 10:15:26","extension":"xml","order_by":27,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":294493,"visible":true,"origin":"","legend":"","description":"","filename":"57a8b1af71144033804bb4c9933ae1f01structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/e10857f04919697b00f2f2c7.xml"},{"id":91843749,"identity":"a483d4bd-27fc-4f1f-9650-a634ccee15d4","added_by":"auto","created_at":"2025-09-22 10:07:26","extension":"html","order_by":28,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":309724,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/476abf3123b5fca1ba946df7.html"},{"id":91845988,"identity":"3f266ae1-faa2-4e67-a01e-0621c20b4bef","added_by":"auto","created_at":"2025-09-22 10:15:25","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":300979,"visible":true,"origin":"","legend":"\u003cp\u003eA) pQSAR is a 2-level stacked model. In level-1, ST-RFR models are trained on available data describing compounds by Morgan 2 substructure fingerprints. These models are used to impute initial missing values. For level-2, PLS models are trained separately for each assay, now describing each compound by its profile of predicted and experimental activities. B) Alchemite is a multiple imputation algorithm that takes all available information about compounds (chemical descriptors and any extant assay information) and simultaneously imputes missing assay information. C) MetaNN is a meta learner that trains a shared model for all assays, then retrains a new model for each assay. D) MT-DNN is an advanced multitask neural network that incorporates shared layers (hatched rectangles) and task-specific layers (dashed rectangles) during model training (in this research, the task-specific layers were omitted). E) Macau is a Bayesian matrix factorization technique that integrates side information and leverages probabilistic inference to improve predictive accuracy in matrix completion tasks.\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/f45e03b1258dc94eac862cf5.png"},{"id":91847564,"identity":"203126f5-b1c3-497f-8055-82a9213b69a1","added_by":"auto","created_at":"2025-09-22 10:23:25","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":654271,"visible":true,"origin":"","legend":"\u003cp\u003eThe realistic split was shown to mirror the novelty of compounds really ordered by projects from virtual screens\u003csup\u003e6\u003c/sup\u003e—hence the name. Note that although these are multitask models, each assay has experimental measurements for different compounds, so each assay has a unique training/test set split. The details, justification and code were previously published. \u003csup\u003e1,6\u003c/sup\u003e\u003cstrong\u003e \u003c/strong\u003eBriefly, the compounds are first clustered. The models are trained on 75% of the data from the largest clusters and tested on the remaining singletons and small clusters.\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/107053231cdbf1c5e1bd79a8.png"},{"id":91843731,"identity":"9c76f221-3223-411c-9a61-aff273abe5c7","added_by":"auto","created_at":"2025-09-22 10:07:25","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":654271,"visible":true,"origin":"","legend":"\u003cp\u003eFigure 2. Random and realistic training/test-set splits for a single assay. A) Red stars represent 25% random test set. Note that every test-set member has a nearby training set member (blue circles) B) Red stars are the 25% “Realistic” test set. I.e., for each assay, models are trained on the 75% of compounds from the largest chemical clusters and tested on the remaining 25% from singletons and small clusters. This split was shown to mirror the extreme novelty of compounds ordered in real virtual screens.\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/ea534917bd7e0d2710533c32.png"},{"id":91845991,"identity":"dc71ab9e-5eaa-4554-b681-eedd0025377d","added_by":"auto","created_at":"2025-09-22 10:15:25","extension":"jpeg","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":337381,"visible":true,"origin":"","legend":"\u003cp\u003eFigure 3. Training-set collections for (A) “all-out”, (B) “subset-out”, (C) “one-out” and (D) final production “none-out” models for assay A1, plus other assays trained along with A1, together called “co-A1” assays. Red assay headers indicate the co-A1 assays. Red markers (beneath red headers) are test-set measurements held-out during co-A1 training. Green markers (both light and dark) are co-A1 training measurements. Dark-green “all-train” markers are measurements from the combined 75% of “realistic” training sets of all individual assays, which are thus in all training set collections. Light-green “some-train” markers are from the realistic test sets of the “co-A1 support” assays (black column headers) that are not modeled along with A1, and can thus be included in that co-A1 training-set collection without test-set leakage. Note that predictions for \u003cu\u003eA1\u003c/u\u003eon compounds C4 and C7 both represent cold-start predictions for the all-out model, whereas only C5 is an imputation in all-out for A1, supported by an all-train (dark green) measurement from assay A3. (Other compounds are not in the A1 test set.) However, for the A1 one-out models, while C7 is still cold-start, C4 is now an imputation, because it is now supported by the A3 some-train (light green) measurement. All algorithms trained all-out and subset-out models. A smaller number also trained some one-out models.\u003c/p\u003e","description":"","filename":"floatimage4.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/0adff2f6546e9e749bca8287.jpeg"},{"id":91847563,"identity":"702a64b8-67d3-47d2-bf0e-c3a711c23a0a","added_by":"auto","created_at":"2025-09-22 10:23:25","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":127903,"visible":true,"origin":"","legend":"\u003cp\u003eFigure 4. Results of the statistical significance test for difference in signed-r\u003csup\u003e2\u003c/sup\u003e between algorithms. Each matrix is for a different training-set collection and assay collection with the color of the cell based on the multiple hypothesis test corrected p-values. Values are the median difference (row – column) in signed-r\u003csup\u003e2\u003c/sup\u003e. The full list of p-values is in Supplementary File tab_analysis1_full.\u003c/p\u003e","description":"","filename":"floatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/bd82846fe83a9ab18097bed3.png"},{"id":91846002,"identity":"0dd9045e-dc1a-4771-bd04-b18516d1b76c","added_by":"auto","created_at":"2025-09-22 10:15:25","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":151457,"visible":true,"origin":"","legend":"\u003cp\u003eFigure 5. Statistical significance tests comparing algorithms for fraction of models with signed-r\u003csup\u003e2\u003c/sup\u003e\u0026gt;0.3. Each matrix is for a different training-set collection and assay collection with the color of the cell based on the multiple hypothesis test corrected p-values. The value in the cell is the difference in the percentage of test-set assays for which the model produced signed-r\u003csup\u003e2\u003c/sup\u003e\u0026gt;0.3. The full list of p- values is in Supplementary file tab_analysis2_pvalues.\u003c/p\u003e","description":"","filename":"floatimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/b740a06cd8e5759d16303161.png"},{"id":91843739,"identity":"0438cd11-a64f-4c59-9679-a9cac2f6bbc4","added_by":"auto","created_at":"2025-09-22 10:07:26","extension":"png","order_by":7,"title":"Figure 7","display":"","copyAsset":false,"role":"figure","size":295848,"visible":true,"origin":"","legend":"\u003cp\u003eFigure 6. Signed-r\u003csup\u003e2\u003c/sup\u003e for one-out vs. all-out models for 5 MMRMs for the 159 kinase assays.\u003c/p\u003e","description":"","filename":"floatimage7.png","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/9e5311b515fddb85ef44c0bc.png"},{"id":91846006,"identity":"e47a3931-9215-43e8-9e2f-bbf7236d0e84","added_by":"auto","created_at":"2025-09-22 10:15:26","extension":"jpeg","order_by":8,"title":"Figure 8","display":"","copyAsset":false,"role":"figure","size":398432,"visible":true,"origin":"","legend":"\u003cp\u003eFigure 7. Hexbin plot of signed-r\u003csup\u003e2\u003c/sup\u003e for one-out vs. all-out pQSAR models for the 4276 diverse assays WHERE THE darker THE HEXAGON the more ASSAYS IN THAT VOLUME.\u003c/p\u003e","description":"","filename":"floatimage8.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/12c272a2118cacbc371a8123.jpeg"},{"id":91843733,"identity":"a1d8a0cf-626f-4c1b-8d54-680a7accb269","added_by":"auto","created_at":"2025-09-22 10:07:25","extension":"jpeg","order_by":9,"title":"Figure 9","display":"","copyAsset":false,"role":"figure","size":834793,"visible":true,"origin":"","legend":"\u003cp\u003eFigure 8 For each assay collection, impact of the number of supporting measurements required for the kinase and diverse assay-collections. Line plot is the per-compound difference between MMRM and ST-RFR prediction absolute errors, averaged across all assays, binned by minimum number of supporting measurements. Histogram underneath is the proportion of measurements in that bin.\u003c/p\u003e","description":"","filename":"floatimage9.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/f273483aaf16bafb4740a201.jpeg"},{"id":102234110,"identity":"8e46d1e7-6ace-4750-8666-477c19f79ccb","added_by":"auto","created_at":"2026-02-09 16:06:31","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":4510679,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/f64c858d-de75-4a09-a842-606718cfd681.pdf"},{"id":91843703,"identity":"aba6cbd5-b6b9-4a60-b5ba-40246e0b3bb5","added_by":"auto","created_at":"2025-09-22 10:07:25","extension":"txt","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":1088,"visible":true,"origin":"","legend":"","description":"","filename":"tabsignedr2goodenough.csv.txt","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/1d9beeeb178b695a544feeec.txt"},{"id":91843757,"identity":"54b270f5-24c1-4cbb-b6a2-6a38fbc7b1a6","added_by":"auto","created_at":"2025-09-22 10:07:26","extension":"txt","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":1695,"visible":true,"origin":"","legend":"","description":"","filename":"tabmediansignedr2.csv.txt","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/3fa536b603ae393ca749042f.txt"},{"id":91843705,"identity":"94f97c98-6fe0-4ed5-bcbd-c28ab8f0ad80","added_by":"auto","created_at":"2025-09-22 10:07:25","extension":"txt","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":7963,"visible":true,"origin":"","legend":"","description":"","filename":"tabanalysis2tables.csv.txt","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/63d7e3e83088cddd98230887.txt"},{"id":91843708,"identity":"7760eeee-1444-470d-89e4-aa0ae8842a57","added_by":"auto","created_at":"2025-09-22 10:07:25","extension":"txt","order_by":4,"title":"","display":"","copyAsset":false,"role":"supplement","size":10428,"visible":true,"origin":"","legend":"","description":"","filename":"tabanalysis2pvalues.csv.txt","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/53422b694392e033661ee9e3.txt"},{"id":91845994,"identity":"060a6c9d-8a3f-4b8f-b898-91d2b4ba8ccf","added_by":"auto","created_at":"2025-09-22 10:15:25","extension":"txt","order_by":5,"title":"","display":"","copyAsset":false,"role":"supplement","size":15579,"visible":true,"origin":"","legend":"","description":"","filename":"tabanalysis1full.csv.txt","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/1219a639dedcd02a6f4ac8c6.txt"},{"id":91846004,"identity":"54a4c37a-71b3-45d6-963a-9a32bf1a5d9d","added_by":"auto","created_at":"2025-09-22 10:15:26","extension":"txt","order_by":6,"title":"","display":"","copyAsset":false,"role":"supplement","size":36994,"visible":true,"origin":"","legend":"","description":"","filename":"cvstables.txt","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/471056fa1324299796900181.txt"},{"id":91843723,"identity":"b9346d73-c930-4423-bbd3-1f97091fe810","added_by":"auto","created_at":"2025-09-22 10:07:25","extension":"docx","order_by":7,"title":"","display":"","copyAsset":false,"role":"supplement","size":14861,"visible":true,"origin":"","legend":"","description":"","filename":"Listofsupplementalmaterials.docx","url":"https://assets-eu.researchsquare.com/files/rs-7482715/v1/7fe14ea09ba37a3d9a804fb7.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Comparing Massively-Multitask Regression Algorithms for Drug Discovery","fulltext":[{"header":"Introduction","content":"\u003cp\u003eQuantitative structure-activity relationships (QSAR), invented by Hansch and Fujita in 1963,\u003csup\u003e\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e,\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e was based on the earlier linear free energy relationships (LFER) of Hammet, first published in 1937.\u003csup\u003e\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u003c/sup\u003e Despite reporting very high correlations with experimental activity, the impact of Hansch analysis on drug discovery was minimal for several reasons: reported correlations were fits, not predictions; models were only useful for congeneric series; descriptors were substituent constants that required experimental determination; model applicability domains were too narrow for virtual screening, mechanism-of-action (MoA) determination, hit-list triaging or off-target prediction; models were simple linear, quadratic or bilinear least-squares regression; and too few observations for the number of descriptors, along with the lack of multiple-hypothesis adjustment when many descriptors were tested, led to overfitting. Even when held-out test sets were eventually employed, they were leave-one-out or random test sets that only tested a much narrower applicability domain than typical real-life use-cases.\u003c/p\u003e\u003cp\u003eOver time, these problems were solved. Despite the many decades-long history of LFER and QSAR, the last obstacle to practical use, applicability domain, fell only recently. Evaluating model accuracy across a relevant applicability domain was addressed by designing held-out test sets wherein the test compounds are realistically novel with respect to the training set. Expanding the applicability domain was first addressed by proteochemometric modeling \u003csup\u003e\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u003c/sup\u003e, which included both protein and compound descriptors and trained a single large model across many protein targets. Because it trains on and predicts target-compound pairs, the number of compounds and measurements informing the models is much larger, greatly enhancing the applicability domain. It is limited, however, by not differentiating between different assays for the same protein target, which can vary widely for legitimate biological reasons.\u003csup\u003e\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u003c/sup\u003e It also cannot effectively include models for phenotypic screens, physicochemical properties, pharmacokinetic properties, or other assay types. Massively-multitask regression models (MMRMs) were subsequently developed. These algorithms train models collectively on up to many thousands of endpoints of any type. In drug discovery, they can combine models for biochemical, cellular, ADME (absorption, distribution, metabolism, excretion), physicochemical, binding or \u003cem\u003ein vivo\u003c/em\u003e assays, greatly expanding the chemical matter and assay data informing all the models. They do not combine assays for the same target, but produce separate models for each assay, including separating activation vs. inhibition, yet all other assays for that target still inform the models with optimal scaling (along with all other assays in the multitask assay collection).\u003c/p\u003e\u003cp\u003eIn multitask models, each measurement and each prediction are for a compound-assay pair. The training matrix for multitask models covering thousands to millions of compounds across hundreds to thousands of assays is generally very sparse, sometimes as much as 99.9% missing values. An important application of MMRMs is “imputing” those missing values, i.e. predicting activities on the full set of assays for compounds already measured for a few of the assays and used in training the MMRM. This distinguishes them from single-task models, where predictions are always on a single assay for new compounds not used to train the model. Thus, in evaluating single-task models a compound in the training set is never in the test set. In evaluating multi-task models, where a compound is associated with many assays, compounds in the test set for one assay are frequently in the training sets for other assays. It was previously reported that the accuracy of compound-target pair predictions in multitask models is heavily influenced by whether the compounds were part of the training data in other assays (i.e. imputations).\u003csup\u003e\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e\u003c/sup\u003e The accuracy of predictions for entirely new compounds, known as “cold-start” predictions, can be significantly worse. One key objective of this study was to compare the accuracy of cold-start predictions to imputations, including the number of supporting assays, and how that affects the uses for which MMRMs are suited.\u003c/p\u003e\u003cp\u003eThe much higher accuracy and wider applicability domain of MMRMs than corresponding single-assay models greatly improves virtual screening (including virtual counter-screens for artifacts or selectivity). The near-quantitative predictions, rather than mere classification, allow sophisticated hit triaging based on predicted potency, ligand efficiency, lipophilic efficiency, selectivity, etc. Most importantly, having 1000s of semiquantitative models for each compound enables previously unavailable predictions, such as discovering mechanisms of action (MoA) for phenotypic screens, promiscuity, off-target activity to identify potential toxicities, identification of polypharmacology or drug repurposing. A recent study included comparison of several multitask kinase models.\u003csup\u003e\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u003c/sup\u003e It differs from this work in that it compares classification models, not regression models. Unlike this study, it uses random tests sets, assumes all assays for the same kinase are equivalent, does not distinguish cold-starts from imputations and does no significance testing. Another recent MMRM study trained on a large ChEMBL dataset and showed good performance on 6 datasets.\u003csup\u003e\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e\u003c/sup\u003e This article compares the MMRMs for 6 algorithms: Profile-QSAR (pQSAR), Alchemite, a meta learner (MetaNN), a multitask feed-forward neural network (MT-DNN), Bayesian matrix factorization (Macau) and inductive matrix completion (IMC). Each model uses the exact same data, including the same “realistic” training/test set splits, and each is trained by an expert, in many cases an author, of the algorithm.\u003c/p\u003e\u003cp\u003eBrief description of MMRM algorithms:\u003c/p\u003e\u003cp\u003epQSAR:\u003c/p\u003e\u003cp\u003eProfile-QSAR (pQSAR), first presented in 2006, was the first general MMRM developed for drug discovery.\u003csup\u003e\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u003c/sup\u003e At that time, pQSAR included models for only 115 Novartis biochemical and cellular kinase assays.\u003csup\u003e\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e Furthermore, it was only tested on random held-out test sets, evaluating an unrealistically small applicability domain. By 2019 pQSAR had expanded to 11,805 Novartis biochemical, cellular and ADME assays covering all protein families, of which 8558 (72%) achieved average accuracy comparable to 4-concentration experimental IC\u003csub\u003e50\u003c/sub\u003es using the “realistic” test set that evaluated a realistically broad applicability domain. Y-scrambling showed a chance-correlation rate of only 0.2% on the 11,805 Novartis assays, and 1.5% on the 4276 assays in this study—the higher rate for ChEMBL likely due to smaller assay sizes.\u003csup\u003e\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e,\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e,\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u003c/sup\u003e pQSAR also enables federated models, where innocuous partial models are shared among collaborating companies without exposing compounds, targets or bioactivities. With these, each partner can build pQSAR models for their internal assays virtually identical to the models they would build if all the data were actually shared.\u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e\u003cp\u003eVery briefly, the pQSAR algorithm is a multi-task, 2-level, stacked model as illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003eA. For level-1, conventional single-task random-forest regression (ST-RFR) models are built on the available experimental data for each individual assay, describing the compounds by Morgan 2 substructural fingerprints. For level-2, a separate PLS model is then built for each assay, one-at-a-time, now describing each compound not by its chemical structure, but by the profile of predicted (or experimental) bioactivities from all the other ST-RFR models besides the one being currently trained—hence the name “Profile-QSAR.”\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eAlchemite:\u003c/p\u003e\u003cp\u003eAlchemite is a commercial machine learning software tool originally developed for use in materials design and discovery\u003csup\u003e\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e,\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u003c/sup\u003e and more recently applied in catalyst\u003csup\u003e\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e\u003c/sup\u003e and drug\u003csup\u003e\u003cspan additionalcitationids=\"CR18 CR19\" citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e–\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e\u003c/sup\u003e discovery among other areas. The general structure of Alchemite is described elsewhere,\u003csup\u003e\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e\u003c/sup\u003e but in brief, it is a multiple imputation approach, where machine learning models of each assay are trained using all other assay and chemical descriptor information; the values missing from each assay are imputed using the appropriate model, all simultaneously (as opposed to sequentially as in some other multiple imputation approaches); and the process is repeated, now utilizing imputed information for the input assays as well as experimental data where available (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003eB). Automated variable selection is carried out to ensure only those assays with high enough correlation and overlap of available data with the target assay are used as input variables at each stage. Typically, two or three cycles of imputation are carried out, as selected through cross-validation within the training set; in the models below two cycles were used.\u003c/p\u003e\u003cp\u003eBeyond making predictions for individual compound-assay pairs, Alchemite also uses bootstrap samples of the training data to generate uncertainty estimates for its predictions. In industrial applications uncertainty quantification is vital to enable confident prioritization of compounds predicted to be performant against desired criteria: it is often a better use of limited resources to experimentally validate a suggestion that is predicted with high confidence to marginally achieve the target criteria rather than a suggestion that has a slightly improved expected performance but much increased uncertainty, as the former suggestion will have a higher overall probability of achieving success.\u003c/p\u003e\u003cp\u003eBeyond making predictions with their corresponding uncertainties, the full Alchemite suite also includes packages for adaptive experimental design, interpretability,\u003csup\u003e\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e\u003c/sup\u003e and cold-start prediction, all within a no-code web-based graphical user interface.\u003c/p\u003e\u003cp\u003eMetaNN:\u003c/p\u003e\u003cp\u003eMetaNN was developed in 2020 and is inspired by the success of gradient-based meta-learning, which uses meta-knowledge learned from previous tasks to facilitate learning for new tasks. MetaNN is designed as a meta-learner that trains a deep neural network for each individual assay, initializing it from a well-generalized consensus DNN optimized across all assays. This algorithm creates task-specific neural networks while requiring minimal learning for each individual task, thus minimizing the risk of overfitting due to limited training data. Figure\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003eC shows an overview of the workflow of gradient-based meta-learning algorithms for predicting bioactivity: we first train a neural network with weight initialization shared across all source assays; then for a source assay, the initialized neural network based on the bioactivity values of its training set compounds is established; then the adapted neural networks on the test set compounds of each source assay are evaluated, and the error rates of all source assays are averaged and backpropagated to improve the generalization ability of the initialization; finally the learned initialization is transferred to facilitate training for each target assay dataset.\u003c/p\u003e\u003cp\u003eMeta-learning has been proven to be a powerful paradigm for transferring knowledge from previous tasks to facilitate learning for new tasks while reducing the risk of overfitting. The key is to optimize the generalization ability of the initialization, which is measured by the performance of the adapted model on each task's query set. Unfortunately, this generalization metric may cause the initialization to overfit on meta-training tasks, severely impairing the ability to generalize and adapt to new tasks. To develop more flexible and powerful algorithm, MetaNN uses task augmentation to increase the dependence of target predictions on the support set and provide additional knowledge to optimize model initialization. In the MetaNN algorithm, specifically, we use MetaMix, which linearly combines features and labels of samples from both the support and query sets and thus can actively use \"more data\" to augment meta-training tasks when evaluating generalization.\u003c/p\u003e\n\u003ch3\u003eMT-DNN:\u003c/h3\u003e\n\u003cp\u003eA Multitask Deep Neural Network (MT-DNN) is an extension of the traditional Deep Neural Network (DNN) architecture, designed to handle multiple learning tasks concurrently. While the conceptual foundation for neural networks was laid in the mid-20th century, significant advancements in computational power, algorithmic development, and data availability in recent decades have enabled the creation of more sophisticated systems like MT-DNNs.\u003c/p\u003e\u003cp\u003eMT-DNNs stand out for their ability to process multiple learning tasks concurrently, utilizing shared representations to both generalize and specialize where necessary. This multitasking capability is especially beneficial in scenarios where individual tasks may have limited data, as the shared structure allows for knowledge transfer and regularization across tasks.\u003c/p\u003e\u003cp\u003eArchitecturally, MT-DNNs share the layered structure of DNNs, comprising input, hidden, and output layers. However, the feature of MT-DNN lies in the organization of these hidden layers: hidden layers can be divided into shared layers for extracting common features across tasks and task-specific layers for improving each task individually. Initially, the network leverages shared layers to learn general features relevant to all tasks. Then, task-specific layers focus on refining this knowledge to address the nuances of each distinct task. This layered organization is adaptable to align with the specific tasks, the nature of the data, and the targeted goals, allowing for a customizable architecture that may, for instance, vary the incorporation or extent of task-specific layers to optimize learning and outcomes. In this research, only shared layers were utilized, emphasizing the extraction and use of common features across multiple tasks without incorporating task-specific layers (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003eD). This nuanced architecture allows MT-DNNs to effectively process and learn from diverse datasets across multiple tasks, representing a significant evolution in the capabilities of neural network models.\u003c/p\u003e\u003cp\u003eMacau:\u003c/p\u003e\u003cp\u003eThe Macau approach, introduced in 2015,\u003csup\u003e\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e represents a significant advancement in matrix factorization techniques, specifically tailored for handling heterogeneous and sparse datasets. Demonstrating its efficacy in recommender systems, particularly with the MovieLens dataset, Macau has also been applied to more complex domains. Its application in biochemistry for tasks such as drug-protein activity prediction underscores its versatility beyond simple recommender system approaches. This versatility is further evidenced by its ability to handle heterogeneous data and incorporate side information across various fields, which is especially valuable in scientific and research-oriented applications. In 2017,\u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e a substantial improvement to the Macau method was introduced, addressing computational limitations.\u003c/p\u003e\u003cp\u003eAt its core, Macau, illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003eE, utilizes an advanced Bayesian matrix factorization technique combined with integrating high-dimensional side information. This Bayesian probabilistic approach treats matrix factorization as a probabilistic model, where priors are assigned to the parameters. During factorization, the posterior distributions of these parameters are updated based on observed data. Macau utilizes Gibbs sampling, a Markov Chain Monte Carlo method, for sampling from these posterior distributions. It allows Macau to navigate the probable parameter space effectively, providing robust predictions while managing the uncertainties inherent in the data and model. A notable advancement in Macau's methodology, highlighted in the 2017 update, is the adoption of Krylov methods and a new prior for the link matrix, designed to scale with the magnitude of latent variables. These enhancements enabled Macau to boost its capability to analyze large and complex datasets significantly.\u003c/p\u003e\u003cp\u003eInductive Matrix Completion (IMC):\u003c/p\u003e\u003cp\u003eInductive Matrix Completion (IMC) is a paradigm often used as a benchmark or baseline when comparing different matrix factorization models. IMC revolutionized traditional matrix completion methods by incorporating auxiliary data (like compound and target features) into the process. Pioneered by Prateek Jain and Inderjit S. Dhillon in the work \"Provable Inductive Matrix Completion\",\u003csup\u003e\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e\u003c/sup\u003e IMC was developed to overcome the shortcomings of conventional matrix completion techniques. Since its introduction, IMC has inspired a wealth of subsequent studies across various domains, establishing it as a valuable methodology for leveraging side information in predictive modeling tasks where matrix completion is applicable. Beyond its initial use in recommender systems, IMC has found widespread application in various domains, including genomics for predicting gene-disease associations, text mining, and social network analysis.\u003c/p\u003e\u003cp\u003eIMC algorithm involves two main steps: the factorization of the observed matrix into latent factors and the utilization of side information through matrix multiplication. Specifically, an observed matrix \u003cb\u003eW\u003c/b\u003e can be approximated as the product of two low-rank matrices of latent features, \u003cb\u003eU\u003c/b\u003e and \u003cb\u003eV\u003c/b\u003e, further multiplied by additional matrices \u003cb\u003eX\u003c/b\u003e and \u003cb\u003eY\u003c/b\u003e, which contain the side information. This process can be formalized as \u003cb\u003eW ≈ X\u003c/b\u003e\u003csup\u003e\u003cb\u003eT\u003c/b\u003e\u003c/sup\u003e\u003cb\u003eUVY\u003c/b\u003eFigure 1. The optimization focuses on minimizing the discrepancy between the observed and predicted entries by iteratively refining the latent factors \u003cb\u003eU\u003c/b\u003e and \u003cb\u003eV\u003c/b\u003e, while keeping the side information constant.\u003c/p\u003e\n\n"},{"header":"Methods","content":"\u003cp\u003eChEMBL assay collections:\u003c/p\u003e\n\u003cp\u003eThe models were compared on 2 previously published ChEMBL bioactivity data sets: a relatively homogeneous set of 159 kinase dose-response assays from ChEMBL version 20 downloaded on July 6, 2015, and a heterogeneous set of 4276 diverse dose-response assays from ChEMBL version 24.1 downloaded on June 22, 2018. The exact compounds, assays, experimental and pQSAR predicted bioactivities and training/test set splits were in the Supporting Information from 2 papers.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e1\u003c/span\u003e,\u003cspan class=\"CitationRef\"\u003e6\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003eTraining/test set splits:\u003c/p\u003e\n\u003cp\u003eWe use the \u0026ldquo;realistic\u0026rdquo; training/test-set split. It is compared to a random split in Fig. \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e Briefly, the compounds are first clustered. The models are trained on 75% of the data from the largest clusters and tested on the remaining singletons and small clusters.\u003c/p\u003e\n\u003cp\u003eTraining-set collections:\u003c/p\u003e\n\u003cp\u003eTraining-set collections are only a concern for multitask models. After the individual assays were each split into training and test sets, three training-set (and corresponding test-set) collections were assembled. These collections group together training and test sets for some of the individual assays into training collections and reserve the remaining test sets for the test-set collections. For models trained together, all test-set measurements for those assays must be held out to avoid test-set leakage. Note that the final production models for actual use are trained on 100% of the data, with no held-out measurements. Differences in training\u003cem\u003e-\u003c/em\u003eset collections thus do not affect final model quality, only the \u003cem\u003eevaluation\u003c/em\u003e of final model quality. The training-set collection affects how similar the evaluation models are to the final production models for an intended use case, here virtual screening of the compound archive used to train the overall multitask models (see below).\u003c/p\u003e\n\u003cp\u003eThe three training-set (and corresponding test-set) collections were named by how many assays\u0026rsquo; test-sets must be held out when assembling the corresponding training-set collections for the multitask models. These are named \u0026ldquo;all-out\u0026rdquo;, \u0026ldquo;subset-out\u0026rdquo; and \u0026ldquo;one-out\u0026rdquo; as described below. Models and training set collections are similarly named by the test-set collections held out. Tables of colored measurements for each compound (rows) and assay (columns) in Fig. \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e illustrate the three training-set collections for assay A1, and any additional assays co-trained simultaneously with A1, together named \u0026ldquo;co-A1\u0026rdquo; assays (indicated with red column headers). Red markers (beneath red headers) are test-set measurements held-out during co-A1 training. Green markers (both light and dark) are co-A1 training measurements. Dark green distinguishes the 75% of \u0026ldquo;All-train\u0026rdquo; measurements, assembled from the combined realistic training sets for all individual assays, which are included in \u003cem\u003eall\u003c/em\u003e training-set collections. Light green indicates \u0026ldquo;Some-train\u0026rdquo; measurements, from the test sets of \u0026ldquo;co-A1 support assays\u0026rdquo; (black column headers) not trained with A1, which thus can be included in the training of co-A1 assays. For each collection, red and light-green markers combined comprise the 25% of compounds from the combined realistic test-sets of all assays. Thus, looking left to right, red markers are replaced with light-green markers as fewer models are trained together. As the co-A1 set is thus reduced, ever fewer test-set measurements need be held out.\u003c/p\u003e\n\u003cul\u003e\n \u003cli\u003e\n \u003cp\u003eAll-out: Methods that train models for all assays simultaneously must leave out the 25% test-set bioactivities for all assays during training as shown in Fig. \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003eA. In this case all assays are in the co-A1 set, and only the 75% of all-train measurements from combining all individual training-set assays (dark green) can be used for training, with no additional (light green) co-A1 support test-set measurements. The advantage of all-out models is that they include all assays while only training one MMRM regardless of algorithm, so all algorithms could train models for all assays, even in the large diverse assay collection. The limitation is that they are least like the \u0026ldquo;none-out\u0026rdquo; production models trained on 100% the data. Trained just on the large clusters for each assay, these all-train training-set collections contain the most redundant information and lack any of the more informative measurements from singletons and small clusters. They also contain the fewest imputations. Note that in Fig. \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003eA, A1 predictions on C4 and C7 represent cold-start predictions that have no supporting green measurements in assays A2 \u0026hellip; An. C5 predictions are imputations, supported by a (necessarily dark green) measurement from A3.\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003eOne-out: Illustrated in Fig. \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003eC, methods that train separate models for each assay one-at-a-time can leave out just the test-set data for that assay\u0026rsquo;s model, while training on both test- and training-set measurements from all other assays. I.e. the co-A1 assay-set consists only of A1 itself, so test-set measurements from assays A2 \u0026hellip; An can all be included in training, as they will not leak into the A1 model. Since 99+% of the measurements are included in each one-out model\u0026rsquo;s training set, these models best reflect the final, production models trained on 100% of the data. Any method can train one-out models, but algorithms that train all the assays simultaneously must build 159 or 4276 separate MMRMs, leaving out just the test set for each assay in turn, and only testing predictions for that assay\u0026mdash;a computationally onerous task. Thus, not all algorithms participated in one or both one-out comparisons. Note that for assay A1, test set compound C7 is still a \u0026ldquo;cold-start\u0026rdquo; prediction, but unlike the all-out case, C4 is now an imputation, because it is supported by a (light green) some-train measurement on A3. Having fewer cold-starts than the all-out models is presumably an important reason for the better performance of one-out models (see below). If there are 10 assays in the multitask model, one-out models are trained on ~\u0026thinsp;90% of the data, 100 assays use 99%, 1000 assays 99.9%, the difference between final production and one-out decreasing as the number of assays in the multitask model increases.\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003eSubset-out: All methods could participate in a compromise comparison, the subset-out models, shown in Fig. \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003eB. These used full-factorial design, based on assay properties, to sample an assay subset from each of the 2 assay collections. 48 assays were selected from the 4276 diverse assays using 4 properties: compound count (\u0026thinsp;\u0026lt;\u0026thinsp;=\u0026thinsp;100 and \u0026gt;\u0026thinsp;100), standard deviation of pIC\u003csub\u003e50\u003c/sub\u003es (\u0026lt;\u0026thinsp;1 and \u0026gt;\u0026thinsp;=\u0026thinsp;1), pQSAR signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e (\u0026lt;\u0026thinsp;0.4, 0.4\u0026thinsp;\u0026lt;\u0026thinsp;signed-r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;\u0026lt;\u0026thinsp;0.6 and \u0026gt;\u0026thinsp;0.6) and family (kinase, GPCR, phenotypic and other). 12 assays were similarly selected from the 159 kinase assays based on 3 properties: compound count (\u0026thinsp;\u0026lt;\u0026thinsp;=\u0026thinsp;680 and \u0026gt;\u0026thinsp;680), standard deviation of pIC\u003csub\u003e50\u003c/sub\u003es (\u0026lt;\u0026thinsp;1 and \u0026gt;\u0026thinsp;=\u0026thinsp;1) and pQSAR signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e (\u0026lt;\u0026thinsp;0.5, 0.5\u0026thinsp;\u0026lt;\u0026thinsp;signed-r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;\u0026lt;\u0026thinsp;0.65 and \u0026gt;\u0026thinsp;0.65). The assay nearest the centroid of each cell was selected for modeling. MMRM evaluation models were trained for just these subsets of 12 or 48 assays. Model training thus included only realistic training-set data (all-train dark-green markers) for these assays, but both realistic training-set data (dark-green markers) and realistic test-set data (some-train, light-green markers) for the remaining support assays, leaving out just the test-set bioactivities for these 12 or 48 assays (held-out, red markers). Because this was a single training, all methods could participate fully. Model quality, however, could only be evaluated for these 12 or 48 assay subsets. Subset-out models generally have more cold-start predictions than one-out models, but fewer than all-out models, although it depends on the particular subset.\u003c/p\u003e\n \u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eOne study objective was to compare cold-start predictions to imputations. To guarantee a minimum of 10 cold-start predictions per assay, we selectively removed some bioactivity data from the training set collection of the 12 kinase subset assays. The following was applied for each of those 12 assays:\u003c/p\u003e\n\u003col\u003e\n \u003cli\u003eCount the cold-start test-set compounds (n).\u003c/li\u003e\n \u003cli\u003eIf n\u0026lt;10, count the IC\u003csub\u003e50\u003c/sub\u003es for each test-set compound in the subset-out kinase training-set collection. Select the 10-n compounds with the fewest subset-out training-set measurements.\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eRemove those compound\u0026rsquo;s measurements from the subset-out training sets to ensure at least 10 cold-start predictions for each of the 12 assays.\u0026nbsp;\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eTable \u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e and Table \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e show the distributions of compounds and measurements described in Fig. \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e. Note that the All-out and Subset-out columns represent single training-set collections, but One-out includes 159 or 4276 separate models and therefore separate training-set collections. The \u0026ldquo;number of training measurements\u0026rdquo; is the total count of light and dark green markers. \u0026ldquo;Unique training compounds\u0026rdquo; is the number of rows with any green markers. \u0026ldquo;Compounds exclusive to training set\u0026rdquo; is the number of rows with no red markers. These are the compounds that are not in the test set for any model. \u0026ldquo;Unique test compounds\u0026rdquo; counts rows with at least one red marker, i.e. for which predictions will be made. \u0026ldquo;Compounds exclusive to test\u0026rdquo; set have only red markers, i.e. cold-start compounds. \u0026ldquo;Number of predictions\u0026rdquo; is the total number of red points. Cold-start predictions is the total number of red markers in cold-start rows. For one-out, cold-start rows equal cold-start predictions, i.e. test-set compounds measured in only 1 assay.\u003c/p\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e\n\u003ctable id=\"Tab1\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003e\u003cspan type=\"SmallCaps\" class=\"SmallCaps\" name=\"Emphasis\"\u003eDistribution of compounds and predictions in Kinase training-set collections (13,190 unique compounds; 159 assays)\u003c/span\u003e\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\u0026nbsp;\u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eAll-out\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eSubset-out (12 assays)\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eOne-out\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eNumber of training measurements\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e85,675\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e108,675\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e114,146 (median)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eUnique training compounds\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e10,134\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e11,205\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e13,190\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eCompounds exclusive to training set\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e9,515 (94%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e10,705 (96%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e9,515 (72%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eUnique test compounds\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3,675\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2,485\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3,675\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eCompounds exclusive to test set (cold-start rows)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3,056 (83%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1,985 (80%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2,780 (76%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eNumber of predictions\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e28,642\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3,427\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e28,642\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eCold-start predictions\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e13,074 (46%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2,046 (60%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2,780 (10%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e\n\u003ctable id=\"Tab2\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003e\u003cspan type=\"SmallCaps\" class=\"SmallCaps\" name=\"Emphasis\"\u003eDistribution of compounds and predictions in Diverse training-set collections (496,946 unique compounds; 4276 assays)\u003c/span\u003e\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\u0026nbsp;\u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eAll-out\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eSubset-out (48 assays)\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eOne-out\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eNumber of training points\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1,024,751\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1,364,594\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1,368,482 (median)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eUnique training compounds\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e409,253\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e494,425\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e496,946\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eCompounds exclusive to training set\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e319,262 (78%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e493,041 (99%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e319,262 (64%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eUnique test compounds\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e177,684\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3,905\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e177,684\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eCompounds exclusive to test set (cold-start)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e87,693 (49%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2,521 (65%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e58,514 (33%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eNumber of predictions\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e343,749\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3,906\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e343,749\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eCold-start predictions\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e146,405 (43%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2,521 (65%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e58,514 (17%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e\n\u003cp\u003eStatistical analyses:\u003c/p\u003e\n\u003cp\u003eWe treat each test-set collection (all-out, subset-out, one-out, by diverse or kinase) as 6 separate statistical analyses. We follow the general recommendations of Dem\u0026scaron;ar\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e26\u003c/span\u003e\u003c/sup\u003e and Benavoli et al.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e27\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003eFor the analysis of signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e, we first use a Friedman test to assess whether there is any difference between the models\u0026rsquo; performances. Friedman\u0026rsquo;s test is a non-parametric test that uses the pairing (i.e. that the models are tested on the same datasets). If the Friedman test suggests there is a difference (p\u0026thinsp;\u0026lt;\u0026thinsp;0.05), we then do post-hoc tests of every model pair with a Wilcoxen signed-rank test. Details are in supplemental file tab_analysis1_full. We chose the Wilcoxen test because it again uses the pairing of the models (as opposed to just looking for a difference in mean signed r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e), and only assumes that differences in signed r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e are ordinal. That is, the test assumes a difference of 0.2 is better than a difference of 0.1, but not exactly twice as good. Because there are many pair predictions, we do a multiple hypothesis testing correction on the p-values using the Simes-Hochberg\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e28\u003c/span\u003e\u003c/sup\u003e step up procedure with a target family-wise error rate (FWER) of 5%. This correction aims for at most 5% false positive rate (detecting a difference in models when in fact there is no difference).\u003c/p\u003e\n\u003cp\u003eThe analysis of the rate of \u0026ldquo;successful\u0026rdquo; models (signed-r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;\u0026gt;\u0026thinsp;0.3) is similar. Each assay-model pair is assigned either a 1 or 0 for meeting the criterion. A Friedman test is followed by post-hoc pairwise tests, but here McNemar\u0026rsquo;s test is used on the 2x2 contingency tables. Detailed results are in supplemental files tab_analysis2_pvalues and tab_analysis2_tables. The same p-value correction is done with the Simes-Hochberg step up procedure.\u003c/p\u003e\n\u003cp\u003eRegression algorithms:\u003c/p\u003e\n\u003cp\u003eSingle-task random forest regression:\u003c/p\u003e\n\u003cp\u003eAs a benchmark, ST-RFR models were trained for each assay using scikit-learn RandomForestRegressor (v0.20.2). Parameters were defaults except that the number of trees was set to 200. Compound descriptors are Rdkit (v2018.09.1.0) Morgan radius 2 substructure fingerprints of 1024 bits.\u003c/p\u003e\n\u003cp\u003eProfile QSAR:\u003c/p\u003e\n\u003cp\u003eThe Profile-QSAR algorithm outlined above was used as previously described in detail,\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e1\u003c/span\u003e\u003c/sup\u003e except that experimental measurements were used where available, rather than always using random-forest regression predictions, as the input to level-2 PLS models (5% of the kinase IC\u003csub\u003e50\u003c/sub\u003es and 0.01% of the diverse-assays IC\u003csub\u003e50\u003c/sub\u003es). The original \u0026ldquo;Max2\u0026rdquo; variant employed a simple variable reduction where three level-2 PLS models were built: one using all the RFR models, one using only those whose predictions correlate with experiment at a threshold of r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;\u0026gt;\u0026thinsp;0.05 and a third using only those with a correlation of r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;\u0026gt;\u0026thinsp;0.2. The model that performed best on the test set was kept. Here, a consensus model is instead created from averaging the 3 predictions, rather than selecting just one. This avoids any bias due to selecting a model based on the test set, and it produced slightly fewer chance correlations.\u003c/p\u003e\n\u003cp\u003eAlchemite:\u003c/p\u003e\n\u003cp\u003eAlchemite was run using the 20211214 version, which was the production version when the analysis was carried out. Cross-validation to optimize the model hyperparameters was carried out using random 5-fold splits of the training data for the kinase subset-out dataset: the same hyperparameters were then used for all the models for consistency, although we note better results might be obtained by optimizing the hyperparameters separately for each assay collection and training set collection. Performance against all assays was weighted equally in the hyperparameter optimization, no weight was given to the quality of the uncertainty predictions, and Tree-structured Parzen Estimators\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e29\u003c/span\u003e\u003c/sup\u003e were used as the optimizer, as part of the overall Alchemite platform.\u003c/p\u003e\n\u003cp\u003eMetaNN:\u003c/p\u003e\n\u003cp\u003eThe MetaNN model was previously described\u003csup\u003e3031\u003c/sup\u003e. A more specific description of the algorithm as applied in this work is in the supplemental materials.\u003c/p\u003e\n\u003cp\u003eIn this work, compound features were Morgan radius 2 substructure fingerprints of 1024 bits from RDkit v2018.03. The neural network for predicting pIC50 activity was a two-layer Multi-layer Perceptron (MLP) with 500 hidden neurons in each layer. Also, each fully connected layer was followed by a batch normalization layer and a non-linear activation of leakyReLU (negative slope is 0.01). We updated the meta-initialization in a batch-wise manner, with each batch consisting of 8 randomly selected source assays, i.e., N_s\u0026thinsp;=\u0026thinsp;8. The learning rates to update the meta-initialization (i.e., \u0026alpha;) and to learn assay-specific weights (i.e., \u0026micro;) were 0.001 and 0.01, respectively. We set the Beta distribution to sample values of \u0026lambda; as Beta(0.5,0.5), i.e., \u0026alpha;\u0026thinsp;=\u0026thinsp;\u0026beta;\u0026thinsp;=\u0026thinsp;0.5. We iteratively ran 50 epochs to update the meta-initialization, with each epoch including 500 iterations, while we took only 5 gradient steps to quickly learn weights for the assay-specific neural networks.\u003c/p\u003e\n\u003ch2\u003eMT-DNN:\u003c/h2\u003e\n\u003cp\u003eWe utilized a feed-forward deep neural network (DNN) using the PyTorch Lightning framework.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e32\u003c/span\u003e\u003c/sup\u003e We employed only shared structure for the hidden layers, eliminating task-specific layers. The architecture concludes with distinct output layers for each task. To optimize prediction performance, we conducted a random search of hyperparameters\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e33\u003c/span\u003e\u003c/sup\u003e varying: i) the number and size of layers, ii) activation functions (ELU, PReLU, and LeakyReLU), and iii) optimizers (Adam, RAdam, and Yogi). Dropout regularization was applied across all hidden layers with dropout values ranging from 0.1 to 0.5. Model performance was assessed by averaging the results from three separate runs using identical hyperparameters. The Python code for the prediction algorithm is openly accessible on GitHub.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e34\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003eCompound features were 2048-bit Morgan 2 fingerprints generated with RDKit v.2021.03.1 for all models. The RAdam optimizer was used with PReLU for activation. Each model operated at a learning rate of 0.0001. The kinase models used an input layer of 2048, three hidden layers (768, 512, and 384 neurons respectively), and an output layer with 149 neurons. Dropout regularization rates for the hidden layers were 0.1, 0.4, and 0.2. The model was trained for 400 epochs. The diverse assays used an input layer with 2048 neurons, four hidden layers (768, 512, 384, and 256 neurons respectively), and an output layer with 4276 neurons. Dropout regularization rates for the hidden layers were 0.1, 0.4, 0.3, and 0.2. The training duration was 240 epochs.\u003c/p\u003e\n\u003cp\u003eMacau:\u003c/p\u003e\n\u003cp\u003eMacau was implemented within the open-source Bayesian Matrix and Tensor Factorization framework, SMURFF.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e35\u003c/span\u003e\u003c/sup\u003e Our use of Macau followed the protocol and examples provided in its official documentation.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e36\u003c/span\u003e\u003c/sup\u003e A pivotal part of employing Macau involved fine-tuning the hyperparameters, specifically adjusting the number of \u0026lsquo;latent dimensions\u0026rsquo; and \u0026lsquo;samples to keep\u0026rsquo;. We determined the best models based on the consensus results from three runs, each using the same set of hyperparameters. This approach was applied to each assay collection and training-set collection, ensuring the identification of the most effective model configuration for each case. Although Macau facilitates the incorporation of features from both compounds and assays, this study exclusively utilized compound features.\u003c/p\u003e\n\u003cp\u003eCompound features were 2048-bit Morgan 2 fingerprints generated with RDKit v.2021.03.1. The optimal hyperparameters for the kinase assays, the optimal number of latent dimensions and samples were [5, 720] for the all-out, [13, 600] for the subset-out, and [14, 680] for the one-out collection. For 4276 diverse assays, the best-performing number of latent dimensions and samples were [50, 150] for the all-out and [40, 730] for the subset-out collection. (Prediction for the diverse one-out collection was not performed due to high computation requirements).\u003c/p\u003e\n\u003ch3\u003eIMC:\u003c/h3\u003e\n\u003cp\u003eThe IMC algorithm was implemented in Python through two primary steps: factorizing the observed matrix into latent factor matrices and utilizing side information via matrix multiplication. Given that only the characteristics of compounds were available, only compound side information, i.e. substructure fingerprints, was used. Therefore, the factorization of the original matrix \u003cstrong\u003eW\u003c/strong\u003e can be represented as \u003cstrong\u003eW\u0026thinsp;\u0026asymp;\u0026thinsp;X\u003c/strong\u003e\u003csup\u003e\u003cstrong\u003eT\u003c/strong\u003e\u003c/sup\u003e\u003cstrong\u003eUV\u003c/strong\u003e, where \u003cstrong\u003eU\u003c/strong\u003e and \u003cstrong\u003eV\u003c/strong\u003e are latent factor matrices, and \u003cstrong\u003eX\u003c/strong\u003e is the matrix containing compounds\u0026apos; side information. The optimization process focused on minimizing the discrepancy between the observed and predicted entries by iteratively refining the latent factors while keeping the side information constant. Compound features were 2048-bit Morgan 2 fingerprints generated with RDKit v.2021.03.1. The ranks of the latent matrices U and V were 100 and 200 for the kinase and diverse assay datasets, respectively.\u003c/p\u003e\n\u003cp\u003eCompound descriptors:\u003c/p\u003e\n\u003cp\u003eAll methods except Alchemite described the compounds by Morgan 2 fingerprints. Alchemite used a collection of 330 calculated physicochemical properties from the StarDrop software.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e37\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e"},{"header":"Results and Discussion","content":"\u003cp\u003eThis study was designed to examine several key questions about MMRMs: for what use cases do MMRMs have an advantage over much simpler single-task models, how large is that advantage, how to evaluate the accuracy of MMRMs, are some MMRM algorithms more accurate than others, and what are other pros and cons of the different MMRM algorithms?\u003c/p\u003e\n\u003cp\u003eUse-cases for MMRMs:\u003c/p\u003e\n\u003cp\u003eAssessing the utility of any modeling method is inextricably tied to how the models will be used. MMRMs have 2 important characteristics that both highlight and limit their unique applications in drug design: the ability to make near-quantitative IC\u003csub\u003e50\u003c/sub\u003e predictions for thousands of assays for each of millions of compounds, and the limitation that this impressive accuracy on compounds unlike a given assays training set only applies for imputations, not for cold-start predictions on very novel compounds (see below). This points MMRMs toward on- or off-target virtual screens of the compound collection on which the MMRMs were trained, where many or most of the predictions are imputations, or at least near-neighbors of compounds that are imputations. Despite this limitation, having high-quality predictions for 100s to 1000s of assays for a large compound collection still addresses many important drug-discovery problems. Highly accurate on-target virtual screens allow experimental screening and model training on a modest fraction of the collection, followed by a virtual screen of the much larger remainder. More importantly, off-target virtual screens of the entire bioactivity profile allow prediction of potential off-target toxicities, mechanisms-of-action, artifactual hits, changes in mechanism, polypharmacologies or drug repurposing for a compound, or better, a medchem series of related compounds. Such information is difficult to obtain by other means, computational or experimental. MMRM\u0026rsquo;s semi-quantitative IC\u003csub\u003e50\u003c/sub\u003e predictions, rather than mere binary categories, means they also excel at triaging virtual or experimental screening hits for advancement to hit exploration or lead optimization using predictions of e.g. lipid efficiency, promiscuity and isoform selectivity. However, while the applicability domains are far larger than single-task models, covering the thousands to millions of compounds used to train the entire MMRM, predictions are still most effective for compounds in, or similar to, those in the data sources on which the MMRMs were trained. Thus, in lead optimization, while they excel for MoA or off-target activity prediction of close analogs, for on-target activity prediction they are recommended mainly for project-focused virtual libraries that stick close to the chemical matter used to train the MMRM. For chemical matter outside that chemical space, any small improvement over single-task models does not justify the much higher computational expense.\u003c/p\u003e\n\u003cp\u003eTraining/test splits:\u003c/p\u003e\n\u003cp\u003eThere are many ways to split training and test sets, and the choice must reflect the use-case. For the use-case of on- or off-target virtual screening of compounds in a corporate archive, the predictions will be for diverse historical compounds from the data source on which the MMRM was trained. A compound the project team selects for experimental testing from a virtual screen of the archive will typically be unlike the already known actives. However, it, or a near analog, will likely have some other historical activity data, i.e. an imputation (or near imputation). For hit-list triaging, off-target, polypharmacology, MoA or drug repurposing prediction, the models will likewise have been trained on compounds with historical data, but structurally unlike those tested in the corresponding project\u0026rsquo;s on-target assay. We therefore use the \u0026ldquo;realistic test-set\u0026rdquo;, which was designed specifically to test performance on compounds from the overall MMRM training set, but very unlike those in the assay being evaluated (see Methods). Additionally, cold-start predictions from an assay\u0026rsquo;s realistic test set will be on held-out compounds unlike anything in the assay being studied, and also not in the rest of the database, and thus will be our best facsimile of the performance on compounds from other external sources like vendor collections or exploratory generative chemistry.\u003c/p\u003e\n\u003cp\u003eTime-gated splits are a common approach to mimic cold-start, on-target, activity prediction on synthesis candidates in lead optimization, a typical use-case for single-task models. This is not one of the important applications of MMRMs as noted above, and time-gating would be unrelated to our use-cases. Random test sets are much too similar to the training set for any drug design use-case. Scaffold splits are closer to our case, but also generally more similar than the real use cases, since small changes in the core can define a new scaffold.\u003c/p\u003e\n\u003cp\u003eAccuracy comparison between algorithms and training-set collections:\u003c/p\u003e\n\u003cp\u003eThe signed square of Pearson\u0026rsquo;s correlation between prediction and experiment (signed-r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;r\u003csub\u003e*\u003c/sub\u003e|r|) was chosen as the primary model-accuracy figure of merit for several reasons. For many assays, most compounds are inactive, so merely guessing low activity for every compound gives a good MAE or RMSD, whereas signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e is sensitive to pulling the handful of precious needles out of the haystack of weak activity. Correlation coefficient was chosen over coefficient of determination, which combines correlation with correct absolute value, because assays vary in scale and coefficient of determination can be very sensitive to the absolute range in the test set. The range of activities for different assays can vary greatly even for the same target. For example, the sensitivity of ATP competitive kinase inhibitors will change if assays differ in protein construct, partner proteins in a larger complex, peptide substrate, degree of phosphorylation, cofactor concentrations, etc. Furthermore, besides EC\u003csub\u003e50\u003c/sub\u003e or Kd, some assays report other quantitative endpoints such as IC\u003csub\u003e90\u003c/sub\u003e, MIC, half-life, reaction rate or physicochemical or ADME endpoints. For comparisons across assays, such as identifying PK risks, off-target activities, promiscuity or MoA, it is thus most important to identify the predicted activities unusually high for that particular assay. I.e. correlation is most relevant, rather than getting the correct absolute value. In fact, for off-target, polypharmacology, ADME or MoA prediction, we generally Z-scale the predictions from each model, thus normalizing the predictions for each individual assay. Here, we compare algorithms by 2 metrics: signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e, and the fraction of models that achieve a heuristic criterion for virtual-screening success of signed-r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;\u0026gt;\u0026thinsp;0.3. Mean absolute error (MAE) was used in one analysis, comparing cold-start predictions to imputations, where point-prediction error was aggregated across many assays, bearing in mind this source of incommensurability.\u003c/p\u003e\n\u003cp\u003eThe results are summarized in 2 ways: Table \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e gives the median signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e, and Table \u003cspan class=\"InternalRef\"\u003e4\u003c/span\u003e the count of assays with signed-r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;\u0026gt;\u0026thinsp;0.3 for each algorithm, for each training-set collection for the kinase and diverse assay collections. More precise values are in supplemental files tab_median_signed_r2.csv and tab_signed_r2_good_enough.csv. The corresponding full rank-order distribution plots in Figure \u003cspan class=\"InternalRef\"\u003eS1\u003c/span\u003e are similar in shape, indicating these 2 metrics summarize the results well. Figure \u003cspan class=\"InternalRef\"\u003e4\u003c/span\u003e and Fig. \u003cspan class=\"InternalRef\"\u003e5\u003c/span\u003e show the results of statistical significance analyses comparing pairs of models for signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e and count (signed-r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;\u0026gt;\u0026thinsp;0.3) as described in the Methods.\u003c/p\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e\n\u003cp\u003eThe 5 \u0026ldquo;advanced\u0026rdquo; methods generally perform better than the baseline single-task ST-RFR and baseline IMC method across the full range of tests. MT-DNN, pQSAR and Alchemite were roughly similar. MetaNN was slightly below those 3 on most tests. Macau\u0026rsquo;s performance was mixed, roughly comparable to the other 4 on the kinase assays, but substantially worse on the diverse assays. Note that the number of assays varies from 12 to 4276 between test-set collections. With a large number of models, even very small differences can be statistically significant, such as those seen in the diverse all-out case. The does not mean the differences are large enough to matter in practice.\u003c/p\u003e\n\u003cp\u003eFour of the 5 advanced MMRM algorithms contributed kinase one-out models: pQSAR, MT-DNN, metaNN, Macau. One-out models are most like the final none-out models that will be used in production. As Table \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e shows, the IMC MMRM baseline model at median signed-r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;0.26, was still substantially better than ST-RFR single-task baseline at 0.09. The 4 advanced MMRMs were highly successful. The order of performance was MT-DNN\u0026thinsp;=\u0026thinsp;pQSAR\u0026thinsp;\u0026gt;\u0026thinsp;metaNN\u0026thinsp;\u0026gt;\u0026thinsp;Macau\u0026thinsp;\u0026gt;\u0026thinsp;\u0026gt;\u0026thinsp;IMC\u0026thinsp;\u0026gt;\u0026thinsp;\u0026gt;\u0026thinsp;ST-RFR. MT-DNN and pQSAR had median signed-r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;0.59 and 0.58 respectively. MetaNN was close behind at 0.56 and then Macau at 0.53, although Fig. \u003cspan class=\"InternalRef\"\u003e4\u003c/span\u003e shows that with 159 assays those differences were statistically significant. As Table \u003cspan class=\"InternalRef\"\u003e4\u003c/span\u003e shows, the top 4 algorithms gave successful models (signed-r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;\u0026gt;\u0026thinsp;0.3) on at least 90% of the 159 assays, and as Fig. \u003cspan class=\"InternalRef\"\u003e5\u003c/span\u003e shows, were statistically equivalent by that measure. By contrast, ST-RFR achieved only 9% successful models with median signed-r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;0.09 and IMC achieved 38% successful models with median signed-r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;0.26.\u003c/p\u003e\n\u003cp\u003eAll methods contributed kinase subset-out models. The results were similar to one-out: all 5 advanced MMRMs achieved 9 of 12 successful models, with median signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e between 0.51 and 0.55. The order of statistical significance was pQSAR\u0026thinsp;=\u0026thinsp;MT-DNN\u0026thinsp;=\u0026thinsp;Alchemite\u0026thinsp;=\u0026thinsp;metaNN\u0026thinsp;=\u0026thinsp;Macau\u0026thinsp;\u0026gt;\u0026thinsp;\u0026gt;\u0026thinsp;IMC\u0026thinsp;\u0026gt;\u0026thinsp;ST-RFR.\u003c/p\u003e\n\u003cp\u003eFor the kinase all-out models, the signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e order of significance was pQSAR\u0026thinsp;=\u0026thinsp;MT-DNN\u0026thinsp;=\u0026thinsp;Alchemite\u0026thinsp;\u0026gt;\u0026thinsp;Macau\u0026thinsp;=\u0026thinsp;metaNN\u0026thinsp;\u0026gt;\u0026thinsp;\u0026gt;\u0026thinsp;ST-RFR\u0026thinsp;\u0026gt;\u0026thinsp;IMC. However, median signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e was less than 0.2 for all algorithms. and no method achieved more than 20% successful models. All-out models were, thus, not generally able to identify the assays for which the final 100% virtual screening models would be useful on these test-sets. This casts suspicion on interpreting any performance differences.\u003c/p\u003e\n\u003cp\u003eOnly pQSAR contributed a one-out MMRM for the diverse assay collection. Median signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e was 0.41 and 2500 of 4276 assays exceeded signed-r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;\u0026gt;\u0026thinsp;0.3, compared to 0.16 and 1300 respectively for ST-RFR. While less dramatic than the 159 kinases, this is a substantial improvement. The decrease likely reflects better transfer of learning among homogeneous kinase assays.\u003c/p\u003e\n\u003cp\u003eThe subset of 48 diverse models showed substantial variation between MMRM algorithms. Median signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e varied between 0.19 and 0.44 for the 5 advanced models, with significance order pQSAR\u0026thinsp;=\u0026thinsp;DNN\u0026thinsp;=\u0026thinsp;ST-RFR\u0026thinsp;=\u0026thinsp;Alchemite\u0026thinsp;\u0026ge;\u0026thinsp;metaNN\u0026thinsp;=\u0026thinsp;Macau\u0026thinsp;\u0026gt;\u0026thinsp;\u0026gt;\u0026thinsp;IMC. Similarly, the proportion of successful models varied from 15% for IMC to 73% for pQSAR. The significance order was pQSAR\u0026thinsp;=\u0026thinsp;DNN\u0026thinsp;=\u0026thinsp;ST-RFR\u0026thinsp;=\u0026thinsp;Alchemite\u0026thinsp;\u0026ge;\u0026thinsp;metaNN\u0026thinsp;=\u0026thinsp;Macau\u0026thinsp;\u0026ge;\u0026thinsp;IMC.\u003c/p\u003e\n\u003cp\u003eThe surprising strong performance of ST-RFR on the diverse subset-out models deserves more scrutiny. The 48 assays were selected by full-factorial design to sample a wide but realistic range of 4 assay properties: compound count, dynamic range, pQSAR signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e and target family (see methods). Unfortunately, the experimental design appears to have selected a very biased sampling of the 4276 assays. Both pQSAR and ST-RFR did better on subset-out than higher-fidelity diverse one-out, but that difference was much greater for the ST-RFR. As Table \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e shows, the median ST-RFR signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e is 0.36 for the 48 subset models vs. 0.15 for the full set of 4276 one-out models, and 58% of the 48 ST-RFR models achieve signed-r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;\u0026gt;\u0026thinsp;0.3 vs. only 30% for the full set of one-out models. For pQSAR, subset-out was much closer to one-out: 0.44 vs. 0.38 and 73% vs. 58%. This suggests that the 48 selected assay were not a representative sampling of the 4276. Thus, the strong ST-RFR subset-out performance appears to be an artifact of the specific selected subset, and it might likely not outperform the final none-out metaNN models, though might still outperform Macau and IMC.\u003c/p\u003e\n\u003cp\u003eThe diverse all-out signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e order was MT-DNN\u0026thinsp;\u0026gt;\u0026thinsp;pQSAR\u0026thinsp;\u0026gt;\u0026thinsp;Alchemite\u0026thinsp;\u0026gt;\u0026thinsp;ST-RFR\u0026thinsp;\u0026gt;\u0026thinsp;MetaNN\u0026thinsp;\u0026gt;\u0026thinsp;Macau\u0026thinsp;\u0026gt;\u0026thinsp;IMC. The order for count of successful models was only slightly different, switching ST-RFR and MetaNN, with DNN\u0026thinsp;\u0026gt;\u0026thinsp;pQSAR\u0026thinsp;\u0026gt;\u0026thinsp;Alchemite\u0026thinsp;\u0026gt;\u0026thinsp;MetaNN\u0026thinsp;=\u0026thinsp;ST-RFR\u0026thinsp;\u0026gt;\u0026thinsp;Macau\u0026thinsp;\u0026gt;\u0026thinsp;\u0026gt;\u0026thinsp;IMC. With so many assays, virtually all differences are statistically significant. Again, since the performance of all-out MMRMs for each algorithm is much worse than subset-out or one-out, with median signed-r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;\u0026lt;\u0026thinsp;0.25 and success count\u0026thinsp;\u0026lt;\u0026thinsp;41%, it is hard to interpret the all-out order. Despite those caveats, the poor performance of Macau on the all-out and subset-out assay collections does stand out.\u003c/p\u003e\n\u003cp\u003eTable\u0026nbsp;3\u0026nbsp;Median Signed-r\u003csup\u003e2\u003c/sup\u003e\u003c/p\u003e\n\u003ctable style=\"border: none;width:468.0pt;border-collapse:collapse;\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 104.15pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(142, 169, 219);background: rgb(217, 225, 242);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eTest Set \\ Method\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 66.15pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(142, 169, 219);background: rgb(217, 225, 242);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003ePQSAR\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 45.55pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(142, 169, 219);background: rgb(217, 225, 242);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eDNN\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 67.65pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(142, 169, 219);background: rgb(217, 225, 242);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eAlchemite\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 51.65pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(142, 169, 219);background: rgb(217, 225, 242);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eMetaNN\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47.35pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(142, 169, 219);background: rgb(217, 225, 242);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eMacau\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 37.5pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(142, 169, 219);background: rgb(217, 225, 242);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eIMC\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 48pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(142, 169, 219);background: rgb(217, 225, 242);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eST-RFR\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 104.15pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eKinase One-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 66.15pt;background: rgb(104, 192, 124);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.58\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 45.55pt;background: rgb(99, 190, 123);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.59\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 67.65pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd style=\"width: 51.65pt;background: rgb(99, 190, 123);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.56\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47.35pt;background: rgb(110, 193, 124);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.53\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 37.5pt;background: rgb(254, 228, 130);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.26\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 48pt;background: rgb(250, 145, 114);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.09\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 104.15pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eKinase Subset-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 66.15pt;background: rgb(139, 202, 126);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.51\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 45.55pt;background: rgb(117, 196, 125);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.55\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 67.65pt;background: rgb(117, 196, 125);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.55\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 51.65pt;background: rgb(121, 197, 125);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.51\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47.35pt;background: rgb(121, 197, 125);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.51\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 37.5pt;background: rgb(249, 234, 132);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.29\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 48pt;background: rgb(248, 105, 107);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.08\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 104.15pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eKinase All-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 66.15pt;background: rgb(252, 181, 121);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.17\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 45.55pt;background: rgb(252, 180, 121);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.17\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 67.65pt;background: rgb(251, 174, 120);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.15\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 51.65pt;background: rgb(248, 123, 110);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.12\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47.35pt;background: rgb(248, 122, 110);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.12\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 37.5pt;background: rgb(248, 120, 110);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.05\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 48pt;background: rgb(250, 145, 114);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.09\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 104.15pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eDiverse One-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 66.15pt;background: rgb(202, 220, 129);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.38\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 45.55pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd style=\"width: 67.65pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd style=\"width: 51.65pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd style=\"width: 47.35pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd style=\"width: 37.5pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd style=\"width: 48pt;background: rgb(239, 231, 132);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.15\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 104.15pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eDiverse Subset-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 66.15pt;background: rgb(176, 212, 128);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.44\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 45.55pt;background: rgb(197, 219, 129);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.39\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 67.65pt;background: rgb(211, 223, 130);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.36\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 51.65pt;background: rgb(239, 231, 132);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.26\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47.35pt;background: rgb(253, 198, 124);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.19\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 37.5pt;background: rgb(248, 106, 107);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.02\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 48pt;background: rgb(99, 190, 123);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.36\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 104.15pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eDiverse All-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 66.15pt;background: rgb(176, 212, 128);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.20\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 45.55pt;background: rgb(197, 219, 129);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.23\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 67.65pt;background: rgb(211, 223, 130);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.18\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 51.65pt;background: rgb(239, 231, 132);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.16\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47.35pt;background: rgb(253, 198, 124);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.10\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 37.5pt;background: rgb(248, 106, 107);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.02\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 48pt;background: rgb(99, 190, 123);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0.15\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e\n\u003cp\u003eTable\u0026nbsp;4\u0026nbsp;Percentage of Models with Signed-r\u003csup\u003e2\u003c/sup\u003e \u0026gt; 0.3\u0026nbsp;\u003c/p\u003e\n\u003ctable style=\"border: none;width:468.0pt;border-collapse:collapse;\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 121.4pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(142, 169, 219);background: rgb(217, 225, 242);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eTest Set \\ Method\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47.2pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(142, 169, 219);background: rgb(217, 225, 242);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003ePQSAR\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 46.9pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(142, 169, 219);background: rgb(217, 225, 242);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eDNN\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 67.95pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(142, 169, 219);background: rgb(217, 225, 242);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eAlchemite\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 49.55pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(142, 169, 219);background: rgb(217, 225, 242);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eMetaNN\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 49.5pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(142, 169, 219);background: rgb(217, 225, 242);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eMacau\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 41.55pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(142, 169, 219);background: rgb(217, 225, 242);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eIMC\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 43.95pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(142, 169, 219);background: rgb(217, 225, 242);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eST-RFR\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 121.4pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eKinase One-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47.2pt;background: rgb(108, 193, 124);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e91%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 46.9pt;background: rgb(99, 190, 123);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e94%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 67.95pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd style=\"width: 49.55pt;background: rgb(106, 192, 124);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e92%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 49.5pt;background: rgb(111, 194, 124);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e90%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 41.55pt;background: rgb(246, 233, 132);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e38%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 43.95pt;background: rgb(249, 137, 113);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e9%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 121.4pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eKinase Subset-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47.2pt;background: rgb(150, 205, 126);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e75%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 46.9pt;background: rgb(150, 205, 126);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e75%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 67.95pt;background: rgb(150, 205, 126);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e75%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 49.55pt;background: rgb(150, 205, 126);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e75%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 49.5pt;background: rgb(150, 205, 126);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e75%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 41.55pt;background: rgb(254, 229, 130);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e33%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 43.95pt;background: rgb(251, 167, 119);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e17%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 121.4pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eKinase All-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47.2pt;background: rgb(150, 205, 126);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e20%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 46.9pt;background: rgb(150, 205, 126);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e13%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 67.95pt;background: rgb(150, 205, 126);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e16%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 49.55pt;background: rgb(150, 205, 126);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e7%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 49.5pt;background: rgb(150, 205, 126);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e11%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 41.55pt;background: rgb(254, 229, 130);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e0%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 43.95pt;background: rgb(251, 167, 119);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e9%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 121.4pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eDiverse One-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47.2pt;background: rgb(196, 218, 129);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e58%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 46.9pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd style=\"width: 67.95pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd style=\"width: 49.55pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd style=\"width: 49.5pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd style=\"width: 41.55pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd style=\"width: 43.95pt;background: rgb(253, 215, 128);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e30%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 121.4pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eDiverse Subset-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47.2pt;background: rgb(156, 207, 127);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e73%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 46.9pt;background: rgb(177, 213, 128);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e65%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 67.95pt;background: rgb(188, 216, 129);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e60%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 49.55pt;background: rgb(221, 226, 131);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e48%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 49.5pt;background: rgb(254, 221, 129);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e31%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 41.55pt;background: rgb(250, 159, 117);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e15%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 43.95pt;background: rgb(194, 218, 129);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e58%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 121.4pt;padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eDiverse All-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 47.2pt;background: rgb(252, 235, 132);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e36%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 46.9pt;background: rgb(239, 231, 132);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e41%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 67.95pt;background: rgb(254, 230, 131);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e34%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 49.55pt;background: rgb(253, 216, 128);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e30%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 49.5pt;background: rgb(252, 183, 122);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e21%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 41.55pt;background: rgb(249, 137, 113);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e9%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 43.95pt;background: rgb(253, 215, 128);padding: 0cm 5.4pt;height: 12.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e30%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eAlthough the one-out multitask models for each algorithm greatly outperform the all-out models, one might hope that the quality of all-out models might correlate with one-out models, so they could at least be used to predict the relative performance of the final none-out models and to compare the performance of the algorithms. Figure \u003cspan class=\"InternalRef\"\u003e6\u003c/span\u003e plots the signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e for models trained with all-out vs. one-out collections for the 159 kinase assays for the 5 MMRMs that contributed kinase one-out models. Figure \u003cspan class=\"InternalRef\"\u003e7\u003c/span\u003e is similar for pQSAR models on the 4276 diverse assays. Unfortunately, the low correlation coefficients and triangular scatter plots indicate that the order of all-out models is not even a relative indicator of one-out, and by extension of none-out, model quality for our use case of measurements on novel members of the compounds used to train the multitask models. This confirms the suspicion that differences among the all-out MMRMs does not reflect the relative performance for the final none-out models.\u003c/p\u003e\n\u003cp\u003eSubset-out MMRM compromise for model evaluation:\u003c/p\u003e\n\u003cp\u003ePresumably due to the limited, redundant training data and excess of cold-start predictions, all-out models do not reflect well the quality of the final none-out production models for our use case of measurements of novel compounds used in training the multitask model. However, for the assays tested, especially the kinases, the subset-out models are close to the one-out models. This suggests that while training a single all-out MMRM will not indicate the expected performance of the final production models\u0026mdash;and training 1000s of separate multitask models is impracticable\u0026mdash;a compromise of training multiple, separate multitask subset models could be a reasonable compromise for feasible computation with useful evaluation of final model accuracy for these algorithms.\u003c/p\u003e\n\u003cp\u003eBy-assay comparisons:\u003c/p\u003e\n\u003cp\u003eTable \u003cspan class=\"InternalRef\"\u003e5\u003c/span\u003e summarizes the signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e correlations across assays between each pair of algorithms for the six training-set collections. Not only were the overall average results similar between several methods, but except for the 2 all-out collections, which did not produce many useful models, which assays produced good or poor models was also very similar between the 5 more-advanced MMRM algorithms. For visual comparison, corresponding trellis plots are presented in Figures \u003cspan class=\"InternalRef\"\u003eS2\u003c/span\u003e-S7. This further supports the hypothesis that these very different algorithms are fundamentally comparable, capturing essentially the same information, and suggests that the quality of the predictions from the MMRMs is limited by the assay data more than by differences between the algorithms.\u003c/p\u003e\n\u003cp\u003eTable\u0026nbsp;5.\u0026nbsp;Correlations of r\u003csup\u003e2\u003c/sup\u003e across assays between each pair of algorithms for each training set.\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"https://myfiles.space/user_files/58895_8739fc6c57c1c19a/58895_custom_files/img1758286081.png\" width=\"655\" height=\"868\"\u003e\u003c/p\u003e\n\u003cdiv class=\"gridtable\"\u003e\n \u003cdiv align=\"left\" class=\"colspec\"\u003e\u003cbr\u003e\u003c/div\u003e\n \u003ch2\u003eImputation vs. cold-start prediction:\u003c/h2\u003e\n \u003cdiv align=\"left\" class=\"colspec\"\u003ePrevious work has shown that MT-DNN MMRM performance on these assay collections is much better for imputations than for cold-start predictions.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e7\u003c/span\u003e\u003c/sup\u003e We studied this further to understand if this is true for all algorithms, how improvement depends on the number of supporting measurements from other assays, how this impacts performance between training-set collections, and how this affects the cases for which multitask models are helpful in drug discovery practice.\u003c/div\u003e\n\u003c/div\u003e\n\u003cp\u003eTable \u003cspan class=\"InternalRef\"\u003e6\u003c/span\u003e shows, for each algorithm and training-set collection, the median signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e and percentage of models with Signed-r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;\u0026gt;\u0026thinsp;0.3 for cold-start predictions and imputations. The full rank-order distribution plots in Figures S8 a-f are similar, indicating these 2 metrics summarize the results well. Only a fraction of models in each training-set collection could be analyzed, because for inclusion of a model we required a minimum number of both cold-start and imputation predictions to compute a meaningful signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e. For kinase all-out, diverse all-out and diverse one-out, the minimum was 8, which still allowed at least 150 assays each. Subset-out training-set collections were small to start with, and with so many supporting assays in the one-out models, their number of cold-start predictions was small. Thus, for diverse subset-out, kinase subset-out and kinase one-out the minimum was reduced to 5 cold-start and 5 imputation predictions. Even so, these training-set collections could only compare from 5 to 9 assays.\u003c/p\u003e\n\u003cp\u003eTable\u0026nbsp;6\u0026nbsp;Table of median signed-r\u003csup\u003e2\u003c/sup\u003e and percentage of models with Signed-r\u003csup\u003e2\u003c/sup\u003e \u0026gt; 0.3 of cold-start predictions and imputations for each method for kinase and diverse assay sets also including number of assays for each case.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e\n\u003cdiv class=\"gridtable\"\u003e\n \u003ctable style=\"border: none;width:541.05pt;border-collapse:collapse;\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"18\" style=\"width:541.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eMedian signed-r2\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;background:#156082;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:white;\"\u003etest-set collection\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:59.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#156082;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:white;\"\u003ecold/imp.\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#156082;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:white;\"\u003ePQSAR\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.2pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#156082;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:white;\"\u003eMT-DNN\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:63.2pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#156082;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:white;\"\u003eAlchemite\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:54.3pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#156082;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:white;\"\u003eMetaNN\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.45pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#156082;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:white;\"\u003eMacau\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.4pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#156082;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:white;\"\u003eIMC\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#156082;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:white;\"\u003eST-RFR\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;background:#156082;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:white;\"\u003eAssay #\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eKinase Subset-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:59.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003ecold\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FBA777;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.06\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.2pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FA9273;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.04\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:63.2pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FCEA84;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.14\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:54.3pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FDC67D;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.09\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.45pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FEE583;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.12\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.4pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#FA9D75;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.05\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F97D6F;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.02\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e10\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eKinase One-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:59.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003ecold\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F8736D;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.01\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.2pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F8696B;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.00\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 63.2pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:54.3pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F8736D;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.01\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.45pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F98871;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.03\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.4pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#F8696B;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.00\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F8736D;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.01\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e6\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eKinase All-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:59.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003ecold\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FBB179;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.07\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.2pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F8736D;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.01\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:63.2pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FBB179;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.07\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:54.3pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F98871;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.03\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.45pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F8736D;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.01\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.4pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#F97D6F;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.02\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FBB179;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.07\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e155\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eDiverse Subset-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:59.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003ecold\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#DFE283;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.26\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.2pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#CADC81;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.35\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:63.2pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FBA777;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.06\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:54.3pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FAEA84;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.15\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.45pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F7E984;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.16\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.4pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#FDD17F;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.10\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#C5DB81;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.37\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e6\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eDiverse One-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:59.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003ecold\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FCBC7B;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.08\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 46.2pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 63.2pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 54.3pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 51.45pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 30.4pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FBA777;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.06\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e245\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eDiverse All-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:59.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003ecold\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FA9D75;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.05\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.2pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FA9273;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.04\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:63.2pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FA9D75;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.05\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:54.3pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F98871;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.03\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.45pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F97D6F;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.02\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.4pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#F8696B;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.00\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FBA777;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.06\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e986\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eKinase Subset-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:59.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eimp.\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#96CD7E;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.57\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.2pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#8ECB7E;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.60\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:63.2pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#93CC7E;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.58\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:54.3pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#9ACE7F;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.55\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.45pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#98CE7F;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.56\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.4pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#D1DE82;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.32\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FEDB81;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.11\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e10\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eKinase One-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:59.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eimp.\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#CFDD82;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.33\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.2pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F2E884;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.18\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 63.2pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);background: rgb(192, 230, 245);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:54.3pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F2E884;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.18\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.45pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#E6E483;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.23\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.4pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#F8696B;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.00\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F2E884;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.18\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e6\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eKinase All-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:59.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eimp.\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#E4E483;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.24\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.2pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#CFDD82;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.33\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:63.2pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#E4E483;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.24\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:54.3pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#DDE283;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.27\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.45pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#DBE182;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.28\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.4pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#FBB179;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.07\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FEEB84;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.13\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e155\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eDiverse Subset-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:59.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eimp.\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#66BF7C;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.77\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.2pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#63BE7B;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.78\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:63.2pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#6BC17C;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.75\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:54.3pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#B2D580;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.45\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.45pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#96CD7E;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.57\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.4pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#F8696B;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.00\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#DBE182;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.28\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e6\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eDiverse One-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:59.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eimp.\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#E9E583;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.22\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 46.2pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);background: rgb(192, 230, 245);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 63.2pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);background: rgb(192, 230, 245);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 54.3pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);background: rgb(192, 230, 245);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 51.45pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);background: rgb(192, 230, 245);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 30.4pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);background: rgb(192, 230, 245);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FEDB81;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.11\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e245\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eDiverse All-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:59.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eimp.\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#DBE182;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.28\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.2pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#CFDD82;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.33\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:63.2pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#DFE283;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.26\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:54.3pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#EEE683;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.20\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:51.45pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#EBE683;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.21\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.4pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#F8696B;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.00\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FAEA84;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0.15\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e986\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"17\" style=\"width:510.65pt;border:none;border-bottom: solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003e% Signed-r2 \u0026gt; 0.3\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"border:none;padding:0cm 0cm 0cm 0cm;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:8.0pt;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;'\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;background:#156082;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:white;\"\u003etest-set collection\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:54.7pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#156082;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:white;\"\u003ecold/imp.\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#156082;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:white;\"\u003ePQSAR\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:45.55pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#156082;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:white;\"\u003eMT-DNN\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:58.6pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#156082;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:white;\"\u003eAlchemite\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:48.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#156082;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:white;\"\u003eMetaNN\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:41.95pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#156082;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:white;\"\u003eMacau\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.35pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#156082;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:white;\"\u003eIMC\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#156082;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:white;\"\u003eST-RFR\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;background:#156082;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cstrong\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:white;\"\u003eAssay #\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"border:none;padding:0cm 0cm 0cm 0cm;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:8.0pt;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;'\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eKinase Subset-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:54.7pt;border:none;border-bottom:solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003ecold\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FDC77D;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e20%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:45.55pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FA9874;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e10%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:58.6pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FA9874;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e10%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:48.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FAEA84;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e30%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:41.95pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FDC77D;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e20%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.35pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#FAEA84;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e30%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F8696B;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e10\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"border:none;padding:0cm 0cm 0cm 0cm;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:8.0pt;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;'\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eKinase One-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:54.7pt;border:none;border-bottom:solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003ecold\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F8696B;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:45.55pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F8696B;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 58.6pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:48.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FCB97A;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e17%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:41.95pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FCB97A;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e17%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.35pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#F8696B;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e0%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FCB97A;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e17%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e6\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"border:none;padding:0cm 0cm 0cm 0cm;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:8.0pt;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;'\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eKinase All-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:54.7pt;border:none;border-bottom:solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003ecold\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F98A71;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e7%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:45.55pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F86D6B;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e1%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:58.6pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F9806F;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e5%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:48.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F86D6B;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e1%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:41.95pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F86D6B;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e1%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.35pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#F86D6B;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e1%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F98570;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e6%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e155\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"border:none;padding:0cm 0cm 0cm 0cm;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:8.0pt;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;'\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eDiverse Subset-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:54.7pt;border:none;border-bottom:solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003ecold\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F4E884;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e33%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:45.55pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#CFDE82;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e50%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:58.6pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F4E884;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e33%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:48.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F4E884;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e33%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:41.95pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FCB97A;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e17%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.35pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#FCB97A;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e17%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#CFDE82;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e50%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e6\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"border:none;padding:0cm 0cm 0cm 0cm;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:8.0pt;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;'\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eDiverse One-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:54.7pt;border:none;border-bottom:solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003ecold\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FDC77D;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e20%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 45.55pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 58.6pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 48.05pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 41.95pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 30.35pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FDD57F;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e23%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e245\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"border:none;padding:0cm 0cm 0cm 0cm;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:8.0pt;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;'\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eDiverse All-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:54.7pt;border:none;border-bottom:solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003ecold\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FCBE7B;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e18%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:45.55pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FCB97A;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e17%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:58.6pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FDC77D;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e20%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:48.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FBA175;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e12%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:41.95pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FA9373;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e9%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.35pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#F9806F;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e5%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FCBE7B;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e18%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e986\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"border:none;padding:0cm 0cm 0cm 0cm;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:8.0pt;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;'\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eKinase Subset-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:54.7pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eimp.\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#8FCB7E;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e80%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:45.55pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#8FCB7E;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e80%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:58.6pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#8FCB7E;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e80%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:48.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#8FCB7E;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e80%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:41.95pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#8FCB7E;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e80%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.35pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#CFDE82;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e50%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FA9874;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e10%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e10\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"border:none;padding:0cm 0cm 0cm 0cm;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:8.0pt;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;'\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eKinase One-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:54.7pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eimp.\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#CFDE82;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e50%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:45.55pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#CFDE82;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e50%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 58.6pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);background: rgb(192, 230, 245);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:48.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#CFDE82;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e50%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:41.95pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#CFDE82;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e50%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.35pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#FCB97A;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e17%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#F4E884;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e33%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e6\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"border:none;padding:0cm 0cm 0cm 0cm;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:8.0pt;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;'\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eKinase All-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:54.7pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eimp.\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#E7E483;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e39%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:45.55pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#C0D981;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e57%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:58.6pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#E5E483;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e40%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:48.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#DCE182;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e44%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:41.95pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#E2E383;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e41%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.35pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#F8726C;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e2%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FBAF78;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e15%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e155\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"border:none;padding:0cm 0cm 0cm 0cm;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:8.0pt;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;'\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eDiverse Subset-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:54.7pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eimp.\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#63BE7B;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e100%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:45.55pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#63BE7B;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e100%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:58.6pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#88C97E;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e83%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:48.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#ABD380;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e67%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:41.95pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#88C97E;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e83%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.35pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#FCB97A;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e17%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#CFDE82;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e50%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e6\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"border:none;padding:0cm 0cm 0cm 0cm;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:8.0pt;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;'\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eDiverse One-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:54.7pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eimp.\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#DAE182;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e45%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 45.55pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);background: rgb(192, 230, 245);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 58.6pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);background: rgb(192, 230, 245);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 48.05pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);background: rgb(192, 230, 245);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 41.95pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);background: rgb(192, 230, 245);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 30.35pt;border-top: none;border-right: none;border-left: none;border-image: initial;border-bottom: 1pt solid rgb(68, 179, 225);background: rgb(192, 230, 245);padding: 0cm 5.4pt;height: 15.75pt;vertical-align: bottom;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-family: Calibri;color:black;\"\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FEE883;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e27%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e245\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"border:none;padding:0cm 0cm 0cm 0cm;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:8.0pt;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;'\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:98.8pt;border-top:none;border-left:solid #44B3E1 1.0pt;border-bottom:solid #44B3E1 1.0pt;border-right:none;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eDiverse All-Out\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:54.7pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;line-height:normal;'\u003e\u003cspan style=\"font-size:13px;font-family:Calibri;color:black;\"\u003eimp.\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:46.4pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#D6DF82;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e47%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:45.55pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#C9DC81;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e53%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:58.6pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#DAE182;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e45%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:48.05pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#E7E483;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e39%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:41.95pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#E5E483;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e40%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:30.35pt;border:none;border-bottom:solid #44B3E1 1.0pt;background:#F98A71;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e7%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:42.5pt;border:none;border-bottom: solid #44B3E1 1.0pt;background:#FEEB84;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e28%\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:43.75pt;border-top:none;border-left:none;border-bottom:solid #44B3E1 1.0pt;border-right:solid #44B3E1 1.0pt;background:#C0E6F5;padding:0cm 5.4pt 0cm 5.4pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:normal;'\u003e\u003cspan style=\"font-family:Calibri;color:black;\"\u003e986\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"border:none;padding:0cm 0cm 0cm 0cm;\"\u003e\n \u003cp style='margin-top:0cm;margin-right:0cm;margin-bottom:8.0pt;margin-left:0cm;font-size:11.0pt;font-family:\"Calibri\",sans-serif;'\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003cdiv align=\"left\" class=\"colspec\"\u003e\u003cbr\u003e\u003c/div\u003e\n \u003cdiv align=\"left\" class=\"colspec\"\u003e\u003cbr\u003e\u003c/div\u003e\n\u003c/div\u003e\n\u003cp\u003eTable \u003cspan class=\"InternalRef\"\u003e6\u003c/span\u003e shows that cold-start MMRM predictions are no better than the corresponding ST-RFRs, although the small number of assays for Diverse Subset-Out, Kinase Subset-Out, and Kinase One-Out makes these comparisons uncertain. Imputations, on the other hand, have substantial improvements. Even the all-out imputations show improvement over ST-RFR. These results suggest that one reason for the improvement seen for one-out over all-out models might be that one-out models have mostly imputations rather than cold-starts. Note that while the cold-start/imputation distinction does not apply to ST-RFR models, for a proper benchmark, the results must still be compared to the corresponding ST-RFR models for the same subset of assays.\u003c/p\u003e\n\u003cp\u003eNote that this analysis was tested on the realistic test set of structures maximally unlike the training set, corresponding to the extremes of each model\u0026rsquo;s applicability domain. Although activity prediction for compounds similar to those in a model\u0026rsquo;s training set was not tested in this study, cold-start predictions for compounds with near-neighbors that are imputations, such as analogs in lead optimization, might still be better than single-task. Since the cold-start test compounds were unlike the training set of the model under evaluation, and also are not in the training sets for any other assay, they are our closest simulation of virtual screens against external compound sources like vendor databases or exploratory generative chemistry. In these use cases we thus conclude that multitask models will have little advantage over single-task models, while costing much more. This result also suggests that models should be updated frequently, e.g. monthly or weekly rather than yearly, so off-target predictions on current compounds and series will at least include supporting measurements from assays near the top of the project testing funnel, and thus will be imputations or at least near neighbors of imputations.\u003c/p\u003e\n\u003cp\u003eBeyond the binary distinction of cold-start vs. imputation, Fig. \u003cspan class=\"InternalRef\"\u003e8\u003c/span\u003e shows the impact of only testing compounds with ever more supporting measurements. Two plots are paired for each test-set/assay collection. The line plots show the average per-prediction difference in absolute errors between each MMRM and the corresponding baseline ST-RFR, aggregated across all assays, binned by the minimum number of supporting experimental measurements required for inclusion. At the bottom is a histogram of the count of predictions contributing to that bin. Error bands are +/- 1.96 \u003csub\u003e*\u003c/sub\u003e SE. The first bin is unique, representing cold-start predictions (exactly zero supporting measurements). The subsequent bins indicate imputations with at least the listed number of supporting measurements. I.e. the \u0026ge;\u0026thinsp;1 bin includes all imputations, \u0026ge;\u0026thinsp;2 is all except those with just 1 supporting assay, etc. Note that the previous Table \u003cspan class=\"InternalRef\"\u003e6\u003c/span\u003e analysis of signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e was based on commensurate predictions from individual models, and each model contributes equal weight. Figure 9 plots are aggregated across diverse models for all predictions with the same range of supporting measurements, so assays with more measurements contribute more. Also note that bin ranges on the X-axis get progressively larger going right, so the downward curve is influenced by the increasingly wider bins. The first 5 points increment by a single additional assay, then by 5, 15, 30 and 50. If all bins increased by just 1 assay, the plots would be more linear, but would go way off the page to the right. The multi-task predictions get better when requiring all imputations have successively more supporting measurements. Conversely, looking right to left, performance gets worse as predictions with fewer supporting measurements are included.\u003c/p\u003e\n\u003cp\u003eAggregating prediction error across all assays is admittedly a crude analysis, and using mean-absolute prediction error rather than signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e can be dominated by minimally helpful correct predictions for large numbers of inactive compounds. Still, the trend shows that a single supporting measurement helps, and that accuracy continues to improve when more supporting measurements are required for inclusion. The upward curve at the right of the kinase one-out and subset-out plots might indicate artifactual frequent hitters that are followed up in many assays, but by non-specific mechanisms that thus are not well predicted.\u003c/p\u003e\n\u003cp\u003eIn summary, for compounds unlike those already tested in a given assay, MMRM cold-start predictions are little better than ST-RFR. For imputations, predictions were better than ST-RFR for compounds tested with just a few supporting assays. The predictions improve as the number of supporting measurements increases until the improvements are substantial. Since MMRMs are expensive, this further supports the high value of employing MMRMs for virtual on- and off-target screens of the compound collection on which the MMRMs were trained, but discourages their use for screens on other compound collections where predictions will be cold starts on novel compounds and cost/benefit will be poor. It also indicates that the improved one-out and subset-out predictions result partly from capturing more supporting training measurements, many of which are missed by the all-out model training sets.\u003c/p\u003e\n\u003cp\u003eModel quality and decisions:\u003c/p\u003e\n\u003cp\u003eUnderstanding the quality of a given model is important for making decisions about how it should best be used. For example, if undertaking a virtual screen using a fairly low-quality model with signed-r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;0.35, where the false positive rate will be high, one might first order and test 10,000\u0026thinsp;+\u0026thinsp;compounds at a single concentration, then order dose-response plates to determine IC\u003csub\u003e50\u003c/sub\u003es just for the single-point hits. A high-quality model with signed-r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;0.6 should be at least as good as a single-concentration percent inhibition. One might, therefore, just order several hundred compounds at 10 concentrations for immediate IC\u003csub\u003e50\u003c/sub\u003e determination, saving maybe 3 months.\u003c/p\u003e\n\u003cp\u003eSimilarly, if an off-target or potential MoA activity is predicted for an assay that is currently running in the company, there is very little cost to add your compound to the queue even if the model only has signed-r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;0.3. If the assay is only running at a CRO, the threshold is somewhat higher. If the assay must be restarted internally, the threshold is higher yet.\u003c/p\u003e\n\u003cp\u003eConsensus affects decisions. If most members of a compound series are predicted to hit a given assay, that consensus strengthens the prediction even if signed-r\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e for the model is low. Similarly, if a compound is predicted to hit multiple assays for the same target, which were run at different times on different compound sets, that strengthens the prediction of even marginal models. If a compound from a luciferase assay is predicted to also hit dozens of other unrelated luciferase assays, odds are it is an artifact even if models are good. If a compound is predicted to hit several cytotoxicity assays, that antimicrobial phenotypic virtual screening result is probably not specific. Similar considerations apply to hit-list triaging based on ligand- or lipid-efficiency, isoform selectivity, virtual counter-screens etc.\u003c/p\u003e\n\u003cp\u003eCompound cost also greatly affects the model quality required. A compound that must be repurchased, or worse resynthesized, requires a higher prediction confidence than one pulled from the company archive. Predictions for project-focused library synthesis require a lower threshold than many one-off syntheses. If a novel compound from generative chemistry requires a new and difficult synthesis, the model should be very good indeed to justify the effort. Because it affects important decisions, it is important to have good estimates of final model quality.\u003c/p\u003e\n\u003cp\u003eStatistical significance vs. meaningful difference:\u003c/p\u003e\n\u003cp\u003eThese were large datasets. Given enough data, almost any minor systematic accuracy difference between algorithms will be statistically higher than chance. A more important question is whether a difference has practical consequences for your use-cases. What constitutes a meaningful accuracy difference depends on context. A slightly more accurate model for a virtual screen of your current projects primary assay might be important. By contrast, if the number of potential off-target assays that reach signed-r\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;\u0026gt;\u0026thinsp;0.3 is 5% higher with one algorithm, is that enough reason to select that method over another? That minor distinction becomes just one practical consideration among many in choosing an algorithm. Additional practical pros and cons of each method are therefore discussed below.\u003c/p\u003e\n\u003cp\u003ePractical pros and cons of the algorithms:\u003c/p\u003e\n\u003cp\u003eMinor differences in prediction accuracy is only one reason to choose one prediction algorithm over another. Each algorithm has strengths and weaknesses.\u003c/p\u003e\n\u003cp\u003ePQSAR pros:\u003c/p\u003e\n\u003cp\u003eThe accuracy of pQSAR is comparable to the best MMRMs tested, so that is neither a reason to prefer or discount it. Profile-QSAR does have several practical advantages. pQSAR uses one-out training sets. One-out training sets most-closely reflect the final production none-out model performance for the use-case of screening the compound collection on which the models were trained. Since understanding a model\u0026rsquo;s quality affects decisions for how the model can be used, the use of one-out training sets constitutes pQSAR\u0026rsquo;s most important advantage.\u003c/p\u003e\n\u003cp\u003eA second strength of pQSAR is facile transfer learning and adaptive learning, i.e., what is required to add a new model to the multitask ensemble, or to update a model by adding or subtracting measurements to an assay with an existing model. Sometimes a new assay becomes available, or one needs an additional endpoint, such as A\u003csub\u003emax\u003c/sub\u003e, rather than just AC\u003csub\u003e50\u003c/sub\u003e. Retraining a large number of hyperparameters is not needed. Referring back to Fig. \u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003eA, using the stored matrix of predictions from the RFR models, it simply requires training one additional PLS model. Similarly, if new data are available for an existing model, or if some measurements should be removed because they hit in a counter screen or fail QC, it simply requires retraining that single PLS model. Since the data for a new or updated assay would not have been used to train its own level 2 PLS model, the new model is supported by all the data in the existing multitask model exactly as if the new model had been in the initial training. The new model will not have contributed its support to the other models in the multitask collection, but a single new model\u0026rsquo;s contribution among 1000s is generally very incremental. Note that if any entirely new compounds are added that had not been tested in any assay (cold-starts), their RFR profiles are missing from the stored matrix, so RFR predictions for those compounds must also be computed before training the new PLS model, which does add a little more time.\u003c/p\u003e\n\u003cp\u003eA third advantage is federated models, i.e. where several companies collaborate by training a super-multitask model covering the union of all their compounds, assays and bioactivity data, without sharing the actual structures or experimental data. Federated models can be built for multitask deep neural networks, but it is difficult. The IMI MELLODDY consortium did it in a 3-year, \u0026euro;18\u0026nbsp;million collaboration of 16 public and private partners.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e38\u003c/span\u003e\u003c/sup\u003e Collaborative modeling with pQSAR is much simpler.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e Each company, behind its own firewall, trains innocuous single-task level-1 models on its own data. It assembles the collection of level-1 models into a compiled \u0026ldquo;black box\u0026rdquo; program that takes chemical structures in and outputs a vector of numbers which are the bioactivity predictions for the unspecified assays. It can anonymize the predictions further by applying a similarity transformation that mixes the activity predictions but preserves the combined information. Each partner exports its black-box level-1 models into a shared location, from which each company imports its collaborator\u0026rsquo;s models back inside its firewall. It runs the predictions from the collaborator\u0026rsquo;s level-1 models on its compounds and adds them to its own level-1 predictions. It then trains the level-2 PLS models for its assays on the expanded level-1 profile. The calculation is virtually the same as if the collaborators had actually shared all the data, except that the work is distributed among the collaborators and each only gets pQSAR models for its own assays.\u003c/p\u003e\n\u003cp\u003epQSAR models are interpretable in terms of biology. The PLS coefficients from pQSAR models have been shown to capture unknown mechanisms of action and polypharmacologies.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e39\u003c/span\u003e\u003c/sup\u003e This indicates that pQSAR is not employing some kind of meaningless curve fitting, but is learning underlying biology from the bioactivity data.\u003c/p\u003e\n\u003cp\u003eProfile-QSAR is a \u0026ldquo;turn-key\u0026rdquo; calculation. It was built with default or standard parameters for the RFR and PLS models. When we apply it to completely different data sets, we don\u0026rsquo;t tune any parameters\u0026mdash;we only change the data.\u003c/p\u003e\n\u003cp\u003eProfile-QSAR is embarrassingly parallel. It runs on conventional CPUs\u0026mdash;GPUs or TPUs are not needed. Training Novartis\u0026rsquo;s very large pQSAR of 20\u0026nbsp;million measurements from 2\u0026nbsp;million compounds by 15,000 assays, and making almost 100\u0026nbsp;billion IC\u003csub\u003e50\u003c/sub\u003e predictions for 6.5 million compounds, takes about 3 days on a modest cluster of 600 CPUs and is repeated every month. This could be an advantage for some users.\u003c/p\u003e\n\u003cp\u003eThe pQSAR source code is available at GitHub\u0026ndash;- Novartis/pQSAR.\u003c/p\u003e\n\u003cp\u003ePQSAR cons:\u003c/p\u003e\n\u003cp\u003epQSAR is the slowest algorithm in this study for making cold-start activity predictions on new compounds for just a few assays. This is because the activity of new compounds must first be computed for all the level-1 RFR models before running the few level-2 PLS models. This is only a disadvantage for predictions on a small number of specific assays, because for full-profile off-target, polypharmacology or MOA predictions the cost of level-1 models is amortized over all the level-2 PLS predictions. Thus, this limits its application in cold-start optimization problems like generative chemistry or interactive design, which as mentioned above is only worthwhile for analogs. A work-around is to train fast, single-task models that approximate pQSAR by augmenting the single-task training sets with synthetic data from pQSAR predictions for that assay using all 10,000s to millions of compounds in the whole MMRM training set.\u003c/p\u003e\n\u003cp\u003eFor generative chemistry, the optimization procedures exploit the artifacts in any model, which leads to false positives. We therefore also use pQSAR as a valuable orthogonal prediction method to confirm the predicted activity of the best compounds from generative chemistry optimized on other models.\u003c/p\u003e\n\u003cp\u003eAnother limitation of pQSAR is that each assay is independent. Unlike some methods, pQSAR has no simple way to explicitly incorporate side information on the outputs, i.e. assay properties such as incubation time or cofactor concentration that might inform the model that some assays are more closely related. This is implicitly incorporated, because assays run at e.g. different incubation times or ATP concentrations are separate assays, but explicit assay properties might improve models in some situations or in other domains besides bioactivity prediction.\u003c/p\u003e\n\u003cp\u003eAlchemite pros:\u003c/p\u003e\n\u003cp\u003eBeyond its performance as generally one of the more accurate multi-task algorithms considered here, Alchemite provides other benefits. Alchemite is available via easy-to-use no-code web graphical user interfaces, including in the drug discovery platform Cerella,\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e40\u003c/span\u003e\u003c/sup\u003e which means that it is much more straightforward to deploy to large teams of non-data scientists than research-level tools that require the user to run programs from the command line, use python or install software on their own machines. Beyond simply making imputations of missing assay data or predictions for cold-start compounds (both of which tasks have specific web pages associated with them), Alchemite also has specific components for other tasks:\u003c/p\u003e\n\u003cul\u003e\n \u003cli\u003e\n \u003cp\u003eUncertainty quantification associated with each prediction. This can enable a user to focus on the most confident predictions, which increases the accuracy of the remaining predictions used to take decisions,\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e41\u003c/span\u003e\u003c/sup\u003e or, conversely, to focus on compounds that are predicted to be relatively highly performing but have high uncertainty and so may be risky but potentially lucrative options. Uncertainty estimates also enable the identification of outlier data points, which could be false positive or false negative points. The uncertainty quantification in Alchemite is non-parametric, enabling it to capture more complex predictive distributions than a standard Gaussian.\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003eAn \u0026ldquo;Importance Matrix\u0026rdquo; that highlights the inter-assay and descriptor-assay relationships the model is leveraging, which is helpful for identifying mechanisms of action or adverse outcome pathways\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003eBayesian experimental design directly integrated with the predictive modelling component, which proposes experiments based on the Importance Matrix, model uncertainty, and predicted performance against a target profile\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003eAlchemite also models categorical data on an equal footing to the continuous/regression data used in this study. This can be useful for example where data is qualified (e.g. \u0026gt;10 uM) and not useful for regression modelling.\u003c/p\u003e\u003cbr\u003e\n \u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eAs Alchemite is used daily in a variety of applications, including in fields beyond drug discovery (e.g. materials science, industrial chemistry, manufacturing), it is flexible enough to apply to many different use-cases beyond assay IC\u003csub\u003e50\u003c/sub\u003e prediction, including pharmacokinetic curve prediction,\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e42\u003c/span\u003e\u003c/sup\u003e toxicology modeling,\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e43\u003c/span\u003e\u003c/sup\u003e and sensory property prediction.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e41\u003c/span\u003e\u003c/sup\u003e These diverse applications generate varied improvements in the Alchemite approach, all contributing to the same tool.\u003c/p\u003e\n\u003cp\u003eAlchemite cons:\u003c/p\u003e\n\u003cp\u003eAlchemite is commerical software, and therefore not available for free, although an academic program is available. Alchemite does not provide a database of existing data, as generally new problems require specific data to generate good models, but this means that existing data on the problem of interest is required to begin using Alchemite (although the software includes a Design of Experiments component to assist with initial data generation). Due to its applicability to a variety of different domains, Alchemite contains seven hyperparameters that need to be optimized for each new application (as different applications generally are best served by slightly different model parameters): the Alchemite software contains Bayesian hyperparameter optimization tools, but this takes time and computational resource relative to hyperparameter-free machine learning methods.\u003c/p\u003e\n\u003cp\u003eMetaNN pros:\u003c/p\u003e\n\u003cp\u003eMetaNN has been proven strong in bioactivity predictions; particularly, MetaNN performs better in smaller datasets, which have already been well predicted by the consensus DNN. We have applied MetaNN models to several drug discovery projects, both in-house and in the collaborations. In one project, we have observed that MetaNN bioactivity predictions provide much better differentiation even among the most active compounds, which can significantly help prioritizing compound designs; and this is the advantage that single task model cannot achieve. In another project, we have demonstrated that using metaNN models can help to select the sub-series for the next phase of compound optimization.\u003c/p\u003e\n\u003cp\u003eMetaNN has high storage efficiency, storing only one consensus DNN optimized across all tests. This feature enables much faster and more convenient calculations in applications.\u003c/p\u003e\n\u003cp\u003eMetaNN cons:\u003c/p\u003e\n\u003cp\u003eOne con of MetaNN is that the consensus DNN must go through a full re-training, which requires certain computational power, and this is needed even when a small fraction of the whole database is updated.\u003c/p\u003e\n\u003cp\u003eMT-DNN pros:\u003c/p\u003e\n\u003cp\u003eMT-DNNs, a sophisticated branch of deep learning techniques, have become widely recognized in machine learning. Besides their excellent predictive capabilities, shown in this study, MT-DNNs possess several advantages that can make them a preferred choice in various applications:\u003c/p\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e\n\u003cul\u003e\n \u003cli\u003e\n \u003cp\u003eAvailability of Ready-to-Use Frameworks: MT-DNNs can be implemented using a range of general-purpose deep learning frameworks, including Theano,\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e44\u003c/span\u003e\u003c/sup\u003e Tensorflow,\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e45\u003c/span\u003e\u003c/sup\u003e Chainer,\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e46\u003c/span\u003e\u003c/sup\u003e Torch,\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e47\u003c/span\u003e\u003c/sup\u003e and PyTorch.\u003csup\u003e\u003cspan class=\"CitationRef\"\u003e48\u003c/span\u003e\u003c/sup\u003e While these tools are not exclusive to MT-DNNs, they facilitate the modeling of such networks by offering robust functionalities, making advanced neural networks accessible to researchers with even basic programming skills.\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003eComputational Efficiency: Although deep neural networks generally benefit from GPU acceleration, the extent of efficiency gains can vary. MT-DNNs are particularly well-suited for GPUs, which can execute thousands of parallel operations to significantly speed up computations and minimize memory access latency, especially beneficial when training with large datasets.\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003eKnowledge Transfer: MT-DNNs leverage extensive datasets across multiple tasks, enhancing the learning process for each by transferring insights from data-rich tasks to those with less data. This capability helps reduce overfitting and improves the robustness and generalization of the model to new, unseen data.\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003eFlexibility in Data Handling: MT-DNNs demonstrate versatility in managing different data types. They can process information related to either the properties of compounds or tasks independently or in combination.\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003eComplex Data Management: MT-DNNs efficiently handle data of varying sizes and complexities. They are adept at working with data in complex tensor forms, which enables them to tackle more sophisticated predictive challenges. In this work, we utilized an MT-DNN capable of processing information presented in a matrix form.\u003c/p\u003eMT-DNN cons:\n \u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eDespite these advantages, MT-DNNs also present several challenges:\u003c/p\u003e\n\u003cul\u003e\n \u003cli\u003e\n \u003cp\u003eLack of Interpretability: MT-DNNs, like many deep learning models, operate as \u0026quot;black boxes,\u0026quot; meaning that the model operation is not easily understandable. It can be a critical drawback in fields requiring transparent decision-making processes.\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003eDemand for Substantial Computational Resources: MT-DNNs require significant computational power, which includes the need for high-performance GPUs and substantial memory. It can escalate costs and limit the feasibility of using MT-DNNs in real time.\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003eRisk of Negative Transfer: While knowledge transfer is a significant strength of MT-DNNs, there is a potential risk of negative transfer. This occurs when the tasks are not sufficiently related, or the shared features do not benefit all tasks, potentially leading to worse performance than if the tasks were trained independently.\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003eDependency on Task Relatedness: The effectiveness of MT-DNNs heavily depends on the relatedness of the tasks involved. When tasks are too dissimilar, the benefits of shared representations and joint training can decrease, leading to suboptimal outcomes.\u003c/p\u003e\u003cbr\u003e\n \u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eMacau pros:\u003c/p\u003e\n\u003cp\u003eMacau did very well on the kinase assays, although less well on the diverse assay collection. Macau is a sophisticated machine learning algorithm that can deal with sparse data and integrate a wide range of features, which makes it a powerful tool for multi-task learning across various domains. Unlike many traditional machine learning models that provide point estimates, Macau, being Bayesian, inherently provides probabilistic predictions. This means that for every prediction it makes, it also estimates the uncertainty associated with that prediction. It leads to the main advantages distinguishing Macau from many other multi-task machine learning methods:\u003c/p\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e\n\u003cul\u003e\n \u003cli\u003e\n \u003cp\u003eImproved Risk Assessment: The uncertainty estimation intrinsic to Macau is invaluable for risk assessment. In drug discovery, the confidence level of a prediction can significantly influence the decision-making process, such as selecting candidates in early drug discovery. This capability allows researchers to make more informed and cautious decisions, considering not only the model\u0026apos;s predictions but also their reliability.\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003eIncorporation of Prior Knowledge: Macau\u0026apos;s Bayesian nature facilitates the incorporation of prior knowledge or expert insights into its modeling process. This ability to integrate existing knowledge effectively reduces prediction uncertainties and refines the learning process, making Macau particularly adept in fields where historical data or expert understanding is pivotal.\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003eGuidance for Data Collection: Macau\u0026rsquo;s approach to uncertainty estimation is particularly useful in pinpointing areas where additional data collection could enhance model accuracy. In multi-task learning, this means accurately identifying specific targets or compounds that, if expanded upon, could significantly reduce uncertainty and improve model performance.\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003eUtility in Exploratory Analysis: The algorithm\u0026rsquo;s capability to quantify uncertainty can also be leveraged for exploratory analysis. This feature enables researchers to either concentrate on predictions with low uncertainty for reliable predictions or explore those with high uncertainty to uncover potential anomalies.\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003eEnhanced Descriptor Integration: Macau\u0026apos;s algorithm excels in simultaneously processing descriptors of both compounds and assays, a critical aspect in drug discovery and chemical informatics. This dual capability allows for a more comprehensive analysis, as it can consider the intricate interplay between chemical properties of compounds and the biological context of assays.\u003c/p\u003e\n \u003c/li\u003e\n \u003cli\u003e\n \u003cp\u003eAdvanced Tensor Data Handling: One of Macau\u0026apos;s standout features is its ability to operate not just with matrices but also with tensors of data. This capability is particularly advantageous in scenarios where data is multi-dimensional and complex, such as in multi-factorial drug response studies or when dealing with time-series data. Tensors allow for the incorporation of additional dimensions, like time, dose-response curves, or multi-layered genetic information, providing a more nuanced and detailed analysis.\u003c/p\u003eIn summary, Macau\u0026apos;s approach to uncertainty calculation, rooted in its Bayesian framework, can provide a more nuanced and comprehensive understanding of predictions, enhance the reliability of decision-making processes, and add a layer of transparency and trust to the model\u0026apos;s outputs.\n \u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eMacau cons:\u003c/p\u003e\n\u003cp\u003eLike many advanced predictive algorithms, Macau, while offering considerable advantages, also presents notable challenges related to its computational demands, the complexity of its implementation, the dependence on feature quality, and interpretability issues. These challenges are based on Macau\u0026apos;s reliance on Bayesian methods and the high level of programming expertise required for its effective application.\u003c/p\u003e\n\u003cp\u003eThe implementation of Macau, with its intricate Bayesian foundation, is inherently more complex than more straightforward machine learning algorithms. Consequently, users must possess advanced programming skills and a deep understanding of the algorithm to utilize Macau\u0026apos;s capabilities fully. This necessity potentially limits its accessibility to a broader user base. While average researchers might employ Macau for basic predictive tasks, more nuanced applications \u0026mdash; such as model tuning that accounts for uncertainty predictions and incorporates expert knowledge \u0026mdash; demand a significantly higher level of technical proficiency.\u003c/p\u003e\n\u003cp\u003eFurthermore, effective operation with Macau extends beyond programming knowledge; it requires a comprehensive understanding of Bayesian statistics. A solid foundation in probabilistic modeling and Bayesian inference and familiarity with techniques such as Markov Chain Monte Carlo (MCMC) are essential.\u003c/p\u003e\n\u003cp\u003eTherefore, while Macau offers a wide array of opportunities for enhancing predictive analytics, its sophisticated nature can hinder its widespread and straightforward use. Mastery of both the theoretical underpinnings and practical implementation aspects of Macau is essential for leveraging its full potential in various practical applications.\u003c/p\u003e"},{"header":"Conclusions","content":"\u003cp\u003e6 MMRM algorithms were compared using realistically novel test sets, trained on large compound clusters and tested on singletons and small clusters, on 2 published assay collections from ChEMBL: a homogenous set of 159 kinase assays and a heterogeneous collection of 4276 diverse assays. The comparison was complicated, because methods that train all models simultaneously must leave out all test-set measurements for all assays to avoid test-set leakage, 25% of the data in our cases. Because realistic test sets are trained on large compound clusters and leave out the most unique compounds, all-out model training-set collections are missing particularly informative measurements. MMRMs which train models one-at-a-time only leave out data for each assay as it is trained, less than 1% and 0.1% of the data in our kinase and diverse assay collections respectively. These one-out models should be very close to the final production models which don\u0026rsquo;t leave out any data, and thus should give the best estimate of the final none-out models\u0026rsquo; predictive performance on the compound collections on which the models were trained. To enable better comparisons across all algorithms, subset-out models were also trained. These only built models for an experimentally designed roughly 10% of kinase or 1% of diverse subsets of the assays, thus leaving out only\u0026thinsp;~\u0026thinsp;10% or 1% of the test data, and including the realistic test-set compound measurements for the remaining\u0026thinsp;~\u0026thinsp;90% or 99% of assays. The analysis found that for the realistic test-set predictions, all-out model predictions were much worse than the one-out models, and by extension the final none-out production models for our use case of predictions on the training archive. The subset-out model accuracy was close to the one-out models, suggesting that for methods for which training thousands of one-out models are impracticable, a compromise of building multiple separate subset-out models could be used to evaluate the likely performance of the final models. Again, this only affects the evaluation of final model quality, not the performance of the final models, which are always none-out models.\u003c/p\u003e\u003cp\u003eGenerally, the 5 \u0026ldquo;advanced\u0026rdquo; MMRMs did substantially better than the benchmark ST-RFR models on the realistic test sets. Comparing algorithms showed that, in most cases, the performance of the better methods, while sometimes statistically distinguishable, were still relatively close, at least on the more homogeneous kinase assay collection. Macau had trouble with the larger diverse assay collection and the MMRM benchmark IMC performed less well, as expected. The 5 very different methods also agreed on which individual assays could be modeled well, and which could not. This supports the hypothesis that many diverse MMRM algorithms are comparable in performance at a fundamental level and suggests that they have reached a limit of what signal can be extracted from the data.\u003c/p\u003e\u003cp\u003eThe advantage of MMRMs over ST-RFR was marginal for cold-start predictions on these realistic test-set compounds very unlike the individual assay\u0026rsquo;s training sets. MMRM imputations were much more accurate than ST-RFR on these challenging predictions. The larger proportion of imputations in one-out vs. all-out models likely contributes to their better performance. Even a few supporting measurements improved performance, and the improvement was especially notable for compounds with large numbers of supporting measurements. This implies that MMRMs are best for predictions on the compound-collection used for MMRM training: for on-target virtual screens to find additional novel chemical matter and for hit-list triaging, off-target, promiscuity, MoA, polypharmacology or drug repurposing predictions of existing compounds or their close analogs. It also implies that models should be updated frequently\u0026mdash;monthly to weekly\u0026mdash;so predictions on recent compounds will include initial measurements and thus be imputations. We expect little advantage for cold-start predictions on compounds outside the multitask training space, such as vendor collections or exploratory generative chemistry, and at significant extra cost.\u003c/p\u003e\u003cp\u003eThe algorithms differ with respect to other practical considerations. We therefore included a detailed discussion of the pros and cons of each method. While this comparison dealt specifically with drug discovery, similar conclusions should apply to modeling other domains. In fact, only pQSAR was originally developed for drug discovery. Alchemite was originally developed for material science, and matrix factorization, meta-learners and DNNs were used in many diverse domains before drug discovery.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\n\u003cp\u003eE.M. arranged the collaboration, wrote the main manuscript text, developed pQSAR;X.Z. performed the pQSAR calculations and analysis;P.R. performed analysis, especially significance testing;S.K. performed analysis;E.S. built MT-DNN, Macau and IMC models, performed calculations and analysis;L.T. and Y.W. developed metaNN;Z.W. performed metaNN calculations;T.W., G.C. and MS performed Alchemite calculations and analysis.\u003c/p\u003e\n\u003ch2\u003eAcknowledgments:\u003c/h2\u003e\n\u003cp\u003eDr. Li Tian and Mr. Zijian Wang acknowledge \u0026quot;Laboratory for Synthetic Chemistry and Chemical Biology\u0026quot; under the Health@InnoHK Program launched by Innovation and Technology Commission, The Government of Hong Kong Special Administrative Region of the People\u0026apos;s Republic of China, for the funding support of this research, as well as for providing the studentship for Mr. Wang from year 2022 to year 2025.\u003c/p\u003e\n\u003ch2\u003eData and Software Availability:\u003c/h2\u003e\n\u003cp\u003eThe compounds, assays, experimental and pQSAR predicted bioactivities and training/test set splits are available in the Supporting Information from 2 papers.\u003csup\u003e1,6\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003eThe pQSAR source code is openly accessible at GitHub\u0026ndash;- Novartis/pQSAR.\u003c/p\u003e\n\u003cp\u003eMacau was implemented within the open-source Bayesian Matrix and Tensor Factorization framework, SMURFF.\u003csup\u003e35\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003eThe Python code for the MT-DNN prediction algorithm is openly accessible on GitHub.\u003csup\u003e34\u003c/sup\u003e\u003c/p\u003e\n\u003ch2\u003eData Availability\u003c/h2\u003e\n\u003cp\u003eThe compounds, assays, experimental and pQSAR predicted bioactivities and training/test set splits are available in the Supporting Information from 2 papers: https://doi.org/10.1021/ACS.JCIM.9B00375 and https://doi.org/10.1021/acs.jcim.7b00166.The pQSAR source code is openly accessible at [GitHub\u0026ndash;- Novartis/pQSAR](https:/github.com/Novartis/pQSAR) .Macau was implemented within the open-source Bayesian Matrix and Tensor Factorization framework, SMURFF.The Python code for the MT-DNN prediction algorithm is openly accessible on GitHub.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eMartin, E. J.; Polyakov, V. R.; Zhu, X. W.; Tian, L.; Mukherjee, P.; Liu, X. All-Assay-Max2 PQSAR: Activity Predictions as Accurate as Four-Concentration IC50s for 8558 Novartis Assays. \u003cem\u003eJ Chem Inf Model\u003c/em\u003e \u003cstrong\u003e2019\u003c/strong\u003e. https://doi.org/10.1021/ACS.JCIM.9B00375.\u003c/li\u003e\n\u003cli\u003eHansch, C.; Fujita Vol, T.; Roberts, R. B.; Cowie, D. B.; Britten, R.; Abelson, P. H.; Moses, V.; Holm-Hansen, O.; Bassham, J. A.; Calvin, M.; Mol Biol, J.; Corwin Hansch, B.; Fujita, T. Analysis. A Method for the Correlation of Biological Activity and Chemical Structure. \u003cem\u003eJ. Biol. Chem\u003c/em\u003e \u003cstrong\u003e1963\u003c/strong\u003e, \u003cem\u003e39\u003c/em\u003e (61), 284.\u003c/li\u003e\n\u003cli\u003eHansch, C.; Fujita Vol, T.; Roberts, R. B.; Cowie, D. B.; Britten, R.; Abelson, P. H.; Moses, V.; Holm-Hansen, O.; Bassham, J. A.; Calvin, M.; Mol Biol, J.; Corwin Hansch, B.; Fujita, T. \u0026rho;-\u0026sigma;-\u0026pi; Analysis. A Method for the Correlation of Biological Activity and Chemical Structure. \u003cem\u003eJ Am Chem Soc\u003c/em\u003e \u003cstrong\u003e1964\u003c/strong\u003e, \u003cem\u003e86\u003c/em\u003e (8), 1616\u0026ndash;1626. https://doi.org/10.1021/JA01062A035.\u003c/li\u003e\n\u003cli\u003eHammett, L. P. The Effect of Structure upon the Reactions of Organic Compounds. Benzene Derivatives. \u003cem\u003eJ Am Chem Soc\u003c/em\u003e \u003cstrong\u003e1937\u003c/strong\u003e, \u003cem\u003e59\u003c/em\u003e (1), 96\u0026ndash;103. https://doi.org/10.1021/JA01280A022/ASSET/JA01280A022.FP.PNG_V03.\u003c/li\u003e\n\u003cli\u003eLapinsh, M.; Prusis, P.; Gutcaits, A.; Lundstedt, T.; Wikberg, J. E. S. Development of Proteo-Chemometrics: A Novel Technology for the Analysis of Drug-Receptor Interactions. \u003cem\u003eBiochim Biophys Acta Gen Subj\u003c/em\u003e \u003cstrong\u003e2001\u003c/strong\u003e, \u003cem\u003e1525\u003c/em\u003e (1\u0026ndash;2), 180\u0026ndash;190. https://doi.org/10.1016/S0304-4165(00)00187-2.\u003c/li\u003e\n\u003cli\u003eMartin, E. J.; Polyakov, V. R.; Tian, L.; Perez, R. C. Profile-QSAR 2.0: Kinase Virtual Screening Accuracy Comparable to Four-Concentration IC50s for Realistically Novel Compounds. \u003cem\u003eJ Chem Inf Model\u003c/em\u003e \u003cstrong\u003e2017\u003c/strong\u003e, \u003cem\u003e57\u003c/em\u003e (8), 2077\u0026ndash;2088. https://doi.org/10.1021/acs.jcim.7b00166.\u003c/li\u003e\n\u003cli\u003eSosnina, E. A.; Sosnin, S.; Fedorov, M. V. Improvement of Multi-Task Learning by Data Enrichment: Application for Drug Discovery. \u003cem\u003eJ Comput Aided Mol Des\u003c/em\u003e \u003cstrong\u003e2023\u003c/strong\u003e, \u003cem\u003e37\u003c/em\u003e (4), 183\u0026ndash;200. https://doi.org/10.1007/s10822-023-00500-w.\u003c/li\u003e\n\u003cli\u003eWu, J.; Chen, Y.; Wu, J.; Zhao, D.; Huang, J.; Lin, M.; Wang, L. Large-Scale Comparison of Machine Learning Methods for Profiling Prediction of Kinase Inhibitors. \u003cem\u003eJ Cheminform\u003c/em\u003e \u003cstrong\u003e2024\u003c/strong\u003e, \u003cem\u003e16\u003c/em\u003e (1), 13. https://doi.org/10.1186/s13321-023-00799-5.\u003c/li\u003e\n\u003cli\u003eFeng, B.; Liu, Z.; Huang, N.; Xiao, Z.; Zhang, H.; Mirzoyan, S.; Xu, H.; Hao, J.; Xu, Y.; Zhang, M.; Wang, S. A Bioactivity Foundation Model Using Pairwise Meta-Learning. \u003cem\u003eNat Mach Intell\u003c/em\u003e \u003cstrong\u003e2024\u003c/strong\u003e, \u003cem\u003e6\u003c/em\u003e (8), 962\u0026ndash;974. https://doi.org/10.1038/s42256-024-00876-w.\u003c/li\u003e\n\u003cli\u003eMartin, E.; Sullivan, D. Iterative Kinase Medium-Throughput Screening with Profile-QSAR and Ensemble Surrogate AutoShim for Fast Predictive Kinase Docking without a Crystal Structure. In \u003cem\u003eKeystone symposium: Structure Based Drug Discovery (D6)\u003c/em\u003e; Burley, S. K., Shoichet, B. K., Bartlett, P. A., Eds.; Whistler, BC, Canada, 2006.\u003c/li\u003e\n\u003cli\u003eMartin, E.; Mukherjee, P.; Sullivan, D.; Jansen, J. Profile-QSAR: A Novel Meta-QSAR Method That Combines Activities across the Kinase Family To Accurately Predict Affinity, Selectivity, and Cellular Activity. \u003cem\u003eJ Chem Inf Model\u003c/em\u003e \u003cstrong\u003e2011\u003c/strong\u003e, \u003cem\u003e51\u003c/em\u003e (8), 1942\u0026ndash;1956. https://doi.org/10.1021/ci1005004.\u003c/li\u003e\n\u003cli\u003eMartin, E.; Mukherjee, P.; Sullivan, D.; Jansen, J. Profile-QSAR: A Novel Meta-QSAR Method That Combines Activities across the Kinase Family To Accurately Predict Affinity, Selectivity, and Cellular Activity. \u003cem\u003eJ. Chem. Inf. Model\u003c/em\u003e \u003cstrong\u003e2011\u003c/strong\u003e, \u003cem\u003e51\u003c/em\u003e, 1942\u0026ndash;1956. https://doi.org/10.1021/ci1005004.\u003c/li\u003e\n\u003cli\u003eMartin, E. J.; Zhu, X. W. Collaborative Profile-QSAR: A Natural Platform for Building Collaborative Models among Competing Companies. \u003cem\u003eJ Chem Inf Model\u003c/em\u003e \u003cstrong\u003e2021\u003c/strong\u003e, \u003cem\u003e61\u003c/em\u003e (4), 1603\u0026ndash;1616. https://doi.org/10.1021/ACS.JCIM.0C01342/ASSET/IMAGES/LARGE/CI0C01342_0016.JPEG.\u003c/li\u003e\n\u003cli\u003eConduit, B. D.; Jones, N. G.; Stone, H. J.; Conduit, G. J. Probabilistic Design of a Molybdenum-Base Alloy Using a Neural Network. \u003cem\u003eScr Mater\u003c/em\u003e \u003cstrong\u003e2018\u003c/strong\u003e, \u003cem\u003e146\u003c/em\u003e, 82\u0026ndash;86. https://doi.org/10.1016/j.scriptamat.2017.11.008.\u003c/li\u003e\n\u003cli\u003eConduit, B. D.; Jones, N. G.; Stone, H. J.; Conduit, G. J. Design of a Nickel-Base Superalloy Using a Neural Network. \u003cem\u003eMater Des\u003c/em\u003e \u003cstrong\u003e2017\u003c/strong\u003e, \u003cem\u003e131\u003c/em\u003e, 358\u0026ndash;365. https://doi.org/10.1016/j.matdes.2017.06.007.\u003c/li\u003e\n\u003cli\u003eWhitehead, T. M.; Chen, F.; Daly, C.; Conduit, G. J. Accelerating the Design of Automotive Catalyst Products Using Machine Learning. \u003cem\u003eJohnson Matthey Technology Review\u003c/em\u003e \u003cstrong\u003e2022\u003c/strong\u003e, \u003cem\u003e66\u003c/em\u003e (2), 130\u0026ndash;136. https://doi.org/10.1595/205651322x16270488736796.\u003c/li\u003e\n\u003cli\u003eObrezanova, O.; Martinsson, A.; Whitehead, T.; Mahmoud, S.; Bender, A.; Miljković, F.; Grabowski, P.; Irwin, B.; Oprisiu, I.; Conduit, G.; Segall, M.; Smith, G. F.; Williamson, B.; Winiwarter, S.; Greene, N. Prediction of in Vivo Pharmacokinetic Parameters and Time-Exposure Curves in Rats Using Machine Learning from the Chemical Structure. \u003cem\u003eMol Pharm\u003c/em\u003e \u003cstrong\u003e2022\u003c/strong\u003e, \u003cem\u003e19\u003c/em\u003e (5), 1488\u0026ndash;1504. https://doi.org/10.1021/acs.molpharmaceut.2c00027.\u003c/li\u003e\n\u003cli\u003eTse, E. G.; Aithani, L.; Anderson, M.; Cardoso-Silva, J.; Cincilla, G.; Conduit, G. J.; Galushka, M.; Guan, D.; Hallyburton, I.; Irwin, B. W. J.; Kirk, K.; Lehane, A. M.; Lindblom, J. C. R.; Lui, R.; Matthews, S.; McCulloch, J.; Motion, A.; Ng, H. L.; \u0026Ouml;eren, M.; Robertson, M. N.; Spadavecchio, V.; Tatsis, V. A.; van Hoorn, W. P.; Wade, A. D.; Whitehead, T. M.; Willis, P.; Todd, M. H. An Open Drug Discovery Competition: Experimental Validation of Predictive Models in a Series of Novel Antimalarials. \u003cem\u003eJ Med Chem\u003c/em\u003e \u003cstrong\u003e2021\u003c/strong\u003e, \u003cem\u003e64\u003c/em\u003e (22), 16450\u0026ndash;16463. https://doi.org/10.1021/acs.jmedchem.1c00313.\u003c/li\u003e\n\u003cli\u003eIrwin, B. W. J.; Whitehead, T. M.; Rowland, S.; Mahmoud, S. Y.; Conduit, G. J.; Segall, M. D. Deep Imputation on Large‐scale Drug Discovery Data. \u003cem\u003eApplied AI Letters\u003c/em\u003e \u003cstrong\u003e2021\u003c/strong\u003e, \u003cem\u003e2\u003c/em\u003e (3). https://doi.org/10.1002/ail2.31.\u003c/li\u003e\n\u003cli\u003eIrwin, B. W. J.; Levell, J. R.; Whitehead, T. M.; Segall, M. D.; Conduit, G. J. Practical Applications of Deep Learning to Impute Heterogeneous Drug Discovery Data. \u003cem\u003eJ Chem Inf Model\u003c/em\u003e \u003cstrong\u003e2020\u003c/strong\u003e, \u003cem\u003e60\u003c/em\u003e (6), 2848\u0026ndash;2857. https://doi.org/10.1021/acs.jcim.0c00443.\u003c/li\u003e\n\u003cli\u003eWhitehead, T.; Irwin, B.; Hunt, P.; Segall, M.; Conduit, G. Imputation of Assay Bioactivity Data Using Deep Learning. \u003cem\u003eJ Chem Inf Model\u003c/em\u003e \u003cstrong\u003e2019\u003c/strong\u003e, \u003cem\u003e59\u003c/em\u003e (3), 1197 \u0026ndash; 1204. https://doi.org/10.1021/acs.jcim.8b00768.\u003c/li\u003e\n\u003cli\u003eStuckner, J.; Whitehead, T. M.; Parini, R. C.; Conduit, G. J.; Benafan, O.; Arnold, S. M. Design of Materials With Alchemite. \u003cem\u003eNASA Tech Memo\u003c/em\u003e \u003cstrong\u003e2022\u003c/strong\u003e, No. July.\u003c/li\u003e\n\u003cli\u003eSimm, J.; Arany, A.; Zakeri, P.; Haber, T.; Wegner, J. K.; Chupakhin, V.; Ceulemans, H.; Moreau, Y. Macau: Scalable Bayesian Multi-Relational Factorization with Side Information Using MCMC. 2015.\u003c/li\u003e\n\u003cli\u003eSimm, J.; Arany, A.; Zakeri, P.; Haber, T.; Wegner, J. K.; Chupakhin, V.; Ceulemans, H.; Moreau, Y. Macau: Scalable Bayesian Factorization with High-Dimensional Side Information Using MCMC. In \u003cem\u003e2017 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP)\u003c/em\u003e; 2017; pp 1\u0026ndash;6. https://doi.org/10.1109/MLSP.2017.8168143.\u003c/li\u003e\n\u003cli\u003eJain, P.; Dhillon, I. S. Provable Inductive Matrix Completion. \u003cstrong\u003e2013\u003c/strong\u003e.\u003c/li\u003e\n\u003cli\u003eDem\u0026scaron;ar, J. Statistical Comparisons of Classifiers over Multiple Data Sets. \u003cem\u003eJournal of Machine Learning Research\u003c/em\u003e \u003cstrong\u003e2006\u003c/strong\u003e, \u003cem\u003e7\u003c/em\u003e (1), 1\u0026ndash;30.\u003c/li\u003e\n\u003cli\u003eBenavoli, A.; Corani, G.; Mangili, F. Should We Really Use Post-Hoc Tests Based on Mean-Ranks? \u003cem\u003eJournal of Machine Learning Research\u003c/em\u003e \u003cstrong\u003e2016\u003c/strong\u003e, \u003cem\u003e17\u003c/em\u003e (5), 1\u0026ndash;10.\u003c/li\u003e\n\u003cli\u003eHOCHBERG, Y. A Sharper Bonferroni Procedure for Multiple Tests of Significance. \u003cem\u003eBiometrika\u003c/em\u003e \u003cstrong\u003e1988\u003c/strong\u003e, \u003cem\u003e75\u003c/em\u003e (4), 800\u0026ndash;802. https://doi.org/10.1093/biomet/75.4.800.\u003c/li\u003e\n\u003cli\u003eBergstra, J.; Bardenet, R.; Bengio, Y.; K\u0026eacute;gl, B. Algorithms for Hyper-Parameter Optimization. In \u003cem\u003eAdvances in Neural Information Processing Systems\u003c/em\u003e; Shawe-Taylor, J., Zemel, R., Bartlett, P., Pereira, F., Weinberger, K. Q., Eds.; Curran Associates, Inc., 2011; Vol. 24.\u003c/li\u003e\n\u003cli\u003eYao, H.; Huang, L.-K.; Zhang, L.; Wei, Y.; Tian, L.; Zou, J.; Huang, J.; Li, Z. (). Improving Generalization in Meta-Learning via Task Augmentation. In \u003cem\u003eProceedings of the 38th International Conference on Machine Learning\u003c/em\u003e; Meila, M., Zhang, T., Eds.; Proceedings of Machine Learning Research; PMLR, 2021; Vol. 139, pp 11887\u0026ndash;11897.\u003c/li\u003e\n\u003cli\u003eTian, L.; Huang, H. H.; Martin, E.; Wei, J. Y.; Xu, S.; Huang, J. Transfer Learning Applied in Predicting Small Molecule Bioactivity. \u003cem\u003ebioRxiv\u003c/em\u003e \u003cstrong\u003e2025\u003c/strong\u003e. https://doi.org/10.1101/2025.01.09.632140.\u003c/li\u003e\n\u003cli\u003ePaszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; Desmaison, A.; Kopf, A.; Yang, E.; DeVito, Z.; Raison, M.; Tejani, A.; Chilamkurthy, S.; Steiner, B.; Fang, L.; Bai, J.; Chintala, S. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In \u003cem\u003eAdvances in Neural Information Processing Systems 32\u003c/em\u003e; Curran Associates, Inc., 2019; pp 8024\u0026ndash;8035.\u003c/li\u003e\n\u003cli\u003eSalakhutdinov, R. M. A. Bayesian Probabilistic Matrix Factorization Using Markov Chain Monte Carlo. In \u003cem\u003eProceedings of the 25th international conference on Machine learning (ICML \u0026rsquo;08)\u003c/em\u003e; ACM: New York, 2008; pp 880\u0026ndash;887.\u003c/li\u003e\n\u003cli\u003eSosnina, E. A.; Sosnin, S.; Fedorov, M. V. ImprovingMTT. \u003cem\u003eGitHub repository\u003c/em\u003e. GitHub 2023. https://github.com/ekaterina-sea/ImprovingMTT.\u003c/li\u003e\n\u003cli\u003eSimm, J. \u003cem\u003eSMURFF - Scalable Matrix Factorization Framework\u003c/em\u003e. https://github.com/ExaScience/smurff.\u003c/li\u003e\n\u003cli\u003eSimm Jaak. \u003cem\u003eSMURFF - Scalable Matrix Factorization Framework\u003c/em\u003e. https://smurff.readthedocs.io/en/release-0.16/index.html.\u003c/li\u003e\n\u003cli\u003e\u003cem\u003eOptibrium StarDrop\u003c/em\u003e. https://optibrium.com/stardrop/.\u003c/li\u003e\n\u003cli\u003eHeyndrickx, W.; Mervin, L.; Morawietz, T.; Sturm, N.; Friedrich, L.; Zalewski, A.; Pentina, A.; Humbeck, L.; Oldenhof, M.; Niwayama, R.; Schmidtke, P.; Fechner, N.; Simm, J.; Arany, A.; Drizard, N.; Jabal, R.; Afanasyeva, A.; Loeb, R.; Verma, S.; Harnqvist, S.; Holmes, M.; Pejo, B.; Telenczuk, M.; Holway, N.; Dieckmann, A.; Rieke, N.; Zumsande, F.; Clevert, D.-A.; Krug, M.; Luscombe, C.; Green, D.; Ertl, P.; Antal, P.; Marcus, D.; Do Huu, N.; Fuji, H.; Pickett, S.; Acs, G.; Boniface, E.; Beck, B.; Sun, Y.; Gohier, A.; Rippmann, F.; Engkvist, O.; G\u0026ouml;ller, A. H.; Moreau, Y.; Galtier, M. N.; Schuffenhauer, A.; Ceulemans, H. MELLODDY: Cross-Pharma Federated Learning at Unprecedented Scale Unlocks Benefits in QSAR without Compromising Proprietary Information. \u003cem\u003eJ Chem Inf Model\u003c/em\u003e \u003cem\u003e0\u003c/em\u003e (0), null. https://doi.org/10.1021/acs.jcim.3c00799.\u003c/li\u003e\n\u003cli\u003eMartin, E. J.; Polyakov, V. R.; Zhu, X.-W.; Tian, L.; Mukherjee, P.; Liu, X. All-Assay-Max2 PQSAR: Activity Predictions as Accurate as Four-Concentration IC\u0026lt;inf\u0026gt;50\u0026lt;/Inf\u0026gt;s for 8558 Novartis Assays. \u003cem\u003eJ Chem Inf Model\u003c/em\u003e \u003cstrong\u003e2019\u003c/strong\u003e. https://doi.org/10.1021/acs.jcim.9b00375.\u003c/li\u003e\n\u003cli\u003e\u003cem\u003eCerella home page\u003c/em\u003e. https://optibriu m.com/cerella/.\u003c/li\u003e\n\u003cli\u003eMahmoud, S.; Irwin, B.; Chekmarev, D.; Vyas, S.; Kattas, J.; Whitehead, T.; Mansley, T.; Bikker, J.; Conduit, G.; Segall, M. Imputation of Sensory Properties Using Deep Learning. \u003cem\u003eJ Comput Aided Mol Des\u003c/em\u003e \u003cstrong\u003e2021\u003c/strong\u003e, \u003cem\u003e35\u003c/em\u003e (11), 1125\u0026ndash;1140. https://doi.org/10.1007/s10822-021-00424-3.\u003c/li\u003e\n\u003cli\u003eObrezanova, O.; Martinsson, A.; Whitehead, T.; Mahmoud, S.; Bender, A.; Miljković, F.; Grabowski, P.; Irwin, B.; Oprisiu, I.; Conduit, G.; Segall, M.; Smith, G. F.; Williamson, B.; Winiwarter, S.; Greene, N. Prediction of In Vivo Pharmacokinetic Parameters and Time\u0026ndash;Exposure Curves in Rats Using Machine Learning from the Chemical Structure. \u003cem\u003eMol Pharm\u003c/em\u003e \u003cstrong\u003e2022\u003c/strong\u003e, \u003cem\u003e19\u003c/em\u003e (5), 1488\u0026ndash;1504. https://doi.org/10.1021/acs.molpharmaceut.2c00027.\u003c/li\u003e\n\u003cli\u003eWhitehead, T. M.; Strickland, J.; Conduit, G. J.; Borrel, A.; Mucs, D.; Baskerville-Abraham, I. Quantifying the Benefits of Imputation over QSAR Methods in Toxicology Data Modeling. \u003cem\u003eJ Chem Inf Model\u003c/em\u003e \u003cstrong\u003e2024\u003c/strong\u003e, \u003cem\u003e64\u003c/em\u003e (7), 2624\u0026ndash;2636. https://doi.org/10.1021/acs.jcim.3c01695.\u003c/li\u003e\n\u003cli\u003eThe Theano Development Team; Al-Rfou, R.; Alain, G.; Almahairi, A.; Angermueller, C.; Bahdanau, D.; Ballas, N.; Bastien, F.; Bayer, J.; Belikov, A.; Belopolsky, A.; Bengio, Y.; Bergeron, A.; Bergstra, J.; Bisson, V.; Snyder, J. B.; Bouchard, N.; Boulanger-Lewandowski, N.; Bouthillier, X.; de Brebisson, A.; Breuleux, O.; Carrier, P.-L.; Cho, K.; Chorowski, J.; Christiano, P.; Cooijmans, T.; Cote, M.-A.; Cote, M.; Courville, A.; Dauphin, Y. N.; Delalleau, O.; Demouth, J.; Desjardins, G.; Dieleman, S.; Dinh, L.; Ducoffe, M.; Dumoulin, V.; Kahou, S. E.; Erhan, D.; Fan, Z.; Firat, O.; Germain, M.; Glorot, X.; Goodfellow, I.; Graham, M.; Gulcehre, C.; Hamel, P.; Harlouchet, I.; Heng, J.-P.; Hidasi, B.; Honari, S.; Jain, A.; Jean, S.; Jia, K.; Korobov, M.; Kulkarni, V.; Lamb, A.; Lamblin, P.; Larsen, E.; Laurent, C.; Lee, S.; Lefrancois, S.; Lemieux, S.; Leonard, N.; Lin, Z.; Livezey, J. A.; Lorenz, C.; Lowin, J.; Ma, Q.; Manzagol, P.-A.; Mastropietro, O.; McGibbon, R. T.; Memisevic, R.; van Merrienboer, B.; Michalski, V.; Mirza, M.; Orlandi, A.; Pal, C.; Pascanu, R.; Pezeshki, M.; Raffel, C.; Renshaw, D.; Rocklin, M.; Romero, A.; Roth, M.; Sadowski, P.; Salvatier, J.; Savard, F.; Schl\u0026uuml;ter, J.; Schulman, J.; Schwartz, G.; Serban, I. V.; Serdyuk, D.; Shabanian, S.; Simon, \u0026Eacute;tienne; Spieckermann, S.; Subramanyam, S. R.; Sygnowski, J.; Tanguay, J.; van Tulder, G.; Turian, J.; Urban, S.; Vincent, P.; Visin, F.; de Vries, H.; Warde-Farley, D.; Webb, D. J.; Willson, M.; Xu, K.; Xue, L.; Yao, L.; Zhang, S.; Zhang, Y. Theano: A Python Framework for Fast Computation of Mathematical Expressions. arXiv May 2016. https://doi.org/10.48550/arXiv.1605.02688.\u003c/li\u003e\n\u003cli\u003eAbadi, M.; Barham, P.; Chen, J.; Chen, Z.; Davis, A.; Dean, J.; Devin, M.; Ghemawat, S.; Irving, G.; Isard, M.; others. Tensorflow: A System for Large-Scale Machine Learning. In \u003cem\u003e12th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16)\u003c/em\u003e; 2016; pp 265\u0026ndash;283.\u003c/li\u003e\n\u003cli\u003eTokui, S.; Okuta, R.; Akiba, T.; Niitani, Y.; Ogawa, T.; Saito, S.; Suzuki, S.; Uenishi, K.; Vogel, B.; Vincent, H. Y. Chainer: A Deep Learning Framework for Accelerating the Research Cycle. arXiv August 2019. https://doi.org/10.48550/arXiv.1908.00213.\u003c/li\u003e\n\u003cli\u003eCollobert, R.; Kavukcuoglu, K.; Farabet, C. Torch7: A Matlab-like Environment for Machine Learning. In \u003cem\u003eBigLearn, NIPS Workshop\u003c/em\u003e; 2011.\u003c/li\u003e\n\u003cli\u003ePaszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; Desmaison, A.; Kopf, A.; Yang, E.; DeVito, Z.; Raison, M.; Tejani, A.; Chilamkurthy, S.; Steiner, B.; Fang, L.; Bai, J.; Chintala, S. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In \u003cem\u003eAdvances in Neural Information Processing Systems 32\u003c/em\u003e; Wallach, H., Larochelle, H., Beygelzimer, A., d Alch\u0026eacute;-Buc, F., Fox, E., Garnett, R., Eds.; Curran Associates, Inc., 2019; pp 8024\u0026ndash;8035. https://doi.org/10.48550/arXiv.1912.01703.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"journal-of-computer-aided-molecular-design","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jcam","sideBox":"Learn more about [Journal of Computer-Aided Molecular Design](http://link.springer.com/journal/10822)","snPcode":"10822","submissionUrl":"https://submission.nature.com/new-submission/10822/3","title":"Journal of Computer-Aided Molecular Design","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"multitask regression, imputation, QSAR, virtual screening, algorithm comparison, drug discovery","lastPublishedDoi":"10.21203/rs.3.rs-7482715/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7482715/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eMassively-multitask regression models (MMRMs) have revolutionized activity prediction for drug discovery. MMRMs trained on millions of compounds and many thousands of assays can predict bioactivity with accuracy comparable to 4-concentration IC\u003csub\u003e50\u003c/sub\u003e experiments. This report compares six MMRMs: pQSAR, Alchemite, MT-DNN, MetaNN, Macau and IMC. Models were trained by experts in each method, on identical sets of 159 kinase and 4276 diverse ChEMBL assays, employing the same, realistically novel, training/test set splits.\u003c/p\u003e\u003cp\u003eMMRMs performed much better than single-task random forest regression (ST-RFR) models for our use-case of imputing full bioactivity profiles for the very sparse compound collection on which the models were trained. Five MMRMs train all models simultaneously, so must leave out test-set measurements for all assays to avoid leakage (i.e. 25% of data). One method trains models one-at-a-time, and trains on all but the test data for that assay (\u0026lt;\u0026thinsp;1% of data). All algorithms were compared both using 75/25 splits, and when possible, 99+/\u0026lt;1 splits. Many evaluations achieved similar accuracy when tested on the same split. When evaluated on 75/25 splits, all MMRMs performed much worse than when evaluated on 99+/\u0026lt;1% splits. Thus, while many produce comparable high-accuracy final production models (trained on all the data), models that require 75/25 splits cannot evaluate the accuracy of those final models.\u003c/p\u003e\u003cp\u003eWhile outstanding for imputations, MMRMs proved little better than ST-RFR for compounds very unlike the training collection. Thus, MMRMs are best for hit-finding, off-target, promiscuity, MoA, polypharmacology or drug-repurposing within the training collection. Besides accuracy, other pros and cons of each method are discussed.\u003c/p\u003e","manuscriptTitle":"Comparing Massively-Multitask Regression Algorithms for Drug Discovery","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-09-22 10:07:20","doi":"10.21203/rs.3.rs-7482715/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-10-31T21:12:47+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-10-31T10:21:43+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-10-28T12:55:29+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"134564517723300270333584081226169613813","date":"2025-10-20T18:21:37+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"286854828965551154037836460523462714679","date":"2025-10-17T05:10:35+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"285206410078608397737103888323683373546","date":"2025-10-16T07:37:47+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"23209039698312325781521377815806539586","date":"2025-10-03T20:07:20+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-09-11T23:06:19+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-09-11T23:03:46+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-09-02T12:46:11+00:00","index":"","fulltext":""},{"type":"submitted","content":"Journal of Computer-Aided Molecular Design","date":"2025-08-28T18:14:20+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"journal-of-computer-aided-molecular-design","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jcam","sideBox":"Learn more about [Journal of Computer-Aided Molecular Design](http://link.springer.com/journal/10822)","snPcode":"10822","submissionUrl":"https://submission.nature.com/new-submission/10822/3","title":"Journal of Computer-Aided Molecular Design","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"4693a313-96e1-4266-82a9-1cd2c88df737","owner":[],"postedDate":"September 22nd, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2026-02-09T16:02:42+00:00","versionOfRecord":{"articleIdentity":"rs-7482715","link":"https://doi.org/10.1007/s10822-026-00761-1","journal":{"identity":"journal-of-computer-aided-molecular-design","isVorOnly":false,"title":"Journal of Computer-Aided Molecular Design"},"publishedOn":"2026-02-05 15:59:19","publishedOnDateReadable":"February 5th, 2026"},"versionCreatedAt":"2025-09-22 10:07:20","video":"","vorDoi":"10.1007/s10822-026-00761-1","vorDoiUrl":"https://doi.org/10.1007/s10822-026-00761-1","workflowStages":[]},"version":"v1","identity":"rs-7482715","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7482715","identity":"rs-7482715","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}