Drug Repurposing Prediction for Immune-Mediated Cutaneous Diseases using a Word-Embedding-Based Machine Learning Approach.

We extracted as a ‘gold standard’ those word embeddings which represent drugs contained in the Comparative Toxicogenomics Database (CTD) ( Davis et al., 2015 ) or the Unified Medical Language System (UMLS) National Drug File-Reference Terminology (NDFRT) ( Carter et al., 2002 ). Both CTD and NDF-RT provide disease-drug associations curated by domain experts. Although CTD also contains information about chemicals that cause adverse effects, we only consider disease-drug associations annotated for therapeutic use. Diseases and drugs are represented by these resources using MeSH and customized identifiers, respectively. We combined the two resources by mapping synonyms using UMLS Metathesaurus concept unique identifiers (CUIs) for the diseases ( Bodenreider, 2004 ) and IDs compiled in our previous work for the drugs ( Raja et al., 2017 ), and then removed any duplicate entries. We also filtered the associations so that they only include drugs (as opposed to other chemicals) by restricting the semantic types to clinical drugs, antibiotics and pharmacological substances. In total, the combined resource contains word embeddings for 2,814 drugs. Various unsupervised learning techniques have been developed to generate word embeddings by reducing the dimensions of word co-occurrence data. The earliest approaches were based on global factorization (e.g. LSA ( Deerwester et al., 1990 )). More recently, techniques have been developed using local context (i.e. words that occur near each other). For example, Mikolov et al. ( Mikolov et al., 2013 ) developed a skip-gram model for predicting words based on the other words nearby. This fine-grained level of analysis allows more complex relationships between words to be captured. However, by only considering one context at a time, it does not take advantage of all the information that is available through global statistics. We combine the benefits of global and local information in our word embeddings, by using Global Vectors for Word Representation (GloVe) ( Pennington et al., 2014 ). GloVe trains embeddings on aggregated (i.e. global) statistics, taking advantage of information more efficiently than in the skip-gram approach. However, it also counts word co-occurrences in their local context, by factorizing a word-word matrix, rather than the document-word matrix used by LSA. Prior to training, we pre-processed the data by tokenizing each word and transforming it into lower case. We then removed all characters that are not in the Roman alphabet or numbers; tokens containing only these characters were removed. GloVe ( Pennington et al., 2014 ) uses matrix factorization to produce word embeddings, and parallel computing was enlisted to provide efficient matrix factorization. Equation 1 describes GloVe's loss function for generating word embeddings ( w i and w̃ j ), such that their product for each pair of words ( i and j ) minimizes error compared to the logarithm of word-word matrix ( X ij ): X ij is produced by counting the number of times word j occurs inside the context of word i (with window size of 15 words) and b i , b̃ j are scalar biases (i.e. errors) learned for w i and ŵ j respectively. Previous research has shown that wider context windows (like the one we used) help to ensure the semantics captured represent topical rather than syntactic similarity ( Levy and Goldberg, 2014 ). We chose to train word embeddings as vectors with 200 dimensions, because accuracy was not found to substantially increase for embeddings beyond this size ( Pennington et al., 2014 ). We improved the accuracy of the final result by pooling (adding) two different sets of word embeddings ( w i and w̃ j ) for each word. Finally, f is a weighting function that helps to prevent learning only from common word pairs, by dividing each value in X ij by a maximum threshold ( x max ) and raising the result to a power of α (we used x max =100 and α =¾); co-occurrence counts greater than x max are given a weight of 1. (1) J = ∑ i , j = 1 V f ( X i j ) ( w i T w ~ j + b i + b ~ j − log ( X i j ) ) 2 Using 500 Intel Xeon cores (1,000 threads), it took 65 hours to train embeddings for all 2.2 million words in our corpus through 500 iterations. This approach performs favorably compared to the other main competitor for generating word embeddings, word2vec ( Mikolov et al., 2013 ), which processes each word in the text one at a time and would have taken far longer. We then stored the decomposed data in our database, to reduce computation time in the future: our word embeddings can be repeatedly reused (both for psoriasis and for other diseases) with no additional cost. Machine learning was applied using MLR ( Bischl et al., 2016 ), a machine learning library from R. We benchmarked our approach to predict new drugs for psoriasis on 23 classification algorithms in MLR. In each case, the positive training dataset was created from embeddings for all drugs marked as being used to treat the disease in the CTD/NDFRT dataset we compiled, whereas the negative dataset was created to contain embeddings for all 2,417 drugs not indicated (in the CTD/NDFRT dataset) as being used to treat any of the immune-mediated diseases listed by the AARDA: www.aarda.org/diseaselist . The remaining drugs (i.e. those used to treat some other immune-mediated disease) were placed into an unlabeled dataset, to which the trained model is applied. For example, in the case of psoriasis, the CTD/NDFRT dataset contains 44 drugs indicated as being used to treat psoriasis (positive set) and 353 drugs indicated as being used to treat some other immune-mediated disease (unlabeled set). Classification models were evaluated using stratified 10-fold cross-validation, ensuring the same proportion of class labels in each (randomly selected) partition. We used the default parameters for each classification algorithm, to make it easier for other researchers to reproduce our results, although by doing so we acknowledge that even greater performance might be achieved through parameter tuning. Partial Least Squares Discriminant Analysis (PLS-DA) achieved the highest mean AUROC (0.928), with a standard deviation of 0.0365, and was used throughout the study. We also compared our approach against three alternative techniques ( Table 1 ). Latent Semantic Analysis (LSA) ( Deerwester et al., 1990 ) is a frequently used technique for encoding the semantics of words. For consistency with our approach, we applied LSA to the same data as our approach, to produce 200-dimension vectors which were used to train PLS-DA. A key difference between LSA and word embedding is that, rather than using a sliding window, it evaluates semantics in the context of a document (i.e. PubMed abstract). We believe this could make the semantic vectors it generates less precise. As a further baseline, we also utilized maximum cosine similarity (between known and candidate drugs) in place of machine learning. It is still possible to compute the AUROC even without machine learning, as we still have the positive and negative labels, so all we need to calculate the rate of true and false positives is a means to order the drugs (we can use maximum cosine similarity for this). To evaluate the drugs our approach predicts for psoriasis, we extracted differential expression data from a RNA-seq transcriptomic study ( Li et al., 2014 ), which included 92 psoriatic and 82 normal samples. Significant differentially expressed genes (DEGs) are defined as yielding p-values <1×10 −6 and ∣log2(Fold Change)∣≥1. If our approach is valid, we should expect the predicted drugs to be enriched for DEGs compared to the rest of the unlabeled set. The gene targets for each drug predicted by our approach were identified by combining drug-gene interaction data from the Comparative Toxicogenomics Database (CTD), DrugBank and Phar-mGKB. Drug-gene interactions were unified according to the drug IDs from our lexicon and then filtered based on three semantic types: clinical drugs, antibiotics and pharmacological substances.

The workflow used in our approach is illustrated in Figure 1 . We trained word embeddings using an unsupervised learning algorithm, Global Vectors for Word Representation (GloVe) ( Pennington et al., 2014 ). By using all the abstracts and their titles in the recent (December 2017) annual data dump from PubMed, our input dataset consists of ∼20 million abstracts, with 3.3 billion words in total and a vocabulary of 2.2 million unique words (∼200GB). Briefly, our approach enumerates the co-occurrences between drugs and other vocabularies in the input dataset (using a sliding window), and then applies GloVe to generate “word embeddings”, which project a lower dimension representation of the drug-vocabulary matrix. We then trained various machine-learning algorithms to model the contribution of different word embeddings in classifying drugs that have been prescribed to treat each of the immune-mediated skin conditions, using existing drug databases as resources (see Methods for detail). We represent the search for drug repurposing opportunities as a series of binary classification tasks, where for each disease: the positive set consists of word embeddings for drugs used to treat the disease; the negative set consists of word embeddings for drugs not used to treat any immune-mediated disease; and the unlabeled dataset consists of word embeddings for all remaining drugs (we aim to predict whether these drugs can be repurposed to treat the disease). Each instance in our machine learning models represents a single drug (2,814 drugs in total) and the size of the positive, negative and unlabeled set depends on the disease being considered. The trained classification models provide a mechanism by which word embeddings may be input for each potential drug and prediction scores output estimating the relative likelihood that they may be used to treat a specific disease. Our word embeddings are produced through matrix factorization in GloVe, to represent the vast number of possible word-word combinations as a finite set of values (fixed vector of weights) in lower dimension. Each weight corresponds to a different abstract concept, such that the unique assignment of values to these weights generates a distinct vector for each word and provides a way to represent and compare its semantics. We use the weights as features for our machine learning approach, whereby their values for a candidate drug are compared with those of known drugs used to treat the disease. Since matrix factorization already reduces the dimensionality (in our case to 200 weights/features) and presents the data in a uniform form that is suitable for machine learning, we have found it to be unnecessary to perform further feature selection. Our approach achieved good performance in evaluating existing drugs for different diseases, with a mean area under the receiver operator curve (AUROC) across the diseases above 0.80 in 10-fold cross-validation for 19 of the 23 classifiers we benchmarked ( Figure 2a shows the results for six distinct classifiers on all the diseases, with psoriasis highlighted in blue; the results for all classifiers are shown in Figure S1 ), illustrating the robustness of our approach. Partial Least Squares Discriminant Analysis (PLS-DA) achieved the highest mean AUROC (0.928) across the diseases, with a standard deviation of 0.0365, and was used throughout the remainder of our study. PLS-DA works by projecting data into a new space, to facilitate the fitting of a regression line ( Perez-Enciso and Tenenhaus, 2003 ). Figure 2b illustrates the performance for psoriasis in terms of its rate of true positives (drugs correctly identified as being able to treat the disease) and false positives (drugs predicted to treat the disease that do not) averaged across the cross-validation datasets. Since AUROC does not depend on the ratio of positive and negative samples, it is valuable for comparing the effectiveness of different classifiers. We also evaluated other metrics ( Figure 2c ) on the left-out samples from crossvalidation for psoriasis: precision (proportion of drugs predicted by our approach that are disease relevant) and recall (proportion of disease relevant drugs identified by our approach). The results suggest that, selecting the top 1% of drugs predicted as being relevant for psoriasis will give a precision of 77% with a recall of 43%. However, it is important to note that our negative set (drugs not known to treat any immune-mediated disease, see Methods ) is our best approximation, since it is difficult to know for certain which drugs cannot be used to treat psoriasis (i.e. false negatives). We also computed the harmonic mean between precision and recall (the F1 metric), which in our case is weighted more towards precision than recall because the negative set (2,417 drugs) is much larger than the positive set (44 drugs known for psoriasis). Notably, our approach outperformed three other baseline/competitive methods, achieving significantly better performance than using i ) cosine similarity for semantic relationship measure in Latent Semantic Analysis (LSA) ( p =7.7×10 −5 ) and ii ) word embeddings ( p =3.2×10 −2 ) on 16 and 12 of the diseases we evaluated, respectively; as well as iii ) LSA with machine learning ( p =4.9×10 −4 ) on 14 out of the 17 immune-mediated diseases ( Table 1 ). It also suggests word embeddings provide the greatest benefit, with machine learning increasing performance by a significant but smaller amount. We applied our trained model to 353 drugs indicated as not being used to treat psoriasis, but used to treat some other immune-mediated disease (the unlabeled dataset; see Methods ), then we evaluated its predictions against genes that are dysregulated in lesional skin. We hypothesized that drugs effective at treating psoriasis are more likely to have gene targets with differential expression profiles in psoriatic skin samples when compared to normal skin. Our results show that drugs ranked higher (i.e. with higher prediction score) by our approach have gene targets significantly enriched among differentially expressed genes in psoriatic lesional skin (see Figure 3a ), whereas those ranked as being less likely to be relevant to psoriasis showed no enrichment. By using transcriptomic data as an independent in vivo validation, these results illustrate the ability of our approach in distinguishing between plausible psoriasis and non-psoriasis drugs in the unlabeled dataset. The highest enrichment value ( p =1×10 −21 ) occurred when selecting the top 18 drugs predicted for psoriasis. We compared the prediction scores of drugs our approach identified as relevant to psoriasis against the positive and negative training datasets. Figure 3b shows the distribution of prediction scores for the positive dataset in blue, the negative dataset in red and the unlabeled dataset in green. There is a clear distinction between the prediction scores for the test and training datasets, with the unlabeled dataset located in an intermediate position. Since the distribution for the unlabeled dataset overlaps that of the negative dataset more than the positive dataset, this suggests most of the drugs in the unlabeled dataset would not be effective against psoriasis (thus supporting the results of our differential expression analysis). We plotted the prediction scores of the top 20 drugs from the unlabeled dataset predicted by our approach for psoriasis as dashed lines. All of these prediction scores lie close to the center of the distribution for the positive dataset, suggesting the similarity of their semantic structure to current established drugs for psoriasis. In addition to the differential expression data, we also evaluated our approach against an independent database of drug-disease associations (DrugBank). The DrugBank database ( Wishart et al., 2018 ) has recently received a major update, more than doubling its size. This provides us with a unique opportunity to validate the drug repurposing opportunities predicted by our approach, which was trained on slightly older data curated from the CTD/NDFRT database. We found that 5 of the top 20 drugs predicted by our approach (colored in black in Figure 3b ) are indicated for psoriasis in DrugBank (prednisone, triamcinolone, methylprednisolone, hydrocortisone and dexamethasone). This confirms our approach is capable of accurately predicting existing psoriasis drugs, and since a number of the drugs not indicated as treating psoriasis in any of the datasets (colored in grey in Figure 3b ) have higher prediction score than those found in DrugBank, we believe it is also effective at predicting new drugs for psoriasis. The top 10 drugs predicted for psoriasis by our repurposing approach are shown in Table 2 ( Table S1 contains results for the other diseases). The drugs with highest prediction score are budesonide (a corticosteroid, currently used to treat asthma and inflammatory bowel disease, IBD) and hydroxychloroquine (an anti-malarial drug that is also used to treat lupus and rheumatoid arthritis). Both of these drugs, along with mesalamine and balsalazide, are known to have anti-inflammatory properties, which could be beneficial to psoriasis; indeed, other corticosteroids have been used to treat psoriasis ( Mason et al., 2002 ). In addition, leflunomide, cyclophosphamide and mercaptopurine have immune suppressing properties. By contrast, the potential modes of action for glatiramer, latanoprost and bimatoprost are less clear. Glatiramer is used to treat multiple sclerosis and is thought to act by promoting the development of Th2 CD4 + T-cells ( Weber et al., 2007 ); skewing CD4 + T-cells from Th1 (IFN- γ producing) to Th2 (IL4 producing) has been found to improve the symptoms of psoriasis ( Ghoreschi et al., 2003 ). Latanoprost and bimatoprost are prostaglandin analogs; prostaglandins are a group of lipid compounds that have complex and diverse effects on the immune system ( Hata and Breyer, 2004 ). It is interesting to note that a substantially higher proportion of the differentially expressed gene targets of the top 10 drugs predicted for psoriasis (see Table 2 ) are up-regulated among psoriasis patients, compared to the rest of the unlabeled set. Among the top 10 drugs, 29 of the differentially expressed gene targets are up-regulated in psoriasis and 19 are down-regulated (i.e. 60% upregulated), as opposed to 119 up-regulated and 198 down-regulated (i.e. 39% up-regulated) in the unlabeled set as a whole. Notably, this significant enrichment of up-regulated genes among the top 10 predictions (p=2.5×10 −3 ; using our −ve set as background) hints at the anti-inflammation properties of these drugs, which work by suppressing inflammatory/immune cell infiltration patterns. This also compares well to the positive dataset, which has 32 genes that are up-regulated and 28 genes that are down-regulated in psoriasis (i.e. 53% up-regulated). This distribution of up and down regulated genes may therefore also be considered as evidence our approach is able to identify potential new drugs for psoriasis through drug repurposing. The predictions made by our approach therefore seem promising and merit further investigation into the potential effects and modes of action for these drugs on psoriasis. Finally, Figure 4 depicts the differentially expressed gene targets shared between at least two of the top 10 drugs (see Figure S2 for the top 20 drugs). Drugs are shown as green circles with size proportional to the number of gene targets, whereas the genes are shown as squares (red for upregulated and blue for down-regulated) with size proportional to expression in psoriasis and darkness proportion to their differential expression compared to controls. Interestingly, the drug targets point to genes that are crucial components in psoriasis pathology, including: IL1B , a cytokine which has been shown to induce the hyperproliferative inflammatory skin lesion in mouse models ( Schon et al., 2001 ); NOS2 , a nitric oxide synthase that is involved in antimicrobial activities, and its expression was found to facilitate perfect classification between psoriasis and another common skin condition, eczema ( Quaranta et al., 2014 ); and IFN γ , a cytokine signature that is induced by Th1 cells. These results further illustrate that our approach can provide mechanistic inference regarding the gene targets these predicted drugs are able to act upon.

Word embeddings have previously been used for named entity recognition ( Habibi et al., 2017 ) and identifying adverse drug reactions from social media ( Nikfarjam et al., 2015 ). Recently, they have also been included along with a variety of other datasets in models for drug repurposing in different human diseases ( Luu Ngo et al., 2016 , Manchanda and Anand, 2017 ). GloVe has been applied to corpora as large as 840 billion words ( Pennington et al., 2014 ) and has been used on tasks as diverse as annotating videos from free text descriptions ( Hendricks et al., 2017 ), to identifying implicit human bias/stereotyping ( Greenwald, 2017 ). In semantic and syntactic comprehension tasks ( Pennington et al., 2014 ) GloVe was found to be consistently more accurate than other approaches, and all the participants in a recent competition for natural language comprehension ( Nangia et al., 2017 ) used it in their approach. Nevertheless, GloVe has three main limitations: 1 ) GloVe cannot handle words outside its vocabulary (unlike character-based approaches, e.g. fastText ( Bojanowski et al., 2017 )); 2) GloVe does not consider semantic ambiguity (AdaGram ( Bartunov et al., 2016 ) identifies multiple meanings for each word); 3 ) GloVe ignores sentence structure (wang2vec ( Wang et al., 2015 ) uses the order of words to identify parts of speech). It may be worthwhile to explore whether approaches such as the ones suggested can be used to further improve performance. In our study, we illustrated that by combining word embeddings with an advanced machine learning technique, we were able to obtain accurate performance (mean AUROC=0.93). The information extracted is not easily available through a manual search of the literature (two of the top 10 drugs predicted by our approach, balsalazide and bimatoprost, are not mentioned with psoriasis in any research paper), and is formed from complex semantic relationships (144 out of the 200 features had a positive importance value in psoriasis drug prediction). Indeed, the components learned by PLS-DA utilize values from across the word embedding vector space ( Figure S3 ), making them difficult to interpret. It is important to stress however our classification scores infer the collected evidence for each drug’s association with the disease based on existing drugs, and are not necessarily correlated with potential drug efficacy, as this depends on multiple different factors (e.g. disease severity, genetic profile of patients). Since our approach only suggests drugs used to treat some other immune-mediated disease (drugs not known to treat any immunemediated disease were used as the negative training set), it may exclude some possible treatments (i.e. it is deliberately conservative). For comparison, we also implemented Positive Unlabeled (PU) learning, which merges the negative and unlabeled set, repeats cross-validation 100 times (classifying positive drugs against unlabeled) and takes the mean rank of the classification scores (for each unlabeled drug). The alternative approach was found to be highly concordant with our approach (Pearson r 2 : 0.89) and four of the top 10 predicted drugs were the same in each approach. As further evidence for the drugs predicted by our approach for psoriasis, we showed that their gene targets are enriched among genes dysregulated by the disease. The expression dataset thus provides an independent validation for our model, and the dysregulated genes targeted by predicted drugs also provide important clues in unraveling the biological mechanism of how the drugs might work. We should note that while we train our approach on drugs that are used to treat immune-mediated diseases, we do not explicitly include mechanisms to avoid toxicity or adverse drug reactions. For example, steroids (e.g. budesonide) can cause side-effects their withdrawal may lead to a relapse of symptoms. Some of the current psoriasis drugs (e.g. methotrexate) have also been used to treat cancer, and interestingly, among the top predictions we also observed drugs used to treat several types of cancers. From a clinical standpoint, some of these drugs could have severe side effects (due to toxicity), so we acknowledge the efficacy of the drugs our approach identifies should be balanced with their potential side effects. Interestingly, our algorithm has revealed chloroquine among the top 10 predictions. Chloroquine along with other anti-malarial drugs has previously been considered contraindicative for psoriasis ( Cornbleet, 1956 ). However, these reactions are not always found to occur ( Baker, 1966 ), and since arthritis and lupus are comorbidities of psoriasis, psoriasis patients occasionally receive treatment with this drug. One interesting case study indicates that chloroquine led to a reduction in symptoms of both lupus and psoriasis ( Sorbara et al., 2006 ). It is important therefore to stress, that whilst we believe the drugs identified by our approach may be effective for psoriasis, this does not eliminate the need for in vivo and in vitro studies. The algorithm we developed cannot be used to determine clinical efficacy, but our approach provides a way to prioritize these drug candidates for future research. Whilst many in silico methods have been developed for predicting drug repurposing, we believe context-specific model training is critical for providing robust results as well as candidates that are biologically relevant. In this study, we compiled comprehensive disease-drug relationships using different resources to train our word embeddings for drugs specific to 17 immune-mediated diseases. The approach we have developed is highly flexible and can easily be applied to predict new drugs for other diseases, just by changing the positive dataset and rerunning the machine learning pipeline. We not only achieved satisfying classification results, but also independently illustrated how the targets of predicted drugs are dysregulated using in vivo RNA-seq data, thus providing inference relating potential drug effects to lesional inflammatory skin.

Developing a new drug typically takes an average 10-15 years, and costs >$2 billion ( DiMasi et al., 2016 ). Notably, ∼90% of drugs that enter clinical evaluation fail at some phase, due to safety concerns or lack of efficacy ( Nagaraj et al., 2018 ), and many drugs are withdrawn/discontinued after being brought to market ( Siramshetty et al., 2016 ). The development process can be made 2-3 times faster by repurposing (aka repositioning) existing drugs ( Hurle et al., 2013 ), since the adverse effects have already been investigated and manufacturing procedures have been established. Compared to newly developed drugs, repurposed drugs are three times more likely to be approved ( Hernandez et al., 2017 ). Historically, most drug repurposing has been serendipitous ( Jin and Wong, 2014 ), applying different drugs to disease cells or animal models ( Issa et al., 2017 ). High throughput screening through robotic automation ( Swamidass, 2011 ), or ‘virtual screening’ using molecular similarity/docking ( Rognan, 2017 ), can be applied when specific target behavior (e.g. activation of a protein) is known. Approaches have also been developed to compare differentially expressed genes from administering a drug with those resulting from the disease ( Dudley et al., 2011 , Jahchan et al., 2013 , Sirota et al., 2011 ) or from existing drugs ( Iorio et al., 2013 ). Other approaches include the use of genome-wide association studies (GWAS) ( Rastegar-Mojarad et al., 2015 , Sanseau et al., 2012 ), eQTL analysis ( Okada et al., 2014 ), GWAS-imputed transcriptome ( Pritchard et al., 2017 , So et al., 2017 ) and health records ( Xu et al., 2015 ). Nevertheless, these approaches are limited by the availability of data with sufficient sample size, particularly for rare diseases. The explosion of information embedded within the ‘free text’ of research publications has led to increasing interest in harnessing the power of text mining techniques ( Henry and McInnes, 2017 ). Most approaches correlate/cluster terms co-mentioned with drugs and diseases at the document (i.e. paper/abstract) level ( Kostoff et al., 2008 , Swanson et al., 2006 , Yetisgen-Yildiz and Pratt, 2006 ). By contrast, word embeddings provide a semantic representation of each word ( Goth, 2016 ), according to its local context (immediately surrounding words). Word embeddings are grounded in ordinary language philosophy ( Wittgenstein, 1953 ) and structuralist linguistics ( Firth, 1957 ), which focus on analyzing and understanding the everyday use of language. Although there are a vast number of possible word-word combinations, it is possible to represent this information using a finite number of values, if each value corresponds to a different abstract concept. Word embeddings embody this concept through fixed length vectors of weights, such that the unique assignment of values to these weights generates a distinct vector for each word and provides a way to represent and compare its semantics. This uniform representation makes word embeddings suitable for training machine learning models. In this paper, we introduce an efficient word embedding-based machine learning approach towards repurposing drugs for cutaneous immune-mediated conditions, and we use psoriasis as a specific case study to provide biological interpretations for the new drugs identified. We assessed our approach independently using expression data from 174 RNA-seq samples ( Li et al., 2014 ). By using existing drug information in model training as well as in vivo data for evaluation, our method identifies candidates that are promising for the treatment of multiple diseases.