Predicting Adverse Drug Reactions Using Machine Learning: A Hybrid Model Stacking Approach

Adverse drug reactions (ADRs) pose a significant challenge to healthcare systems worldwide. Accurate prediction of ADRs is necessary to prevent the patient from contracting unwanted side effects and improve vigilance within the pharmaceutical industry. This paper proposes a hybrid, ensemble-based machine learning model to predict the probability of adverse reactions based on drug-side effect relationships, patient indications, and co-occurrence patterns. The approach integrates extensive fea- ture engineering, such as vectorization, calculation of side effect frequencies, indication-drug relevance scores, and co-occurrence metrics. Three state-of-the-art gradient boosting models, LightGBM, XGBoost, and CatBoost, were fine-tuned, trained, and evaluated on the engineered dataset. The predictions from each base model were stacked into a random forest regressor to build a hybrid meta-model. Upon evaluating the models using the mean absolute error as the primary metric alongside the mean squared error and the R 2 score as additional metrics, the stacked meta-model was found to have outperformed all base models, achieving a mean squared error of 0.000826, an R ² score of 0.8789, and a mean absolute error of 0.01833. The obtained

highlight the potential of using hybrid models over individual machine learning systems whilst dealing with datasets that encompass deep, complex, non-linear relations and advanced feature engineering in predicting ADRs. 1 Predicting Adverse Drug Reactions Using Machine Learning: A Hybrid Model Stacking Approach Tharun A 1 [email protected] Bhuvaneshwari P2 [email protected] 1,2Department of Computer Science and Engineering, Manipal Institute of Technology Bengaluru, Manipal Academy of Higher Education, Manipal, India Corresponding Author:Tharun A Author Note.This is the postprint version of the article published in2025 Second International Con- ference on Cognitive Robotics and Intelligent Systems (ICC-ROBINS).Final version available at IEEE Xplore: https://doi.org/10.1109/ICC-ROBINS64345.2025.11086311. To cite the IEE version use: T. A and B. P, “Predicting Adverse Drug Reactions Using Machine Learning: A Hybrid Model Stacking Approach,” ICC-ROBINS, Coimbatore, India, 2025.

Adverse drug reactions (ADRs) pose a significant challenge to healthcare systems worldwide. Accurate prediction of ADRs is necessary to prevent the patient from contracting unwanted side effects and improve vigilance within the pharmaceutical industry. This paper proposes a hybrid, ensemble-based machine learning model to predict the probability of adverse reactions based on drug-side effect relationships, patient indications, and co-occurrence patterns. The approach integrates extensive feature engineering, such as vectorization, calculation of side effect frequencies, indication-drug relevance scores, and co-occurrence metrics. Three state-of-the-art gradient boosting models, LightGBM, XGBoost, and CatBoost, were fine-tuned, trained, and evaluated on the engineered dataset. The predictions from each base model were stacked into a random forest regressor to build a hybrid meta-model. Upon evaluating the models using the mean absolute error as the primary metric alongside the mean squared error and theR 2 score as additional metrics, the stacked meta-model was found to have outperformed all base models, achieving a mean squared error of 0.000826, an R²score of 0.8789, and a mean absolute error of 0.01833. The obtained results highlight the potential of using hybrid models over individual machine learning systems whilst dealing with datasets that encompass deep, complex, non-linear relations and advanced feature engineering in predicting ADRs. Keywords:Adverse drug reactions,Ensemble Learning, LightGBM, XGBoost, CatBoost, Random Forest regressor. 1 Introduction Adverse drug reactions (ADRs) are unexpected side effects or indications often observed in patients after the consumption of a pharmaceutical drug. These ADRs can range from mild and harmless to life-threatening problems if not checked at the right time. ADRs account for one of the lead- ing post-consumption causes of patient mortality worldwide, thereby posing significant challenges 1 to the healthcare system. Early detection of such ADRs is imperative for patient safety, reduc- ing healthcare costs, and enhancing pharmacovigilance. Traditional methods for ADR detection rely heavily on post-market surveillance and human expertise, which, considering the volume and complexity of medical datasets, can cause significant delays and increase the required costs. There is a need for automated tools capable of predicting adverse reactions to drugs, thereby reducing both costs and the required manual intervention. Machine learning tools have emerged as potential solutions for dealing with such tasks; they have demonstrated the ability to analyze and learn complex relationships among drugs, side effects, and indications. However, the datasets used for such studies tend to exhibit high cardinality and complex non-linear relationships; individual machine learning models often struggle or fail to perform adequately when handling such data. This study proposes a comprehensive machine learning pipeline to predict ADRs to mitigate this issue. The proposed model was built and evaluated on the SIDER dataset [1] [2] and integrates advanced feature engineering techniques, TF-IDF vectorization, frequency-based metrics, and co- occurrence analysis to enrich the dataset along with strategic downsampling to handle the class imbalance. The proposed methodology creates a hybrid random forest metamodel by stacking the predictions obtained from three extensively fine-tuned individual gradient boosting models, viz. the LightGBM, CatBoost, and XGBoost, into a random forest regressor, to leverage the individual strengths of the base models while mitigating the effects of their limitations. Upon evaluating the models with the mean absolute error as the primary metric, and the mean squared error andR 2 score as additional metrics, the proposed hybrid model was seen to significantly outperform the individual models in all three metrics. 2 Related Works Galeano et al. [3] employed a matrix decomposition technique to estimate how frequently drugs ex- hibit side effects. Their results highlighted that specific elements of drug signatures correlated with anatomical categories and administration routes, providing valuable insights into the underlying mechanisms of drug action. Mohsen et al. [4] have published a deep learning-based model that pre- dicts ADRs, using drug-induced gene expression profiles, obtained by feature engineering the FDA Adverse Events Reporting System. They incorporate extensive feature engineering, preprocessing, and hyperparameter tuning to achieve remarkable results. Zhang et al., [5] have put forth a novel method for predicting drug side effects by identifying key feature dimensions and building highly accurate prediction models. They introduced an en- semble learning framework that combines models based on distinct feature sets, achieving superior performance compared to existing state-of-the-art techniques on standard benchmark datasets. Galeano and Paccanaro [6] have developed a machine learning framework capable of predict- ing side effects of drugs that are undergoing clinical trials. By leveraging pharmacological graph networks, their model learns globally optimal self-representations for both drugs and side effects. Analyzing data from 505 therapeutically diverse drugs and 904 side effects, their work demonstrated the model’s effectiveness in identifying adverse effects that might emerge post-market approval. Dey et al. [7] have built a deep learning framework to predict ADRs and identify the associated molecular substructures without needing any predefined requisites. Their study finds that neural model fingerprints tend to outperform other models. Kim et al. [8] have conducted a review of published articles and analyzed several statistical and machine learning methods for detecting ADRs. Their findings revealed that regression methods were predominantly used for electronic medical record (EMR) data, while disproportionality meth- ods were preferred for FDA Adverse Event Reporting System data. Yang and Kar [9] have put out 2 a review highlighting the use of artificial intelligence and machine learning in the early detection of ADRs. Das and Mazumder [10] have published the results of an extensive study on the use of supervised machine learning to predict drug side effects over the past two decades. Their study emphasizes the importance of predicting side effects to reduce time, chemical waste, design com- plexity, and costs in drug development. They summarized drug descriptors, computational models, and their performances, highlighting the challenges and open problems in the field. Their findings underscore the need for further advancements in supervised learning-based side effect prediction to address existing gaps and improve drug safety evaluation. 3 Methodology Figure 1 depicts the proposed methodology of this paper. As seen from the figure, the datasets were first acquired and consolidated before being subjected to preprocessing and then fed into the individual base models before being stacked and fed into the meta-model. The subsequent subsections explain the dataset and the methodology in detail before delving into the architectures of the base and meta models. Figure 1: Proposed Methodology 3.1 Data Preprocessing 3.1.1 Data Acquisition The study utilizes datasets from SIDER, a comprehensive online repository that records informa- tion on marketed medicines and their recorded adverse drug reactions [1] [2]. Three datasets were chosen for this study, viz. the Drug Names dataset, the Indications dataset, and the Side Effects dataset. The Drug Names dataset contained two columns: STITCH ID, which serves as a unique identifier for drugs, and Drug Name, which specifies the name of the drug. The Indications dataset contained details about the indications that usage of a drug could show, these included columns such as Detection Method, Indication Name, MedDRA Type, and MedDRA Indication Name. The Side Effects dataset contained information about drug side effects, including Concept Type and Side Effect Name. 3.1.2 Cleaning and Preprocessing The column names of all the datasets were standardized, and a single dataset was created by merging the individual datasets. These operations consolidated all relevant information into a single analytical framework, enabling comprehensive analysis and further modeling. Handling missing values was imperative to the preprocessing, any problematic entry was sys- tematically dropped from the dataset to maintain its integrity and ensure that the models were 3 reliable. Once this procedure was implemented, the dataset contained a little over 10 million unique rows. The dataset was observed to show class imbalance, with certain side effects being overrepresented while others were very rare. This posed a risk to the credibility of the model’s predictions. Hence a strategic downsampling approach was introduced, wherein rare side effects, defined as those that occur fewer than 1,550 times, were retained without modification to ensure that clinically significant but infrequent adverse reactions were not overlooked whereas, common side effects were randomly sampled to reduce their dominance, thus balancing the data set and preventing bias toward frequently occurring side effects, this ensured that the data remained sensitive to under represented medical side effects, while also ensuring that its diversity was preserved. The dataset was left with around 9.3 million unique values once the downsampling was done. 3.2 Feature Engineering Although the dataset was now cleaned and risks of bias were reduced, it still lacked some key features that were needed to make predictions, whereas a couple of the existing features were not in a form that could directly be used by machine learning models. The dataset required intensive transformations before it could be useful in making the desired predictions. For this purpose, extensive feature engineering was used. This phase dealt with the creation of new features and the transformation of existing ones to ensure that the models could capture complex relationships between drugs, indications, and side effects. The features engineered were as follows: First, TF- IDF vectorization was applied to capture textual patterns from the data. TF-IDF is a statistical measure that computes the importance of a term within a corpus, relative to the entire document. Text data is not directly usable by machine learning models; TF-IDF converts the textual data into numerical vectors that can be used directly by such models. The obtained vectors were then concatenated with the original dataset, significantly enriching the feature space and enabling the models to leverage textual information effectively. Next, frequency-based metrics were added to quantify the prevalence of side effects and indi- cations associated with each drug. The frequency of each side effect and indication was calculated and normalized to represent their relative occurrence in the dataset. These normalized frequencies provided the necessary insights to show the distribution of side effects and indications, thus allowing the models to grasp their relative importance during prediction. Extensive analysis was performed to quantify the relationships between drugs, their indications and associated side effects. This analysis aimed to quantify the frequency with which specific drugs were linked to particular side effects, as well as to assess the number of indications associated with each drug. 3.3 Model Architectures The proposed model was a hybrid ensemble that combined the predictive capabilities of three base models into a single meta-model. The predictive capabilities of each base model were integrated to increase overall accuracy and robustness. This section highlights the features and architectures of each base model and the proposed model. 3.3.1 LightGBM LightGBM is a gradient boosting model that is popular for its efficiency, scalability, and rapid generalization, particularly when dealing with large-scale and high-dimensional data. The model 4 itself is built upon decision trees and is commonly applied to regression problems. What differ- entiates LightGBM from traditional boosting algorithms is that it makes use of techniques like Gradient-based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB), which further help lower the computational overhead without sacrificing accuracy. The LightGBM uses histogram-based algorithms to discretize continuous values into finite bins to speed up training and save space. Furthermore, it employs a leaf-wise tree development approach, rather than the level-wise approach that most other gradient boosting algorithms follow. These trees build upon sequentially, and each successor tries to correct any errors made by their predecessors to minimize the loss. The hyperparameters tuned within the LightGBM model are as follows: the number of iterations was set to 100 to balance model complexity and training time. To prevent overfitting, the L2 regularization was set to 0.1. The minimum number of samples required to create a leaf node was set to 20, the maximum depth of each tree was limited to 5 to control model complexity, and the learning rate was fixed at 0.1. Overall, LightGBM was chosen for its ability to model complex relationships efficiently and its support for parallel processing, which enables faster training compared to other boosting algorithms. 3.3.2 XGBoost XGBoost, or Extreme Gradient Boosting, is a widely used gradient boosting framework known for its scalability, flexibility, and high performance. It employs a regularized learning objective that combines both L1 and L2 regularization terms to prevent overfitting and improve general- ization. XGBoost utilizes a level-wise tree growth strategy, which ensures balanced trees but can be computationally expensive for large datasets. To address this, XGBoost incorporates advanced techniques such as parallel processing, cache-aware access patterns, and sparsity-aware splitting, making it highly efficient for structured data with missing values. These features make XGBoost particularly suitable for tasks dealing with complex relationships between features, such as predict- ing adverse drug reactions. The hyperparameters fine-tuned in this work were as follows: The number of boosting iterations was set to 30 to balance model complexity and training time. The maximum depth of each tree was limited to 5 to control model complexity and avoid overfitting. Regularization parameters were added to mitigate any overfitting, the L2 regularization value was been set to 0.2, while the L1 regularization was set to 0.1. these regularization parameters helped mitigate the risk of overfitting by penalizing overly complex models. Finally, the learning rate was set to 0.1 to ensure stable convergence during training. The above hyperparameters were chosen using Bayesian Optimization, a probabilistic technique that treats the objective function as a distribution and iteratively selects the next set of hyperparameters to maximize expected improvement. XGBoost was selected for its reliability in handling high-dimensional datasets and its flexibil- ity in processing both numerical and categorical variables. Its built-in regularization mechanisms and ability to perform sparsity-aware tree splitting make it particularly suitable for datasets with missing entries and class imbalance—conditions present in the dataset used in this work. 3.3.3 CatBoost CatBoost is a powerful gradient boosting framework, developed to work with categorical data with very little preprocessing. Unlike traditional models that need encoding, CatBoost uses a technique 5 called Ordered Target Encoding that handles categorical variables internally, this technique com- putes category-specific statistics while ensuring that the target leakage is avoided by maintaining a strict order of data during training. Bayesian Optimization was used to choose the hyperparameters for the CatBoost model; the values chosen were as follows: The boosting rounds were set to 600, providing the model with enough iterations to capture nuanced patterns in the data. The depth of the trees was limited to 5 to prevent overfitting. The L2 regularization term was set to 0.2. A learning rate of 0.1 was used to ensure the training process remained stable and gradual. The CatBoost was chosen for use because its ability to internally handle categorical variables, while also managing missing data and class imbalance, perfectly suited the nature of the dataset, which included multiple categorical attributes such as drug names and indications. The inter- pretability and efficiency of the model further contributed to its role in uncovering key associations among drugs, therapeutic uses, and potential side effects. 3.3.4 Proposed Hybrid Model The proposed model was designed to leverage the complementary strengths of multiple base models, combining their predictions into a single framework to achieve superior performance. This approach addressed the limitations of individual models, such as sensitivity to hyperparameters, overfitting, or biases toward specific feature types. By integrating LightGBM, XGBoost, and CatBoost into a single meta-model, the framework captured a broader range of patterns and relationships within the dataset, enhancing its predictive power and robustness. The architecture of the hybrid model follows a two-phase structure: base-level model prediction and meta-level aggregation. In the initial phase, each of the three base models was independently trained using the same preprocessed data. Their outputs were then stacked together and fed in as new input features for the second phase. For the aggregation phase, a Random Forest regressor was used as a meta-learner. The meta learner takes in aggregated outputs yielded by the previous stage to make the final predictions. The random forest model was chosen for its resilience against overfitting, combined with its ability to manage high-dimensional data and maintain interpretabil- ity. The meta model was trained not only on the predictions of the base learners but also on a set of engineered features, including normalized occurrence rates, relevance scores, and TF-IDF representations. These additional features enriched the input space, allowing the meta-model to capture the complex non-linear relationships between drugs, their indications, and associated side effects. 4 Results, Evaluation, and Discussion 4.1 Analysis 4.1.1 Distribution of Side Effects and Indications per Drug The distribution of side effects and indications across each drug was studied to understand their prevalence. Figure 6. shows fifty of the most occurring side effects in the dataset. Pregabalin was seen to have the highest number of side effects (839), followed by aripiprazole (827) and citalopram (823). On the other hand, pramipexole had only 3 indications, while paroxetine had 99. These findings highlight the variability in the number of side effects and indications across different drugs. Similarly, Figure 3 illustrates the top 20 drugs with the most indications. Dexamethasone had the highest number of indications (195), followed by prednisolone (146) and methylprednisolone 6 Figure 2: Drugs with Most Side Effects (142). These results indicate that certain drugs are associated with a wide range of medical condi- tions, which may influence their side effect profiles. Figure 3: Top 20 Drugs with the Most Indications Figure 4. further shows the relationship between the number of side effects and indications per drug via a scatterplot. As seen from the graph, there appears to be no strong linear correlation, and some drugs exhibit a higher number of both side effects and indications, suggesting a complex interplay between their therapeutic uses and adverse reactions. 4.1.2 Top Indication and Side Effect Names The frequency of indication names was also analyzed to identify common medical conditions associ- ated with the drugs. Figure 5. shows the top 30 indication names. Renal impairment was the most frequent indication, occurring nearly 100,000 times, followed by infection (85,000) and malignant neoplasm (80,000). These findings suggest that the dataset contains a significant number of drugs used to treat serious medical conditions. Figure 6 shows the frequency of side effects. As seen from the figure, Nausea was the most frequent side effect, occurring over 60,000 times, followed by dizziness (55,000) and rash (50,000). Overall, the Analysis provided a comprehensive overview of the dataset. The insights gained from this analysis were crucial in understanding the complexity of adverse drug reactions and informing the design of the hybrid meta-model. 7 Figure 4: Side Effects vs Indications Figure 5: Top 30 Indication Names 4.2 Result Evaluation The proposed hybrid meta-model was evaluated using MAE as the primary metric, with the MSE andR 2 score as the additional metrics. These metrics were chosen to provide a comprehensive assessment of the models’ accuracy, consistency, and explanatory power. The results obtained are summarized in Table 1 and are depicted graphically in Figure 7. The individual base models—LightGBM, XGBoost, and CatBoost—each demonstrated strong predictive capabilities but had distinct strengths and limitations. LightGBM achieved an MSE of 0.003706, an MAE of 0.041977, and anR 2 score of 0.4562. While it performed well in terms of MSE and MAE, itsR 2 score indicated room for improvement in explaining the variance in the data. This suggests that while LightGBM can make relatively accurate predictions, it may not capture all the underlying patterns in the dataset. XGBoost showed similar performance with an MSE of 0.003745, an MAE of 0.042241, and an R2 score of 0.4506. Its performance was comparable to LightGBM, indicating that both models have similar strengths and weaknesses when applied independently. However, the slight differences in their scores suggest that one of them might be capturing certain aspects of the data differently from the other. 8 Figure 6: Most frequently observed side effects Table 1: Comparison of Model Performance Model MSE MAER 2 LightGBM 0.003706 0.041977 0.4562 XGBoost 0.003745 0.042241 0.4506 CatBoost 0.003153 0.038657 0.5374 Hybrid Meta-Model 0.000826 0.018332 0.8789 The CatBoost model was seen to outperform the other two base models with an MSE of 0.003153, an MAE of 0.038657, and anR 2 score of 0.5374. These results show that CatBoost has better overall performance compared to LightGBM and XGBoost. Its higherR 2 score indi- cates that it explains more of the variance in the data, suggesting that it captures a broader range of relationships between the features and the target variable. However, the hybrid meta-model significantly outperformed all individual base models. It achieved an MSE of 0.000826, an MAE of 0.018332, and anR 2 score of 0.8789. These results highlight the effectiveness of the stacking approach in capturing intricate relationships within the data and improving predictive performance. By combining the predictions from LightGBM, XG- Boost, and CatBoost, the hybrid model leverages the complementary strengths of each algorithm while mitigating their individual weaknesses. The improvement in measurements across all eval- uation metrics underscores the value of ensemble methods in complex prediction tasks like drug reaction forecasting. 5 Conclusion This study aimed to propose a robust hybrid, ensemble based model, to predict the probability of adverse reactions that could occur due to consumption of a pharmaceutical drug.. The proposed model was built upon the SIDER dataset and leverages the strengths of three base models- the 9 Figure 7: Graphs of the Results 10 LightGBM, XGBoost, and CatBoost. A Random Forest regressor serves as the meta-model, which takes the output predictions from each base model as input features and generates the final pre- diction. The models were evaluated using the MAE as the primary evaluation metric, along with the MSE and R 2 score as additional metrics. Experimental results showed that the individual base models exhibited strong predictive capabilities by themselves, but the proposed hybrid model was seen to outperform them by a margin. The LightGBM model was seen to score a MAE of 0.041977, MSE of 0.003706, and aR 2 score of 0.4562. The XGBoost model was seen to show similar perfor- mance, with a MAE of 0.04224, MSE of 0.003745, andR 2 score of 0.4506. The CatBoost model was seen to outperform the other base models with a MAE of 0.038657, MSE of 0.003153, and a R2 score of 0.5374. The proposed model outperformed the individual base models and yielded the lowest MAE of 0.018332, lowest MSE of 0.000826, and the highestR 2 score of 0.8789. The achieved

demonstrate the advantage that using ensemble techniques can provide whilst dealing with complex datasets with highly non-linear relations. It can be said that the approach adopted in this methodology refined the performance of the base models and yielded more reliable predictions. The performance of the hybrid meta-model can be attributed to the fact that it leverages the complementary predictive strengths of multiple learning algorithms. The integration of a meta- model, which took in the aggregated predictions from each of the base model, ensures that the weaknesses of the individual learners is overcome and allows the proposed model to capture a broader range of relations from the dataset Future studies should look at integrating more real-word sources of data, genomic data or patient-specific information, to further personalize the model for the needs of each individual. Additionally, integration of explainable AI techniques could be studied in order to give the user detailed insights into the predictions.

[1] Kuhn, Michael, Ivica Letunic, Lars Juhl Jensen, and Peer Bork. 2015. “The SIDER Database of Drugs and Side Effects.” Nucleic Acids Research, October 19. https://doi.org/10.1093/nar/gkv1075. [2] Kuhn, Michael, M. Campillos, Ivica Letunic, Lars Juhl Jensen, and Peer Bork. 2010. “A Side Effect Resource to Capture the Phenotypic Effects of Drugs.” Molecular Systems Biology 6 (January): 343. https://doi.org/10.1038/msb.2010.19. [3] Galeano, Diego, Shan Li, Mark Gerstein, et al. 2020. “Predicting the Frequencies of Drug Side Effects.” Nature Communications 11: 4575. https://doi.org/10.1038/s41467-020-18305-y. [4] Mohsen, Ahmed, Lokesh P. Tripathi, and Kazuo Mizuguchi. 2021. “Deep Learn- ing Prediction of Adverse Drug Reactions in Drug Discovery Using Open TG–GATEs and FAERS Databases.” Frontiers in Drug Discovery 1 (October). https://doi.org/10.3389/fddsv.2021.768792. [5] Zhang, Wei, Fang Liu, Ling Luo, et al. 2015. “Predicting Drug Side Effects by Multi-Label Learning and Ensemble Learning.” BMC Bioinformatics 16: 365. https://doi.org/10.1186/s12859-015-0774-y. [6] Galeano, Diego, and Alberto Paccanaro. 2022. “Machine Learning Prediction of Side Effects for Drugs in Clinical Trials.” Cell Reports Medicine 3 (10): 100358. https://doi.org/10.1016/j.crmeth.2022.100358. 11 [7] Dey, Sudipta, Hao Luo, Alok Fokoue, et al. 2018. “Predicting Adverse Drug Reactions through Interpretable Deep Learning Framework.” BMC Bioinformatics 19 (Suppl. 21): 476. https://doi.org/10.1186/s12859-018-2544-0. [8] Kim, Hye Rin, Min Sung, Ji Ae Park, et al. 2022. “Analyzing Adverse Drug Reaction Using Statistical and Machine Learning Methods: A Systematic Review.” Medicine 101 (25): e29387. https://doi.org/10.1097/MD.0000000000029387. [9] Yang, Sritama, and Sarbani Kar. 2023. “Application of Artificial Intelligence and Machine Learning in Early Detection of Adverse Drug Reactions (ADRs) and Drug-Induced Toxicity.” Artificial Intelligence in Chemistry. https://doi.org/10.1016/j.aichem.2023.100011. [10] Das, Priyanka, and Debarshi H. Mazumder. 2023. “An Extensive Survey on the Use of Su- pervised Machine Learning Techniques in the Past Two Decades for Prediction of Drug Side Effects.” Artificial Intelligence Review 56: 9809–9836. https://doi.org/10.1007/s10462-023- 10413-7. 12