optRF: Optimising random forest stability by determining the optimal number of trees

Abstract Machine learning is frequently used to make decisions based on big data. Among these techniques, random forest is particularly prominent in genomic research, where it is used for selecting the best individuals within a test population or for identifying the most important genomic markers. Although random forest is known to have many advantages, one aspect that is often overseen is that it is a non-deterministic method that can produce different models using the same input data. This can have severe consequences on decision-making processes. In this study, we introduce a method to quantify the impact of non-determinism on predictions, variable importance estimates, and the selection process. Our findings demonstrate that increasing the number of trees in random forests enhances the stability in a non-linear way while computation time increases linearly. Consequently, we conclude that there exists an optimal number of trees for any given data set that maximises the stability without unnecessarily extending the computation time. Based on these results, we have developed the R package optRF which models the relationship between the number of trees and the stability of random forest, providing recommendations for the optimal number of trees for any given data set.