A RS-BOLGBM Algorithm Based on LightGBM for Supercomputing Center Job Running Status Prediction

In recent years, more and more large-scale high-performance applications have been submitted to the Supercomputing Center for processing. These jobs often require intensive computations, and any running failure will lead to a lot of wasted costs. Therefore, the effective prediction of job running status will help to improve the resource management efficiency of the Supercomputing Center. In this paper, we first propose a combinatorial feature selection method combining ReliefF and SelectKBest, which can effectively obtain job running state features. Then, an improved LightGBM algorithm based on Bayesian hyperparameter optimization is proposed. This approach can further enhance the prediction accuracy and tuning efficiency. Thirdly, we achieve a RS-BOLGBM algorithm based on LightGBM for the Supercomputing Center job running status prediction on Spark. The experimental results show that our proposed RS-BOLGBM algorithm has good prediction accuracy on the common SWF (standard workloads format) dataset of the Supercomputing Center. Our proposed RS-BOLGBM also has good running efficiency on the Spark cluster.