Best Practices in Supervised Machine Learning: A Tutorial for Psychologists

Supervised machine learning (ML) is becoming an influential analytical method in psychology and other social sciences. However, theoretical ML concepts and predictive modeling techniques are not yet widely taught in psychology programs. This tutorial is intended to provide an intuitive but thorough primer and introduction to supervised ML for psychologists in four consecutive modules. After introducing the basic terminology and mindset of supervised ML, Module I covers how to use resampling methods to evaluate the performance of ML models (bias-variance tradeoff, performance measures, k-fold cross-validation). Module II introduces the nonlinear random forest, a type of ML model that is particularly user-friendly and well-suited to predicting psychological outcomes. Module III is about performing empirical benchmark experiments (comparing the performance of several ML models on multiple datasets). Finally, Module IV discusses the interpretation of ML models, including permutation variable importance measures, effect plots (partial dependence plots, individual conditional expectation profiles), and the concept of model fairness. Throughout the tutorial, intuitive descriptions of theoretical concepts are provided, with as few mathematical formulas as possible, and followed by code examples, using the mlr3 and companion packages in R. Key practical analysis steps are demonstrated on the publicly available PhoneStudy dataset (N = 624), which includes over 1800 variables from smartphone sensing to predict Big Five personality trait scores. The manuscript contains a checklist to be used as a reminder of important elements when performing, reporting, or reviewing ML analyses in psychology. Additional examples and more advanced concepts are demonstrated in online materials (https://osf.io/9273g/).