Comparing Training Window Selection Methods for Prediction in Non-Stationarity Time Series

The widespread adoption of smartphones creates the possibility to passively monitor everyday behavior via sensors. Sensor data has been linked to moment-tomoment psychological symptoms and mood of individuals and thus could alleviate the burden associated with repeated measurement. Additionally, psychological care could be improved by predicting moments of high psychopathology and providing immediate interventions. Current research assumes that the relationship between sensor data and psychological symptoms is constant over time - or changes with a fixed rate: Models are trained on all past data or on a fixed window, without comparing different window sizes with each other. This is problematic as choosing the wrong training window can negatively impact prediction accuracy, especially if the underlying rate of change is varying. As a potential solution we compare different methodologies for choosing the correct window size ranging from frequent practicebased on heuristics to super learning approaches. In a simulation study, we vary the rate of change in the underlying relationship form over time as well as the model type (OLS and RF). We show that even computing a simple average across different windows can reduce the prediction error rather than selecting a single best window for both simulated and real-world data.