Assessing the Internal Consistency Reliability of Ecological Momentary Assessment Measures: Insights from the WARN-D Study

Intensive longitudinal research has become increasingly popular in the social and clinical sciences in recent years. However, this popularity has brought about many challenges for both methodological and empirical researchers, including challenges regarding measurement. In this study, we are particularly interested in the assessment of the reliability when multiple items are used to measure the same construct in intensive longitudinal data. This is important because reliability estimates are necessary (albeit not sufficient) to evaluate the quality of measures. Here, we evaluate the internal consistency reliability of scales used during Stage 2 of the WARN-D study, a 3-month period of daily and weekly measurements. The WARN-D study is a prospective 2-year study of approximately 1750 students conducted in the Netherlands, aiming at building an early warning system for depression. Stage 2 includes three months of data on positive and negative affect measured four times a day, and depression and anxiety measured once a week. To assess the reliability of each scale, we use six different statistical approaches including three simpler approaches that estimate the reliability at the between-person and within-person level, and three idiographic approaches that estimate person-specific reliability coefficients. This paper also serves as a tutorial guide for substantive researchers, providing coded examples to facilitate the reporting of the reliability of Ecological Momentary Assessement (EMA) measures. We encourage all researchers to report the reliability of their data when applying the introduced statistical approaches, contributing to a collaborative effort toward developing more reliable measures in psychological and behavioral science.