A simulation-based test to investigate interrater agreement for binary time series

In observational studies, intensive longitudinal data are often collected by coding the presence/absence of behavior across time. Having a sufficiently high interrater agreement is then quintessential. Although a host of alternatives exist, Cohen’s Kappa has been most popular ever since the measure has been introduced. In many cases, the obtained interrater agreement values are interpreted using the benchmarks provided by, for example, Landis and Koch. This is, however, problematic because of two reasons: First, the value of Cohen’s Kappa (and many other interrater agreement measures) is impacted by the prevalence of the coded behavior. Second, these measures ignore the serial dependence that typically characterizes time series data. In this study, we aim to develop a test that allows interpreting a Kappa value in a given situation. Therefore, we will make use of the error rate committed by the coders during that coding process. For good coding reliability, we will, of course, aim for data where only a few mistakes have been committed. We will introduce a simulation paradigm that accounts for the serial dependence and relative frequency of the true data but also implements several mistakes corresponding with the benchmarks of Landis & Koch. On this data, we calculate Cohen’s Kappa. This sampling distribution which contains a fixed number of mistakes is used for a one-sided significance test that investigates whether the values calculated on the rater-reported data are significantly higher than could be expected given the implemented error rate (and therefore are of acceptable quality).