Beyond global estimates of sample-specific reliability: A quantile sensitive approach.

Reliable measures are necessary for successful research studies, with unreliable measures significantly reducing the internal validity of studies when they are used. To combat this, measures are often developed using complex techniques, and ensuring they are reliable is of paramount importance. Despite this, reliability, is always estimated at the mean and assumed to be consistent across the distribution, but heterogeneity in this reliability may exist based on the portion of the distribution analyzed, with reliability differing when the measure is given to low, average, or high achievers. To address this, the present study focuses on the development of a novel method for estimating quantile-sensitive heterogeneity in reliability. This method was then applied to a set of measures from a publicly available dataset and was applied to both item-level and sum-score data, highlighting the flexibility of the approach. Results of the study found that the newly developed approach was effective in detecting the presence of heterogeneous reliability, finding statistically significant differences between the reliability at the lowest versus the highest quantiles within one of the measures tested.