Sensitivity of dispersion measures to distributional patterns and corpus design

While the purpose of dispersion measures is to quantify how evenly an item (or structure) is distributed in a corpus, recent work has shown that indices also respond to other features in the data: Juilland’s D varies systematically with the number of texts (or corpus parts) underlying the analysis, and all commonly used measures respond systematically to the frequency of an item. The present study aims to provide further insights into the sensitivity (or fragility) of dispersion measures to aspects of corpus design and data distribution. Using a simulation study that mimics distributional settings observed in natural language data, we explore how measures respond to differences in corpus design (number of texts, average text length, distribution of text lengths) and distributional milieu (frequency and evenness of distribution). Our results suggest that, within the settings covered by our analysis, the factors frequency and evenness of distribution have roughly the same impact on the observed variability in scores, though there is some variation among measures. The average text length emerges as another feature that leaves its mark on the observed scores. Finally, we note that D2 exhibits the same weakness as D – it varies with the number of units (texts or corpus parts) that enter the analysis.