Evaluation of text-level measures of lexical dispersion: Robustness and consistency

The traditional approach to measuring lexical dispersion is to form corpus parts of equal size and then compare the occurrence rate of an item across these units. In recent methodological work, this strategy has met with criticism due to its ignorance to corpus structure. Dispersion, it is argued, should be measured across linguistically meaningful units such as the individual text files constituting the corpus. Though desirable on linguistic grounds, a shift to texts as the unit of analysis raises new methodological issues. While the ability of dispersion measures to handle unevenly-sized corpus units has received attention in the literature, the question of how existing metrics perform in these novel settings has only been partly addressed. This paper aims to shed light on relevant statistical properties of a wide range of text-level dispersion measures. Specifically, we consider the robustness of different indicators, i.e. whether they are (overly) sensitive to data situations that can arise when texts differ (considerably) in length. We use hypothetical data scenarios to identify weak spots in existing measures, and then propose modifications to DP- and DA-related indexes to implement useful statistical properties and effect more resistant estimators. Along with the other measures, these are then evaluated against actual corpus data drawn from the BNC. We observe that adapted DP- and DA-variants perform at least as well as their original versions. Our permutation-based simulation study also demonstrates that Carroll’s D2 shows the same weakness as Juilland’s D, i.e. a noticeable sensitivity to the number of units that enter the analysis.