Typical Yet Unlikely: An Information Theoretic Approach to the Quantification of 'Normality'

Normality has historically been considered an aspirational trait, synonymous with harmony and ideality. The arithmetic average has been used to define normality, and is often used both productively and unproductively as a blunt way to characterize samples and populations. A number of prior commentaries in the fields of psychology and social science have highlighted the need for caution when reducing complex phenomena to a single mean value. However, to the best of our knowledge, none have described and explained why the mean provides such a poor characterization of normality. We demonstrate that even for datasets with a relatively low number of dimensions (<10), data start to exhibit a number of peculiarities which become progressively severe as the number of dimensions increases. One such peculiarity is that the mean is both the most likely as well as one of the least typical points in multi-dimensional space. The availability of large, multi-dimensional datasets is increasing, and it is therefore especially important that researchers understand the peculiar characteristics of such data. We show that normality can be better characterized with `typicality', an information theoretic concept relating to the entropy of a distribution. An application of typicality to both synthetic and real-world data reveals that in multi-dimensional space, to be normal is actually to be highly atypical.