Mathematical constraints onFST: biallelic markers in arbitrarily many populations

ABSTRACT F ST is one of the most widely used statistics in population genetics. Recent mathematical studies have identified constraints on F ST that challenge interpretations of F ST as a measure with potential to range from 0 for genetically similar populations to 1 for divergent populations. We generalize results obtained for population pairs to arbitrarily many populations, characterizing the mathematical relationship between F ST , the frequency M of the more frequent allele at a polymorphic biallelic marker, and the number of subpopulations K . We show that for fixed K , F ST has a peculiar constraint as a function of M , with a maximum of 1 only if M = i / K for integers i with ⌈ K /2⌉ ≤ i ≤ K − 1. For fixed M , as K grows large, the range of F ST becomes the full closed or half-open unit interval. For fixed K , however, some M < ( K − 1)/ K always exists at which the upper bound on F ST is constrained to be below . In each of three migration models—island, rectangular stepping-stone, and linear stepping-stone—we use coalescent simulations to show that under weak migration, F ST depends strongly on the allele frequency M when K is small, but not when K is large. Finally, using data on human genetic variation, we employ our results to explain the generally smaller F ST values between pairs of continents relative to global F ST values. We discuss implications for the interpretation and use of F ST .