False Positive Poisson

When analyzing count data (such as number of questions answered correctly), psychologists often use Poisson regressions. We show through simulations that violating the assumptions of a Poisson distribution even slightly can lead to false positive rates more than doubling, and illustrate this issue with a study that finds a clearly spurious but highly significant connection between seeing blue and eating fish candies. In additional simulations we test alternate methods for analyzing count-data and show that these generally do not suffer from the same inflated false positive rate, nor do they result in much higher false negatives in situations where Poisson would be appropriate.