Monte Carlo approaches to model selection: Application to the prototype and exemplar models of categorization

Computational cognitive models offer powerful means for testing competing theoretical frameworks. The question of how one should choose the best among competing explanations of data is a central focus of the scientific enterprise. The success of one model versus another depends on robust methods guiding evaluation of a model’s ability to capture the data, balanced with a requirement of a model’s parsimony. Several fruitful approaches for model comparison have been used in the areas of cognitive and mathematical psychology, focusing on identifying the balance of model fit with model complexity measured by the number of free parameters. However, model complexity and flexibility may not be always captured by the number of free parameters. Here, we review traditional approaches to model selection on a sample case of the prototype and exemplar models of categorization. We demonstrate several weaknesses of approaches based solely on parameter counts and demonstrate how computationally intensive methods, such as Monte Carlo simulations, may help overcome those weaknesses. We demonstrate advantages of the Monte Carlo approach for comparison of each model to chance, comparison of two models with equal number of parameters, comparison of two models with differing number of parameters, as well as quantification of uncertainty during model selection. We conclude by discussing the practicalities of model selection using computationally-intensive approaches.