A Survey of Model Evaluation Approaches With a Tutorial on Hierarchical Bayesian Methods

• Published in: Cognitive science 2008-12, Vol.32 (8), p.1248-1284
• Main Authors: Shiffrin, Richard M., Lee, Michael D., Kim, Woojae, Wagenmakers, Eric‐Jan
• Format: Article
• Language: English
• Subjects: Bayesian analysis; Bayesian model selection; Bayesian Statistics; Cognition & reasoning; Cognitive Science; Comparative Analysis; Computation; Evaluation Methods; Generalization; Hierarchical Bayesian modeling; Individual Differences; Memory; Minimum description length; Model evaluation; Model mimicry; Model selection; Prediction; Prequential analysis; Probability; Retention (Psychology); Sciences; Simulation
• ISSN: 0364-0213; 1551-6709
• DOI: 10.1080/03640210802414826

This article reviews current methods for evaluating models in the cognitive sciences, including theoretically based approaches, such as Bayes factors and minimum description length measures; simulation approaches, including model mimicry evaluations; and practical approaches, such as validation and generalization measures. This article argues that, although often useful in specific settings, most of these approaches are limited in their ability to give a general assessment of models. This article argues that hierarchical methods, generally, and hierarchical Bayesian methods, specifically, can provide a more thorough evaluation of models in the cognitive sciences. This article presents two worked examples of hierarchical Bayesian analyses to demonstrate how the approach addresses key questions of descriptive adequacy, parameter interference, prediction, and generalization in principled and coherent ways.