Evaluating Probabilistic Forecasts with Bayesian Signal Detection Models

• Published in: Risk analysis 2014-03, Vol.34 (3), p.435-452
• Main Authors: Steyvers, Mark, Wallsten, Thomas S., Merkle, Edgar C., Turner, Brandon M.
• Format: Article
• Language: English
• Subjects: AUC; Bayesian analysis; Bayesian method; Bayesian methods; Biological and medical sciences; combining forecasts; Data collection; Decision making; Economic forecasts; Estimates; evaluating forecasts; Forecasting; Forecasting techniques; judgmental forecasting; Mathematical models; Medical sciences; Probabilistic methods; probability forecasting; Probability theory; Public health. Hygiene-occupational medicine; Representations; Risk assessment; Risk theory; ROC analysis; Signal detection; signal detection theory; Studies
• ISSN: 0272-4332; 1539-6924
• DOI: 10.1111/risa.12127

We propose the use of signal detection theory (SDT) to evaluate the performance of both probabilistic forecasting systems and individual forecasters. The main advantage of SDT is that it provides a principled way to distinguish the response from system diagnosticity, which is defined as the ability to distinguish events that occur from those that do not. There are two challenges in applying SDT to probabilistic forecasts. First, the SDT model must handle judged probabilities rather than the conventional binary decisions. Second, the model must be able to operate in the presence of sparse data generated within the context of human forecasting systems. Our approach is to specify a model of how individual forecasts are generated from underlying representations and use Bayesian inference to estimate the underlying latent parameters. Given our estimate of the underlying representations, features of the classic SDT model, such as the receiver operating characteristic (ROC) curve and the area under the ROC curve (AUC), follow immediately. We show how our approach allows ROC curves and AUCs to be applied to individuals within a group of forecasters, estimated as a function of time, and extended to measure differences in forecastability across different domains. Among the advantages of this method is that it depends only on the ordinal properties of the probabilistic forecasts. We conclude with a brief discussion of how this approach might facilitate decision making.