On policymakers’ loss functions and the evaluation of early warning systems

• Published in: Economics letters 2013-04, Vol.119 (1), p.1-7
• Main Author: Sarlin, Peter
• Format: Article
• Language: English
• Subjects: Classification; Decision making models; Early warning systems; Economic models; Mathematical functions; Misclassification costs; Policymakers’ loss functions; Policymakers’ preferences; Probability; Risk assessment; Risk management; Studies
• ISSN: 0165-1765; 1873-7374
• DOI: 10.1016/j.econlet.2012.12.030

This paper introduces a new loss function and Usefulness measure for evaluating early warning systems (EWSs) that incorporate policymakers’ preferences between issuing false alarms and missing crises, and individual observations. The novelty derives from three enhancements: (i) accounting for unconditional probabilities of the classes, (ii) computing the proportion of available Usefulness that the model captures, and (iii) weighting observations by their importance for the policymaker. The proposed measures are model free such that they can be used to assess early warning signals issued by any type of EWS, and flexible for any type of crisis. Applications to two renowned EWSs, and comparisons to two common evaluation measures, illustrate the importance of an objective criterion for choosing a final specification and threshold value, and for models to be useful, the need to be more concerned about the rare class and the importance of correctly classifying observations of most relevant entities. ► This paper introduces novel measures for evaluating early warning systems (EWSs). ► A loss function is augmented to account for class-specific and observation-specific misclassification costs. ► The loss function is further augmented into an absolute and relative Usefulness measure. ► Incorporating unconditional probabilities leads to the need to be more concerned about the rare class. ► Observation-specific weights lead to the need to be more concerned about systemically important entities.