Performance Metrics, Error Modeling, and Uncertainty Quantification

• Published in: Monthly weather review 2016-02, Vol.144 (2), p.607-613
• Main Authors: Tian, Yudong, Nearing, Grey S., Peters-Lidard, Christa D., Harrison, Kenneth W., Tang, Ling
• Format: Article
• Language: English
• Subjects: Bias; Business metrics; Climatology; Correlation coefficient; Decomposition; Error correction; Errors; Hypothesis testing; Mean square errors; Modelling; Nonlinearity; Performance measurement; Philosophy; Statistics And Probability; Time series; Uncertainty; Weather
• ISSN: 0027-0644; 1520-0493
• DOI: 10.1175/MWR-D-15-0087.1

A common set of statistical metrics has been used to summarize the performance of models or measurements-­ the most widely used ones being bias, mean square error, and linear correlation coefficient. They assume linear, additive, Gaussian errors, and they are interdependent, incomplete, and incapable of directly quantifying un­certainty. The authors demonstrate that these metrics can be directly derived from the parameters of the simple linear error model. Since a correct error model captures the full error information, it is argued that the specification of a parametric error model should be an alternative to the metrics-based approach. The error-modeling meth­odology is applicable to both linear and nonlinear errors, while the metrics are only meaningful for linear errors. In addition, the error model expresses the error structure more naturally, and directly quantifies uncertainty. This argument is further explained by highlighting the intrinsic connections between the performance metrics, the error model, and the joint distribution between the data and the reference.