Shootout–89, A Comparative Evaluation of Knowledge-based Systems That Forecast Severe Weather

• Published in: Bulletin of the American Meteorological Society 1991-09, Vol.72 (9), p.1339-1354
• Main Authors: Moninger, W.R., Bullas, J., de Lorenzis, B., Ellison, E., Flueck, J., McLeod, J.C., Lusk, C., Lampru, P.D., Phillips, R.S., Roberts, W.F., Shaw, R., Stewart, T.R., Weaver, J., Young, K.C., Zubrick, S.M.
• Format: Article
• Language: English
• Subjects: Acoustic data; Analytical forecasting; Climate models; Earth, ocean, space; Exact sciences and technology; Expert systems; External geophysics; Forecasting models; Geophysics. Techniques, methods, instrumentation and models; Meteorology; Probability forecasts; Statistical weather forecasting; Weather; Weather forecasting
• ISSN: 0003-0007; 1520-0477
• DOI: 10.1175/1520-0477(1991)072<1339:SACEOK>2.0.CO;2

During the summer of 1989, the Forecast Systems Laboratory of the National Oceanic and Atmospheric Administration sponsored an evaluation of artificial-intelligence-based systems that forecast severe convective storms. The evaluation experiment, called Shootout-89, took place in Boulder, Colorado, and focused on storms over the northeastern Colorado foothills and plains. Six systems participated in Shootout-89: three traditional expert systems, a hybrid system including a linear model augmented by a small expert system, an analogue-based system, and a system developed using methods from the cognitive science/judgment analysis tradition. Each day of the exercise, the systems generated 2–9-h forecasts of the probabilities of occurrence of nonsignificant weather, significant weather, and severe weather in each of four regions in northeastern Colorado. A verification coordinator working at the Denver Weather Service Forecast Office gathered ground-truth data from a network of observers. The systems were evaluated on several measures of forecast skill, on timeliness, on ease of learning, and on ease of use. They were generally easy to operate; however, they required substantially different levels of meteorological expertise on the part of their users, reflecting the various operational environments for which they had been designed. The systems varied in their statistical behavior, but on this difficult forecast problem, they generally showed a skill approximately equal to that of persistence forecasts and climatological forecasts.