The RAPIDD ebola forecasting challenge: Synthesis and lessons learnt

• Published in: Epidemics 2018-03, Vol.22, p.13-21
• Main Authors: Viboud, Cécile, Sun, Kaiyuan, Gaffey, Robert, Ajelli, Marco, Fumanelli, Laura, Merler, Stefano, Zhang, Qian, Chowell, Gerardo, Simonsen, Lone, Vespignani, Alessandro
• Format: Article
• Language: English
• Subjects: Bayes Theorem; Data accuracy; Ebola epidemic; Epidemics - statistics & numerical data; Forecasting; Forecasting challenge; Hemorrhagic Fever, Ebola - epidemiology; Humans; Liberia - epidemiology; Mathematical modeling; Model comparison; Models, Statistical; Prediction horizon; Prediction performance; Reproducibility of Results; Synthetic data
• ISSN: 1755-4365; 1878-0067
• DOI: 10.1016/j.epidem.2017.08.002

•We report the results of a forecasting challenge based on synthetic, model-generated, outbreak data inspired by the Ebola epidemic in Liberia in 2014–2015.•Eight participating teams representing 16 institutions and government agencies competed in the challenge.•For short-term 1–4 week ahead incidences, prediction performance did not scale with model complexity.•.•Ensemble predictions consistently outperformed any individual model.•Forecasting challenge are important tools to coordinate the modeling community in peace times.•Synthetic forecasting challenges provide a deep understanding of model accuracy and data requirements under controlled environment and could be extended to other known and unknown pathogens. Infectious disease forecasting is gaining traction in the public health community; however, limited systematic comparisons of model performance exist. Here we present the results of a synthetic forecasting challenge inspired by the West African Ebola crisis in 2014–2015 and involving 16 international academic teams and US government agencies, and compare the predictive performance of 8 independent modeling approaches. Challenge participants were invited to predict 140 epidemiological targets across 5 different time points of 4 synthetic Ebola outbreaks, each involving different levels of interventions and “fog of war” in outbreak data made available for predictions. Prediction targets included 1–4 week-ahead case incidences, outbreak size, peak timing, and several natural history parameters. With respect to weekly case incidence targets, ensemble predictions based on a Bayesian average of the 8 participating models outperformed any individual model and did substantially better than a null auto-regressive model. There was no relationship between model complexity and prediction accuracy; however, the top performing models for short-term weekly incidence were reactive models with few parameters, fitted to a short and recent part of the outbreak. Individual model outputs and ensemble predictions improved with data accuracy and availability; by the second time point, just before the peak of the epidemic, estimates of final size were within 20% of the target. The 4th challenge scenario − mirroring an uncontrolled Ebola outbreak with substantial data reporting noise − was poorly predicted by all modeling teams. Overall, this synthetic forecasting challenge provided a deep understanding of model performance under controlled data and epidemiological conditi