Targeting the uncertainty of predictions at patient-level using an ensemble of classifiers coupled with calibration methods, Venn-ABERS, and Conformal Predictors: A case study in AD

• Published in: Journal of biomedical informatics 2020-01, Vol.101, p.103350-103350, Article 103350
• Main Authors: Pereira, Telma, Cardoso, Sandra, Guerreiro, Manuela, Mendonça, Alexandre, Madeira, Sara C.
• Format: Article
• Language: English
• Subjects: Alzheimer’s disease; Conformal prediction; Mild cognitive impairment; Prognostic prediction; Uncertainty at patient-level; Venn-ABERS
• ISSN: 1532-0464; 1532-0480
• DOI: 10.1016/j.jbi.2019.103350

[Display omitted] •Providing a measure of the uncertainty of predictions at patient-level is paramount to guide clinicians in risk-sensitive decisions.•We compared calibration methods (Platt Scaling and Isotonic Regression), Conformal Predictors, and Venn-ABERS predictors on assessing the uncertainty of predictions at patient-level, using different classifiers.•We propose an ensemble-based model where calibrated predictions from multiple pairs (classifier, uncertainty method) are combined to predict whether a given MCI patient will convert to AD. Despite being able to make accurate predictions, most existing prognostic models lack a proper indication about the uncertainty of each prediction, that is, the risk of prediction error for individual patients. This hampers their translation to primary care settings through decision support systems. To address this problem, we studied different methods for transforming classifiers into probabilistic/confidence-based predictors (here called uncertainty methods), where predictions are complemented with probability estimates/confidence regions reflecting their uncertainty (uncertainty estimates). We tested several uncertainty methods: two well-known calibration methods (Platt Scaling and Isotonic Regression), Conformal Predictors, and Venn-ABERS predictors. We evaluated whether these methods produce valid predictions, where uncertainty estimates reflect the ground truth probabilities. Furthermore, we assessed the proportion of valid predictions made at high-certainty thresholds (predictions with uncertainty measures above a given threshold) since this impacts their usefulness in clinical decisions. Finally, we proposed an ensemble-based approach where predictions from multiple pairs of (classifier, uncertainty method) are combined to predict whether a given MCI patient will convert to AD. This ensemble should putatively provide predictions for a larger number of patients while releasing users from deciding which pair of (classifier, uncertainty method) is more appropriate for data under study. The analysis was performed with a Portuguese cohort (CCC) of around 400 patients and validated in the publicly available ADNI cohort. Despite our focus on MCI to AD prognosis, the proposed approach can be applied to other diseases and prognostic problems.