An automatic robust Bayesian approach to principal component regression

• Published in: Journal of applied statistics 2021, Vol.48 (1), p.84-104
• Main Authors: Gagnon, Philippe, Bédard, Mylène, Desgagné, Alain
• Format: Article
• Language: English
• Subjects: Bayesian analysis; Dependent variables; Dimension reduction; linear regression; outliers; Outliers (statistics); principal component analysis; reversible jump algorithms; Robustness (mathematics); Statistical analysis; Statistical methods; whole robustness
• ISSN: 0266-4763; 1360-0532
• DOI: 10.1080/02664763.2019.1710478

Principal component regression uses principal components (PCs) as regressors. It is particularly useful in prediction settings with high-dimensional covariates. The existing literature treating of Bayesian approaches is relatively sparse. We introduce a Bayesian approach that is robust to outliers in both the dependent variable and the covariates. Outliers can be thought of as observations that are not in line with the general trend. The proposed approach automatically penalises these observations so that their impact on the posterior gradually vanishes as they move further and further away from the general trend, corresponding to a concept in Bayesian statistics called whole robustness. The predictions produced are thus consistent with the bulk of the data. The approach also exploits the geometry of PCs to efficiently identify those that are significant. Individual predictions obtained from the resulting models are consolidated according to model-averaging mechanisms to account for model uncertainty. The approach is evaluated on real data and compared to its nonrobust Bayesian counterpart, the traditional frequentist approach and a commonly employed robust frequentist method. Detailed guidelines to automate the entire statistical procedure are provided. All required code is made available, see ArXiv:1711.06341 .