Completeness of reporting of clinical prediction models developed using supervised machine learning: a systematic review

• Published in: BMC medical research methodology 2022-01, Vol.22 (1), p.12-12, Article 12
• Main Authors: Andaur Navarro, Constanza L, Damen, Johanna A A, Takada, Toshihiko, Nijman, Steven W J, Dhiman, Paula, Ma, Jie, Collins, Gary S, Bajpai, Ram, Riley, Richard D, Moons, Karel G M, Hooft, Lotty
• Format: Article
• Language: English
• Subjects: Artificial intelligence; Biological models; Checklist; Development; Diagnosis; Generalized linear models; Humans; Machine Learning; Methods; Models, Statistical; Prediction (Logic); Prediction model; Prognosis; Report writing; Reporting adherence; Statistical methods; Supervised Machine Learning; Support vector machines; Systematic review; Validation
• ISSN: 1471-2288; 1471-2288
• DOI: 10.1186/s12874-021-01469-6

While many studies have consistently found incomplete reporting of regression-based prediction model studies, evidence is lacking for machine learning-based prediction model studies. We aim to systematically review the adherence of Machine Learning (ML)-based prediction model studies to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Statement. We included articles reporting on development or external validation of a multivariable prediction model (either diagnostic or prognostic) developed using supervised ML for individualized predictions across all medical fields. We searched PubMed from 1 January 2018 to 31 December 2019. Data extraction was performed using the 22-item checklist for reporting of prediction model studies ( www.TRIPOD-statement.org ). We measured the overall adherence per article and per TRIPOD item. Our search identified 24,814 articles, of which 152 articles were included: 94 (61.8%) prognostic and 58 (38.2%) diagnostic prediction model studies. Overall, articles adhered to a median of 38.7% (IQR 31.0-46.4%) of TRIPOD items. No article fully adhered to complete reporting of the abstract and very few reported the flow of participants (3.9%, 95% CI 1.8 to 8.3), appropriate title (4.6%, 95% CI 2.2 to 9.2), blinding of predictors (4.6%, 95% CI 2.2 to 9.2), model specification (5.2%, 95% CI 2.4 to 10.8), and model's predictive performance (5.9%, 95% CI 3.1 to 10.9). There was often complete reporting of source of data (98.0%, 95% CI 94.4 to 99.3) and interpretation of the results (94.7%, 95% CI 90.0 to 97.3). Similar to prediction model studies developed using conventional regression-based techniques, the completeness of reporting is poor. Essential information to decide to use the model (i.e. model specification and its performance) is rarely reported. However, some items and sub-items of TRIPOD might be less suitable for ML-based prediction model studies and thus, TRIPOD requires extensions. Overall, there is an urgent need to improve the reporting quality and usability of research to avoid research waste. PROSPERO, CRD42019161764.