Systematic metareview of prediction studies demonstrates stable trends in bias and low PROBAST inter-rater agreement

• Published in: Journal of clinical epidemiology 2023-07, Vol.159, p.159-173
• Main Authors: Langenhuijsen, Liselotte F.S., Janse, Roemer J., Venema, Esmee, Kent, David M., van Diepen, Merel, Dekker, Friedo W., Steyerberg, Ewout W., de Jong, Ype
• Format: Article
• Language: English
• Subjects: Bias; Datasets; Diagnosis; Epidemiology; Inter-rater agreement; Literature reviews; Medical diagnosis; Medical prognosis; Prediction; Prediction models; PROBAST; Prognosis; Reviews; Trends
• ISSN: 0895-4356; 1878-5921
• DOI: 10.1016/j.jclinepi.2023.04.012

To (1) explore trends of risk of bias (ROB) in prediction research over time following key methodological publications, using the Prediction model Risk Of Bias ASsessment Tool (PROBAST) and (2) assess the inter-rater agreement of the PROBAST. PubMed and Web of Science were searched for reviews with extractable PROBAST scores on domain and signaling question (SQ) level. ROB trends were visually correlated with yearly citations of key publications. Inter-rater agreement was assessed using Cohen's Kappa. One hundred and thirty nine systematic reviews were included, of which 85 reviews (containing 2,477 single studies) on domain level and 54 reviews (containing 2,458 single studies) on SQ level. High ROB was prevalent, especially in the Analysis domain, and overall trends of ROB remained relatively stable over time. The inter-rater agreement was low, both on domain (Kappa 0.04–0.26) and SQ level (Kappa −0.14 to 0.49). Prediction model studies are at high ROB and time trends in ROB as assessed with the PROBAST remain relatively stable. These results might be explained by key publications having no influence on ROB or recency of key publications. Moreover, the trend may suffer from the low inter-rater agreement and ceiling effect of the PROBAST. The inter-rater agreement could potentially be improved by altering the PROBAST or providing training on how to apply the PROBAST. •Risk of bias (ROB) trends in prediction research remain stable over time.•High ROB is prevalent in all PROBAST domains, but especially in the Analysis domain.•Most PROBAST domains and signaling questions suffer from poor inter-rater agreement.•The PROBAST cannot differentiate between high and very high ROB (ceiling effect).