Methodologically rigorous risk of bias tools for nonrandomized studies had low reliability and high evaluator burden

To assess the real-world interrater reliability (IRR), interconsensus reliability (ICR), and evaluator burden of the Risk of Bias (RoB) in Nonrandomized Studies (NRS) of Interventions (ROBINS-I), and the ROB Instrument for NRS of Exposures (ROB-NRSE) tools. A six-center cross-sectional study with se...

Full description

Saved in:
Bibliographic Details
Published in:Journal of clinical epidemiology 2020-12, Vol.128, p.140-147
Main Authors: Jeyaraman, Maya M., Rabbani, Rasheda, Copstein, Leslie, Robson, Reid C., Al-Yousif, Nameer, Pollock, Michelle, Xia, Jun, Balijepalli, Chakrapani, Hofer, Kimberly, Mansour, Samer, Fazeli, Mir S., Ansari, Mohammed T., Tricco, Andrea C., Abou-Setta, Ahmed M.
Format: Article
Language:English
Subjects:
Agreements
Bias
Consensus
Cross-Sectional Studies
Domains
Epidemiologic Research Design
Epidemiology
Evaluation
Evaluator burden
Humans
Interconsensus reliability
Interrater reliability
Investigations
Nonrandomized studies
Observer Variation
Reliability analysis
Reproducibility of Results
Research Personnel - statistics & numerical data
Risk Assessment
RoB instrument for NRS of exposures
ROBINS-I
User training
Validity
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:To assess the real-world interrater reliability (IRR), interconsensus reliability (ICR), and evaluator burden of the Risk of Bias (RoB) in Nonrandomized Studies (NRS) of Interventions (ROBINS-I), and the ROB Instrument for NRS of Exposures (ROB-NRSE) tools. A six-center cross-sectional study with seven reviewers (2 reviewer pairs) assessing the RoB using ROBINS-I (n = 44 NRS) or ROB-NRSE (n = 44 NRS). We used Gwet’s AC1 statistic to calculate the IRR and ICR. To measure the evaluator burden, we assessed the total time taken to apply the tool and reach a consensus. For ROBINS-I, both IRR and ICR for individual domains ranged from poor to substantial agreement. IRR and ICR on overall RoB were poor. The evaluator burden was 48.45 min (95% CI 45.61 to 51.29). For ROB-NRSE, the IRR and ICR for the majority of domains were poor, while the rest ranged from fair to perfect agreement. IRR and ICR on overall RoB were slight and poor, respectively. The evaluator burden was 36.98 min (95% CI 34.80 to 39.16). We found both tools to have low reliability, although ROBINS-I was slightly higher. Measures to increase agreement between raters (e.g., detailed training, supportive guidance material) may improve reliability and decrease evaluator burden.
ISSN:0895-4356
1878-5921
DOI:10.1016/j.jclinepi.2020.09.033