Customized guidance/training improved the psychometric properties of methodologically rigorous risk of bias instruments for non-randomized studies

To evaluate the impact of guidance and training on the inter-rater reliability (IRR), inter-consensus reliability (ICR) and evaluator burden of the Risk of Bias (RoB) in Non-randomized Studies (NRS) of Interventions (ROBINS-I) tool, and the RoB instrument for NRS of Exposures (ROB-NRSE). In a before...

Full description

Saved in:
Bibliographic Details
Published in:Journal of clinical epidemiology 2021-08, Vol.136, p.157-167
Main Authors: Jeyaraman, Maya M., Robson, Reid C., Copstein, Leslie, Al-Yousif, Nameer, Pollock, Michelle, Xia, Jun, Balijepalli, Chakrapani, Hofer, Kimberly, Mansour, Samer, Fazeli, Mir S., Ansari, Mohammed T., Tricco, Andrea C., Rabbani, Rasheda, Abou-Setta, Ahmed M.
Format: Article
Language:English
Subjects:
Bias
Decision making
Domains
Epidemiology
Evaluation
Evaluator burden
Inter-consensus reliability
Inter-rater reliability
Intervention
Non-randomized studies
Quantitative psychology
Reliability analysis
RoB instrument for NRS of exposures
ROBINS-I
Systematic review
Training
User training
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:To evaluate the impact of guidance and training on the inter-rater reliability (IRR), inter-consensus reliability (ICR) and evaluator burden of the Risk of Bias (RoB) in Non-randomized Studies (NRS) of Interventions (ROBINS-I) tool, and the RoB instrument for NRS of Exposures (ROB-NRSE). In a before-and-after study, seven reviewers appraised the RoB using ROBINS-I (n = 44) and ROB-NRSE (n = 44), before and after guidance and training. We used Gwet's AC1 statistic to calculate IRR and ICR. After guidance and training, the IRR and ICR of the overall bias domain of ROBINS-I and ROB-NRSE improved significantly; with many individual domains showing either a significant (IRR and ICR of ROB-NRSE; ICR of ROBINS-I), or nonsignificant improvement (IRR of ROBINS-I). Evaluator burden significantly decreased after guidance and training for ROBINS-I, whereas for ROB-NRSE there was a slight nonsignificant increase. Overall, there was benefit for guidance and training for both tools. We highly recommend guidance and training to reviewers prior to RoB assessments and that future research investigate aspects of guidance and training that are most effective.
ISSN:0895-4356
1878-5921
DOI:10.1016/j.jclinepi.2021.04.017