The risk of bias in systematic reviews tool showed fair reliability and good construct validity

There is a movement from generic quality checklists toward a more domain-based approach in critical appraisal tools. This study aimed to report on a first experience with the newly developed risk of bias in systematic reviews (ROBIS) tool and compare it with A Measurement Tool to Assess Systematic R...

Full description

Saved in:
Bibliographic Details
Published in:Journal of clinical epidemiology 2017-11, Vol.91, p.121-128
Main Authors: Bühn, Stefanie, Mathes, Tim, Prengel, Peggy, Wegewitz, Uta, Ostermann, Thomas, Robens, Sibylle, Pieper, Dawid
Format: Article
Language:English
Subjects:
AMSTAR
Bias
Epidemiology
Humans
Occupational Health
Quality assessment
Ratings
Reliability
Reliability analysis
Reproducibility of Results
Review Literature as Topic
Risk assessment
Risk of bias
ROBIS
Systematic review
Systematic reviews
Validation
Validation studies
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:There is a movement from generic quality checklists toward a more domain-based approach in critical appraisal tools. This study aimed to report on a first experience with the newly developed risk of bias in systematic reviews (ROBIS) tool and compare it with A Measurement Tool to Assess Systematic Reviews (AMSTAR), that is, the most common used tool to assess methodological quality of systematic reviews while assessing validity, reliability, and applicability. Validation study with four reviewers based on 16 systematic reviews in the field of occupational health. Interrater reliability (IRR) of all four raters was highest for domain 2 (Fleiss' kappa κ = 0.56) and lowest for domain 4 (κ = 0.04). For ROBIS, median IRR was κ = 0.52 (range 0.13–0.88) for the experienced pair of raters compared to κ = 0.32 (range 0.12–0.76) for the less experienced pair of raters. The percentage of “yes” scores of each review of ROBIS ratings was strongly correlated with the AMSTAR ratings (rs = 0.76; P = 0.01). ROBIS has fair reliability and good construct validity to assess the risk of bias in systematic reviews. More validation studies are needed to investigate reliability and applicability, in particular.
ISSN:0895-4356
1878-5921
DOI:10.1016/j.jclinepi.2017.06.019