Training a Diverse Team on Critical Appraisal Using the AGREE II Instrument

• Published in: The journal of the Canadian Health Libraries Association 2016-01, Vol.37 (2), p.90-91
• Main Authors: Lang-Robertson, Kelly, Loshak, Hannah, Warner-Romano, Claudia, Shirodkar, Apurva, Kandasamy, Sharmilaa, Hunter, Katie, Bevan, Lindsay, Rogers, Jess, Stapon, Claire
• Format: Article
• Language: English
• Subjects: Content analysis; Librarians
• ISSN: 1708-6892

Introduction: Centre for Effective Practice works to enable appropriate health care through the development and implementation of relevant, evidence-based programs and tools. To support these efforts, our information team developed a comprehensive training program designed to educate staff on critical appraisal and to ensure consistency in guideline evaluation. Methods: Training was developed by an information manager in order to support staff learning, current work in guideline evaluation and development, and projects that require critical appraisal of evidence. Participants receive in-depth education on the AGREE II Instrument and complete practice exercises that highlight areas that are commonly difficult to appraise. A qualified librarian reviews the exercises, and team members' scores are compared to generate discussion on differences in understanding and interpretation of the AGREE II Instrument's evaluation measures. AGREE II scores are aggregated through a concordance process that flags areas of inconsistencies. Results: To date, 7 team members successfully completed the training and moved forward to complete over 550 appraisals of guidelines to support a project with the Canadian Partnership Against Cancer. After completing training, 100% of the guideline evaluations reviewed met the predetermined concordance criteria. The training improved the consistency of AGREE II scores and reduced the number of third, independent reviews required to assess the guidelines. Discussion: Appraising guideline quality through a validated process with the AGREE II Instrument is widely accepted. However, the consistency of AGREE scores between reviewers can vary due to differing interpretations and understanding of the criteria, leading to a lack of concordance when comparing scores from different reviewers. Comprehensive training by the information specialist team increased critical appraisal skills and led to a more consistent appraisal of clinical practice guidelines. This program has wider implications for health librarianship, highlighting opportunities to develop training programs that drive greater and more consistent adoption of evidence-based practice.