Estimating Interrater Agreement with the Average Deviation Index: A User’s Guide

The authors present guidelines for establishing a useful range for interrater agreement and a cutoff for acceptable interrater agreement when using Burke, Finkelstein, and Dusig’s average deviation (AD) index as well as critical values for tests of statistical significance with the AD index. Under t...

Full description

Saved in:
Bibliographic Details
Published in:Organizational research methods 2002-04, Vol.5 (2), p.159-172
Main Authors: Burke, Michael J., Dunlap, William P.
Format: Article
Language:English
Subjects:
Acceptability
Agreements
Approximation
Assessments
Categories
Criteria
Deviation
Guidelines
Job performance
Likert scale
Measurement
Organizational behavior
Psychological tests
Quantitative psychology
Research methodology
Standard deviation
Statistical analysis
Statistical significance
Studies
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The authors present guidelines for establishing a useful range for interrater agreement and a cutoff for acceptable interrater agreement when using Burke, Finkelstein, and Dusig’s average deviation (AD) index as well as critical values for tests of statistical significance with the AD index. Under the assumption that judges respond randomly to an item or set of items in a measure, the authors show that a criterion for acceptable interrater agreement or practical significance when using the AD index can be approximated as c/6, where c is the number of response options for a Likert-type item. The resulting values of 0.8, 1.2, 1.5, and 1.8 are discussed as standards for acceptable interrater agreement when using the AD index with 5-, 7-, 9-, and 11-point items, respectively. Using similar logic, the AD agreement index and interpretive standard are generalized to the case of a response scale that involves percentages or proportions, rather than discrete categories, or at the other extreme, the assessment of interrater agreement with respect to the rating of a single target on a dichotomous item (e.g., yes-no, agree-disagree, true-false item formats). Finally, the usefulness of these guidelines for judging acceptable levels of interrater agreement with respect to the metric (or units) of the original response scale is discussed.
ISSN:1094-4281
1552-7425
DOI:10.1177/1094428102005002002