Testing the Difference of Correlated Agreement Coefficients for Statistical Significance

This article addresses the problem of testing the difference between two correlated agreement coefficients for statistical significance. A number of authors have proposed methods for testing the difference between two correlated kappa coefficients, which require either the use of resampling methods...

Full description

Saved in:
Bibliographic Details
Published in:Educational and psychological measurement 2016-08, Vol.76 (4), p.609-637
Main Author: Gwet, Kilem L.
Format: Article
Language:English
Subjects:
Correlation
Differences
Kappa coefficient
Monte Carlo Methods
Monte Carlo simulation
Statistical Analysis
Statistical Significance
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This article addresses the problem of testing the difference between two correlated agreement coefficients for statistical significance. A number of authors have proposed methods for testing the difference between two correlated kappa coefficients, which require either the use of resampling methods or the use of advanced statistical modeling techniques. In this article, we propose a technique similar to the classical pairwise t test for means, which is based on a large-sample linear approximation of the agreement coefficient. We illustrate the use of this technique with several known agreement coefficients including Cohen’s kappa, Gwet’s AC1, Fleiss’s generalized kappa, Conger’s generalized kappa, Krippendorff’s alpha, and the Brenann–Prediger coefficient. The proposed method is very flexible, can accommodate several types of correlation structures between coefficients, and requires neither advanced statistical modeling skills nor considerable computer programming experience. The validity of this method is tested with a Monte Carlo simulation.
ISSN:0013-1644
1552-3888
DOI:10.1177/0013164415596420