Large-Sample Variance of Fleiss Generalized Kappa

Cohen’s kappa coefficient was originally proposed for two raters only, and it later extended to an arbitrarily large number of raters to become what is known as Fleiss’ generalized kappa. Fleiss’ generalized kappa and its large-sample variance are still widely used by researchers and were implemente...

Full description

Saved in:
Bibliographic Details
Published in:Educational and psychological measurement 2021-08, Vol.81 (4), p.781-790
Main Author: Gwet, Kilem L.
Format: Article
Language:English
Subjects:
Computation
Confidence intervals
Educational tests & measurements
Interrater Reliability
Kappa coefficient
Sample Size
Sample variance
Statistical Analysis
Statistical Inference
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Cohen’s kappa coefficient was originally proposed for two raters only, and it later extended to an arbitrarily large number of raters to become what is known as Fleiss’ generalized kappa. Fleiss’ generalized kappa and its large-sample variance are still widely used by researchers and were implemented in several software packages, including, among others, SPSS and the R package “rel.” The purpose of this article is to show that the large-sample variance of Fleiss’ generalized kappa is systematically being misused, is invalid as a precision measure for kappa, and cannot be used for constructing confidence intervals. A general-purpose variance expression is proposed, which can be used in any statistical inference procedure. A Monte-Carlo experiment is presented, showing the validity of the new variance estimation procedure.
ISSN:0013-1644
1552-3888
DOI:10.1177/0013164420973080