Large-Sample Variance of Fleiss Generalized Kappa

Cohen’s kappa coefficient was originally proposed for two raters only, and it later extended to an arbitrarily large number of raters to become what is known as Fleiss’ generalized kappa. Fleiss’ generalized kappa and its large-sample variance are still widely used by researchers and were implemente...

Full description

Saved in:
Bibliographic Details
Published in:Educational and psychological measurement 2021-08, Vol.81 (4), p.781-790
Main Author: Gwet, Kilem L.
Format: Article
Language:English
Subjects:
Computation
Confidence intervals
Educational tests & measurements
Interrater Reliability
Kappa coefficient
Sample Size
Sample variance
Statistical Analysis
Statistical Inference
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
cited_by cdi_FETCH-LOGICAL-c461t-42fb08dca9851a30bd8d3c1fe87c81c5721c888ad0567f8fabe339728a0bd2e63
cites cdi_FETCH-LOGICAL-c461t-42fb08dca9851a30bd8d3c1fe87c81c5721c888ad0567f8fabe339728a0bd2e63
container_end_page 790
container_issue 4
container_start_page 781
container_title Educational and psychological measurement
container_volume 81
creator Gwet, Kilem L.
description Cohen’s kappa coefficient was originally proposed for two raters only, and it later extended to an arbitrarily large number of raters to become what is known as Fleiss’ generalized kappa. Fleiss’ generalized kappa and its large-sample variance are still widely used by researchers and were implemented in several software packages, including, among others, SPSS and the R package “rel.” The purpose of this article is to show that the large-sample variance of Fleiss’ generalized kappa is systematically being misused, is invalid as a precision measure for kappa, and cannot be used for constructing confidence intervals. A general-purpose variance expression is proposed, which can be used in any statistical inference procedure. A Monte-Carlo experiment is presented, showing the validity of the new variance estimation procedure.
doi_str_mv 10.1177/0013164420973080
format article
fullrecord <record><control><sourceid>proquest_pubme</sourceid><recordid>TN_cdi_pubmedcentral_primary_oai_pubmedcentral_nih_gov_8243202</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ericid>EJ1299724</ericid><sage_id>10.1177_0013164420973080</sage_id><sourcerecordid>2552982809</sourcerecordid><originalsourceid>FETCH-LOGICAL-c461t-42fb08dca9851a30bd8d3c1fe87c81c5721c888ad0567f8fabe339728a0bd2e63</originalsourceid><addsrcrecordid>eNp1kctLw0AQxhdRbK3evQgBL16i-0qyuQhS2vooePBxXSabSU3Jy91W0L_erS0VC85lD99vvvlmh5BTRi8ZS5IrSplgsZScpomgiu6RPosiHgql1D7pr-RwpffIkXNz6ksydkh6QvI4kZT2CZuCnWH4BHVXYfAKtoTGYNAWwbjC0rlggg1aqMovzIMH6Do4JgcFVA5PNu-AvIxHz8PbcPo4uRveTEMjY7YIJS8yqnIDqYoYCJrlKheGFagSo5iJEs6MTwk5jeKkUAVkKESacAUe5RiLAble-3bLrMbcYLPwOXRnyxrsp26h1H-VpnzTs_ZDKy4Fp9wbXGwMbPu-RLfQdekMVhU02C6d5v6nUsUVTT16voPO26Vt_HqeklEqhfwxpGvK2NY5i8U2DKN6dQ-9ew_fcrZuQVuaLT66Zzz1u0qvh2vdwQx_h_7r9w19XJBT</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2545943402</pqid></control><display><type>article</type><title>Large-Sample Variance of Fleiss Generalized Kappa</title><source>SAGE:Jisc Collections:SAGE Journals Read and Publish 2023-2024:2025 extension (reading list)</source><source>ERIC</source><source>PubMed Central</source><creator>Gwet, Kilem L.</creator><creatorcontrib>Gwet, Kilem L.</creatorcontrib><description>Cohen’s kappa coefficient was originally proposed for two raters only, and it later extended to an arbitrarily large number of raters to become what is known as Fleiss’ generalized kappa. Fleiss’ generalized kappa and its large-sample variance are still widely used by researchers and were implemented in several software packages, including, among others, SPSS and the R package “rel.” The purpose of this article is to show that the large-sample variance of Fleiss’ generalized kappa is systematically being misused, is invalid as a precision measure for kappa, and cannot be used for constructing confidence intervals. A general-purpose variance expression is proposed, which can be used in any statistical inference procedure. A Monte-Carlo experiment is presented, showing the validity of the new variance estimation procedure.</description><identifier>ISSN: 0013-1644</identifier><identifier>EISSN: 1552-3888</identifier><identifier>DOI: 10.1177/0013164420973080</identifier><identifier>PMID: 34267400</identifier><language>eng</language><publisher>Los Angeles, CA: SAGE Publications</publisher><subject>Computation ; Confidence intervals ; Educational tests &amp; measurements ; Interrater Reliability ; Kappa coefficient ; Sample Size ; Sample variance ; Statistical Analysis ; Statistical Inference</subject><ispartof>Educational and psychological measurement, 2021-08, Vol.81 (4), p.781-790</ispartof><rights>The Author(s) 2020</rights><rights>The Author(s) 2020 2020 SAGE Publications</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c461t-42fb08dca9851a30bd8d3c1fe87c81c5721c888ad0567f8fabe339728a0bd2e63</citedby><cites>FETCH-LOGICAL-c461t-42fb08dca9851a30bd8d3c1fe87c81c5721c888ad0567f8fabe339728a0bd2e63</cites><orcidid>0000-0001-7968-1432</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktopdf>$$Uhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC8243202/pdf/$$EPDF$$P50$$Gpubmedcentral$$H</linktopdf><linktohtml>$$Uhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC8243202/$$EHTML$$P50$$Gpubmedcentral$$H</linktohtml><link.rule.ids>230,314,723,776,780,881,27901,27902,53766,53768</link.rule.ids><backlink>$$Uhttp://eric.ed.gov/ERICWebPortal/detail?accno=EJ1299724$$DView record in ERIC$$Hfree_for_read</backlink></links><search><creatorcontrib>Gwet, Kilem L.</creatorcontrib><title>Large-Sample Variance of Fleiss Generalized Kappa</title><title>Educational and psychological measurement</title><description>Cohen’s kappa coefficient was originally proposed for two raters only, and it later extended to an arbitrarily large number of raters to become what is known as Fleiss’ generalized kappa. Fleiss’ generalized kappa and its large-sample variance are still widely used by researchers and were implemented in several software packages, including, among others, SPSS and the R package “rel.” The purpose of this article is to show that the large-sample variance of Fleiss’ generalized kappa is systematically being misused, is invalid as a precision measure for kappa, and cannot be used for constructing confidence intervals. A general-purpose variance expression is proposed, which can be used in any statistical inference procedure. A Monte-Carlo experiment is presented, showing the validity of the new variance estimation procedure.</description><subject>Computation</subject><subject>Confidence intervals</subject><subject>Educational tests &amp; measurements</subject><subject>Interrater Reliability</subject><subject>Kappa coefficient</subject><subject>Sample Size</subject><subject>Sample variance</subject><subject>Statistical Analysis</subject><subject>Statistical Inference</subject><issn>0013-1644</issn><issn>1552-3888</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2021</creationdate><recordtype>article</recordtype><sourceid>7SW</sourceid><recordid>eNp1kctLw0AQxhdRbK3evQgBL16i-0qyuQhS2vooePBxXSabSU3Jy91W0L_erS0VC85lD99vvvlmh5BTRi8ZS5IrSplgsZScpomgiu6RPosiHgql1D7pr-RwpffIkXNz6ksydkh6QvI4kZT2CZuCnWH4BHVXYfAKtoTGYNAWwbjC0rlggg1aqMovzIMH6Do4JgcFVA5PNu-AvIxHz8PbcPo4uRveTEMjY7YIJS8yqnIDqYoYCJrlKheGFagSo5iJEs6MTwk5jeKkUAVkKESacAUe5RiLAble-3bLrMbcYLPwOXRnyxrsp26h1H-VpnzTs_ZDKy4Fp9wbXGwMbPu-RLfQdekMVhU02C6d5v6nUsUVTT16voPO26Vt_HqeklEqhfwxpGvK2NY5i8U2DKN6dQ-9ew_fcrZuQVuaLT66Zzz1u0qvh2vdwQx_h_7r9w19XJBT</recordid><startdate>20210801</startdate><enddate>20210801</enddate><creator>Gwet, Kilem L.</creator><general>SAGE Publications</general><general>SAGE PUBLICATIONS, INC</general><scope>7SW</scope><scope>BJH</scope><scope>BNH</scope><scope>BNI</scope><scope>BNJ</scope><scope>BNO</scope><scope>ERI</scope><scope>PET</scope><scope>REK</scope><scope>WWN</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7X8</scope><scope>5PM</scope><orcidid>https://orcid.org/0000-0001-7968-1432</orcidid></search><sort><creationdate>20210801</creationdate><title>Large-Sample Variance of Fleiss Generalized Kappa</title><author>Gwet, Kilem L.</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c461t-42fb08dca9851a30bd8d3c1fe87c81c5721c888ad0567f8fabe339728a0bd2e63</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2021</creationdate><topic>Computation</topic><topic>Confidence intervals</topic><topic>Educational tests &amp; measurements</topic><topic>Interrater Reliability</topic><topic>Kappa coefficient</topic><topic>Sample Size</topic><topic>Sample variance</topic><topic>Statistical Analysis</topic><topic>Statistical Inference</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Gwet, Kilem L.</creatorcontrib><collection>ERIC</collection><collection>ERIC (Ovid)</collection><collection>ERIC</collection><collection>ERIC</collection><collection>ERIC (Legacy Platform)</collection><collection>ERIC( SilverPlatter )</collection><collection>ERIC</collection><collection>ERIC PlusText (Legacy Platform)</collection><collection>Education Resources Information Center (ERIC)</collection><collection>ERIC</collection><collection>CrossRef</collection><collection>MEDLINE - Academic</collection><collection>PubMed Central (Full Participant titles)</collection><jtitle>Educational and psychological measurement</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Gwet, Kilem L.</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><ericid>EJ1299724</ericid><atitle>Large-Sample Variance of Fleiss Generalized Kappa</atitle><jtitle>Educational and psychological measurement</jtitle><date>2021-08-01</date><risdate>2021</risdate><volume>81</volume><issue>4</issue><spage>781</spage><epage>790</epage><pages>781-790</pages><issn>0013-1644</issn><eissn>1552-3888</eissn><abstract>Cohen’s kappa coefficient was originally proposed for two raters only, and it later extended to an arbitrarily large number of raters to become what is known as Fleiss’ generalized kappa. Fleiss’ generalized kappa and its large-sample variance are still widely used by researchers and were implemented in several software packages, including, among others, SPSS and the R package “rel.” The purpose of this article is to show that the large-sample variance of Fleiss’ generalized kappa is systematically being misused, is invalid as a precision measure for kappa, and cannot be used for constructing confidence intervals. A general-purpose variance expression is proposed, which can be used in any statistical inference procedure. A Monte-Carlo experiment is presented, showing the validity of the new variance estimation procedure.</abstract><cop>Los Angeles, CA</cop><pub>SAGE Publications</pub><pmid>34267400</pmid><doi>10.1177/0013164420973080</doi><tpages>10</tpages><orcidid>https://orcid.org/0000-0001-7968-1432</orcidid><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier ISSN: 0013-1644
ispartof Educational and psychological measurement, 2021-08, Vol.81 (4), p.781-790
issn 0013-1644
1552-3888
language eng
recordid cdi_pubmedcentral_primary_oai_pubmedcentral_nih_gov_8243202
source SAGE:Jisc Collections:SAGE Journals Read and Publish 2023-2024:2025 extension (reading list); ERIC; PubMed Central
subjects Computation
Confidence intervals
Educational tests & measurements
Interrater Reliability
Kappa coefficient
Sample Size
Sample variance
Statistical Analysis
Statistical Inference
title Large-Sample Variance of Fleiss Generalized Kappa
url http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-02-13T21%3A09%3A13IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_pubme&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Large-Sample%20Variance%20of%20Fleiss%20Generalized%20Kappa&rft.jtitle=Educational%20and%20psychological%20measurement&rft.au=Gwet,%20Kilem%20L.&rft.date=2021-08-01&rft.volume=81&rft.issue=4&rft.spage=781&rft.epage=790&rft.pages=781-790&rft.issn=0013-1644&rft.eissn=1552-3888&rft_id=info:doi/10.1177/0013164420973080&rft_dat=%3Cproquest_pubme%3E2552982809%3C/proquest_pubme%3E%3Cgrp_id%3Ecdi_FETCH-LOGICAL-c461t-42fb08dca9851a30bd8d3c1fe87c81c5721c888ad0567f8fabe339728a0bd2e63%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_pqid=2545943402&rft_id=info:pmid/34267400&rft_ericid=EJ1299724&rft_sage_id=10.1177_0013164420973080&rfr_iscdi=true