Bayesian approaches to the weighted kappa-like inter-rater agreement measures

Inter-rater agreement measures are used to estimate the degree of agreement between two or more assessors. When the agreement table is ordinal, different weight functions that incorporate row and column scores are used along with the agreement measures. The selection of row and column scores is effe...

Full description

Saved in:
Bibliographic Details
Published in:Statistical methods in medical research 2021-10, Vol.30 (10), p.2329-2351
Main Authors: Tran, Quoc Duyet, Demirhan, Haydar, Dolgun, Anil
Format: Article
Language:English
Subjects:
Agreements
Anomalies
Assessors
Bayes Theorem
Bayesian analysis
Classification
Computer Simulation
Elicitation
Humans
Monte Carlo Method
Monte Carlo simulation
Observer Variation
Reproducibility of Results
Simulation
Weighting functions
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Inter-rater agreement measures are used to estimate the degree of agreement between two or more assessors. When the agreement table is ordinal, different weight functions that incorporate row and column scores are used along with the agreement measures. The selection of row and column scores is effectual on the estimated degree of agreement. The weighted measures are prone to the anomalies frequently seen in agreement tables such as unbalanced table structures or grey zones due to the assessment behaviour of the raters. In this study, Bayesian approaches for the estimation of inter-rater agreement measures are proposed. The Bayesian approaches make it possible to include prior information on the assessment behaviour of the raters in the analysis and impose order restrictions on the row and column scores. In this way, we improve the accuracy of the agreement measures and mitigate the impact of the anomalies in the estimation of the strength of agreement between the raters. The elicitation of prior distributions is described theoretically and practically for the Bayesian estimation of five agreement measures with three different weights using an agreement table having two grey zones. A Monte Carlo simulation study is conducted to assess the classification accuracy of the Bayesian and classical approaches for the considered agreement measures for a given level of agreement. Recommendations for the selection of the highest performing agreement measure and weight combination are made in the breakdown of the table structure and sample size.
ISSN:0962-2802
1477-0334
DOI:10.1177/09622802211037068