A BAYESIAN ANALYSIS FOR INTER-RATER AGREEMENT

An analysis of inter-rater agreement is presented. We study the problem with several raters using a Bayesian model based on the Dirichlet distribution. Inter-rater agreement, including global and partial agreement, is studied by determining the joint posterior distribution of the raters. Posterior d...

Full description

Saved in:
Bibliographic Details
Published in:Communications in statistics. Simulation and computation 2001-08, Vol.30 (3), p.437-446
Main Author: Broemeling, Lyle D.
Format: Article
Language:English
Subjects:
Exact sciences and technology
Inter-rater agreement
Mathematics
Multinomial-Dirichlet model
Nonparametric inference
Parametric inference
Probability and statistics
Resampling
Sciences and techniques of general use
Statistics
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:An analysis of inter-rater agreement is presented. We study the problem with several raters using a Bayesian model based on the Dirichlet distribution. Inter-rater agreement, including global and partial agreement, is studied by determining the joint posterior distribution of the raters. Posterior distributions are computed with a direct resampling technique. Our method is illustrated with an example involving four residents, who are diagnosing 12 psychiatric patients suspected of having a thought disorder. Initially employing analytical and resampling methods, total agreement between the four is examined with a Bayesian testing technique. Later, partial agreement is examined by determining the posterior probability of certain orderings among the rater means. The power of resampling is revealed by its ability to compute complex multiple integrals that represent various posterior probabilities of agreement and disagreement between several raters.
ISSN:0361-0918
1532-4141
DOI:10.1081/SAC-100105072