Meta-Analytical Estimates of Interrater Reliability for Direct Supervisor Performance Ratings: Optimism Under Optimal Measurement Designs

Performance appraisal (PA) is used for various organizational purposes and is vital to human resources practices. Despite this, current estimates of PA reliability are low, leading to decades of criticism regarding the use of PA in organizational contexts. In this article, we argue that current meta...

Full description

Saved in:
Bibliographic Details
Published in:Journal of applied psychology 2024-03, Vol.109 (3), p.456-467
Main Authors: Speer, Andrew B., Delacruz, Angie Y., Wegmeyer, Lauren J., Perrotta, James
Format: Article
Language:English
Subjects:
Employee Performance Appraisal
Human
Human resources
Humans
Interrater Reliability
Job Performance
Measurement
Moderators
Optimism
Organizational and Occupational Measures
Performance appraisal
Performance Management
Personnel Evaluation
Psychometrics
Rating
Ratings & rankings
Reproducibility of Results
Supervisors
Work Performance
Citations: Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Performance appraisal (PA) is used for various organizational purposes and is vital to human resources practices. Despite this, current estimates of PA reliability are low, leading to decades of criticism regarding the use of PA in organizational contexts. In this article, we argue that current meta-analytical interrater reliability (IRR) coefficients are underestimates and do not reflect the reliability of interest to most practitioners and researchers-the reliability of an employee's direct supervisor. To establish the reliability of direct supervisor ratings, those making PA ratings must directly supervise employee job performance instead of nonparallel rater designs (e.g., direct supervisor ratings correlated with ratings from a more senior leader). The current meta-analysis identified 22 independent samples that met this more restrictive study inclusion criterion, finding an average observed IRR of .65. We also report reliability estimates for several important contextual moderators, including whether ratings were completed in operational settings (.60) or for research purposes (.67). In sum, we argue that this study's meta-analytical IRR estimates are the best available estimates of direct supervisor reliability and should be used to guide future research and practice.
ISSN:0021-9010
1939-1854
1939-1854
DOI:10.1037/apl0001146