Selecting Null Distributions When Calculating r^sub wg^ : A Tutorial and Review

• Published in: Organizational research methods 2014-07, Vol.17 (3), p.324
• Main Authors: Meyer, Rustin D, Mumford, Troy V, Burrus, Carla J, Campion, Michael A, James, Lawrence R
• Format: Article
• Language: English
• Subjects: Agreements; Organization theory; Probability distribution; Studies
• ISSN: 1094-4281; 1552-7425

r^sub wg^ is a common metric used to quantify interrater agreement in the organizational sciences. Finn developed r^sub wg^ is but based it on the assumption that raters' deviations from their true perceptions are influenced by random chance only. James, Demaree, and Wolf extended Finn's work by describing procedures to account for the additional influence of response biases. We demonstrate that organizational scientists have relied largely on Finn's procedures, at least in part because of a lack of specific guidance regarding the conditions under which various response biases might be present. In an effort to address this gap in the literature, we introduce the concept of target-irrelevant, nonrandom forces (those aspects of the research context that are likely to lead to response biases), then describe how the familiar "5Ws and an H" framework (i.e., who, what, when, where, why, and how) can be used to identify these biases a priori. It is our hope that this system will permit those who calculate r^sub wg^ to account for the effects of response biases in a manner that is simultaneously rigorous, consistent, and transparent.