The Influence of Status on Evaluations: Evidence from Online Coding Contests

• Published in: MIS quarterly 2022-12, Vol.46 (4), p.2085-2110
• Main Authors: Deodhar, Swanand, Babar, Yash, Burtch, Gordon
• Format: Article
• Language: English
• Subjects: Color coding; Competition; Computer programming; Contests; Datasets; Interviews; Quantitative analysis
• ISSN: 0276-7783; 2162-9730
• DOI: 10.25300/MISQ/2022/16178

In many instances, online contest platforms rely on contestants to ensure submission quality. This scalable evaluation mechanism offers a collective benefit. However, contestants may also leverage it to achieve personal, competitive benefits. Our study examines this tension from a status-theoretic perspective, suggesting that the conflict between competitive and collective benefits, and the net implication for evaluation efficacy, is influenced by contestants’ status. On the one hand, contestants of lower status may be viewed as less skilled and hence more likely to make mistakes. Therefore, low-status contestants may attract more evaluations if said evaluations are driven predominantly by an interest in collective benefits. On the other hand, if evaluations are driven largely by an interest in personal, competitive benefits, a low-status contestant makes for a less attractive target and hence may attract fewer evaluations. We empirically test these competing possibilities using a dataset of coding contests from Codeforces. The platform allows contestants to assess others’ submissions and improve evaluations (a collective benefit) by devising test cases (hacks) in addition to those defined by the contest organizer. If a submission is successfully hacked, the hacker earns additional points, and the target submission is eliminated from the contest (a competitive benefit). We begin by providing qualitative evidence based on semi-structured interviews conducted with contestants spanning the status spectrum at Codeforces. Next, we present quantitative evidence exploiting a structural change at Codeforces wherein many contestants experienced an arbitrary status reduction unrelated to their performance because of sudden changes to the platform’s color-coding system around contestant ratings. We show that status-loser contestants received systematically more evaluations from other contestants, absent changes in their short-run submission quality. Finally, we show that the excess evaluations allocated toward affected contestants were less effective, indicating status-driven evaluations as potentially less efficacious. We discuss the implications of our findings for managing evaluation processes in online contests.