Matched Pair Calibration for Ranking Fairness

We propose a test of fairness in score-based ranking systems called matched pair calibration. Our approach constructs a set of matched item pairs with minimal confounding differences between subgroups before computing an appropriate measure of ranking error over the set. The matching step ensures th...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2023-11
Main Authors: Korevaar, Hannah, McConnell, Chris, Tong, Edmund, Brinkman, Erik, Shine, Alana, Misam Abbas, Metevier, Blossom, Corbett-Davies, Sam, El-Arini, Khalid
Format: Article
Language:English
Subjects:
Calibration
Error analysis
Ranking
Subgroups
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We propose a test of fairness in score-based ranking systems called matched pair calibration. Our approach constructs a set of matched item pairs with minimal confounding differences between subgroups before computing an appropriate measure of ranking error over the set. The matching step ensures that we compare subgroup outcomes between identically scored items so that measured performance differences directly imply unfairness in subgroup-level exposures. We show how our approach generalizes the fairness intuitions of calibration from a binary classification setting to ranking and connect our approach to other proposals for ranking fairness measures. Moreover, our strategy shows how the logic of marginal outcome tests extends to cases where the analyst has access to model scores. Lastly, we provide an example of applying matched pair calibration to a real-word ranking data set to demonstrate its efficacy in detecting ranking bias.
ISSN:2331-8422