Evidence of fairness: Twenty-five years of research in Assessing Writing

• Published in: Assessing writing 2019-10, Vol.42, p.100418, Article 100418
• Main Authors: Poe, Mya, Elliot, Norbert
• Format: Article
• Language: English
• Subjects: Bias; Ethics; Fairness; Justice; Literacies; Validity
• ISSN: 1075-2935; 1873-5916
• DOI: 10.1016/j.asw.2019.100418

•Five fairness reseach trends identified: bias, validity, social context, legal responsibility, and ethical obligation.•Findings show researchers have diverse categories in gathering evidence of fairness.•Findings reveal that we do not share a single taxonomy for researching fairness.•Individual score use and contextual factors invite future research that works to structure opportunity for students. When Assessing Writing (ASW) was founded 25 years ago, conversations about fairness were very much in the air and illustrated sharp divides between teachers and educational measurement researchers. For teachers, fairness was typically associated with consistency and access. For educational measurement researchers, fairness was a technical issue: an assessment that did not identify the presence of β (the bias factor) was fair. Since its founding, ASW has continued to be a space where evolving discussions about fairness play out. In this article, we examine a selection of 73 ASW research studies published from 1994 to 2018 that use fairness as a category of evidence. In tracing the use of fairness and related terms across these research articles, our goal is to understand how the conversation about fairness has changed in the last quarter century. Following a literature review that situates fairness within generational, standards-based, and evidential scholarship, we analyze five trends in the journal: fairness as the elimination of bias; fairness as the pursuit of validity; fairness as acknowledgement of social context; fairness as legal responsibility; and fairness as ethical obligation. A tidy narrative that theoretical conceptualization of fairness has deepened over the ASW lifespan is not born out by our findings. Instead, evidence suggests that the disparate stances and methodological challenges that informed early research on fairness remain. As well, the textual record suggests that we have not developed or shared taxonomies for systematically investigating questions of fairness. In our desire to make the research we present actionable, we close by calling attention to the need for theorization of fairness, the advantages nuanced of research methods, and the benefits of non-Western perspectives.