Revisiting rating scale development for rater-mediated language performance assessments : Modelling construct and contextual choices made by scale developers

Rating scale development in the field of language assessment is often considered in dichotomous ways: It is assumed to be guided either by expert intuition or by drawing on performance data. Even though quite a few authors have argued that rating scale development is rarely so easily classifiable, t...

Full description

Saved in:
Bibliographic Details
Published in:Language testing 2021-10, Vol.38 (4), p.602-626
Main Authors: Knoch, Ute, Deygers, Bart, Khamboonruang, Apichat
Format: Article
Language:English
Subjects:
Applied Linguistics
Corpus linguistics
Evaluation Research
Language assessment
Language Tests
Measures
Rating Scales
Rubrics
Scores
Student assessment
Systematic review
Systematic reviews
Test Construction
Test Reliability
Test Use
Test validity
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Rating scale development in the field of language assessment is often considered in dichotomous ways: It is assumed to be guided either by expert intuition or by drawing on performance data. Even though quite a few authors have argued that rating scale development is rarely so easily classifiable, this dyadic view has dominated language testing research for over a decade. In this paper we refine the dominant model of rating scale development by drawing on a corpus of 36 studies identified in a systematic review. We present a model showing the different sources of scale construct in the corpus. In the discussion, we argue that rating scale designers, just like test developers more broadly, need to start by determining the purpose of the test, the relevant policies that guide test development and score use, and the intended score use when considering the design choices available to them. These include considering the impact of such sources on the generalizability of the scores, the precision of the post-test predictions that can be made about test takers' future performances and scoring reliability. The most important contributions of the model are that it gives rating scale developers a framework to consider prior to starting scale development and validation activities. [Author abstract]
ISSN:0265-5322
1477-0946
DOI:10.1177/0265532221994052