Investigating Design Features of Descriptive Graphic Rating Scales

• Published in: Applied measurement in education 2002-01, Vol.15 (2), p.187-215
• Main Author: Myford, Carol M.
• Format: Article
• Language: English
• Subjects: Evaluators; Graphic Representation; Rasch Model; Rater Effects; Rating Scales; Reliability; Test Construction
• ISSN: 0895-7347; 1532-4818
• DOI: 10.1207/S15324818AME1502_04

Trained raters used descriptive graphic rating scales (i.e., continuous score scales) to evaluate students' work. The scales incorporated different combinations of design features (i.e., presence or absence of a defined midpoint or hatchmarks or both). The study's purposes were to (a) decide how many points each continuous scale could be considered to have and (b) determine whether different design features affected reliability. The data were analyzed using Facets (Linacre, 1996), a Rasch-based rating scale analysis computer program. In addition, intraclass correlational analyses were run. All the continuous scales could be considered to have at least 5 points, whereas some of the scales could be viewed as having as many as 7 to 10 points. If the scales were considered to have 5 points, then student separation reliabilities ranged from .79 to .91, whereas the intraclass correlation coefficients ranged from .73 to .85. Scales with defined midpoints did not have higher student separation reliabilities or intraclass correlations than scales with no defined midpoints. Similarly, scales with more hatchmarks were no more or less reliable than scales with fewer hatchmarks.