What Are Panelists Thinking When They Participate in Standard-Setting Studies?

Panelists in an operational standard-setting study were asked to share their thoughts in written form at important points in the process itself-before the meeting started, after training, after completing the 1st and 2nd sets of ratings, immediately following the discussion between Rounds 1 and 2, a...

Full description

Saved in:
Bibliographic Details
Published in:Applied measurement in education 2005-07, Vol.18 (3), p.233-256
Main Authors: Skorupski, William P., Hambleton, Ronald K.
Format: Article
Language:English
Subjects:
Academic Standards
Cognition & reasoning
English (Second Language)
Grade 5
Grade 6
Language Tests
Scoring
Standard Setting
Standardized Tests
State Standards
Student Evaluation
Teacher education
Test Items
Test methods
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Panelists in an operational standard-setting study were asked to share their thoughts in written form at important points in the process itself-before the meeting started, after training, after completing the 1st and 2nd sets of ratings, immediately following the discussion between Rounds 1 and 2, and so on. The item mapping method of Plake and Impara (1997) was implemented with a Grade 5 and 6 English as a Second Language Test intended for statewide use. Several conclusions follow from the research: (a) Panelists' understanding of the purpose for the standard-setting meeting, the performance categories, and details of training is important; (b) panelists' feedback about the process may not be consistent across modes of student assessment; (c) performance tasks with polytomous scoring create special challenges for panelists; and (d) panelists' rate of work may impact their understanding of the tasks and ratings.
ISSN:0895-7347
1532-4818
DOI:10.1207/s15324818ame1803_3