Using Content Relevance and Representativeness Indices in Instrument Revision

Combining methods from earlier content validity and more contemporary content alignment studies may allow a more complete evaluation of the meaning of test scores than if either set of methods is used on its own. This article distinguishes item relevance indices in the content validity literature fr...

Full description

Saved in:
Bibliographic Details
Published in:Applied measurement in education 2024-04, Vol.37 (2), p.132-147
Main Authors: Traynor, Anne, Christopherson, Sara C.
Format: Article
Language:English
Subjects:
Educational tests & measurements
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Combining methods from earlier content validity and more contemporary content alignment studies may allow a more complete evaluation of the meaning of test scores than if either set of methods is used on its own. This article distinguishes item relevance indices in the content validity literature from test representativeness indices in the subsequent content alignment literature. We then demonstrate that setting minimum thresholds for both types of indices in a sequential process can affect conclusions about test content representativeness, which are integral to validity evaluation. We illustrate this outcome using test item-to-target domain objective match data from subject-matter expert panelists who reviewed sixteen US state achievement tests. Although the illustration incorporates educational achievement test data, the process has broader applicability to development of survey questionnaires, workplace tests, and other instruments.
ISSN:0895-7347
1532-4818
DOI:10.1080/08957347.2024.2347518