A Propensity Score Method for Investigating Differential Item Functioning in Performance Assessment

This study introduces a novel differential item functioning (DIF) method based on propensity score matching that tackles two challenges in analyzing performance assessment data, that is, continuous task scores and lack of a reliable internal variable as a proxy for ability or aptitude. The proposed...

Full description

Saved in:
Bibliographic Details
Published in:Educational and psychological measurement 2020-06, Vol.80 (3), p.476-498
Main Authors: Chen, Michelle Y., Liu, Yan, Zumbo, Bruno D.
Format: Article
Language:English
Subjects:
Educational evaluation
Educational tests & measurements
English proficiency
Evaluation Methods
High Stakes Tests
Language proficiency
Language Tests
Performance Based Assessment
Probability
Scores
Student writing
Test Items
Writing Tests
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This study introduces a novel differential item functioning (DIF) method based on propensity score matching that tackles two challenges in analyzing performance assessment data, that is, continuous task scores and lack of a reliable internal variable as a proxy for ability or aptitude. The proposed DIF method consists of two main stages. First, propensity score matching is used to eliminate preexisting group differences before the test, ideally creating equivalent groups as in a randomized experimental study. Then, linear mixed effects models are adopted to perform DIF analysis based on the matched data set. We demonstrate this propensity DIF method using a high-stakes functional English language proficiency test. DIF due to education was investigated in the writing component, which consists of two continuously scored performance-based tasks. Although the proposed method is demonstrated in the context of language testing, it can be applied to other types of performance assessments.
ISSN:0013-1644
1552-3888
DOI:10.1177/0013164419878861