Item Response Models for Multiple Attempts With Incomplete Data

• Published in: Journal of educational measurement 2019-06, Vol.56 (2), p.415-436
• Main Authors: Bergner, Yoav, Choi, Ikkyu, Castellano, Katherine E.
• Format: Article
• Language: English
• Subjects: Academic Ability; Computer Assisted Instruction; Computer Assisted Testing; Formative Evaluation; Freedoms; Goodness of Fit; Homework; Item Analysis; Item Response Theory; Simulation; Tests
• ISSN: 0022-0655; 1745-3984
• DOI: 10.1111/jedm.12214

Allowance for multiple chances to answer constructed response questions is a prevalent feature in computer-based homework and exams. We consider the use of item response theory in the estimation of item characteristics and student ability when multiple attempts are allowed but no explicit penalty is deductedfor extra tries. This is common practice in online formative assessments, where the number of attempts is often unlimited. In these environments, some students may not always answer until-correct, but may rather terminate a response process after one or more incorrect tries. We contrast the cases of graded and sequential item response models, both unidimensional models which do not explicitly account for factors other than ability. These approaches differ not only in terms of log-odds assumptions but, importantly, in terms of handling incomplete data. We explore the consequences of model misspecification through a simulation study and with four online homework data sets. Our results suggest that model selection is insensitive for complete data, but quite sensitive to whether missing responses are regarded as informative (of inability) or not (e.g., missing at random). Under realistic conditions, a sequential model with similar parametric degrees of freedom to a graded model can account for more response patterns and outperforms the latter in terms of model fit.