Using Natural Language Processing to Predict Item Response Times and Improve Test Construction

In this article, it is shown how item text can be represented by (a) 113 features quantifying the text's linguistic characteristics, (b) 16 measures of the extent to which an information‐retrieval‐based automatic question‐answering system finds an item challenging, and (c) through dense word re...

Full description

Saved in:
Bibliographic Details
Published in:Journal of educational measurement 2021-03, Vol.58 (1), p.4-30
Main Authors: Baldwin, Peter, Yaneva, Victoria, Mee, Janet, Clauser, Brian E., Ha, Le An
Format: Article
Language:English
Subjects:
Algorithms
Educational tests & measurements
Information retrieval
Item Response Theory
Natural Language Processing
Prediction
Prediction models
Reaction Time
Test Construction
Test Items
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this article, it is shown how item text can be represented by (a) 113 features quantifying the text's linguistic characteristics, (b) 16 measures of the extent to which an information‐retrieval‐based automatic question‐answering system finds an item challenging, and (c) through dense word representations (word embeddings). Using a random forests algorithm, these data then are used to train a prediction model for item response times and predicted response times then are used to assemble test forms. Using empirical data from the United States Medical Licensing Examination, we show that timing demands are more consistent across these specially assembled forms than across forms comprising randomly‐selected items. Because an exam's timing conditions affect examinee performance, this result has implications for exam fairness whenever examinees are compared with each other or against a common standard.
ISSN:0022-0655
1745-3984
DOI:10.1111/jedm.12264