A General Method of Empirical Q-matrix Validation

• Published in: Psychometrika 2016-06, Vol.81 (2), p.253-273
• Main Authors: de la Torre, Jimmy, Chiu, Chia-Yi
• Format: Article
• Language: English
• Subjects: Adolescent; Algorithms; Assessment; Behavioral Science and Psychology; Child; Cognition; Educational Measurement; Educational Psychology; Feasibility Studies; Humanities; Humans; Inferences; Item Response Theory; Law; Methods; Models, Psychological; Models, Statistical; Psychology; Psychometrics; Quantitative psychology; Random variables; Reproducibility of Results; Resistance (Psychology); Statistical Theory and Methods; Statistics as Topic; Statistics for Social Sciences; Test Items; Test Theory; Testing and Evaluation
• ISSN: 0033-3123; 1860-0980
• DOI: 10.1007/s11336-015-9467-8

In contrast to unidimensional item response models that postulate a single underlying proficiency, cognitive diagnosis models (CDMs) posit multiple, discrete skills or attributes, thus allowing CDMs to provide a finer-grained assessment of examinees’ test performance. A common component of CDMs for specifying the attributes required for each item is the Q-matrix. Although construction of Q-matrix is typically performed by domain experts, it nonetheless, to a large extent, remains a subjective process, and misspecifications in the Q-matrix, if left unchecked, can have important practical implications. To address this concern, this paper proposes a discrimination index that can be used with a wide class of CDM subsumed by the generalized deterministic input, noisy “and” gate model to empirically validate the Q-matrix specifications by identifying and replacing misspecified entries in the Q-matrix. The rationale for using the index as the basis for a proposed validation method is provided in the form of mathematical proofs to several relevant lemmas and a theorem. The feasibility of the proposed method was examined using simulated data generated under various conditions. The proposed method is illustrated using fraction subtraction data.