Using a Projection IRT Method for Vertical Scaling When Construct Shift Is Present

• Published in: Journal of educational measurement 2021-06, Vol.58 (2), p.211-235
• Main Authors: Strachan, Tyler, Cho, Uk Hyun, Kim, Kyung Yong, Willse, John T., Chen, Shyh‐Huei, Ip, Edward H., Ackerman, Terry A., Weeks, Jonathan P.
• Format: Article
• Language: English
• Subjects: Construct Validity; Educational tests & measurements; Item Response Theory; Reading Skills; Scaling; Simulation; Tests
• ISSN: 0022-0655; 1745-3984
• DOI: 10.1111/jedm.12278

In vertical scaling, results of tests from several different grade levels are placed on a common scale. Most vertical scaling methodologies rely heavily on the assumption that the construct being measured is unidimensional. In many testing situations, however, such an assumption could be problematic. For instance, the construct measured at one grade level may differ from that measured in another grade (e.g., construct shift). On the other hand, dimensions that involve low‐level skills are usually mastered by almost all students as they progress to higher grades. These types of changes in the multidimensional structure, within and across grades, create challenges for developing a vertical scale. In this article, we propose the use of projective IRT (PIRT) as a potential solution to the problem. Assuming that a test measures a primary dimension of substantive interest as well as some peripheral dimensions, the idea underlying PIRT is to integrate out the secondary dimensions such that the model provides both item parameters and ability estimates for the primary dimension. A simulation study was conducted to evaluate the effectiveness of the PIRT as a method for vertical scaling. An example using empirical data from a measure of foundational reading skills is also presented.