The Effect of Method Characteristics on Retest Score Gains and Criterion-Related Validity

• Published in: Journal of business and psychology 2016-06, Vol.31 (2), p.233-248
• Main Authors: Villado, Anton J., Randall, Jason G., Zimmer, Christina U.
• Format: Article
• Language: English
• Subjects: Analysis; Behavioral Science and Psychology; Business and Management; Cognitive ability; Community and Environmental Psychology; Employment; Industrial and Organizational Psychology; Organization theory; ORIGINAL PAPER; Personality and Social Psychology; Personality tests; Personnel selection; Psychology; Skills; Social Sciences; Studies; Testing; Validity
• ISSN: 0889-3268; 1573-353X
• DOI: 10.1007/s10869-015-9408-7

Purpose We sought to empirically assess the effect of predictor method characteristics (test form, item-type, and test-type) on retest score change associated with an invariant construct—general mental ability (GMA)—and to evaluate the effect of retesting on the criterion-related validity of assessments that vary in their susceptibility to retest effects. Design Three hundred seven individuals completed a battery of GMA assessments. After a 6-week interval, participants returned to the testing site to retest using both alternate and identical forms of the initial assessments. Findings Greater score gains were observed on assessments comprising heterogeneous item-types than homogeneous item-types, and on performance-based assessments than self-report assessments. However, despite variations in score gains, the relationships between the initial test scores and criterion scores were no different than the relationships between retest scores and criterion scores for all assessments. Implications Tests and procedures that reduce reliance on test- or item-specific knowledge and skill may help minimize score changes due to retesting across multiple administrations. Moreover, under the boundary conditions present in this study, the criterion-related validity of ability assessments may not be affected by increases in test-specific knowledge and skills. Originality/Value Despite the prevalence and industry support of retesting, a comprehensive understanding of retest score change still eludes researchers and practitioners. This ambiguity may be due in part to neglecting the method-construct distinctions in the retest literature. This is the first report to explicitly utilize the method-construct distinction in an effort to examine the causes and consequences of retest effects.