Consequences of Ignoring Ideal Point Items for Applied Decisions and Criterion-Related Validity Estimates

• Published in: Journal of business and psychology 2015-09, Vol.30 (3), p.483-498
• Main Authors: Dalal, Dev K., Carter, Nathan T.
• Format: Article
• Language: English
• Subjects: Behavioral Science and Psychology; Business and Management; Community and Environmental Psychology; Decision analysis; Decision making; Employee attitude; Estimates; Industrial and Organizational Psychology; Item response theory; Monte Carlo simulation; Occupational psychology; Personality and Social Psychology; Personality tests; Personality traits; Psychology; Social Sciences; Studies; Validity
• ISSN: 0889-3268; 1573-353X
• DOI: 10.1007/s10869-014-9377-2

Purpose Although the popularity of ideal point measurement continues to grow, the implications of mis-scoring ideal point scales by computing a sum-score have received little empirical attention. The possibility of incorrectly scoring these scales is quite high given that many scales developed under dominance assumptions have been shown to follow ideal point assumptions. Unfortunately, a systematic investigation of these implications is missing; this study looked to fill this gap. Design/Methodology/Approach A Monte Carlo simulation was conducted to determine the impact of sum-scoring an ideal point scale on the rank ordering of simulees, bias in criterion-related validity estimates, and selection outcomes. The simulation varied the number of meaningfully unfolding items and the true criterion-related validities. Analyses were conducted within the full sample and varying selection ratios. Findings With all simulees, and when 20 % or more of the items meaningfully unfolded, rank ordering of simulees changed, criterion-related validity estimates showed downward bias, selection hit rates reduced, and per-hire utility was reduced. The patterns of results were similar, though of stronger magnitude, as the sample analyzed became more restrictive. Implications When respondents follow the ideal point response process and the scale contains even a small portion of meaningfully neutral items, sum-scores are inappropriate indices of attribute standing. The consequences of ignoring meaningfully neutral items were much worse in typical application situations. Originality/Value Although some studies have shown strong correlations among the sum-score and IRT-based attribute estimates for ideal point scales, other studies have shown attenuate relations. We present the first systematic investigation of what has, heretofore, been demonstrated incidentally.