Standard error in the Jacobson and Truax Reliable Change Index: The “classical approach” leads to poor estimates

• Published in: Journal of the International Neuropsychological Society 2004-10, Vol.10 (6), p.899-901
• Main Author: TEMKIN, NANCY R.
• Format: Article
• Language: English
• Subjects: Data Interpretation, Statistical; DIALOGUE; Estimates; Humans; Individuality; Neuropsychological Tests - statistics & numerical data; Normal distribution; Outcome and Process Assessment (Health Care) - statistics & numerical data; Practice (Psychology); Practice effects; Psychometrics; Regression Analysis; Reliable change; Reproducibility of Results; Standard error of difference scores; Standard error of measurement of difference scores; Studies; Test–retest data
• ISSN: 1355-6177; 1469-7661
• DOI: 10.1017/S1355617704106115

Different authors have used different estimates of variability in the denominator of the Reliable Change Index (RCI). Maassen attempts to clarify some of the differences and the assumptions underlying them. In particular he compares the ‘classical’ approach using an estimate SEd supposedly based on measurement error alone with an estimate SDiff based on the variability of observed differences in a population that should have no true change. Maassen concludes that not only is SEd based on classical theory, but it properly estimates variability due to measurement error and practice effect while SDiff overestimates variability by accounting twice for the variability due to practice. Simulations show Maassen to be wrong on both accounts. With an error rate nominally set to 10%, RCI estimates using SDiff wrongly declare change in 10.4% and 9.4% of simulated cases without true change while estimates using SEd wrongly declare change in 17.5% and 12.3% of the simulated cases (p < .000000001 and p < .008, respectively). In the simulation that separates measurement error and practice effects, SEd estimates the variability of change due to measurement error to be .34, when the true variability due to measurement error was .014. Neuropsychologists should not use SEd in the denominator of the RCI. (JINS, 2004, 10, 899–901.)