The Case for Use of Simple Difference Scores to Test the Significance of Differences in Mean Rates of Change in Controlled Repeated Measurements Designs

• Published in: Multivariate behavioral research 2010-09, Vol.45 (5), p.806-827
• Main Authors: Overall, John E., Tonidandel, Scott
• Format: Article
• Language: English
• Subjects: Behavioral Sciences; Correlation; Depression (Psychology); Difference Scores; Evaluation Methods; Longitudinal Studies; Measurement Techniques; Medical treatment; Mental depression; Monte Carlo Methods; Monte Carlo simulation; Outcomes of Treatment; Regression (Statistics); Regression analysis; Research Design; Scores; Statistical Inference; Statistical Significance
• ISSN: 0027-3171; 1532-7906
• DOI: 10.1080/00273171.2010.519266

A previous Monte Carlo study examined the relative powers of several simple and more complex procedures for testing the significance of difference in mean rates of change in a controlled, longitudinal, treatment evaluation study. Results revealed that the relative powers depended on the correlation structure of the simulated repeated measurements. Tests on dropout-weighted linear slope coefficients fitted to all of the available measurements for each participant were found to provide superior power in the presence of compound symmetry (CS), but tests of significance applied to simple baseline-to-endpoint difference scores provided superior power in the presence of a strongly autoregressive (AR) correlation structure. Type I error rates appeared in an acceptable range for both of those analyses. Insofar as the previous study considered only two widely disparate correlation structures, the present work was undertaken to examine where along a continuum of correlation structures lying between strongly AR and CS the power balance shifts from favoring the simple endpoint difference-score analysis to favoring a regression analysis that utilizes all of the available repeated measurements for each participant. With power calculated from the relative frequencies of rejecting H o at different levels of autoregression, the results indicate superior power for the simple endpoint analysis across more than half the distance from strongly AR to CS. To examine replicability of the simulation results using real data from a previously published study, sampling with replacement from a double-blind controlled study examining the treatment of depression was used to create a Monte Carlo data set from which power could be calculated from relative frequencies of rejecting H o .