Detecting and Differentiating the Direction of Change and Intervention Effects in Randomized Trials

• Published in: Journal of applied psychology 2018-01, Vol.103 (1), p.37-53
• Main Authors: Bodner, Todd E, Bliese, Paul D
• Format: Article
• Language: English
• Subjects: Clinical Trials; Employment; Health psychology; Human; Humans; Intervention; Mathematical Modeling; Models, Statistical; Organizational Innovation; Psychology; Randomized Controlled Trials as Topic - standards; Statistical Estimation; Statistical Power; Studies; Terminology; Workplaces
• ISSN: 0021-9010; 1939-1854
• DOI: 10.1037/apl0000251

Calls continue for randomized interventions in organizational settings. In many cases, however, practical constraints require researchers to use 2-wave randomized pretest-posttest control group designs. We discuss the importance of randomized trials for theory development with a focus on analytic options for 2-wave designs. Our discussion has implications for both designing studies and interpreting results. We review 23 published work and organizational health psychology intervention studies and find that a majority of studies featured a statistical model known to have low statistical power relative to other options. Furthermore, a majority of studies invoked terminology implying the direction of change without providing explicit statistical tests. To improve research practice, we detail statistical power differences in 3 commonly used statistical models and emphasize the distinction between (a) intervention effects and (b) the size and direction of change over time. We encourage researchers to provide inferential evidence for both types of information and show that only 1 of the 3 reviewed models provides information on the direction of change over time, but at a potential expense for statistical power to detect intervention effects. A reanalysis of data from a published work-family workplace intervention illustrates these nuances and supports recommendations for research practice. We conclude by providing recommendations.