A scale for distinguishing efficacy from effectiveness was adapted and applied to stroke rehabilitation studies

• Published in: Journal of clinical epidemiology 2010, Vol.63 (1), p.11-18
• Main Authors: Zettler, Laura L, Speechley, Mark R, Foley, Norine C, Salter, Katherine L, Teasell, Robert W
• Format: Article
• Language: English
• Subjects: Biological and medical sciences; Clinical medicine; Confidence intervals; Epidemiology; Evidence-Based Medicine - methods; Humans; Internal Medicine; Intervention; Medical research; Medical sciences; Miscellaneous; Neurology; Observer Variation; Public health. Hygiene; Public health. Hygiene-occupational medicine; Randomized controlled trials; Randomized Controlled Trials as Topic - standards; Rehabilitation; Reliability; Reproducibility of Results; Research Design; Stroke; Stroke Rehabilitation; Studies; Systematic review; Treatment Outcome; Validity; Vascular diseases and vascular malformations of the nervous system
• ISSN: 0895-4356; 1878-5921
• DOI: 10.1016/j.jclinepi.2009.06.007

Abstract Objectives To apply a tool that purports to differentiate between efficacy and effectiveness studies to stroke rehabilitation trials and to evaluate its applicability and reliability. Study Design and Setting Three raters developed item operational definitions before independently applying the seven-item scale to 151 randomized controlled trials (RCT), published during or after 1997, that evaluated either a pharmacologic (P, n = 78) or a nonpharmacologic (NP, n = 73) intervention. Inter-rater reliability was assessed for both individual items and total scores, separately for P and NP trials. Results Item inter-rater reliability (multiple-rater kappa) ranged from 0.00 (95% CI [confidence interval]: −0.13, 0.13) to 0.85 (95% CI: 0.73, 0.98) and from 0.21 (95% CI: 0.08, 0.34) to 0.79 (95% CI: 0.66, 0.92) for P and NP RCTs, respectively. For the total score (dichotomized), kappa values were 0.43 (95% CI: 0.31, 0.56) and 0.51 (95% CI: 0.37, 0.64) for P and NP trials, respectively. Conclusions The tool provides a solid foundation on which to base further discussion of the differential criteria of efficacy–effectiveness trial design. Scale items should be properly operationalized depending on the research question of interest and evaluated for reliability before the scale is used for definitively judging a given study's design or the external validity of its results.