Same difference? Understanding variation in the estimation of effect sizes from educational trials

•Analytical models should respect study designs and the nested structure of data.•MLM with total variance should be used for cluster and multisite randomised trials.•Models with different covariates differ in their total variances to be explained.•Both regression models and their outcomes and covari...

Full description

Saved in:
Bibliographic Details
Published in:International journal of educational research 2016, Vol.77, p.1-14
Main Authors: Xiao, ZhiMin, Kasim, Adetayo, Higgins, Steve
Format: Article
Language:English
Subjects:
Bayesian MLM
Effect size
Evaluation
Multilevel modelling
RCT
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:•Analytical models should respect study designs and the nested structure of data.•MLM with total variance should be used for cluster and multisite randomised trials.•Models with different covariates differ in their total variances to be explained.•Both regression models and their outcomes and covariates should be pre-specified. By applying four analytic models with comparable outcomes and covariates to a dataset of 20 outcomes from 17 educational trials, we found results closely matching in well-powered studies without serious implementation problems. The interventions and evaluations were all funded by the Education Endowment Foundation and independently evaluated. We demonstrated that when an analysis takes little account of research design, or where there were difficulties with implementation and data collection, point estimates of effect differ and estimates of precision vary. This adds to the challenge of understanding the comparative impact of interventions and deciding which are worth scaling up.
ISSN:0883-0355
1873-538X
DOI:10.1016/j.ijer.2016.02.001