Causal inferences with large scale assessment data: using a validity framework

• Published in: Large-scale assessments in education 2016-12, Vol.4 (1), p.1-18, Article 6
• Main Authors: Rutkowski, David, Delandshere, Ginette
• Format: Article
• Language: English
• Subjects: Achievement Tests; Assessment; Attribution Theory; Causal inferences with cross-sectional large scale assessment data; Control Groups; Data Analysis; Education; Educational Policy; Educational Policy and Politics; Educational Research; Educational Researchers; Elementary Secondary Education; Foreign Countries; Guidelines; Humanities; Inferences; International Assessment; Law; Longitudinal Studies; Mathematics Achievement; Mathematics Tests; Measurement; Quasiexperimental Design; Science Achievement; Science Tests; Statistics for Social Sciences; Testing and Evaluation; Validity
• ISSN: 2196-0739; 2196-0739
• DOI: 10.1186/s40536-016-0019-1

To answer the calls for stronger evidence by the policy community, educational researchers and their associated organizations increasingly demand more studies that can yield causal inferences. International large scale assessments (ILSAs) have been targeted as a rich data sources for causal research. It is in this context that we take up a discussion around causal inferences and ILSAs. Although these rich, carefully developed studies have much to offer in terms of understanding educational systems, we argue that the conditions for making strong causal inferences are rarely met. To develop our argument we first discuss, in general, the nature of causal inferences and then suggest and apply a validity framework to evaluate the tenability of claims made in two well-cited studies. The cited studies exemplify interesting design features and advances in methods of data analysis and certainly contribute to the knowledge base in educational research; however, methodological shortcomings, some of which are unavoidable even in the best of circumstances, urge a more cautious interpretation than that of strict “cause and effect.” We then discuss how findings from causal-focused research may not provide answers to the often broad questions posed by the policy community. We conclude with examples of the importance of the validity framework for the ILSA research community and a suggestion of what should be included in studies that wish to employ quasi-experimental methods with ILSA data.