How to interpret discrepancies in empirical results from educational intervention studies

• Published in: Scholarship of teaching and learning in psychology 2024-10
• Main Authors: Oppenheimer, Daniel M., Ellefson, Michelle R.
• Format: Article
• Language: English
• Subjects: Classrooms; Education; Empirical Methods; Experimental Replication; Human; School Based Intervention; School Learning; Teaching
• ISSN: 2332-2101; 2332-211X
• DOI: 10.1037/stl0000427

When Mueller and Oppenheimer (2014) documented the longhand advantage—the finding that students learn better when they take notes using pen and paper rather than a laptop—the study went viral, influencing classroom laptop policies around the world. However, more recently Urry et al. (2021) followed up on the original finding but found no evidence for a longhand advantage. Similarly, the literature on whether interventions to improve executive functions lead to improved classroom performance is full of mixed and contradictory findings. Instructors who hope to use evidence-based interventions in the classroom are confronted with a dilemma when they encounter discrepant studies regarding whether or not an intervention is effective. One helpful approach to navigating this challenge is to consider the differences between direct replications and generalization studies. While direct replications recreate the conditions of the original study as closely as possible to determine whether the results are reliable and trustworthy, generalization studies investigate whether the results of the original study are robust to novel conditions, populations, or environments. Because no study can perfectly replicate the conditions of another study, it is not always clear whether a study should be considered a replication or generalization. To resolve this, we describe a framework based in the intervention literature that proposes five principles for evaluating the match between original studies and their follow-ups. We illustrate the utility of this framework using the discrepant findings from Urry et al. (2021) and Mueller and Oppenheimer (2014), and the mixed evidence from the executive control interventions literature as case studies. Broader implications and practicalities for interpreting findings from replication attempts are discussed. (PsycInfo Database Record (c) 2024 APA, all rights reserved) (Source: journal abstract)