Concerns, Considerations, and New Ideas for Data Collection and Research in Educational Technology Studies

• Published in: Journal of research on technology in education 2010-01, Vol.43 (1), p.29-52
• Main Authors: Bebell, Damian, O'Dwyer, Laura M., Russell, Michael, Hoffmann, Tom
• Format: Article
• Language: English
• Subjects: Accountability; Classrooms; Computer Use; Computers; Data Collection; Definitions; Economic indicators; Educational Research; Educational Researchers; Educational Technology; educational technology research; Elementary Secondary Education; Evidence; Instructional Leadership; Internet; Leaders; Learning; measurement; Measurement Techniques; Multimedia Instruction; Outcome Measures; Questionnaires; Research Design; Research Methodology; Researchers; Schools; Software; Students; survey design; Surveys; Teachers; Teaching; Teaching Methods; Technology; Technology Uses in Education; Variables; Word Processing
• ISSN: 1539-1523; 1945-0818
• DOI: 10.1080/15391523.2010.10782560

In the following pages, we examine some common methodological challenges in educational technology research and highlight new data collection approaches using examples from the literature and our own work. Given that surveys and questionnaires remain widespread and dominant tools across nearly all studies of educational technology, we first discuss the background and limitations of how researchers have traditionally used surveys to define and measure technology use (as well as other variables and outcomes). Through this discussion, we introduce our own work with a visual analog "sliding" scale as an example of a new approach to survey design and data collection that capitalizes on the technology resources increasingly available in schools. Next, we highlight other challenges and opportunities inherent in the study of educational technology, including the potential for computer adaptive surveying, and discuss the critical importance of aligning outcome measures with the technological innovation, concerns with computer-based versus paper-based measures of achievement, and the need to consider the hierarchical structure of educational data in the analysis of data for evaluating the impact of technology interventions.