On the difficulty of replicating human subjects studies in software engineering

Replications play an important role in verifying empirical results. In this paper, we discuss our experiences performing a literal replication of a human subjects experiment that examined the relationship between a simple test for consistent use of mental models, and success in an introductory progr...

Full description

Saved in:
Bibliographic Details
Main Authors: Lung, Jonathan, Aranda, Jorge, Easterbrook, Steve M., Wilson, Gregory V.
Format: Conference Proceeding
Language:English
Subjects:
Cognitive science
Collaboration
Computing methodologies > Artificial intelligence > Philosophical/theoretical foundations of artificial intelligence > Cognitive science
Data analysis
empirical
experience report
human subjects
Human-centered computing > Human computer interaction (HCI) > Empirical studies in HCI
Humans
Lungs
Packaging
Performance evaluation
Permission
replication
Software engineering
Testing
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Replications play an important role in verifying empirical results. In this paper, we discuss our experiences performing a literal replication of a human subjects experiment that examined the relationship between a simple test for consistent use of mental models, and success in an introductory programming course. We encountered many difficulties in achieving comparability with the original experiment, due to a series of apparently minor differences in context. Based on this experience, we discuss the relative merits of replication, and suggest that, for some human subjects studies, literal replication may not be the the most effective strategy for validating the results of previous studies.
ISSN:0270-5257
1558-1225
DOI:10.1145/1368088.1368115