Mean scores for self-report surveys completed using paper-and-pencil and computers: A meta-analytic test of equivalence

• Published in: Computers in human behavior 2018-09, Vol.86, p.153-164
• Main Authors: Weigold, Arne, Weigold, Ingrid K., Natera, Sara N.
• Format: Article
• Language: English
• Subjects: Computer; Data acquisition; Design analysis; Equivalence; Equivalence testing; Heterogeneity; Meta-analysis; paper-and-pencil; Polls & surveys; Self report; Variance analysis
• ISSN: 0747-5632; 1873-7692
• DOI: 10.1016/j.chb.2018.04.038

The large body of literature on the comparability of mean scores for self-report survey responses gathered using paper-and-pencil and computer data collection methodologies has yielded inconclusive results. However, no comprehensive meta-analysis has been conducted in this field, and those that are available for specific measures have typically not differentiated between studies using between-groups and within-subjects designs. Also, few individual studies, and no meta-analyses, have used correct statistical procedures to determine the equivalence of the two methodologies. Consequently, we conducted two meta-analyses assessing quantitative equivalence (i.e., mean scores), with the first consisting of 144 independent effect sizes from studies with between-groups designs and the second including 70 independent effect sizes from studies using within-subjects designs. Both meta-analyses assessing mean scores indicated equivalence across conditions, with large heterogeneity of variance in the between-groups analysis. Presence of others in both the paper-and-pencil and computer conditions accounted for a significant portion of this variance. Heterogeneity of variance was small for the within-subjects design analysis. Overall, results indicated that the mean scores for self-report surveys using paper-and-pencil and the computer are comparable, although heterogeneity differs for the study designs. Equivalence testing was demonstrated to be the recommended statistical procedure for this type of research.