A Systematic Assessment of ‘None of the Above’ on Multiple Choice Tests in a First Year Psychology Classroom

• Published in: The Canadian journal for the scholarship of teaching and learning 2015-01, Vol.6 (3), p.1-17
• Main Authors: Pachai, Matthew V, DiBattista, David, Kim, Joseph A
• Format: Article
• Language: English
• Subjects: assessment; Difficulty Level; discrimination; Foreign Countries; Introductory Courses; multiple choice; Multiple Choice Tests; none of the above; Ontario; Psychology; Statistical Analysis; Test Construction; Test Items; Undergraduate Students
• ISSN: 1918-2902; 1918-2902
• DOI: 10.5206/cjsotl-rcacea.2015.3.2

Multiple choice writing guidelines are decidedly split on the use of ‘none of the above’ (NOTA), with some authors discouraging and others advocating its use. Moreover, empirical studies of NOTA have produced mixed results. Generally, these studies have utilized NOTA as either the correct response or a distractor and assessed its effect on difficulty and discrimination. In these studies, NOTA commonly yields increased difficulty when it is used as the correct response, and no change in discrimination regardless of usage. However, when NOTA is implemented as a distractor, rarely is consideration given to the distractor that could have been written in its place. Here, we systematically replaced each distractor in a series of questions with NOTA across different versions of an Introductory Psychology examination. This approach allowed us to quantify the quality of each distractor based on its relative discrimination index and assess the effect of NOTA relative to the quality of distractor it replaced. Moreover, our use of large Introductory Psychology examinations afforded highly stable difficulty and discrimination estimates. We found that NOTA increased question difficulty only when it was the correct response, with no effect on difficulty of replacing any distractor type with NOTA. Moreover, we found that NOTA decreased discrimination when it replaced the most effective distractors, with no effect on discrimination of replacing either the correct response or lowest quality distractor with NOTA. These results replicate the common finding that inclusion of NOTA as the correct response increases question difficulty by equally luring high-performing and low-performing students toward distractors. Moreover, we have shown that including NOTA as a distractor can reduce discrimination if used in lieu of a well written alternative, suggesting that multiple choice authors should avoid using NOTA on multiple choice tests. Les guides de rédaction pour les questions à choix multiple sont incontestablement partagés sur l’usage de la réponse « aucune des situations ci-dessus ». Certains auteurs déconseillent de l’employer alors que d’autres préconisent de le faire. De plus, des études empiriques de l’emploi de cette expression ont mené à des résultats mitigés. En général, ces études ont utilisé l’option « aucune des situations ci-dessus » soit comme étant la réponse correcte soit comme distracteur et ont évalué ses effets sur la difficulté et le discernement. Dans ces ét