The use of intercoder reliability in qualitative interview data analysis in science education

• Published in: Research in science & technological education 2023-07, Vol.41 (3), p.1155-1175
• Main Authors: Cheung, Kason Ka Ching, Tai, Kevin W. H.
• Format: Article
• Language: English
• Subjects: Coding; Data Analysis; Education; intercoder reliability; Interrater Reliability; interview; Interviews; percentage agreement; Principles; qualitative; Qualitative analysis; Reliability analysis; Replication (Evaluation); Research Methodology; Research methods; Researchers; Science Education; Science Instruction; Test Results; Tests; Transcripts (Written Records)
• ISSN: 0263-5143; 1470-1138
• DOI: 10.1080/02635143.2021.1993179

Intercoder reliability is a statistic commonly reported by researchers to demonstrate the rigour of coding procedures during data analysis. Its importance is debatable in the analysis of qualitative interview data. It raises a question on whether researchers should identify the same codes and themes in a transcript or they should produce different accounts in analyzing the transcript. This study reports how articles in four science education journals, International Journal of Science Education, Research in Science Education, Journal of Research in Science Teaching and Science Education report intercoder reliability in their analysis of interview data. This article explores whether 103 papers published in these science education journals in a single year (2019) have reported intercoder reliability test when the authors analyse their interview data. It was found that 19 papers have reported the test results. The authors of these studies have different interpretation towards a similar value of intercoder reliability. Moreover, the percentage of data used in the intercoder reliability test and the identity of intercoder vary across the studies. As a result, this paper aims to raise an issue on whether a replicability of coding can show the reliability of the results when researchers analyze interview data. We propose two major principles when authors report the reliability of the analysis of interview data: transparency and explanatory. We also argue that only when the authors report intercoder reliability test results that are based on these two principles, the reliability statistics of studies are convincing to readers. Some suggestions are offered to authors regarding how to carry out, analyze and report the intercoder reliability test.