The inter-rater agreement of retrospective assessments of adverse events does not improve with two reviewers per patient record

• Published in: Journal of clinical epidemiology 2010, Vol.63 (1), p.94-102
• Main Authors: Zegers, Marieke, de Bruijne, Martine C, Wagner, Cordula, Groenewegen, Peter P, van der Wal, Gerrit, de Vet, Henrica C.W
• Format: Article
• Language: English
• Subjects: Adverse events; Biological and medical sciences; Consensus; Epidemiology; General aspects; Hospital; Hospitalization; Hospitals; Humans; Inter-rater agreement; Internal Medicine; Medical Errors - prevention & control; Medical Errors - statistics & numerical data; Medical Records - standards; Medical sciences; Netherlands; Nurses; Observer Variation; Patient safety; Planification. Prevention (methods). Intervention. Evaluation; Public health. Hygiene; Public health. Hygiene-occupational medicine; Record review; Reliability; Reproducibility of Results; Safety Management - methods; Studies
• ISSN: 0895-4356; 1878-5921
• DOI: 10.1016/j.jclinepi.2009.03.004

Abstract Objective To evaluate the inter-rater agreement of the record review process of the Dutch Adverse Event study, which we aimed to improve by the involvement of two independent physician reviewers per record instead of one including a consensus procedure in case of disagreement. Methods The inter-rater agreement within pairs of physicians (independent review between physician A+B) and between pairs of physicians (independent review between physician A+B and C+D) was measured to evaluate the record review process with two physicians including a consensus procedure, with 4,272 and 119 records, respectively. Results The inter-rater agreement within pairs of physicians was substantial for the determination of adverse events (AEs) with a kappa of 0.64 (95% confidence interval [CI]: 0.61, 0.68). The inter-rater agreement between pairs of physicians was fair for the determination of AEs with a kappa of 0.25 (95% CI: 0.05, 0.45). Conclusion A record review process with two physicians per record including a consensus procedure to assess AEs is not more reliable than a record review process with one physician. Retrospective estimates of incidence of AEs from record review studies should be interpreted with caution. Improvement of the method is necessary for monitoring incidence of AEs over time at a national level.