Effect of Standardised Scoring Conventions on Inter-rater Reliability in the Endoscopic Evaluation of Crohn’s Disease

• Published in: Journal of Crohn's and colitis 2016-09, Vol.10 (9), p.1006-1014
• Main Authors: Dubcenco, Elena, Zou, Guangyong, Stitt, Larry, Baker, Jeffrey P., Jeejeebhoy, Khursheed N., Kandel, Gabor, Kim, Young-in, Grover, Samir C., McDonald, John W. D., Shackelton, Lisa M., Khanna, Reena, D’Haens, Geert, Sandborn, William J., Feagan, Brian G., Levesque, Barrett G.
• Format: Article
• Language: English
• Subjects: Adolescent; Adult; Aged; Aged, 80 and over; Clinical Decision-Making - methods; Colonoscopy; Crohn Disease - diagnostic imaging; Crohn Disease - pathology; Education, Medical, Continuing; Female; Gastroenterology - education; Humans; Male; Middle Aged; Observer Variation; Ontario; Reproducibility of Results; Severity of Illness Index; Single-Blind Method; Young Adult
• ISSN: 1873-9946; 1876-4479
• DOI: 10.1093/ecco-jcc/jjw120

Background and Aims: The Crohn’s Disease Endoscopic Index of Severity [CDEIS] and Simplified Endoscopic Score for Crohn’s Disease [SES-CD] demonstrate consistent overall intra- and inter-rater reliability. However, the reliability of some index items is relatively poor. We evaluated scoring conventions to improve the reliability of these items. Methods: Five gastroenterologists with no previous experience scoring the CDEIS or SES-CD were trained on their use. A total of 65 video recordings of colonoscopies were scored blindly by each gastroenterologist before and after additional training on index scoring conventions. Intra-class correlation coefficients [ICCs] assessed the effect of application of these conventions on the reliability of the CDEIS, SES-CD, and a Global Evaluation of Lesion Severity [GELS] score. Results: Following training on scoring conventions, inter-rater ICCs (95% confidence interval [CI]) for the total SES-CD score increased from 0.78 [0.71, 0.85] to 0.85 [0.79, 0.89]. The ICCs for the total CDEIS and GELS scores were not affected: corresponding inter-rater ICCs were 0.74 [0.65, 0.81] and 0.49, [0.38, 0.61] before and 0.73 [0.65, 0.81] and 0.53 [0.42, 0.64] following application of scoring conventions. Estimations of ulcer depth, surface area, anatomical location, and stenosis were important sources of variability. Conclusions: Use of scoring conventions previously developed by expert central readers enhanced the reliability of the SES-CD but did not similarly affect the CDEIS or GELS. As the SES-CD is more likely to be reliable than the CDEIS and can be optimised with targeted training, it is the preferred instrument for use in clinical trials.