Characterizing the limitations of using diagnosis codes in the context of machine learning for healthcare

• Published in: BMC medical informatics and decision making 2024-02, Vol.24 (1), p.51-51, Article 51
• Main Authors: Guo, Lin Lawrence, Morse, Keith E, Aftandilian, Catherine, Steinberg, Ethan, Fries, Jason, Posada, Jose, Fleming, Scott Lanyon, Lemmon, Joshua, Jessa, Karim, Shah, Nigam, Sung, Lillian
• Format: Article
• Language: English
• Subjects: Age groups; Anemia; Canada; Child; Clinical coding; Cohort identification; Confidence intervals; Data mining; Delivery of Health Care; Diagnosis; Diagnostic coding practice; Diagnostic systems; Electronic health records; Hospitals; Humans; Hyperkalemia; Hypoglycemia; Hyponatremia; Institutions; Labels; Learning algorithms; Machine Learning; Machine learning for health; Medical diagnosis; Methods; Neutropenia; Outcome identification; Patients; Pediatrics; Prediction models; Quality improvement; Quality management; Sensitivity; Sensitivity and Specificity; Technology application; Thrombocytopenia; United States
• ISSN: 1472-6947; 1472-6947
• DOI: 10.1186/s12911-024-02449-8

Diagnostic codes are commonly used as inputs for clinical prediction models, to create labels for prediction tasks, and to identify cohorts for multicenter network studies. However, the coverage rates of diagnostic codes and their variability across institutions are underexplored. The primary objective was to describe lab- and diagnosis-based labels for 7 selected outcomes at three institutions. Secondary objectives were to describe agreement, sensitivity, and specificity of diagnosis-based labels against lab-based labels. This study included three cohorts: SickKids from The Hospital for Sick Children, and StanfordPeds and StanfordAdults from Stanford Medicine. We included seven clinical outcomes with lab-based definitions: acute kidney injury, hyperkalemia, hypoglycemia, hyponatremia, anemia, neutropenia and thrombocytopenia. For each outcome, we created four lab-based labels (abnormal, mild, moderate and severe) based on test result and one diagnosis-based label. Proportion of admissions with a positive label were presented for each outcome stratified by cohort. Using lab-based labels as the gold standard, agreement using Cohen's Kappa, sensitivity and specificity were calculated for each lab-based severity level. The number of admissions included were: SickKids (n = 59,298), StanfordPeds (n = 24,639) and StanfordAdults (n = 159,985). The proportion of admissions with a positive diagnosis-based label was significantly higher for StanfordPeds compared to SickKids across all outcomes, with odds ratio (99.9% confidence interval) for abnormal diagnosis-based label ranging from 2.2 (1.7-2.7) for neutropenia to 18.4 (10.1-33.4) for hyperkalemia. Lab-based labels were more similar by institution. When using lab-based labels as the gold standard, Cohen's Kappa and sensitivity were lower at SickKids for all severity levels compared to StanfordPeds. Across multiple outcomes, diagnosis codes were consistently different between the two pediatric institutions. This difference was not explained by differences in test results. These results may have implications for machine learning model development and deployment.