Monitoring Approaches for a Pediatric Chronic Kidney Disease Machine Learning Model

• Published in: Applied clinical informatics 2022-03, Vol.13 (2), p.431-438
• Main Authors: Morse, Keith E., Brown, Conner, Fleming, Scott, Todd, Irene, Powell, Austin, Russell, Alton, Scheinker, David, Sutherland, Scott M., Lu, Jonathan, Watkins, Brendan, Shah, Nigam H., Pageler, Natalie M., Palma, Jonathan P.
• Format: Article
• Language: English
• Subjects: Benchmarking; Child; Computer Simulation; Female; Hospitalization; Humans; Machine Learning; Male; Models, Biological; Prospective Studies; Renal Insufficiency, Chronic - diagnosis; Renal Insufficiency, Chronic - physiopathology; Research Article; Retrospective Studies; Risk Factors; ROC Curve
• ISSN: 1869-0327; 1869-0327
• DOI: 10.1055/s-0042-1746168

Abstract Objective  The purpose of this study is to evaluate the ability of three metrics to monitor for a reduction in performance of a chronic kidney disease (CKD) model deployed at a pediatric hospital. Methods  The CKD risk model estimates a patient's risk of developing CKD 3 to 12 months following an inpatient admission. The model was developed on a retrospective dataset of 4,879 admissions from 2014 to 2018, then run silently on 1,270 admissions from April to October, 2019. Three metrics were used to monitor its performance during the silent phase: (1) standardized mean differences (SMDs); (2) performance of a “membership model”; and (3) response distribution analysis. Observed patient outcomes for the 1,270 admissions were used to calculate prospective model performance and the ability of the three metrics to detect performance changes. Results  The deployed model had an area under the receiver-operator curve (AUROC) of 0.63 in the prospective evaluation, which was a significant decrease from an AUROC of 0.76 on retrospective data ( p  = 0.033). Among the three metrics, SMDs were significantly different for 66/75 (88%) of the model's input variables ( p