Development and Validation of a Machine Learning Algorithm Using Clinical Pages to Predict Imminent Clinical Deterioration

• Published in: Journal of general internal medicine : JGIM 2024-01, Vol.39 (1), p.27-35
• Main Authors: Steitz, Bryan D., McCoy, Allison B., Reese, Thomas J., Liu, Siru, Weavind, Liza, Shipley, Kipp, Russo, Elise, Wright, Adam
• Format: Article
• Language: English
• Subjects: Acuity; Algorithms; Clinical Deterioration; Critical Care; Female; Health care facilities; Hospitalization; Humans; Intensive care; Internal Medicine; Learning algorithms; Long short-term memory; Machine Learning; Male; Medicine; Medicine & Public Health; Messages; Middle Aged; Observational studies; Original Research; Patients; Retrospective Studies; ROC Curve; Vital signs; Workflow
• ISSN: 0884-8734; 1525-1497; 1525-1497
• DOI: 10.1007/s11606-023-08349-3

Background Early detection of clinical deterioration among hospitalized patients is a clinical priority for patient safety and quality of care. Current automated approaches for identifying these patients perform poorly at identifying imminent events. Objective Develop a machine learning algorithm using pager messages sent between clinical team members to predict imminent clinical deterioration. Design We conducted a large observational study using long short-term memory machine learning models on the content and frequency of clinical pages. Participants We included all hospitalizations between January 1, 2018 and December 31, 2020 at Vanderbilt University Medical Center that included at least one page message to physicians. Exclusion criteria included patients receiving palliative care, hospitalizations with a planned intensive care stay, and hospitalizations in the top 2% longest length of stay. Main Measures Model classification performance to identify in-hospital cardiac arrest, transfer to intensive care, or Rapid Response activation in the next 3-, 6-, and 12-hours. We compared model performance against three common early warning scores: Modified Early Warning Score, National Early Warning Score, and the Epic Deterioration Index. Key Results There were 87,783 patients (mean [SD] age 54.0 [18.8] years; 45,835 [52.2%] women) who experienced 136,778 hospitalizations. 6214 hospitalized patients experienced a deterioration event. The machine learning model accurately identified 62% of deterioration events within 3-hours prior to the event and 47% of events within 12-hours. Across each time horizon, the model surpassed performance of the best early warning score including area under the receiver operating characteristic curve at 6-hours (0.856 vs. 0.781), sensitivity at 6-hours (0.590 vs. 0.505), specificity at 6-hours (0.900 vs. 0.878), and F-score at 6-hours (0.291 vs. 0.220). Conclusions Machine learning applied to the content and frequency of clinical pages improves prediction of imminent deterioration. Using clinical pages to monitor patient acuity supports improved detection of imminent deterioration without requiring changes to clinical workflow or nursing documentation.