Classifying Firearm Injury Intent in Electronic Hospital Records Using Natural Language Processing

• Published in: JAMA network open 2023-04, Vol.6 (4), p.e235870-e235870
• Main Authors: MacPhaul, Erin, Zhou, Li, Mooney, Stephen J, Azrael, Deborah, Bowen, Andrew, Rowhani-Rahbar, Ali, Yenduri, Ravali, Barber, Catherine, Goralnick, Eric, Miller, Matthew
• Format: Article
• Language: English
• Subjects: Accuracy; Adult; Classification; Cross-Sectional Studies; Electronic Health Records; Female; Firearms; Hospital Records; Humans; Injuries; Male; Medical records; Natural Language Processing; Online Only; Original Investigation; Public Health; Retrospective Studies; Wounds, Gunshot - epidemiology
• ISSN: 2574-3805; 2574-3805
• DOI: 10.1001/jamanetworkopen.2023.5870

International Classification of Diseases-coded hospital discharge data do not accurately reflect whether firearm injuries were caused by assault, unintentional injury, self-harm, legal intervention, or were of undetermined intent. Applying natural language processing (NLP) and machine learning (ML) techniques to electronic health record (EHR) narrative text could be associated with improved accuracy of firearm injury intent data. To assess the accuracy with which an ML model identified firearm injury intent. A cross-sectional retrospective EHR review was conducted at 3 level I trauma centers, 2 from health care institutions in Boston, Massachusetts, and 1 from Seattle, Washington, between January 1, 2000, and December 31, 2019; data analysis was performed from January 18, 2021, to August 22, 2022. A total of 1915 incident cases of firearm injury in patients presenting to emergency departments at the model development institution and 769 from the external validation institution with a firearm injury code assigned according to International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) or International Statistical Classification of Diseases and Related Health Problems, 10th Revision, Clinical Modification (ICD-10-CM), in discharge data were included. Classification of firearm injury intent. Intent classification accuracy by the NLP model was compared with ICD codes assigned by medical record coders in discharge data. The NLP model extracted intent-relevant features from narrative text that were then used by a gradient-boosting classifier to determine the intent of each firearm injury. Classification accuracy was evaluated against intent assigned by the research team. The model was further validated using an external data set. The NLP model was evaluated in 381 patients presenting with firearm injury at the model development site (mean [SD] age, 39.2 [13.0] years; 348 [91.3%] men) and 304 patients at the external development site (mean [SD] age, 31.8 [14.8] years; 263 [86.5%] men). The model proved more accurate than medical record coders in assigning intent to firearm injuries at the model development site (accident F-score, 0.78 vs 0.40; assault F-score, 0.90 vs 0.78). The model maintained this improvement on an external validation set from a second institution (accident F-score, 0.64 vs 0.58; assault F-score, 0.88 vs 0.81). While the model showed some degradation between institutions, retraining the model using data from the