Machine learning natural language processing for identifying venous thromboembolism: systematic review and meta-analysis

• Published in: Blood advances 2024-06, Vol.8 (12), p.2991-3000
• Main Authors: Lam, Barbara D., Chrysafi, Pavlina, Chiasakul, Thita, Khosla, Harshit, Karagkouni, Dimitra, McNichol, Megan, Adamski, Alys, Reyes, Nimia, Abe, Karon, Mantha, Simon, Vlachos, Ioannis S., Zwicker, Jeffrey I., Patell, Rushad
• Format: Article
• Language: English
• Subjects: Electronic Health Records; Humans; Machine Learning; Natural Language Processing; Thrombosis and Hemostasis; Venous Thromboembolism - diagnosis
• ISSN: 2473-9529; 2473-9537; 2473-9537
• DOI: 10.1182/bloodadvances.2023012200

•ML-NLP can be an effective method for identifying venous thromboembolism in free-text reports.•The highest performing models use vectorization rather than bag-of-words and deep-learning techniques, like convolutional neural networks. [Display omitted] Venous thromboembolism (VTE) is a leading cause of preventable in-hospital mortality. Monitoring VTE cases is limited by the challenges of manual medical record review and diagnosis code interpretation. Natural language processing (NLP) can automate the process. Rule-based NLP methods are effective but time consuming. Machine learning (ML)-NLP methods present a promising solution. We conducted a systematic review and meta-analysis of studies published before May 2023 that use ML-NLP to identify VTE diagnoses in the electronic health records. Four reviewers screened all manuscripts, excluding studies that only used a rule-based method. A meta-analysis evaluated the pooled performance of each study’s best performing model that evaluated for pulmonary embolism and/or deep vein thrombosis. Pooled sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) with confidence interval (CI) were calculated by DerSimonian and Laird method using a random-effects model. Study quality was assessed using an adapted TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) tool. Thirteen studies were included in the systematic review and 8 had data available for meta-analysis. Pooled sensitivity was 0.931 (95% CI, 0.881-0.962), specificity 0.984 (95% CI, 0.967-0.992), PPV 0.910 (95% CI, 0.865-0.941) and NPV 0.985 (95% CI, 0.977-0.990). All studies met at least 13 of the 21 NLP-modified TRIPOD items, demonstrating fair quality. The highest performing models used vectorization rather than bag-of-words and deep-learning techniques such as convolutional neural networks. There was significant heterogeneity in the studies, and only 4 validated their model on an external data set. Further standardization of ML studies can help progress this novel technology toward real-world implementation.