Towards trustworthy seizure onset detection using workflow notes

• Published in: NPJ digital medicine 2024-02, Vol.7 (1), p.42-42, Article 42
• Main Authors: Saab, Khaled, Tang, Siyi, Taha, Mohamed, Lee-Messer, Christopher, Ré, Christopher, Rubin, Daniel L.
• Format: Article
• Language: English
• Subjects: 692/617/375/178; 692/700/139/1449/1450; Artificial intelligence; Biomedicine; Biotechnology; Convulsions & seizures; Deep learning; Electroencephalography; Error analysis; Hyperventilation; Medicine; Medicine & Public Health; Patients
• ISSN: 2398-6352; 2398-6352
• DOI: 10.1038/s41746-024-01008-9

A major barrier to deploying healthcare AI is trustworthiness. One form of trustworthiness is a model’s robustness across subgroups: while models may exhibit expert-level performance on aggregate metrics, they often rely on non-causal features, leading to errors in hidden subgroups. To take a step closer towards trustworthy seizure onset detection from EEG, we propose to leverage annotations that are produced by healthcare personnel in routine clinical workflows—which we refer to as workflow notes—that include multiple event descriptions beyond seizures. Using workflow notes, we first show that by scaling training data to 68,920 EEG hours, seizure onset detection performance significantly improves by 12.3 AUROC (Area Under the Receiver Operating Characteristic) points compared to relying on smaller training sets with gold-standard labels. Second, we reveal that our binary seizure onset detection model underperforms on clinically relevant subgroups (e.g., up to a margin of 6.5 AUROC points between pediatrics and adults), while having significantly higher FPRs (False Positive Rates) on EEG clips showing non-epileptiform abnormalities (+19 FPR points). To improve model robustness to hidden subgroups, we train a multilabel model that classifies 26 attributes other than seizures (e.g., spikes and movement artifacts) and significantly improve overall performance (+5.9 AUROC points) while greatly improving performance among subgroups (up to +8.3 AUROC points) and decreasing false positives on non-epileptiform abnormalities (by 8 FPR points). Finally, we find that our multilabel model improves clinical utility (false positives per 24 EEG hours) by a factor of 2×.