Closed-domain event extraction for hard news event monitoring: a systematic study

• Published in: PeerJ. Computer science 2024-10, Vol.10, p.e2355, Article e2355
• Main Authors: Dukic, David, Dosilovic, Filip Karlo, Pluscec, Domagoj, Snajder, Jan
• Format: Article
• Language: English
• Subjects: Computational linguistics; Event extraction; Hard news; Language processing; Natural language interfaces; Natural language processing; News event monitoring
• ISSN: 2376-5992; 2376-5992
• DOI: 10.7717/peerj-cs.2355

News event monitoring systems allow real-time monitoring of a large number of events reported in the news, including the urgent and critical events comprising the so-called hard news. These systems heavily rely on natural language processing (NLP) to perform automatic event extraction at scale. While state-of-the-art event extraction models are readily available, integrating them into a news event monitoring system is not as straightforward as it seems due to practical issues related to model selection, robustness, and scale. To address this gap, we present a study on the practical use of event extraction models for news event monitoring. Our study focuses on the key task of closed-domain main event extraction (CDMEE), which aims to determine the type of the story's main event and extract its arguments from the text. We evaluate a range of state-of-the-art NLP models for this task, including those based on pre-trained language models. Aiming at a more realistic evaluation than done in the literature, we introduce a new dataset manually labeled with event types and their arguments. Additionally, we assess the scalability of CDMEE models and analyze the trade-off between accuracy and inference speed. Our results give insights into the performance of state-of-the-art NLP models on the CDMEE task and provide recommendations for developing effective, robust, and scalable news event monitoring systems.