Automatic Identification of Narratives: Evaluation framework, annotation methodology and dataset creation

One of the fundamental components of understanding online discourse in social networks is the identification of narratives. For example, the analysis of disinformation campaigns requires some inference about their communication goals that, in turn, requires the identification of the narratives that...

Full description

Saved in:
Bibliographic Details
Published in:IEEE access 2024-10, p.1-1
Main Authors: Fraile-Hernandez, Jesus M., Penas, Anselmo, Moral, Pablo
Format: Article
Language:English
Subjects:
Annotations
Blogs
Costs
Datasets
Evaluation Methodology
Labeling
Measurement
Media
Narrative Identification
Natural language processing
Object recognition
Performance evaluation
Social Media Analysis
Social networking (online)
System performance
Upper bound
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:One of the fundamental components of understanding online discourse in social networks is the identification of narratives. For example, the analysis of disinformation campaigns requires some inference about their communication goals that, in turn, requires the identification of the narratives that they promote. The research in this task involves a number of challenges such as the limited availability of labelled datasets, the subjectivity of the annotators and the time cost of annotation. This article present a definition of the Narrative Identification task, proposes an evaluation framework for Narrative Identification, and a methodology for the creation and annotation of Narrative Identification datasets taking into account the subjectivity of the task. Keeping in mind the goal of comparing systems performance, we explore how to reduce the annotation time while maintaining the reliability of the evaluation. Following this methodology, a set of eight tasks for narrative identification in the political domain has been developed in Spanish and English. Finally, we validated the evaluation framework by analysing its application to DIPROMATS 2024 shared task, together with the performance analysis of baseline and participant systems.
ISSN:2169-3536
2169-3536
DOI:10.1109/ACCESS.2024.3475579