ASRNN: A recurrent neural network with an attention model for sequence labeling

• Published in: Knowledge-based systems 2021-01, Vol.212, p.106548, Article 106548
• Main Authors: Lin, Jerry Chun-Wei, Shao, Yinan, Djenouri, Youcef, Yun, Unil
• Format: Article
• Language: English
• Subjects: Attention mechanism; Conditional random fields; Labeling; Labels; Natural language processing; Neural network; Neural networks; Recurrent neural networks; Representations; Semi-CRF; Sequence labeling; Structural hierarchy
• ISSN: 0950-7051; 1872-7409
• DOI: 10.1016/j.knosys.2020.106548

Natural language processing (NLP) is useful for handling text and speech, and sequence labeling plays an important role by automatically analyzing a sequence (text) to assign category labels to each part. However, the performance of these conventional models depends greatly on hand-crafted features and task-specific knowledge, which is a time consuming task. Several conditional random fields (CRF)-based models for sequence labeling have been presented, but the major limitation is how to use neural networks for extracting useful representations for each unit or segment in the input sequence. In this paper, we propose an attention segmental recurrent neural network (ASRNN) that relies on a hierarchical attention neural semi-Markov conditional random fields (semi-CRF) model for the task of sequence labeling. Our model uses a hierarchical structure to incorporate character-level and word-level information and applies an attention mechanism to both levels. This enables our method to differentiate more important information from less important information when constructing the segmental representation. We evaluated our model on three sequence labeling tasks, including named entity recognition (NER), chunking, and reference parsing. Experimental results show that the proposed model benefited from the hierarchical structure, and it achieved competitive and robust performance on all three sequence labeling tasks.