Adaptive named entity recognition based on conditional random fields with automatic updated dynamic gazetteers

• Main Authors: Xixin Wu, Zhiyong Wu, Jia Jia, Lianhong Cai
• Format: Conference Proceeding
• Language: English
• Subjects: Computational linguistics; Conditional random fields (CRFs); Dynamic gazetteers (DGs); Educational institutions; Feature extraction; Hidden Markov models; Named entity recognition (NER); Organizations; Training; Training data
• DOI: 10.1109/ISCSLP.2012.6423495

This paper presents a hybrid model which combines conditional random fields (CRFs) with dynamic gazetteers (DGs) for the task of Chinese named entity recognition (NER). In the previous work of NER, gazetteers were widely used. But their gazetteers were all static ones which cannot adapt themselves to the new domains and new out-of-vocabulary named entities (OOVNEs). In this work, we build and maintain DGs to solve the problems and propose a method to automatically update DGs along with the recognition process of the named entities (NEs). With this method, the DGs can be updated to contain more and more new NEs and features of NEs that are not found in the training data. These newly added items make the DGs become more aware of the knowledge about new domains and hence be more adaptive to new domains for the recognition of OOVNEs. Experiments on the People's Daily corpus demonstrate that our method is effective, and can improve the average F-score by 1%~2%.