Automated Phrase Mining from Massive Text Corpora

• Published in: IEEE transactions on knowledge and data engineering 2018-10, Vol.30 (10), p.1825-1837
• Main Authors: Shang, Jingbo, Liu, Jialu, Jiang, Meng, Ren, Xiang, Voss, Clare R., Han, Jiawei
• Format: Article
• Language: English
• Subjects: Analyzers; Automatic phrase mining; Automation; distant training; Domains; Electronic publishing; Encyclopedias; Information retrieval; Internet; Knowledge base; Knowledge based systems; Labeling; Mining; multiple languages; part-of-speech tag; phrase mining; Pragmatics; State of the art; Taxonomy
• ISSN: 1041-4347; 1558-2191
• DOI: 10.1109/TKDE.2018.2812203

As one of the fundamental tasks in text analysis, phrase mining aims at extracting quality phrases from a text corpus and has various downstream applications including information extraction/retrieval, taxonomy construction, and topic modeling. Most existing methods rely on complex, trained linguistic analyzers, and thus likely have unsatisfactory performance on text corpora of new domains and genres without extra but expensive adaption. None of the state-of-the-art models, even data-driven models, is fully automated because they require human experts for designing rules or labeling phrases. In this paper, we propose a novel framework for automated phrase mining, AutoPhrase, which supports any language as long as a general knowledge base (e.g., Wikipedia) in that language is available, while benefiting from, but not requiring, a POS tagger. Compared to the state-of-the-art methods, AutoPhrase has shown significant improvements in both effectiveness and efficiency on five real-world datasets across different domains and languages. Besides, AutoPhrase can be extended to model single-word quality phrases.