Understand Short Texts by Harvesting and Analyzing Semantic Knowledge

• Published in: IEEE transactions on knowledge and data engineering 2017-03, Vol.29 (3), p.499-512
• Main Authors: Hua, Wen, Wang, Zhongyuan, Wang, Haixun, Zheng, Kai, Zhou, Xiaofang
• Format: Article
• Language: English
• Subjects: Coherence; concept labeling; Context; Hidden Markov models; Labeling; semantic knowledge; Semantics; Short text understanding; Tagging; text segmentation; type detection; Vocabulary
• ISSN: 1041-4347; 1558-2191
• DOI: 10.1109/TKDE.2016.2571687

Understanding short texts is crucial to many applications, but challenges abound. First, short texts do not always observe the syntax of a written language. As a result, traditional natural language processing tools, ranging from part-of-speech tagging to dependency parsing, cannot be easily applied. Second, short texts usually do not contain sufficient statistical signals to support many state-of-the-art approaches for text mining such as topic modeling. Third, short texts are more ambiguous and noisy, and are generated in an enormous volume, which further increases the difficulty to handle them. We argue that semantic knowledge is required in order to better understand short texts. In this work, we build a prototype system for short text understanding which exploits semantic knowledge provided by a well-known knowledgebase and automatically harvested from a web corpus. Our knowledge-intensive approaches disrupt traditional methods for tasks such as text segmentation, part-of-speech tagging, and concept labeling, in the sense that we focus on semantics in all these tasks. We conduct a comprehensive performance evaluation on real-life data. The results show that semantic knowledge is indispensable for short text understanding, and our knowledge-intensive approaches are both effective and efficient in discovering semantics of short texts.