MISNIS: An intelligent platform for twitter topic mining

• Published in: Expert systems with applications 2017-12, Vol.89, p.374-388
• Main Authors: Carvalho, Joao P., Rosa, Hugo, Brogueira, Gaspar, Batista, Fernando
• Format: Article
• Language: English
• Subjects: Data mining; Fuzzy fingerprints; Influence; Information dissemination; Intelligent topic mining; Sentiment analysis; Social networks; Society; Text analytics; Twitter
• ISSN: 0957-4174; 1873-6793
• DOI: 10.1016/j.eswa.2017.08.001

•An intelligent platform to efficiently collect and manage large Twitter corpora.•Circumvents Twitter restrictions that limit free access to 1% of all flowing tweets.•An add-on implementing intelligent methods for Twitter topic mining.•Intelligent retrieval of tweets related to a given topic.•A case study is presented as a demonstration example. Twitter has become a major tool for spreading news, for dissemination of positions and ideas, and for the commenting and analysis of current world events. However, with more than 500 million tweets flowing per day, it is necessary to find efficient ways of collecting, storing, managing, mining and visualizing all this information. This is especially relevant if one considers that Twitter has no ways of indexing tweet contents, and that the only available categorization “mechanism” is the #hashtag, which is totally dependent of a user's will to use it. This paper presents an intelligent platform and framework, named MISNIS - Intelligent Mining of Public Social Networks’ Influence in Society - that facilitates these issues and allows a non-technical user to easily mine a given topic from a very large tweet's corpus and obtain relevant contents and indicators such as user influence or sentiment analysis. When compared to other existent similar platforms, MISNIS is an expert system that includes specifically developed intelligent techniques that: (1) Circumvent the Twitter API restrictions that limit access to 1% of all flowing tweets. The platform has been able to collect more than 80% of all flowing portuguese language tweets in Portugal when online; (2) Intelligently retrieve most tweets related to a given topic even when the tweets do not contain the topic #hashtag or user indicated keywords. A 40% increase in the number of retrieved relevant tweets has been reported in real world case studies. The platform is currently focused on Portuguese language tweets posted in Portugal. However, most developed technologies are language independent (e.g. intelligent retrieval, sentiment analysis, etc.), and technically MISNIS can be easily expanded to cover other languages and locations.