Evaluation of clustering and topic modeling methods over health-related tweets and emails

• Published in: Artificial intelligence in medicine 2021-07, Vol.117, p.102096-102096, Article 102096
• Main Authors: Lossio-Ventura, Juan Antonio, Gonzales, Sergio, Morzan, Juandiego, Alatrista-Salas, Hugo, Hernandez-Boussard, Tina, Bian, Jiang
• Format: Article
• Language: English
• Subjects: Cluster Analysis; Clustering; Communication; Electronic Mail; External cluster indices; Humans; Internal cluster indices; Machine Learning; Natural language processing; Social Media; Topic modeling
• ISSN: 0933-3657; 1873-2860
• DOI: 10.1016/j.artmed.2021.102096

•Evaluation of topic modeling and clustering on health-related tweets and emails.•Topic modeling: LSI, LDA, BTM, GibbsLDA, Online LDA, Online Twitter LDA, and GSDMM.•Clustering: k -means with two feature representations (TF-IDF and Doc2Vec).•The evaluation is based on two internal and five external cluster validity indices. Internet provides different tools for communicating with patients, such as social media (e.g., Twitter) and email platforms. These platforms provided new data sources to shed lights on patient experiences with health care and improve our understanding of patient-provider communication. Several existing topic modeling and document clustering methods have been adapted to analyze these new free-text data automatically. However, both tweets and emails are often composed of short texts; and existing topic modeling and clustering approaches have suboptimal performance on these short texts. Moreover, research over health-related short texts using these methods has become difficult to reproduce and benchmark, partially due to the absence of a detailed comparison of state-of-the-art topic modeling and clustering methods on these short texts. We trained eight state-of- the-art topic modeling and clustering algorithms on short texts from two health-related datasets (tweets and emails): Latent Semantic Indexing (LSI), Latent Dirichlet Allocation (LDA), LDA with Gibbs Sampling (GibbsLDA), Online LDA, Biterm Model (BTM), Online Twitter LDA, and Gibbs Sampling for Dirichlet Multinomial Mixture (GSDMM), as well as the k-means clustering algorithm with two different feature representations: TF-IDF and Doc2Vec. We used cluster validity indices to evaluate the performance of topic modeling and clustering: two internal indices (i.e. assessing the goodness of a clustering structure without external information) and five external indices (i.e. comparing the results of a cluster analysis to an externally known provided class labels). In overall, for number of clusters (k) from 2 to 50, Online Twitter LDA and GSDMM achieved the best performance in terms of internal indices, while LSI and k-means with TF-IDF had the highest external indices. Also, of all tweets (N = 286, 971; HPV represents 94.6% of tweets and lynch syndrome represents 5.4%), for k = 2, most of the methods could respect this initial clustering distribution. However, we found model performance varies with the source of data and hyper-parameters such as the number of topics and the number of iter