Experiments with LDA and Top2Vec for embedded topic discovery on social media data—A case study of cystic fibrosis

• Published in: Frontiers in artificial intelligence 2022-08, Vol.5, p.948313-948313
• Main Authors: Karas, Bradley, Qu, Sue, Xu, Yanji, Zhu, Qian
• Format: Article
• Language: English
• Subjects: Artificial Intelligence; cystic fibrosis; LDA; rare disease; Reddit; Top2vec; topic modeling
• ISSN: 2624-8212; 2624-8212
• DOI: 10.3389/frai.2022.948313

Social media has become an important resource for discussing, sharing, and seeking information pertinent to rare diseases by patients and their families, given the low prevalence in the extraordinarily sparse populations. In our previous study, we identified prevalent topics from Reddit via topic modeling for cystic fibrosis (CF). While we were able to derive/access concerns/needs/questions of patients with CF, we observed challenges and issues with the traditional techniques of topic modeling, e.g., Latent Dirichlet Allocation (LDA), for fulfilling the task of topic extraction. Thus, here we present our experiments to extend the previous study with an aim of improving the performance of topic modeling, by experimenting with LDA model optimization and examination of the Top2Vec model with different embedding models. With the demonstrated results with higher coherence and qualitatively higher human readability of derived topics, we implemented the Top2Vec model with doc2vec as the embedding model as our final model to extract topics from a subreddit of CF (“r/CysticFibrosis”) and proposed to expand its use with other types of social media data for other rare diseases for better assessing patients' needs with social media data.