Word sense induction in bengali using parallel corpora and distributional semantics

• Published in: Journal of intelligent & fuzzy systems 2019-01, Vol.36 (5), p.4821-4832
• Main Authors: Sengupta, Saptarshi, Pandit, Rajat, Mitra, Parag, Naskar, Sudip Kumar, Sardar, Mohini Mohan
• Format: Article
• Language: English
• Subjects: Algorithms; Bengali; Natural language processing; Parallel corpora; Semantics; Word sense disambiguation
• ISSN: 1064-1246; 1875-8967
• DOI: 10.3233/JIFS-179030

One of the most challenging research problems in natural language processing (NLP) is that of word sense induction (WSI). It involves discovering senses of a word given its contexts of usage without the use of a sense inventory which differentiates it from traditional word sense disambiguation (WSD). This paper reports a work on sense induction in Bengali, a less-resourced language, based on distributional semantics and translation based context vectors learned from parallel corpora to improve the task performance. The performance of the proposed method of sense induction was compared with the k-means algorithm, which was considered as the baseline in our work. A dataset for sense induction was created for 15 Bengali words, encompassing a total of 111 contexts. The proposed model, in both mono and cross-lingual settings, outperformed k-means in precision (P), recall (R) and F-scores. K-means based sense induction produced average P, R and F-scores of 0.71, 0.73 and 0.66 respectively. The average P, R and F-scores produced by the mono-and cross-lingual settings of the proposed algorithm are 0.77, 0.73, 0.68 and 0.81, 0.77 and 0.72 respectively.