Deep Learning based Semantic Similarity Detection using Text Data

• Published in: Information technology and control 2020-01, Vol.49 (4), p.495-510
• Main Authors: Mansoor, Muhammad, Rehman, Zahoor ur, Shaheen, Muhammad, Khan, Muhammad Attique, Habib, Mohamed
• Format: Article
• Language: English
• ISSN: 1392-124X; 2335-884X
• DOI: 10.5755/j01.itc.49.4.27118

Similarity detection in the text is the main task for a number of Natural Language Processing (NLP) applications. As textual data is comparatively large in quantity and huge in volume than the numeric data, therefore measuring textual similarity is one of the important problems. Most of the similarity detection algorithms are based upon word to word matching, sentence/paragraph matching, and matching of the whole document. In this research, a novel approach is proposed using deep learning models, combining Long Short Term Memory network (LSTM) with Convolutional Neural Network (CNN) for measuring semantics similarity between two questions. The proposed model takes sentence pairs as input to measure the similarity between them. The model is tested on publicly available Quora’s dataset. The model in comparison to the existing techniques gave 87.50 % accuracy which is better than the previous approaches.