Application of Deep Belief Networks for Natural Language Understanding

• Published in: IEEE/ACM transactions on audio, speech, and language processing speech, and language processing, 2014-04, Vol.22 (4), p.778-784
• Main Authors: Sarikaya, Ruhi, Hinton, Geoffrey E., Deoras, Anoop
• Format: Article
• Language: English
• Subjects: Boosting; Call-Routing; Classification; DBN; Deep Learning; Deep Neural Nets; Hidden Markov models; Natural language Understanding; Neural networks; RBM; Speech; Speech processing; Support vector machines; Training; Vectors
• ISSN: 2329-9290; 2329-9304
• DOI: 10.1109/TASLP.2014.2303296

Applications of Deep Belief Nets (DBN) to various problems have been the subject of a number of recent studies ranging from image classification and speech recognition to audio classification. In this study we apply DBNs to a natural language understanding problem. The recent surge of activity in this area was largely spurred by the development of a greedy layer-wise pretraining method that uses an efficient learning algorithm called Contrastive Divergence (CD). CD allows DBNs to learn a multi-layer generative model from unlabeled data and the features discovered by this model are then used to initialize a feed-forward neural network which is fine-tuned with backpropagation. We compare a DBN-initialized neural network to three widely used text classification algorithms: Support Vector Machines (SVM), boosting and Maximum Entropy (MaxEnt). The plain DBN-based model gives a call-routing classification accuracy that is equal to the best of the other models. However, using additional unlabeled data for DBN pre-training and combining DBN-based learned features with the original features provides significant gains over SVMs, which, in turn, performed better than both MaxEnt and Boosting.