Effective sparsity control in deep belief networks using normal regularization term

Nowadays the use of deep network architectures has become widespread in machine learning. Deep belief networks (DBNs) have deep network architectures to create a powerful generative model using training data. Deep belief networks can be used in classification and feature learning. A DBN can be learn...

Full description

Saved in:
Bibliographic Details
Published in:Knowledge and information systems 2017-11, Vol.53 (2), p.533-550
Main Authors: Keyvanrad, Mohammad Ali, Homayounpour, Mohammad Mehdi
Format: Article
Language:English
Subjects:
Belief networks
Classifiers
Computer Science
Data Mining and Knowledge Discovery
Database Management
Information Storage and Retrieval
Information Systems and Communication Service
Information Systems Applications (incl.Internet)
IT in Business
Machine learning
Regular Paper
Regularization
Representations
Sampling
Sparsity
Test sets
Training
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Nowadays the use of deep network architectures has become widespread in machine learning. Deep belief networks (DBNs) have deep network architectures to create a powerful generative model using training data. Deep belief networks can be used in classification and feature learning. A DBN can be learned unsupervised, and then the learned features are suitable for a simple classifier (like a linear classifier) with a few labeled data. In addition, according to researches, by using sparsity in DBNs we can learn useful low-level feature representations for unlabeled data. In sparse representation, we have the property that learned features can be interpreted, i.e., correspond to meaningful aspects of the input, and capture factors of variation in the data. Different methods are proposed to build sparse DBNs. In this paper, we proposed a new method that has different behavior according to deviation of the activation of the hidden units from a (low) fixed value. In addition, our proposed regularization term has a variance parameter that can control the force degree of sparseness. According to the results, our new method achieves the best recognition accuracy on the test sets in different datasets with different applications (image, speech and text) and we can achieve incredible results when using a different number of training samples, especially when we have a few samples for training.
ISSN:0219-1377
0219-3116
DOI:10.1007/s10115-017-1049-x