Bi-firing deep neural networks

Deep neural networks provide more expressive power in comparison to shallow ones. However, current activation functions can not propagate error using gradient descent efficiently with the increment of the number of hidden layers. Current activation functions, e.g. sigmoid, have large saturation regi...

Full description

Saved in:
Bibliographic Details
Published in:International journal of machine learning and cybernetics 2014-02, Vol.5 (1), p.73-83
Main Authors: Li, Jin-Cheng, Ng, Wing W. Y., Yeung, Daniel S., Chan, Patrick P. K.
Format: Article
Language:English
Subjects:
Algorithms
Artificial Intelligence
Artificial neural networks
Back propagation
Complex Systems
Computational Intelligence
Control
Engineering
Mechatronics
Methods
Neural networks
Original Article
Pattern Recognition
Propagation
Robotics
Systems Biology
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Deep neural networks provide more expressive power in comparison to shallow ones. However, current activation functions can not propagate error using gradient descent efficiently with the increment of the number of hidden layers. Current activation functions, e.g. sigmoid, have large saturation regions which are insensitive to changes of hidden neuron’s input and yield gradient diffusion. To relief these problems, we propose a bi-firing activation function in this work. The bi-firing function is a differentiable function with a very small saturation region. Experimental results show that deep neural networks with the proposed activation functions yield faster training, better error propagation and better testing accuracies on seven image datasets.
ISSN:1868-8071
1868-808X
DOI:10.1007/s13042-013-0198-9