A Fast Learning Algorithm for Deep Belief Nets

• Published in: Neural computation 2006-07, Vol.18 (7), p.1527-1554
• Main Authors: Hinton, Geoffrey E, Osindero, Simon, Teh, Yee-Whye
• Format: Article
• Language: English
• Subjects: Algorithmics. Computability. Computer arithmetics; Algorithms; Animals; Applied sciences; Artificial intelligence; Biological and medical sciences; Computer science; control theory; systems; Exact sciences and technology; Fundamental and applied biological sciences. Psychology; General aspects; Humans; Information systems; Learning - physiology; Learning and adaptive systems; Letters; Mathematics in biology. Statistical analysis. Models. Metrology. Data processing in biology (general aspects); Memory; Neural Networks (Computer); Neurology; Neurons - physiology; Theoretical computing; Theory
• ISSN: 0899-7667; 1530-888X
• DOI: 10.1162/neco.2006.18.7.1527

We show how to use “complementary priors” to eliminate the explaining-away effects that make inference difficult in densely connected belief nets that have many hidden layers. Using complementary priors, we derive a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory. The fast, greedy algorithm is used to initialize a slower learning procedure that fine-tunes the weights using a contrastive version of the wake-sleep algorithm. After fine-tuning, a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels. This generative model gives better digit classification than the best discriminative learning algorithms. The low-dimensional manifolds on which the digits lie are modeled by long ravines in the free-energy landscape of the top-level associative memory, and it is easy to explore these ravines by using the directed connections to display what the associative memory has in mind.