Extracting deep bottleneck features using stacked auto-encoders

In this work, a novel training scheme for generating bottleneck features from deep neural networks is proposed. A stack of denoising auto-encoders is first trained in a layer-wise, unsupervised manner. Afterwards, the bottleneck layer and an additional layer are added and the whole network is fine-t...

Full description

Saved in:
Bibliographic Details
Main Authors: Gehring, Jonas, Miao, Yajie, Metze, Florian, Waibel, Alex
Format: Conference Proceeding
Language:English
Subjects:
Acoustics
Auto-encoders
Bottleneck features
Deep learning
Feature extraction
Hidden Markov models
Neural networks
Speech
Training
Vectors
Citations: Items that cite this one
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this work, a novel training scheme for generating bottleneck features from deep neural networks is proposed. A stack of denoising auto-encoders is first trained in a layer-wise, unsupervised manner. Afterwards, the bottleneck layer and an additional layer are added and the whole network is fine-tuned to predict target phoneme states. We perform experiments on a Cantonese conversational telephone speech corpus and find that increasing the number of auto-encoders in the network produces more useful features, but requires pre-training, especially when little training data is available. Using more unlabeled data for pre-training only yields additional gains. Evaluations on larger datasets and on different system setups demonstrate the general applicability of our approach. In terms of word error rate, relative improvements of 9.2% (Cantonese, ML training), 9.3% (Tagalog, BMMI-SAT training), 12% (Tagalog, confusion network combinations with MFCCs), and 8.7% (Switchboard) are achieved.
ISSN:1520-6149
2379-190X
DOI:10.1109/ICASSP.2013.6638284