Investigations on End- to-End Audiovisual Fusion

Audiovisual speech recognition (AVSR) is a method to alleviate the adverse effect of noise in the acoustic signal. Leveraging recent developments in deep neural network-based speech recognition, we present an AVSR neural network architecture which is trained end-to-end, without the need to separatel...

Full description

Saved in:
Bibliographic Details
Main Authors: Wand, Michael, Schmidhuber, Jurgen, Vu, Ngoc Thang
Format: Conference Proceeding
Language:English
Subjects:
Audio-visual speech recognition
Deep Neural networks
Feature Fusion
Hidden Markov models
Neural networks
Neurons
Noise level
Saliency
Speech recognition
Training
Citations: Items that cite this one
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Audiovisual speech recognition (AVSR) is a method to alleviate the adverse effect of noise in the acoustic signal. Leveraging recent developments in deep neural network-based speech recognition, we present an AVSR neural network architecture which is trained end-to-end, without the need to separately model the process of decision fusion as in conventional (e.g. HMM-based) systems. The fusion system outperforms single-modality recognition under all noise conditions. Investigation of the saliency of the input features shows that the neural network automatically adapts to different noise levels in the acoustic signal.
ISSN:2379-190X
DOI:10.1109/ICASSP.2018.8461900