Audiovisual Synchronization and Fusion Using Canonical Correlation Analysis

It is well-known that early integration (also called data fusion) is effective when the modalities are correlated, and late integration (also called decision or opinion fusion) is optimal when modalities are uncorrelated. In this paper, we propose a new multimodal fusion strategy for open-set speake...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on multimedia 2007-11, Vol.9 (7), p.1396-1403
Main Authors: Sargin, M.E., Yemez, Y., Erzin, E., Tekalp, A.M.
Format: Article
Language:English
Subjects:
Applied sciences
Audiovisual synchronization
Computer science
control theory
systems
Computer systems and distributed systems. User interface
correlation
Correlation analysis
Discrete cosine transforms
Error analysis
Exact sciences and technology
Independent component analysis
Linear discriminant analysis
Mel frequency cepstral coefficient
Multimedia
multimodal fusion
Mutual information
Shape
Software
Speaker recognition
Speech
Speech analysis
Speech recognition
Strategy
Studies
Surface layer
Synchronism
Synchronization
Texture
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:It is well-known that early integration (also called data fusion) is effective when the modalities are correlated, and late integration (also called decision or opinion fusion) is optimal when modalities are uncorrelated. In this paper, we propose a new multimodal fusion strategy for open-set speaker identification using a combination of early and late integration following canonical correlation analysis (CCA) of speech and lip texture features. We also propose a method for high precision synchronization of the speech and lip features using CCA prior to the proposed fusion. Experimental results show that i) the proposed fusion strategy yields the best equal error rates (EER), which are used to quantify the performance of the fusion strategy for open-set speaker identification, and ii) precise synchronization prior to fusion improves the EER; hence, the best EER is obtained when the proposed synchronization scheme is employed together with the proposed fusion strategy. We note that the proposed fusion strategy outperforms others because the features used in the late integration are truly uncorrelated, since they are output of the CCA analysis.
ISSN:1520-9210
1941-0077
DOI:10.1109/TMM.2007.906583