Cross-modal Variational Auto-encoder with Distributed Latent Spaces and Associators

In this paper, we propose a novel structure for a cross-modal data association, which is inspired by the recent research on the associative learning structure of the brain. We formulate the cross-modal association in Bayesian inference framework realized by a deep neural network with multiple variat...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2019-05
Main Authors: Jo, Dae Ung, Lee, ByeongJu, Choi, Jongwon, Yoo, Haanju, Jin Young Choi
Format: Article
Language:English
Subjects:
Bayesian analysis
Brain
Coders
Machine learning
Modal data
Neural networks
Statistical inference
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this paper, we propose a novel structure for a cross-modal data association, which is inspired by the recent research on the associative learning structure of the brain. We formulate the cross-modal association in Bayesian inference framework realized by a deep neural network with multiple variational auto-encoders and variational associators. The variational associators transfer the latent spaces between auto-encoders that represent different modalities. The proposed structure successfully associates even heterogeneous modal data and easily incorporates the additional modality to the entire network via the proposed cross-modal associator. Furthermore, the proposed structure can be trained with only a small amount of paired data since auto-encoders can be trained by unsupervised manner. Through experiments, the effectiveness of the proposed structure is validated on various datasets including visual and auditory data.
ISSN:2331-8422