Deep Multimodal Sparse Representation-Based Classification

• Main Authors: Abavisani, Mahdi, Patel, Vishal M.
• Format: Conference Proceeding
• Language: English
• Subjects: Data models; Decoding; deep multimodal sparse representation classification; Encoding; Image reconstruction; multimodal sparse representation; multimodal sparse representation classification; Sparse matrices; Sparse representation; Testing; Training
• ISSN: 2381-8549
• DOI: 10.1109/ICIP40778.2020.9191317

In this paper, we present a deep sparse representation based fusion method for classifying multimodal signals. Our proposed model consists of multimodal encoders and decoders with a shared fully-connected layer. The multimodal encoders learn separate latent space features for each modality. The latent space features are trained to be discriminative and suitable for sparse representation. The shared fully-connected layer serves as a common sparse coefficient matrix that can simultaneously reconstruct all the latent space features from different modalities. We employ discriminator heads to make the latent features discriminative. The reconstructed latent space features are then fed to the multimodal decoders to reconstruct the multimodal signals. We introduce a new classification rule by using the sparse coefficient matrix along with the predictions of the discriminator heads. Experimental results on various multimodal datasets show the effectiveness of our method.