Multi-modal embeddings using multi-task learning for emotion recognition

General embeddings like word2vec, GloVe and ELMo have shown a lot of success in natural language tasks. The embeddings are typically extracted from models that are built on general tasks such as skip-gram models and natural language generation. In this paper, we extend the work from natural language...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2020-09
Main Authors: Khare, Aparna, Parthasarathy, Srinivas, Sundaram, Shiva
Format: Article
Language:English
Subjects:
Audio data
Automatic speech recognition
Coders
Cognitive tasks
Emotion recognition
Emotions
Machine learning
Natural language
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:General embeddings like word2vec, GloVe and ELMo have shown a lot of success in natural language tasks. The embeddings are typically extracted from models that are built on general tasks such as skip-gram models and natural language generation. In this paper, we extend the work from natural language understanding to multi-modal architectures that use audio, visual and textual information for machine learning tasks. The embeddings in our network are extracted using the encoder of a transformer model trained using multi-task training. We use person identification and automatic speech recognition as the tasks in our embedding generation framework. We tune and evaluate the embeddings on the downstream task of emotion recognition and demonstrate that on the CMU-MOSEI dataset, the embeddings can be used to improve over previous state of the art results.
ISSN:2331-8422
DOI:10.48550/arxiv.2009.05019