Video2vec Embeddings Recognize Events When Examples Are Scarce

• Published in: IEEE transactions on pattern analysis and machine intelligence 2017-10, Vol.39 (10), p.2089-2103
• Main Authors: Habibian, Amirhossein, Mensink, Thomas, Snoek, Cees G. M.
• Format: Article
• Language: English
• Subjects: Annotations; Correlation; Embedding; Event recognition; Feature extraction; Multimedia; NIST; Optimization; Recognition; representation learning; Representations; semantic video representation; Semantics; State of the art; Training; Vehicles; Visualization; Webs
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2016.2627563

This paper aims for event recognition when video examples are scarce or even completely absent. The key in such a challenging setting is a semantic video representation. Rather than building the representation from individual attribute detectors and their annotations, we propose to learn the entire representation from freely available web videos and their descriptions using an embedding between video features and term vectors. In our proposed embedding, which we call Video2vec, the correlations between the words are utilized to learn a more effective representation by optimizing a joint objective balancing descriptiveness and predictability. We show how learning the Video2vec embedding using a multimodal predictability loss, including appearance, motion and audio features, results in a better predictable representation. We also propose an event specific variant of Video2vec to learn a more accurate representation for the words, which are indicative of the event, by introducing a term sensitive descriptiveness loss. Our experiments on three challenging collections of web videos from the NIST TRECVID Multimedia Event Detection and Columbia Consumer Videos datasets demonstrate: i) the advantages of Video2vec over representations using attributes or alternative embeddings, ii) the benefit of fusing video modalities by an embedding over common strategies, iii) the complementarity of term sensitive descriptiveness and multimodal predictability for event recognition. By its ability to improve predictability of present day audiovisual video features, while at the same time maximizing their semantic descriptiveness, Video2vec leads to state-of-the-art accuracy for both fewand zero-example recognition of events in video.