Multimedia event detection with multimodal feature fusion and temporal concept localization

We present a system for multimedia event detection. The developed system characterizes complex multimedia events based on a large array of multimodal features, and classifies unseen videos by effectively fusing diverse responses. We present three major technical innovations. First, we explore novel...

Full description

Saved in:
Bibliographic Details
Published in:Machine vision and applications 2014, Vol.25 (1), p.49-69
Main Authors: Oh, Sangmin, McCloskey, Scott, Kim, Ilseo, Vahdat, Arash, Cannons, Kevin J., Hajimirsadeghi, Hossein, Mori, Greg, Perera, A. G. Amitha, Pandey, Megha, Corso, Jason J.
Format: Article
Language:English
Subjects:
Algorithms
Arrays
Communications Engineering
Computer Science
Image Processing and Computer Vision
Localization
Machine learning
Multimedia
Networks
Pattern Recognition
Semantics
Special Issue Paper
Video data
Vision systems
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We present a system for multimedia event detection. The developed system characterizes complex multimedia events based on a large array of multimodal features, and classifies unseen videos by effectively fusing diverse responses. We present three major technical innovations. First, we explore novel visual and audio features across multiple semantic granularities, including building, often in an unsupervised manner, mid-level and high-level features upon low-level features to enable semantic understanding. Second, we show a novel Latent SVM model which learns and localizes discriminative high-level concepts in cluttered video sequences. In addition to improving detection accuracy beyond existing approaches, it enables a unique summary for every retrieval by its use of high-level concepts and temporal evidence localization. The resulting summary provides some transparency into why the system classified the video as it did. Finally, we present novel fusion learning algorithms and our methodology to improve fusion learning under limited training data condition. Thorough evaluation on a large TRECVID MED 2011 dataset showcases the benefits of the presented system.
ISSN:0932-8092
1432-1769
DOI:10.1007/s00138-013-0525-x