Integrated Mining of Visual Features, Speech Features, and Frequent Patterns for Semantic Video Annotation

To support effective multimedia information retrieval, video annotation has become an important topic in video content analysis. Existing video annotation methods put the focus on either the analysis of low-level features or simple semantic concepts, and they cannot reduce the gap between low-level...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on multimedia 2008-02, Vol.10 (2), p.260-267
Main Authors: Tseng, V.S., Ja-Hwung Su, Jhih-Hong Huang, Chih-Jen Chen
Format: Article
Language:English
Subjects:
Annotations
Association rule
Content based retrieval
Data mining
frequent semantic patterns
Image retrieval
Information analysis
Information retrieval
Layout
Mathematical models
Mining
Multimedia
NIST
Pattern analysis
Phases
Predictive models
Semantics
sequential patterns
Speech
Speech analysis
Studies
video annotation
Visual
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:To support effective multimedia information retrieval, video annotation has become an important topic in video content analysis. Existing video annotation methods put the focus on either the analysis of low-level features or simple semantic concepts, and they cannot reduce the gap between low-level features and high-level concepts. In this paper, we propose an innovative method for semantic video annotation through integrated mining of visual features, speech features, and frequent semantic patterns existing in the video. The proposed method mainly consists of two main phases: 1) Construction of four kinds of predictive annotation models, namely speech-association, visual-association, visual-sequential, and statistical models from annotated videos. 2) Fusion of these models for annotating un-annotated videos automatically. The main advantage of the proposed method lies in that all visual features, speech features, and semantic patterns are considered simultaneously. Moreover, the utilization of high-level rules can effectively complement the insufficiency of statistics-based methods in dealing with complex and broad keyword identification in video annotation. Through empirical evaluation on NIST TRECVID video datasets, the proposed approach is shown to enhance the performance of annotation substantially in terms of precision, recall, and F-measure.
ISSN:1520-9210
1941-0077
DOI:10.1109/TMM.2007.911832