Object Priors for Classifying and Localizing Unseen Actions

This work strives for the classification and localization of human actions in videos, without the need for any labeled video training examples. Where existing work relies on transferring global attribute or object information from seen to unseen action videos, we seek to classify and spatio-temporal...

Full description

Saved in:
Bibliographic Details
Published in:International journal of computer vision 2021-06, Vol.129 (6), p.1954-1971
Main Authors: Mettes, Pascal, Thong, William, Snoek, Cees G. M.
Format: Article
Language:English
Subjects:
Artificial Intelligence
Computer Imaging
Computer Science
Detectors
Image classification
Image Processing and Computer Vision
Localization
Matching
Object recognition
Pattern Recognition
Pattern Recognition and Graphics
Semantics
Tubes
Video
Vision
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This work strives for the classification and localization of human actions in videos, without the need for any labeled video training examples. Where existing work relies on transferring global attribute or object information from seen to unseen action videos, we seek to classify and spatio-temporally localize unseen actions in videos from image-based object information only. We propose three spatial object priors, which encode local person and object detectors along with their spatial relations. On top we introduce three semantic object priors, which extend semantic matching through word embeddings with three simple functions that tackle semantic ambiguity, object discrimination, and object naming. A video embedding combines the spatial and semantic object priors. It enables us to introduce a new video retrieval task that retrieves action tubes in video collections based on user-specified objects, spatial relations, and object size. Experimental evaluation on five action datasets shows the importance of spatial and semantic object priors for unseen actions. We find that persons and objects have preferred spatial relations that benefit unseen action localization, while using multiple languages and simple object filtering directly improves semantic matching, leading to state-of-the-art results for both unseen action classification and localization.
ISSN:0920-5691
1573-1405
DOI:10.1007/s11263-021-01454-y