Modeling 4D Human-Object Interactions for Joint Event Segmentation, Recognition, and Object Localization

In this paper, we present a 4D human-object interaction (4DHOI) model for solving three vision tasks jointly: i) event segmentation from a video sequence, ii) event recognition and parsing, and iii) contextual object localization. The 4DHOI model represents the geometric, temporal, and semantic rela...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on pattern analysis and machine intelligence 2017-06, Vol.39 (6), p.1165-1179
Main Authors: Wei, Ping, Zhao, Yibiao, Zheng, Nanning, Zhu, Song-Chun
Format: Article
Language:English
Subjects:
Context modeling
Dynamic programming
event recognition
Hidden Markov models
Human-object interaction
Localization
object affordance
object localization
Object recognition
Robots
Search algorithms
Segmentation
Semantics
sequence segmentation
Solid modeling
Three-dimensional displays
Video sequences
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this paper, we present a 4D human-object interaction (4DHOI) model for solving three vision tasks jointly: i) event segmentation from a video sequence, ii) event recognition and parsing, and iii) contextual object localization. The 4DHOI model represents the geometric, temporal, and semantic relations in daily events involving human-object interactions. In 3D space, the interactions of human poses and contextual objects are modeled by semantic co-occurrence and geometric compatibility. On the time axis, the interactions are represented as a sequence of atomic event transitions with coherent objects. The 4DHOI model is a hierarchical spatial-temporal graph representation which can be used for inferring scene functionality and object affordance. The graph structures and parameters are learned using an ordered expectation maximization algorithm which mines the spatial-temporal structures of events from RGB-D video samples. Given an input RGB-D video, the inference is performed by a dynamic programming beam search algorithm which simultaneously carries out event segmentation, recognition, and object localization. We collected a large multiview RGB-D event dataset which contains 3,815 video sequences and 383,036 RGB-D frames captured by three RGB-D cameras. The experimental results on three challenging datasets demonstrate the strength of the proposed method.
ISSN:0162-8828
1939-3539
2160-9292
DOI:10.1109/TPAMI.2016.2574712