Modeling 4D Human-Object Interactions for Joint Event Segmentation, Recognition, and Object Localization

• Published in: IEEE transactions on pattern analysis and machine intelligence 2017-06, Vol.39 (6), p.1165-1179
• Main Authors: Wei, Ping, Zhao, Yibiao, Zheng, Nanning, Zhu, Song-Chun
• Format: Article
• Language: English
• Subjects: Context modeling; Dynamic programming; event recognition; Hidden Markov models; Human-object interaction; Localization; object affordance; object localization; Object recognition; Robots; Search algorithms; Segmentation; Semantics; sequence segmentation; Solid modeling; Three-dimensional displays; Video sequences
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2016.2574712

In this paper, we present a 4D human-object interaction (4DHOI) model for solving three vision tasks jointly: i) event segmentation from a video sequence, ii) event recognition and parsing, and iii) contextual object localization. The 4DHOI model represents the geometric, temporal, and semantic relations in daily events involving human-object interactions. In 3D space, the interactions of human poses and contextual objects are modeled by semantic co-occurrence and geometric compatibility. On the time axis, the interactions are represented as a sequence of atomic event transitions with coherent objects. The 4DHOI model is a hierarchical spatial-temporal graph representation which can be used for inferring scene functionality and object affordance. The graph structures and parameters are learned using an ordered expectation maximization algorithm which mines the spatial-temporal structures of events from RGB-D video samples. Given an input RGB-D video, the inference is performed by a dynamic programming beam search algorithm which simultaneously carries out event segmentation, recognition, and object localization. We collected a large multiview RGB-D event dataset which contains 3,815 video sequences and 383,036 RGB-D frames captured by three RGB-D cameras. The experimental results on three challenging datasets demonstrate the strength of the proposed method.