Cognition Guided Human-Object Relationship Detection

Human-object relationship detection reveals the fine-grained relationship between humans and objects, helping the comprehensive understanding of videos. Previous human-object relationship detection approaches are mainly developed with object features and relation features without exploring the speci...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on image processing 2023-01, Vol.32, p.1-1
Main Authors: Zeng, Zhitao, Dai, Pengwen, Zhang, Xuan, Zhang, Lei, Cao, Xiaochun
Format: Article
Language:English
Subjects:
Benchmarking
body pose
Coders
Cognition
eye-head-body movements
Head movement
Head Movements
head pose
Human-object relationship
Humans
Object Attachment
Skeleton
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Human-object relationship detection reveals the fine-grained relationship between humans and objects, helping the comprehensive understanding of videos. Previous human-object relationship detection approaches are mainly developed with object features and relation features without exploring the specific information of humans. In this paper, we propose a novel Relation-Pose Transformer (RPT) for human-object relationship detection. Inspired by the coordination of eye-head-body movements in cognitive science, we employ the head pose to find those crucial objects that humans focus on and use the body pose with skeleton information to represent multiple actions. Then, we utilize the spatial encoder to capture spatial contextualized information of the relation pair, which integrates the relation features and pose features. Next, the temporal decoder aims to model the temporal dependency of the relationship. Finally, we adopt multiple classifiers to predict different types of relationships. Extensive experiments on the benchmark Action Genome validate the effectiveness of our proposed method and show the state-of-the-art performance compared with related methods.
ISSN:1057-7149
1941-0042
DOI:10.1109/TIP.2023.3270040