Exploring Rich Semantics for Open-set Action Recognition

Open-set action recognition (OSAR) aims to learn a recognition framework capable of both classifying known classes and identifying unknown actions in open-set scenarios. Existing OSAR methods typically reside in a data-driven paradigm, which ignore the rich semantics in both known and unknown catego...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on multimedia 2024-01, Vol.26, p.1-13
Main Authors: Hu, Yufan, Gao, Junyu, Dong, Jianfeng, Fan, Bin, Liu, Hongmin
Format: Article
Language:English
Subjects:
Activity recognition
Explicit knowledge
Knowledge graphs
Knowledge representation
Open-set action recognition
Prototypes
semantic relation modeling
Semantics
Task analysis
Training
Uncertainty
video action recognition
Visualization
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Open-set action recognition (OSAR) aims to learn a recognition framework capable of both classifying known classes and identifying unknown actions in open-set scenarios. Existing OSAR methods typically reside in a data-driven paradigm, which ignore the rich semantics in both known and unknown categories. In fact, we humans have the capability of leveraging the captured semantic information, i.e., knowledge and experience, to incisively distinguish samples from known and unknown classes. Motivated by this observation, in this paper, we propose a Unified Semantic Exploration (USE) framework for recognizing actions in openset scenarios. Specifically, we explore the explicit knowledge semantics by simulating the unknown classes with knowledge-guided virtual classes based on an external knowledge graph, which enables the model to simulate open-set perception during model training. Besides, we propose to learn the implicit data semantics by transferring the knowledge structure of action categories to the visual prototype space for semantic structure preservation. Extensive experiments on several action recognition benchmarks validate the effectiveness of our proposed method.
ISSN:1520-9210
1941-0077
DOI:10.1109/TMM.2023.3333206