Multi-feature and Multi-branch Action Segmentation Framework for Modeling Long-Short-Term Dependencies

Pioneer efforts have been dedicated to action segmentation that predicts what step is occurring in a video frame. Existing studies focus on improving the accuracy of video segmentation, but neglect the temporal continuity of intersegments and semantic consistency of intra-segments, which are necessa...

Full description

Saved in:
Bibliographic Details
Main Authors: Hong, Junkun, Long, Yitian, Luo, Yueyi, Qi, Qianqian, Long, Jun
Format: Conference Proceeding
Language:English
Subjects:
Accuracy
action segmentation
Aggregates
attention mechanism
Computational modeling
contrast learning
Data mining
Feature extraction
Measurement
Semantics
temporal convolutional network
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Pioneer efforts have been dedicated to action segmentation that predicts what step is occurring in a video frame. Existing studies focus on improving the accuracy of video segmentation, but neglect the temporal continuity of intersegments and semantic consistency of intra-segments, which are necessary for developing computer-assisted systems. Meanwhile, Temporal Convolutional Networks have shown good performance in action segmentation tasks, but their high layers tend to lose fine-grained information and impact the results. Toward this end, we devise a multi-feature and multi-branch action segmentation framework for modeling long-term and short-term dependencies. Specifically, we present a multi-feature fusion to enhance temporal video representation and design a multi-branch predictor for extracting both segment-level and frame-level information. We justify our framework over three datasets and experimental results demonstrate its superiority, especially in Edit and F1 metrics, which means our framework is more applicable to computer-assisted systems.
ISSN:1945-788X
DOI:10.1109/ICME57554.2024.10688242