Adaptation-Oriented Feature Projection for One-Shot Action Recognition

• Published in: IEEE transactions on multimedia 2020-12, Vol.22 (12), p.3166-3179
• Main Authors: Zou, Yixiong, Shi, Yemin, Shi, Daochen, Wang, Yaowei, Liang, Yongsheng, Tian, Yonghong
• Format: Article
• Language: English
• Subjects: Adaptation; Adaptation models; adaptation-oriented feature projection; AOF; Computational modeling; Data integration; Datasets; fast adaptation; Feature recognition; Image recognition; Learning; Object recognition; One-shot action recognition; Projection; Task analysis; Training; Training data
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2020.2972128

One-shot action recognition aims at recognizing actions in unseen classes in cases where only one training video is provided. Compared with one-shot image recognition, one-shot learning on videos is more difficult due to the fact that the temporal dimension of video may lead to greater variation. To handle this variation, it is important to conduct further adaptation in the one-shot training process, despite the scarcity of the training data. While meta-learning is an option for facilitating this adaptation, it cannot be directly applied for two reasons: first, deep networks for action recognition can make current meta-learning methods infeasible to run because of their high computational complexity; second, due to the greater variation in actions, the adapted performance may not be higher than the un-adapted one, making it difficult to train the model by means of meta-learning. To address these problems and facilitate the adaptation, we propose the Adaptation-Oriented Feature (AOF) projection for one-shot action recognition. We first pre-train the base network on seen classes. The output of the network is projected to the adaptation-oriented feature space by fusing the important feature dimensions that are sensitive to adaptation. Subsequently, a small dataset (a.k.a. task) is sampled from seen classes to simulate the unseen-class training and testing settings. The feature adaptation is performed on the training data of this task to integrate the distribution information of the adapted feature. In order to reduce over-fitting, the triplet loss is applied to handle temporal variation with fewer parameters during the adaptation. On the testing data of this task, the losses on both adapted and un-adapted features are calculated to train the projection matrix. This sampling-adaptation-training procedure is then repeated on seen classes until convergence. Extensive experimental results on two challenging one-shot action recognition datasets demonstrate that our proposed method outperforms state-of-the-art methods.