Pseudo-labeling with keyword refining for few-supervised video captioning

• Published in: Pattern recognition 2025-03, Vol.159, p.111176, Article 111176
• Main Authors: Li, Ping, Wang, Tao, Zhao, Xinkui, Xu, Xianghua, Song, Mingli
• Format: Article
• Language: English
• Subjects: Few supervision; Gated fusion; Keyword refiner; Pseudo-labeling; Video captioning
• ISSN: 0031-3203
• DOI: 10.1016/j.patcog.2024.111176

Video captioning generate a sentence that describes the video content. Existing methods always require a number of captions (e.g., 10 or 20) per video to train the model, which is quite costly. In this work, we explore the possibility of using only one or very few ground-truth sentences, and introduce a new task named few-supervised video captioning. Specifically, we propose a few-supervised video captioning framework that consists of lexically constrained pseudo-labeling module and keyword-refined captioning module. Unlike the random sampling in natural language processing that may cause invalid modifications (i.e., edit words), the former module guides the model to edit words using some actions (e.g., copy, replace, insert, and delete) by a pretrained token-level classifier, and then fine-tunes candidate sentences by a pretrained language model. Meanwhile, the former employs the repetition penalized sampling to encourage the model to yield concise pseudo-labeled sentences with less repetition, and selects the most relevant sentences upon a pretrained video-text model. Moreover, to keep semantic consistency between pseudo-labeled sentences and video content, we develop the transformer-based keyword refiner with the video-keyword gated fusion strategy to emphasize more on relevant words. Extensive experiments on several benchmarks demonstrate the advantages of the proposed approach in both few-supervised and fully-supervised scenarios. •A new task named few-supervised video captioning that uses only one human-sentence is introduced.•A pseudo labeling strategy with lexical constraint is proposed to augment knowledge.•A keyword-refined captioning module with video-text gated fusion is designed generating high-quality sentences by modeling global context.•Empirical studies demonstrate the satisfying quality of the generated captions by proposed method.