Reliable shot identification for complex event detection via visual-semantic embedding

• Published in: Computer vision and image understanding 2021-12, Vol.213, p.103300, Article 103300
• Main Authors: Luo, Minnan, Chang, Xiaojun, Gong, Chen
• Format: Article
• Language: English
• Subjects: Complex event detection; Machine learning; Reliable shot identification; Visual-semantic guidance
• ISSN: 1077-3142; 1090-235X
• DOI: 10.1016/j.cviu.2021.103300

Multimedia event detection is the task of detecting a specific event of interest in an user-generated video on websites. The most fundamental challenge facing this task lies in the enormously varying quality of the video as well as the high-level semantic abstraction of event inherently. In this paper, we decompose the video into several segments and intuitively model the task of complex event detection as a multiple instance learning problem by representing each video as a “bag” of segments in which each segment is referred to as an instance. Instead of treating the instances equally, we associate each instance with a reliability variable to indicate its importance and then select reliable instances for training. To measure the reliability of the varying instances precisely, we propose a visual-semantic guided loss by exploiting low-level feature from visual information together with instance-event similarity based high-level semantic feature. Motivated by curriculum learning, we introduce a negative elastic-net regularization term to start training the classifier with instances of high reliability and gradually taking the instances with relatively low reliability into consideration. An alternative optimization algorithm is developed to solve the proposed challenging non-convex non-smooth problem. Experimental results on standard datasets, i.e., TRECVID MEDTest 2013 and TRECVID MEDTest 2014, demonstrate the effectiveness and superiority of the proposed method to the baseline algorithms. •A visual-semantic guided loss is proposed to measure reliability of instance for event detection.•Training begins with high-reliability instances and gradually added instances of low reliability.•Promising experimental results show the effectiveness and superiority of the proposed method.