Multi-Stage Image-Language Cross-Generative Fusion Network for Video-Based Referring Expression Comprehension

• Published in: IEEE transactions on image processing 2024, Vol.33, p.3256-3270
• Main Authors: Zhang, Yujia, Li, Qianzhong, Pan, Yi, Zhao, Xiaoguang, Tan, Min
• Format: Article
• Language: English
• Subjects: Algorithms; Annotations; consistency loss; Feature extraction; Frames (data processing); Grounding; Humans; Image Processing, Computer-Assisted - methods; image-language cross-generative fusion; Language; Linguistics; Location awareness; Modules; multi-stage learning; Object recognition; Representation learning; Similarity; Task analysis; Tracking problem; Video Recording - methods; Video-based referring expression comprehension; Visualization
• ISSN: 1057-7149; 1941-0042
• DOI: 10.1109/TIP.2024.3394260

Video-based referring expression comprehension is a challenging task that requires locating the referred object in each video frame of a given video. While many existing approaches treat this task as an object-tracking problem, their performance is heavily reliant on the quality of the tracking templates. Furthermore, when there is not enough annotation data to assist in template selection, the tracking may fail. Other approaches are based on object detection, but they often use only one adjacent frame of the key frame for feature learning, which limits their ability to establish the relationship between different frames. In addition, improving the fusion of features from multiple frames and referring expressions to effectively locate the referents remains an open problem. To address these issues, we propose a novel approach called the Multi-Stage Image-Language Cross-Generative Fusion Network (MILCGF-Net), which is based on one-stage object detection. Our approach includes a Frame Dense Feature Aggregation module for dense feature learning of adjacent time sequences. Additionally, we propose an Image-Language Cross-Generative Fusion module as the main body of multi-stage learning to generate cross-modal features by calculating the similarity between video and expression, and then refining and fusing the generated features. To further enhance the cross-modal feature generation capability of our model, we introduce a consistency loss that constrains the image-language similarity and language-image similarity matrices during feature generation. We evaluate our proposed approach on three public datasets and demonstrate its effectiveness through comprehensive experimental results.