Clustering swap prediction for image-text pre-training

• Published in: Scientific reports 2024-05, Vol.14 (1), p.11879-16, Article 11879
• Main Authors: Fayou, Sun, Ngo, Hea Choon, Sek, Yong Wee, Meng, Zuqiang
• Format: Article
• Language: English
• Subjects: 639/705/117; 639/705/258; Cluster number; Clustering learning; Distillation; Embedding; Humanities and Social Sciences; Image processing; Learning; Model pre-training; multidisciplinary; Performance evaluation; Predictions; Science; Science (multidisciplinary); Swap prediction; Training
• ISSN: 2045-2322; 2045-2322
• DOI: 10.1038/s41598-024-60832-x

It is essential to delve into the strategy of multimodal model pre-training, which is an obvious impact on downstream tasks. Currently, clustering learning has achieved noteworthy benefits in multiple methods. However, due to the availability of open image-text pairs, it is challenging for multimodal with clustering learning. In this paper, we propose an approach that utilizes clustering swap prediction strategy to learn image-text clustering embedding space by interaction prediction between image and text features. Unlike existing models with clustering learning, our method (Clus) allows for an open number of clusters for web-scale alt-text data. Furthermore, in order to train the image and text encoders efficiently, we introduce distillation learning approach and evaluate the performance of the image-encoder in downstream visual tasks. In addition, Clus is pre-trained end-to-end by using large-scale image-text pairs. Specifically, both text and image serve as ground truth for swap prediction, enabling effective representation learning. Concurrently, extensive experiments demonstrate that Clus achieves state-of-the-art performance on multiple downstream fine-tuning and zero-shot tasks (i.e., Image-Text Retrieval, VQA, NLVR 2 , Image Captioning, Object Detection, and Semantic Segmentation).