Multi-modal graph contrastive encoding for neural machine translation

• Published in: Artificial intelligence 2023-10, Vol.323, p.103986, Article 103986
• Main Authors: Yin, Yongjing, Zeng, Jiali, Su, Jinsong, Zhou, Chulun, Meng, Fandong, Zhou, Jie, Huang, Degen, Luo, Jiebo
• Format: Article
• Language: English
• Subjects: Contrastive learning; Graph neural networks; Multi-modal neural machine translation
• ISSN: 0004-3702; 1872-7921
• DOI: 10.1016/j.artint.2023.103986

As an important extension of conventional text-only neural machine translation (NMT), multi-modal neural machine translation (MNMT) aims to translate input source sentences paired with images into the target language. Although a lot of MNMT models have been proposed to perform multi-modal semantic fusion, they do not consider fine-grained semantic correspondences between semantic units of different modalities (i.e., words and visual objects), which can be exploited to refine multi-modal representation learning via fine-grained semantic interactions. To address this issue, we propose a graph-based multi-modal fusion encoder for NMT. Concretely, we first employ a unified multi-modal graph to represent the input sentence and image, in which the multi-modal semantic units are considered as the nodes in the graph, connected by two kinds of edges with different semantic relationships. Then, we stack multiple graph-based multi-modal fusion layers that iteratively conduct intra- and inter-modal interactions to learn node representations. Finally, via an attention mechanism, we induce a multi-modal context from the top node representations for the decoder. Particularly, we introduce a progressive contrastive learning strategy based on the multi-modal graph to refine the training of our proposed model, where hard negative samples are introduced gradually. To evaluate our model, we conduct experiments on commonly-used datasets. Experimental results and analysis show that our MNMT model obtains significant improvements over competitive baselines, achieving state-of-the-art performance on the Multi30K dataset.