Iterative Adversarial Attack on Image-Guided Story Ending Generation

• Published in: IEEE transactions on multimedia 2024, Vol.26, p.6117-6130
• Main Authors: Wang, Youze, Hu, Wenbo, Hong, Richang
• Format: Article
• Language: English
• Subjects: adversarial attack; Artificial neural networks; Computational modeling; Data models; Data processing; Fuses; Iterative methods; Machine learning; Machine translation; Multimodal; multimodal text generation; Perturbation methods; Task analysis; Visualization
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2023.3345167

Multimodal learning involves developing models that can integrate information from various sources like images and texts. In this field, multimodal text generation is a crucial aspect that involves processing data from multiple modalities and outputting text. The image-guided story ending generation (IgSEG) is a particularly significant task, targeting on an understanding of complex relationships between text and image data with a complete story text ending. Unfortunately, deep neural networks, which are the backbone of recent IgSEG models, are vulnerable to adversarial samples. Current adversarial attack methods mainly focus on single-modality data and do not analyze adversarial attacks for multimodal text generation tasks that use cross-modal information. To this end, we propose an iterative adversarial attack method (Iterative-attack) that fuses image and text modality attacks, allowing for an attack search for adversarial text and image in a more effective iterative way. Experimental results demonstrate that the proposed method outperforms existing single-modal and non-iterative multimodal attack methods, indicating the potential for improving the adversarial robustness of multimodal text generation models, such as multimodal machine translation, multimodal question answering, etc.