Locally controllable network based on visual–linguistic relation alignment for text-to-image generation

• Published in: Multimedia systems 2024-02, Vol.30 (1), Article 34
• Main Authors: Li, Zaike, Liu, Li, Zhang, Huaxiang, Liu, Dongmei, Song, Yu, Li, Boqun
• Format: Article
• Language: English
• Subjects: Alignment; Computer Communication Networks; Computer Graphics; Computer Science; Controllability; Cryptology; Data Storage Representation; Image processing; Linguistics; Multimedia Information Systems; Operating Systems; Regular Paper; Similarity
• ISSN: 0942-4962; 1432-1882
• DOI: 10.1007/s00530-023-01222-7

Since locally controllable text-to-image generation cannot achieve satisfactory results in detail, a novel locally controllable text-to-image generation network based on visual–linguistic relation alignment is proposed. The goal of the method is to complete image processing and generation semantically through text guidance. The proposed method explores the relationship between text and image to achieve local control of text-to-image generation. The visual–linguistic matching learns the similarity weights between image and text through semantic features to achieve the fine-grained correspondence between local images and words. The instance-level optimization function is introduced into the generation process to accurately control the weight with low similarity and combine with text features to generate new visual attributes. In addition, a local control loss is proposed to preserve the details of the text and local regions of the image. Extensive experiments demonstrate the superior performance of the proposed method and enable more accurate control of the original image.