A Comprehensive Pipeline for Complex Text-to-Image Synthesis

• Published in: Journal of computer science and technology 2020-05, Vol.35 (3), p.522-537
• Main Authors: Fang, Fei, Luo, Fei, Zhang, Hong-Pan, Zhou, Hua-Jian, Chow, Alix L. H., Xiao, Chun-Xia
• Format: Article
• Language: English
• Subjects: Artificial Intelligence; Computational linguistics; Computer Science; Computer vision; Cost function; Data Structures and Information Theory; Datasets; Generative adversarial networks; Image quality; Information Systems Applications (incl.Internet); Language processing; Machine vision; Markov chains; Markov processes; Monte Carlo method; Natural language interfaces; Natural language processing; Object recognition; Pipe lines; Regular Paper; Retrieval; Software Engineering; Synthesis; Theory of Computation
• ISSN: 1000-9000; 1860-4749
• DOI: 10.1007/s11390-020-0305-9

Synthesizing a complex scene image with multiple objects and background according to text description is a challenging problem. It needs to solve several difficult tasks across the fields of natural language processing and computer vision. We model it as a combination of semantic entity recognition, object retrieval and recombination, and objects’ status optimization. To reach a satisfactory result, we propose a comprehensive pipeline to convert the input text to its visual counterpart. The pipeline includes text processing, foreground objects and background scene retrieval, image synthesis using constrained MCMC, and post-processing. Firstly, we roughly divide the objects parsed from the input text into foreground objects and background scenes. Secondly, we retrieve the required foreground objects from the foreground object dataset segmented from Microsoft COCO dataset, and retrieve an appropriate background scene image from the background image dataset extracted from the Internet. Thirdly, in order to ensure the rationality of foreground objects’ positions and sizes in the image synthesis step, we design a cost function and use the Markov Chain Monte Carlo (MCMC) method as the optimizer to solve this constrained layout problem. Finally, to make the image look natural and harmonious, we further use Poisson-based and relighting-based methods to blend foreground objects and background scene image in the post-processing step. The synthesized results and comparison results based on Microsoft COCO dataset prove that our method outperforms some of the state-of-the-art methods based on generative adversarial networks (GANs) in visual quality of generated scene images.