Interpreting the Rhetoric of Visual Advertisements

• Published in: IEEE transactions on pattern analysis and machine intelligence 2021-04, Vol.43 (4), p.1308-1323
• Main Authors: Ye, Keren, Nazari, Narges Honarvar, Hahn, James, Hussain, Zaeem, Zhang, Mingda, Kovashka, Adriana
• Format: Article
• Language: English
• Subjects: Advertisements; atypicality; Cognition; Computer vision; Decoding; Media; representation learning; Rhetoric; Task analysis; video understanding; vision and language; Visual reasoning; visual rhetoric; Visualization
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2019.2947440

Visual media have important persuasive power, but prior computer vision approaches have predominantly ignored the persuasive aspects of images. In this work, we propose a suite of data and techniques that enable progress on understanding the messages that visual advertisements convey. We make available a dataset of 64,832 image ads and 3,477 video ads, annotated with ten types of information: the topic and sentiment of the ad; whether it is funny, exciting, or effective; what action it prompts the viewer to do, and what arguments it provides for why this action should be taken; symbolic associations that the ad relies on; the metaphorical object transformations on which especially creative ads rely; and the climax in video ads. We develop methods that use multimodal cues, i.e., both visuals and slogans, for both the image and video domains. Our methods rely on finding poignant content spatially and temporally. We also examine the creative story construction in ads: for videos, we learn to predict when the climax occurs (if any), and how effective the story is; for images, we analyze how object transformations in ads metaphorically depict product properties.