Recipe2Video: Synthesizing Personalized Videos from Recipe Texts

Procedural texts are a special type of documents that contain complex textual descriptions for carrying out a sequence of instructions. Due to the lack of visual cues, it often becomes difficult to consume the textual information effectively. In this paper, we focus on recipes - a particular type of...

Full description

Saved in:
Bibliographic Details
Main Authors: Udhayanan, Prateksha, Bv, Suryateja, Laturia, Parth, Chauhan, Dev, Khandelwal, Darshan, Petrangeli, Stefano, Srinivasan, Balaji Vasan
Format: Conference Proceeding
Language:English
Subjects:
Applications: Visualization
Computer vision
Layout
Measurement
Quality assessment
Semantics
Task analysis
Vision + language and/or other modalities
Visualization
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Procedural texts are a special type of documents that contain complex textual descriptions for carrying out a sequence of instructions. Due to the lack of visual cues, it often becomes difficult to consume the textual information effectively. In this paper, we focus on recipes - a particular type of procedural document and introduce a novel deep-learning driven system - Recipe2Video that automatically converts a recipe document into a multimodal illustrative video. Our method employs novel retrieval and re-ranking methods to select the best set of images and videos that can provide the desired illustration. We formulate a Viterbi-based optimization algorithm to stitch together a coherent video that combines the visual cues, text and voice-over to present an enhanced mode of consumption. We design automated metrics and compare performance across several baselines on two recipe datasets (RecipeQA, Tasty Videos). Our results on downstream tasks and human studies indicate that Recipe2Video captures the semantic and sequential information of the input in the generated video.
ISSN:2642-9381
DOI:10.1109/WACV56688.2023.00230