PreSTU: Pre-Training for Scene-Text Understanding

The ability to recognize and reason about text embedded in visual inputs is often lacking in vision-and-language (V&L) models, perhaps because V&L pre-training methods have often failed to include such an ability in their training objective. In this paper, we propose PreSTU, a novel pre-trai...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2023-08
Main Authors: Kil, Jihyung, Changpinyo, Soravit, Chen, Xi, Hu, Hexiang, Goodman, Sebastian, Wei-Lun, Chao, Soricut, Radu
Format: Article
Language:English
Subjects:
Datasets
Image resolution
Training
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The ability to recognize and reason about text embedded in visual inputs is often lacking in vision-and-language (V&L) models, perhaps because V&L pre-training methods have often failed to include such an ability in their training objective. In this paper, we propose PreSTU, a novel pre-training recipe dedicated to scene-text understanding (STU). PreSTU introduces OCR-aware pre-training objectives that encourage the model to recognize text from an image and connect it to the rest of the image content. We implement PreSTU using a simple transformer-based encoder-decoder architecture, combined with large-scale image-text datasets with scene text obtained from an off-the-shelf OCR system. We empirically demonstrate the effectiveness of this pre-training approach on eight visual question answering and four image captioning benchmarks.
ISSN:2331-8422