TULIP: Token-length Upgraded CLIP

• Published in: arXiv.org 2024-10
• Main Authors: Najdenkoska, Ivona, Mohammad Mahdi Derakhshani, Asano, Yuki M, Nanne van Noord, Worring, Marcel, Snoek, Cees G M
• Format: Article
• Language: English
• Subjects: Coders; Image processing
• ISSN: 2331-8422

We address the challenge of representing long captions in vision-language models, such as CLIP. By design these models are limited by fixed, absolute positional encodings, restricting inputs to a maximum of 77 tokens and hindering performance on tasks requiring longer descriptions. Although recent work has attempted to overcome this limit, their proposed approaches struggle to model token relationships over longer distances and simply extend to a fixed new token length. Instead, we propose a generalizable method, named TULIP, able to upgrade the token length to any length for CLIP-like models. We do so by improving the architecture with relative position encodings, followed by a training procedure that (i) distills the original CLIP text encoder into an encoder with relative position encodings and (ii) enhances the model for aligning longer captions with images. By effectively encoding captions longer than the default 77 tokens, our model outperforms baselines on cross-modal tasks such as retrieval and text-to-image generation.