LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioning

• Published in: arXiv.org 2024-10
• Main Authors: Li, Zhe, Yuan, Weihao, He, Yisheng, Qiu, Lingteng, Zhu, Shenhao, Gu, Xiaodong, Shen, Weichao, Yuan, Dong, Dong, Zilong, Yang, Laurence T
• Format: Article
• Language: English
• Subjects: Human motion; Image enhancement; Language; Large language models; Retrieval; Semantics; Transformers
• ISSN: 2331-8422

Language plays a vital role in the realm of human motion. Existing methods have largely depended on CLIP text embeddings for motion generation, yet they fall short in effectively aligning language and motion due to CLIP's pretraining on static image-text pairs. This work introduces LaMP, a novel Language-Motion Pretraining model, which transitions from a language-vision to a more suitable language-motion latent space. It addresses key limitations by generating motion-informative text embeddings, significantly enhancing the relevance and semantics of generated motion sequences. With LaMP, we advance three key tasks: text-to-motion generation, motion-text retrieval, and motion captioning through aligned language-motion representation learning. For generation, we utilize LaMP to provide the text condition instead of CLIP, and an autoregressive masked prediction is designed to achieve mask modeling without rank collapse in transformers. For retrieval, motion features from LaMP's motion transformer interact with query tokens to retrieve text features from the text transformer, and vice versa. For captioning, we finetune a large language model with the language-informative motion features to develop a strong motion captioning model. In addition, we introduce the LaMP-BertScore metric to assess the alignment of generated motions with textual descriptions. Extensive experimental results on multiple datasets demonstrate substantial improvements over previous methods across all three tasks. The code of our method will be made public.