AniSora: Exploring the Frontiers of Animation Video Generation in the Sora Era

• Published in: arXiv.org 2024-12
• Main Authors: Jiang, Yudong, Xu, Baohan, Yang, Siqian, Yin, Mingyu, Liu, Jing, Xu, Chao, Wang, Siqi, Wu, Yidi, Zhu, Bingwen, Zhang, Xinwen, Zheng, Xingyu, Xu, Jixuan, Zhang, Yue, Hou, Jinlong, Sun, Huyang
• Format: Article
• Language: English
• Subjects: Animation; Benchmarks; Controllability; Data processing; Datasets; Human motion; Image quality; Spatiotemporal data; Video data
• ISSN: 2331-8422

Animation has gained significant interest in the recent film and TV industry. Despite the success of advanced video generation models like Sora, Kling, and CogVideoX in generating natural videos, they lack the same effectiveness in handling animation videos. Evaluating animation video generation is also a great challenge due to its unique artist styles, violating the laws of physics and exaggerated motions. In this paper, we present a comprehensive system, AniSora, designed for animation video generation, which includes a data processing pipeline, a controllable generation model, and an evaluation dataset. Supported by the data processing pipeline with over 10M high-quality data, the generation model incorporates a spatiotemporal mask module to facilitate key animation production functions such as image-to-video generation, frame interpolation, and localized image-guided animation. We also collect an evaluation benchmark of 948 various animation videos, the evaluation on VBench and human double-blind test demonstrates consistency in character and motion, achieving state-of-the-art results in animation video generation. Our evaluation benchmark will be publicly available at https://github.com/bilibili/Index-anisora.