Continual few-shot patch-based learning for anime-style colorization

• Published in: Computational visual media (Beijing) 2024-08, Vol.10 (4), p.705-723
• Main Authors: Maejima, Akinobu, Shinagawa, Seitaro, Kubo, Hiroyuki, Funatomi, Takuya, Yotsukura, Tatsuo, Nakamura, Satoshi, Mukaigawa, Yasuhiro
• Format: Article
• Language: English
• Subjects: Anime; Artificial Intelligence; Artificial neural networks; Artists; Barriers; Colorization; Computer Graphics; Computer Science; Image Processing and Computer Vision; Machine learning; Pipelines; Research Article; Sampling methods; Teaching methods; User Interfaces and Human Computer Interaction
• ISSN: 2096-0433; 2096-0662
• DOI: 10.1007/s41095-024-0414-4

The automatic colorization of anime line drawings is a challenging problem in production pipelines. Recent advances in deep neural networks have addressed this problem; however, collectingmany images of colorization targets in novel anime work before the colorization process starts leads to chicken-and-egg problems and has become an obstacle to using them in production pipelines. To overcome this obstacle, we propose a new patch-based learning method for few-shot anime-style colorization. The learning method adopts an efficient patch sampling technique with position embedding according to the characteristics of anime line drawings. We also present a continuous learning strategy that continuously updates our colorization model using new samples colorized by human artists. The advantage of our method is that it can learn our colorization model from scratch or pre-trained weights using only a few pre- and post-colorized line drawings that are created by artists in their usual colorization work. Therefore, our method can be easily incorporated within existing production pipelines. We quantitatively demonstrate that our colorizationmethod outperforms state-of-the-art methods.