A Survey of Visual Transformers

Transformer, an attention-based encoder-decoder model, has already revolutionized the field of natural language processing (NLP). Inspired by such significant achievements, some pioneering works have recently been done on employing Transformer-liked architectures in the computer vision (CV) field, w...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transaction on neural networks and learning systems 2024-06, Vol.35 (6), p.7478-7498
Main Authors: Liu, Yang, Zhang, Yao, Wang, Yixin, Hou, Feng, Yuan, Jin, Tian, Jiang, Zhang, Yang, Shi, Zhongchao, Fan, Jianping, He, Zhiqiang
Format: Article
Language:English
Subjects:
Artificial neural networks
Attention
Benchmarks
Classification
Computational modeling
Computer vision
computer vision (CV)
Convolutional neural networks
Data transmission
Decoding
detection
Natural language processing
Neural networks
point clouds
segmentation
self-supervision
Surveys
Task analysis
Taxonomy
Transformers
Visual perception
visual Transformer
visual-linguistic pretraining
Visualization
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Transformer, an attention-based encoder-decoder model, has already revolutionized the field of natural language processing (NLP). Inspired by such significant achievements, some pioneering works have recently been done on employing Transformer-liked architectures in the computer vision (CV) field, which have demonstrated their effectiveness on three fundamental CV tasks (classification, detection, and segmentation) as well as multiple sensory data stream (images, point clouds, and vision-language data). Because of their competitive modeling capabilities, the visual Transformers have achieved impressive performance improvements over multiple benchmarks as compared with modern convolution neural networks (CNNs). In this survey, we have reviewed over 100 of different visual Transformers comprehensively according to three fundamental CV tasks and different data stream types, where taxonomy is proposed to organize the representative methods according to their motivations, structures, and application scenarios. Because of their differences on training settings and dedicated vision tasks, we have also evaluated and compared all these existing visual Transformers under different configurations. Furthermore, we have revealed a series of essential but unexploited aspects that may empower such visual Transformers to stand out from numerous architectures, e.g., slack high-level semantic embeddings to bridge the gap between the visual Transformers and the sequential ones. Finally, two promising research directions are suggested for future investment. We will continue to update the latest articles and their released source codes at https://github.com/liuyang-ict/awesome-visual-transformers .
ISSN:2162-237X
2162-2388
DOI:10.1109/TNNLS.2022.3227717