Driver Gaze Zone Estimation Based on Three-Channel Convolution-Optimized Vision Transformer With Transfer Learning

• Published in: IEEE sensors journal 2024-12, Vol.24 (24), p.42064-42078
• Main Authors: Li, Zhao, Jiang, Siyang, Fu, Rui, Guo, Yingshi, Wang, Chang
• Format: Article
• Language: English
• Subjects: Accuracy; Computational modeling; Convolutional neural network (CNN); deep learning; driver gaze zone; Estimation; Face recognition; Feature extraction; Head; Magnetic heads; Mathematical models; transfer learning; Transformers; Vehicles; vision transformer (ViT); visual interpretability
• ISSN: 1530-437X
• DOI: 10.1109/JSEN.2024.3486373

Driver gaze zone estimation (DGZE) is essential for detecting the driver's state and taking over rule-making in intelligent driving systems. However, convolutional neural network (CNN)-based multichannel models lack global feature extraction capability, with a large number of parameters and high computational complexity. Therefore, this article proposes a novel method that uses a three-channel convolution-optimized vision transformer (3C-CoViT) to estimate the driver's gaze zone. The method replaces the linear projection in the pure ViT structure with convolutional projection, converts the input images of different channels into image sequences, and then adds a convolutional feed-forward network to extract the local features of the markers, enhance the correlation of adjacent tokens in spatial dimensions, and improve the performance and efficiency of the model. We then pretrained the model on the GazeCapture dataset based on transfer learning and then fine-tuned the model on the dataset built in the actual road experiment. To enhance the interpretability of the model, we presented a novel visualization method. Experimental results show that the proposed method can accurately identify driver gaze zones (98.04% average accuracy) and outperform state-of-the-art methods in terms of accuracy and reliability. Ablation studies proved the effectiveness of our proposed method over the pure ViT and the beneficial effects of transfer learning and three-channel information input.