Assessing Brain-Like Characteristics of DNNs With Spatiotemporal Features: A Study Based on the Müller-Lyer Illusion

• Published in: IEEE access 2024, Vol.12, p.147192-147208
• Main Authors: Zhang, Hongtao, Matsuzaki, Kiminori, Yoshida, Shinichi
• Format: Article
• Language: English
• Subjects: Artificial neural networks; Biological system modeling; Brain modeling; Computational modeling; computer vision; Datasets; Deep learning; deep neural networks; Human performance; Illusions; Image color analysis; Impact analysis; Machine learning; Perception; Perceptual errors; Predictive models; Self-supervised learning; Sensitivity analysis; Similarity; Spatiotemporal data; Spatiotemporal phenomena; Static models; Three-dimensional displays; Training; visual illusion; Visualization
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2024.3475478

This study explores the capabilities and limitations of deep neural networks (DNNs) in simulating human visual illusions, particularly the Müller-Lyer illusion. Visual illusions are often overlooked in DNN research, which tends to neglect the inherent temporal dynamics and complex context dependencies of human visual processing. By integrating self-supervised learning and teacher-student network models, we examined the performance of DNNs combining spatiotemporal dynamics on visual illusion phenomena. The study employed various video classification models including ResNet3D(R3D-18), Multiscale Vision Transformers(MViT-V1-B), 3DCNN(S3D), and 3D Swin Transformer(Swin3D-T), as well as PredNet for experiments to explore their perception abilities for the Müller-Lyer illusion. The experimental results were visualized using representational dissimilarity matrices (RDMs) and gradient-weighted class activation mapping (Grad-CAM), showing that DNNs considering spatiotemporal characteristics can simulate perceptual errors similar to those of humans in handling these types of visual illusions. Specifically, R3D-18, MViT-V1-B, and S3D exhibited high similarity on the diagonal in both Type A and Type B RDMs, indicating similar length perception for inward and outward pointing lines within the models. In the control group RDMs, the high similarity distribution slightly shifted upwards from the diagonal, suggesting that outward-pointing lines need to be longer to match inward-pointing lines, mirroring human perception. Additionally, we found significant differences in the sensitivity and response patterns to visual illusions among different model architectures, emphasizing the impact of dataset selection and model structure on the performance of DNNs in visual illusions. On the contrary, while spatiotemporal DNNs showed advantages in RDM analysis, static models like AlexNet, Vgg19, and ResNet101 demonstrated more focused attention on arrows in Grad-CAM analysis, similar to human visual processing. The significant differences in the sensitivity and response patterns to visual illusions among different model architectures emphasize the impact of dataset selection and model structure on the performance of DNNs in visual illusions.