The representational hierarchy in human and artificial visual systems in the presence of object-scene regularities

• Published in: PLoS computational biology 2023-04, Vol.19 (4), p.e1011086-e1011086
• Main Authors: Bracci, Stefania, Mraz, Jakob, Zeman, Astrid, Leys, Gaëlle, Op de Beeck, Hans
• Format: Article
• Language: English
• Subjects: Analysis; Animals; Artificial neural networks; Biology and Life Sciences; Brain; Brain Mapping; Computer and Information Sciences; Computer vision; Data processing; Design of experiments; Experimental design; Human-machine systems; Humans; Information processing; Machine vision; Magnetic Resonance Imaging; Medicine and Health Sciences; Methods; Neural networks; Object recognition; Pattern recognition; Pattern Recognition, Visual; Photic Stimulation; Representations; Similarity; Social Sciences; Visual Cortex; Visual Perception; Visual stimuli
• ISSN: 1553-7358; 1553-734X; 1553-7358
• DOI: 10.1371/journal.pcbi.1011086

Human vision is still largely unexplained. Computer vision made impressive progress on this front, but it is still unclear to which extent artificial neural networks approximate human object vision at the behavioral and neural levels. Here, we investigated whether machine object vision mimics the representational hierarchy of human object vision with an experimental design that allows testing within-domain representations for animals and scenes, as well as across-domain representations reflecting their real-world contextual regularities such as animal-scene pairs that often co-occur in the visual environment. We found that DCNNs trained in object recognition acquire representations, in their late processing stage, that closely capture human conceptual judgements about the co-occurrence of animals and their typical scenes. Likewise, the DCNNs representational hierarchy shows surprising similarities with the representational transformations emerging in domain-specific ventrotemporal areas up to domain-general frontoparietal areas. Despite these remarkable similarities, the underlying information processing differs. The ability of neural networks to learn a human-like high-level conceptual representation of object-scene co-occurrence depends upon the amount of object-scene co-occurrence present in the image set thus highlighting the fundamental role of training history. Further, although mid/high-level DCNN layers represent the category division for animals and scenes as observed in VTC, its information content shows reduced domain-specific representational richness. To conclude, by testing within- and between-domain selectivity while manipulating contextual regularities we reveal unknown similarities and differences in the information processing strategies employed by human and artificial visual systems.