Computational reconstruction of mental representations using human behavior

• Published in: Nature communications 2024-05, Vol.15 (1), p.4183-4183, Article 4183
• Main Authors: Caplette, Laurent, Turk-Browne, Nicholas B.
• Format: Article
• Language: English
• Subjects: 631/378/2613/2616; 631/477/2811; Adult; Artificial neural networks; Behavior; Cognition - physiology; Cognitive science; Female; Human behavior; Humanities and Social Sciences; Humans; Male; multidisciplinary; Neural networks; Neural Networks, Computer; Photic Stimulation - methods; Representations; Science; Science (multidisciplinary); Semantics; Visual Perception - physiology; Young Adult
• ISSN: 2041-1723; 2041-1723
• DOI: 10.1038/s41467-024-48114-6

Revealing how the mind represents information is a longstanding goal of cognitive science. However, there is currently no framework for reconstructing the broad range of mental representations that humans possess. Here, we ask participants to indicate what they perceive in images made of random visual features in a deep neural network. We then infer associations between the semantic features of their responses and the visual features of the images. This allows us to reconstruct the mental representations of multiple visual concepts, both those supplied by participants and other concepts extrapolated from the same semantic space. We validate these reconstructions in separate participants and further generalize our approach to predict behavior for new stimuli and in a new task. Finally, we reconstruct the mental representations of individual observers and of a neural network. This framework enables a large-scale investigation of conceptual representations. Revealing how the human mind represents information is a longstanding goal of cognitive science. Here, the authors develop a method to reconstruct the mental representations of multiple visual concepts using behavioral judgments.