Wills Aligner: Multi-Subject Collaborative Brain Visual Decoding

Decoding visual information from human brain activity has seen remarkable advancements in recent research. However, the diversity in cortical parcellation and fMRI patterns across individuals has prompted the development of deep learning models tailored to each subject. The personalization limits th...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-12
Main Authors: Bao, Guangyin, Zhang, Qi, Gong, Zixuan, Zhou, Jialei, Fan, Wei, Yi, Kun, Usman Naseem, Hu, Liang, Miao, Duoqian
Format: Article
Language:English
Subjects:
Brain
Cognition
Cognition & reasoning
Cognitive tasks
Commonality
Learning
Performance evaluation
Representations
Robustness
Visual tasks
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Decoding visual information from human brain activity has seen remarkable advancements in recent research. However, the diversity in cortical parcellation and fMRI patterns across individuals has prompted the development of deep learning models tailored to each subject. The personalization limits the broader applicability of brain visual decoding in real-world scenarios. To address this issue, we introduce Wills Aligner, a novel approach designed to achieve multi-subject collaborative brain visual decoding. Wills Aligner begins by aligning the fMRI data from different subjects at the anatomical level. It then employs delicate mixture-of-brain-expert adapters and a meta-learning strategy to account for individual fMRI pattern differences. Additionally, Wills Aligner leverages the semantic relation of visual stimuli to guide the learning of inter-subject commonality, enabling visual decoding for each subject to draw insights from other subjects' data. We rigorously evaluate our Wills Aligner across various visual decoding tasks, including classification, cross-modal retrieval, and image reconstruction. The experimental results demonstrate that Wills Aligner achieves promising performance.
ISSN:2331-8422