Synthesizing Images From Spatio-Temporal Representations Using Spike-Based Backpropagation

Spiking neural networks (SNNs) offer a promising alternative to current artificial neural networks to enable low-power event-driven neuromorphic hardware. Spike-based neuromorphic applications require processing and extracting meaningful information from spatio-temporal data, represented as series o...

Full description

Saved in:
Bibliographic Details
Published in:Frontiers in neuroscience 2019-06, Vol.13, p.621
Main Authors: Roy, Deboleena, Panda, Priyadarshini, Roy, Kaushik
Format: Article
Language:English
Subjects:
Algorithms
Artificial intelligence
audio to image conversion
autoencoders
Back propagation
backpropagataon
Firing pattern
Information processing
International conferences
Machine learning
Membrane potential
multimodal
Neural networks
Neurons
Neuroscience
spiking neural networks
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Spiking neural networks (SNNs) offer a promising alternative to current artificial neural networks to enable low-power event-driven neuromorphic hardware. Spike-based neuromorphic applications require processing and extracting meaningful information from spatio-temporal data, represented as series of spike trains over time. In this paper, we propose a method to synthesize images from multiple modalities in a spike-based environment. We use spiking auto-encoders to convert image and audio inputs into compact spatio-temporal representations that is then decoded for image synthesis. For this, we use a direct training algorithm that computes loss on the membrane potential of the output layer and back-propagates it by using a sigmoid approximation of the neuron's activation function to enable differentiability. The spiking autoencoders are benchmarked on MNIST and Fashion-MNIST and achieve very low reconstruction loss, comparable to ANNs. Then, spiking autoencoders are trained to learn meaningful spatio-temporal representations of the data, across the two modalities-audio and visual. We synthesize images from audio in a spike-based environment by first generating, and then utilizing such shared multi-modal spatio-temporal representations. Our audio to image synthesis model is tested on the task of converting TI-46 digits audio samples to MNIST images. We are able to synthesize images with high fidelity and the model achieves competitive performance against ANNs.
ISSN:1662-4548
1662-453X
1662-453X
DOI:10.3389/fnins.2019.00621