Sparse and silent coding in neural circuits

• Published in: Neurocomputing (Amsterdam) 2012-03, Vol.79, p.115-124
• Main Authors: Lőrincz, András, Palotai, Zsolt, Szirtes, Gábor
• Format: Article
• Language: English
• Subjects: Cross-entropy method; formula omitted; Sparse coding
• ISSN: 0925-2312; 1872-8286
• DOI: 10.1016/j.neucom.2011.10.017

Sparse coding algorithms find a linear basis in which signals can be represented by a small number of non-zero coefficients. Such coding may play an important role in neural information processing and metabolically efficient natural solutions serve as an inspiration for algorithms employed in various areas of computer science. In particular, finding non-zero coefficients in overcomplete sparse coding is a computationally hard problem, for which different approximate solutions have been proposed. Methods that minimize the magnitude of the coefficients (‘ ℓ 1 - norm ’) instead of minimizing the size of the active subset of features (‘ ℓ 0 - norm ’) may find the optimal solutions, but they do not scale well with the problem size and use centralized algorithms. Iterative, greedy methods, on the other hand are fast, but require a priori knowledge of the number of non-zero features, often find suboptimal solutions and they converge to the final sparse form through a series of non-sparse representations. In this article we propose a neurally plausible algorithm which efficiently integrates an ℓ 0 - norm based probabilistic sparse coding model with ideas inspired by novel iterative solutions. Furthermore, the resulting algorithm does not require an exactly defined sparseness level thus it is suitable for representing natural stimuli with a varying number of features. We demonstrate that our combined method can find optimal solutions in cases where other, ℓ 1 - norm based algorithms already fail.