Trainable back-propagated functional transfer matrices

Connections between nodes of fully connected neural networks are usually represented by weight matrices. In this article, functional transfer matrices are introduced as alternatives to the weight matrices: Instead of using real weights, a functional transfer matrix uses real functions with trainable...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2017-10
Main Authors: Cheng-Hao, Cai, Xu, Yanyan, Ke, Dengfeng, Su, Kaile, Sun, Jing
Format: Article
Language:English
Subjects:
Back propagation networks
Digits
Functionals
Mathematical analysis
Matrix methods
Neural networks
Nodes
Transfer matrices
Weight
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Connections between nodes of fully connected neural networks are usually represented by weight matrices. In this article, functional transfer matrices are introduced as alternatives to the weight matrices: Instead of using real weights, a functional transfer matrix uses real functions with trainable parameters to represent connections between nodes. Multiple functional transfer matrices are then stacked together with bias vectors and activations to form deep functional transfer neural networks. These neural networks can be trained within the framework of back-propagation, based on a revision of the delta rules and the error transmission rule for functional connections. In experiments, it is demonstrated that the revised rules can be used to train a range of functional connections: 20 different functions are applied to neural networks with up to 10 hidden layers, and most of them gain high test accuracies on the MNIST database. It is also demonstrated that a functional transfer matrix with a memory function can roughly memorise a non-cyclical sequence of 400 digits.
ISSN:2331-8422
DOI:10.48550/arxiv.1710.10403