Partially pre-calculated weights for the backpropagation learning regime and high accuracy function mapping using continuous input RAM-based sigma–pi nets

• Published in: Neural networks 2000, Vol.13 (1), p.91-110
• Main Authors: Neville, R.S., Stonham, T.J., Glover, R.J.
• Format: Article
• Language: English
• Subjects: Applied sciences; Artificial Intelligence; Backpropagation; Electric, optical and optoelectronic circuits; Electronics; Exact sciences and technology; Higher order; Models, Neurological; n-Tuple; Neural networks; Neural Networks (Computer); RAM nets; Sigma–pi; Training
• ISSN: 0893-6080; 1879-2782
• DOI: 10.1016/S0893-6080(99)00102-1

In this article we present a methodology that partially pre-calculates the weight updates of the backpropagation learning regime and obtains high accuracy function mapping. The paper shows how to implement neural units in a digital formulation which enables the weights to be quantised to 8-bits and the activations to 9-bits. A novel methodology is introduced to enable the accuracy of sigma–pi units to be increased by expanding their internal state space. We, also, introduce a novel means of implementing bit-streams in ring memories instead of utilising shift registers. The investigation utilises digital “Higher Order” sigma–pi nodes and studies continuous input RAM-based sigma–pi units. The units are trained with the backpropagation learning regime to learn functions to a high accuracy. The neural model is the sigma–pi units which can be implemented in digital microelectronic technology. The ability to perform tasks that require the input of real-valued information, is one of the central requirements of any cognitive system that utilises artificial neural network methodologies. In this article we present recent research which investigates a technique that can be used for mapping accurate real-valued functions to RAM-nets. One of our goals was to achieve accuracies of better than 1% for target output functions in the range Y∈[0,1], this is equivalent to an average Mean Square Error (MSE) over all training vectors of 0.0001 or an error modulus of 0.01. We present a development of the sigma–pi node which enables the provision of high accuracy outputs. The sigma–pi neural model was initially developed by Gurney (Learning in nets of structured hypercubes. PhD Thesis, Department of Electrical Engineering, Brunel University, Middlessex, UK, 1989; available as Technical Memo CN/R/144). Gurney's neuron models, the Time Integration Node (TIN), utilises an activation that was derived from a bit-stream. In this article we present a new methodology for storing sigma–pi node's activations as single values which are averages. In the course of the article we state what we define as a real number; how we represent real numbers and input of continuous values in our neural system. We show how to utilise the bounded quantised site-values (weights) of sigma–pi nodes to make training of these neurocomputing systems simple, using pre-calculated look-up tables to train the nets. In order to meet our accuracy goal, we introduce a means of increasing the bandwidth capability of sig