Reduced HyperBF Networks: Regularization by Explicit Complexity Reduction and Scaled Rprop-Based Training

• Published in: IEEE transactions on neural networks 2011-05, Vol.22 (5), p.673-686
• Main Authors: Mahdi, R N, Rouchka, E C
• Format: Article
• Language: English
• Subjects: Algorithms; Applied sciences; Artificial Intelligence; Automatic Data Processing - methods; Automatic Data Processing - standards; Bridge regression; Complexity theory; Computer science; control theory; systems; Computer Simulation - standards; Connectionism. Neural networks; Data processing. List processing. Character string processing; Exact sciences and technology; generalized RBF; Humans; HyperBF; Information systems. Data bases; localized dimension reduction; Mathematical Concepts; Memory organisation. Data processing; Neural Networks (Computer); Neurons; Normal Distribution; Optimization; Pattern Recognition, Automated - methods; Pattern Recognition, Automated - standards; Radial basis function networks; reduced HyperBF; regularized HyperBF; Shape; Software; Software Design; Training; Training data; weight decay
• ISSN: 1045-9227; 1941-0093
• DOI: 10.1109/TNN.2011.2109736

Hyper basis function (HyperBF) networks are generalized radial basis function neural networks (where the activation function is a radial function of a weighted distance. Such generalization provides HyperBF networks with high capacity to learn complex functions, which in turn make them susceptible to overfitting and poor generalization. Moreover, training a HyperBF network demands the weights, centers, and local scaling factors to be optimized simultaneously. In the case of a relatively large dataset with a large network structure, such optimization becomes computationally challenging. In this paper, a new regularization method that performs soft local dimension reduction in addition to weight decay is proposed. The regularized HyperBF network is shown to provide classification accuracy competitive to a support vector machine while requiring a significantly smaller network structure. Furthermore, a practical training to construct HyperBF networks is presented. Hierarchal clustering is used to initialize neurons followed by a gradient optimization using a scaled version of the Rprop algorithm with a localized partial backtracking step. Experimental results on seven datasets show that the proposed training provides faster and smoother convergence than the regular Rprop algorithm.