Regularized Negative Correlation Learning for Neural Network Ensembles

• Published in: IEEE transaction on neural networks and learning systems 2009-12, Vol.20 (12), p.1962-1979
• Main Authors: Chen, Huanhuan, Yao, Xin
• Format: Article
• Language: English
• Subjects: Algorithm design and analysis; Algorithms; Application software; Applied sciences; Artificial intelligence; Automatic Data Processing; Bayes Theorem; Bayesian analysis; Bayesian methods; Computational intelligence; Computer science; control theory; systems; Connectionism. Neural networks; Correlation; Cost function; Ensembles; Exact sciences and technology; Humans; Inference algorithms; Learning; Learning - physiology; Machine learning; Mean square error methods; negative correlation learning (NCL); neural network ensembles; Neural networks; Neural Networks (Computer); Noise level; Nonlinear Dynamics; probabilistic model; Regularization; Signal Processing, Computer-Assisted; Training
• ISSN: 1045-9227; 2162-237X; 1941-0093; 2162-2388
• DOI: 10.1109/TNN.2009.2034144

Negative correlation learning (NCL) is a neural network ensemble learning algorithm that introduces a correlation penalty term to the cost function of each individual network so that each neural network minimizes its mean square error (MSE) together with the correlation of the ensemble. This paper analyzes NCL and reveals that the training of NCL (when ¿ = 1) corresponds to training the entire ensemble as a single learning machine that only minimizes the MSE without regularization. This analysis explains the reason why NCL is prone to overfitting the noise in the training set. This paper also demonstrates that tuning the correlation parameter ¿ in NCL by cross validation cannot overcome the overfitting problem. The paper analyzes this problem and proposes the regularized negative correlation learning (RNCL) algorithm which incorporates an additional regularization term for the whole ensemble. RNCL decomposes the ensemble's training objectives, including MSE and regularization, into a set of sub-objectives, and each sub-objective is implemented by an individual neural network. In this paper, we also provide a Bayesian interpretation for RNCL and provide an automatic algorithm to optimize regularization parameters based on Bayesian inference. The RNCL formulation is applicable to any nonlinear estimator minimizing the MSE. The experiments on synthetic as well as real-world data sets demonstrate that RNCL achieves better performance than NCL, especially when the noise level is nontrivial in the data set.