EM-Based Hyperparameter Optimization for Regularized Volterra Kernel Estimation

• Published in: IEEE control systems letters 2017-10, Vol.1 (2), p.388-393
• Main Authors: Stoddard, Jeremy G., Welsh, James S., Hjalmarsson, Hakan
• Format: Article
• Language: English
• Subjects: Bayes methods; Bayesian approaches; Bayesian networks; Covariance matrices; Covariance structures; Estimation; Expectation Maximization; Finite impulse response filters; Global optimization; Hyper-parameter optimizations; Kernel; Marginal likelihood; Maximum likelihood estimation; Maximum principle; Noise measurement; Nonlinear systems; Nonlinear systems identification; Numerical methods; Optimization; Optimization problems; Regularized least squares
• ISSN: 2475-1456; 2475-1456
• DOI: 10.1109/LCSYS.2017.2719766

In nonlinear system identification, Volterra kernel estimation based on regularized least squares can be performed by taking a Bayesian approach. In this framework, covariance structures which describe the Gaussian kernels are represented by a set of hyperparameters. The hyperparameters are traditionally tuned through a global optimization which maximizes their marginal likelihood with respect to the measured data. The global optimization is computationally intensive for high-order estimates, as the number of hyperparameters increases quadratically with the Volterra series order. In this letter, we propose a new method of hyperparameter tuning based on expectation-maximization (EM). The technique allows the global optimization to be split into smaller components such that the search space of any given optimization problem is not prohibitively large. The main advantage of the proposed EM method is improved computation time scaling with respect to Volterra series order. The computation time benefits of the EM-based method are demonstrated through a numerical example for the case where the maximum nonlinear order is known.