Sparse multinomial logistic regression: fast algorithms and generalization bounds

• Published in: IEEE transactions on pattern analysis and machine intelligence 2005-06, Vol.27 (6), p.957-968
• Main Authors: Krishnapuram, B., Carin, L., Figueiredo, M.A.T., Hartemink, A.J.
• Format: Article
• Language: English
• Subjects: Algorithms; Applied sciences; Artificial Intelligence; Basis functions; Bayesian inference; Bayesian methods; bound optimization; classification; Classification algorithms; Classifiers; Cluster Analysis; Computer science; control theory; systems; Computer Simulation; Exact sciences and technology; expectation maximization (EM); generalization bounds; Index Terms- Supervised learning; Inference algorithms; Information Storage and Retrieval - methods; Intelligence; Kernel; Laplace equations; Learning; Learning and adaptive systems; learning theory; Logistics; Models, Biological; Models, Statistical; multinomial logistic regression; Pattern Recognition, Automated - methods; Regression; Regression Analysis; Sparsity; Studies; Supervised learning; Support vector machine classification; Support vector machines; Training data
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2005.127

Recently developed methods for learning sparse classifiers are among the state-of-the-art in supervised learning. These methods learn classifiers that incorporate weighted sums of basis functions with sparsity-promoting priors encouraging the weight estimates to be either significantly large or exactly zero. From a learning-theoretic perspective, these methods control the capacity of the learned classifier by minimizing the number of basis functions used, resulting in better generalization. This paper presents three contributions related to learning sparse classifiers. First, we introduce a true multiclass formulation based on multinomial logistic regression. Second, by combining a bound optimization approach with a component-wise update procedure, we derive fast exact algorithms for learning sparse multiclass classifiers that scale favorably in both the number of training samples and the feature dimensionality, making them applicable even to large data sets in high-dimensional feature spaces. To the best of our knowledge, these are the first algorithms to perform exact multinomial logistic regression with a sparsity-promoting prior. Third, we show how nontrivial generalization bounds can be derived for our classifier in the binary case. Experimental results on standard benchmark data sets attest to the accuracy, sparsity, and efficiency of the proposed methods.