Spectral Algorithms for Supervised Learning

• Published in: Neural computation 2008-07, Vol.20 (7), p.1873-1897
• Main Authors: Gerfo, L. Lo, Rosasco, L., Odone, F., Vito, E. De, Verri, A.
• Format: Article
• Language: English
• Subjects: Algorithms; Applied sciences; Artificial Intelligence; Biological and medical sciences; Computer science; control theory; systems; Databases as Topic; Exact sciences and technology; Fundamental and applied biological sciences. Psychology; General aspects; Learning; Learning and adaptive systems; Least-Squares Analysis; Mathematics; Mathematics in biology. Statistical analysis. Models. Metrology. Data processing in biology (general aspects); Matrix; Miscellaneous; Numerical analysis; Numerical analysis. Scientific computation; Numerical linear algebra; Performance evaluation; Problem solving; Regression Analysis; Sciences and techniques of general use; Time Factors
• ISSN: 0899-7667; 1530-888X
• DOI: 10.1162/neco.2008.05-07-517

We discuss how a large class of regularization methods, collectively known as spectral regularization and originally designed for solving ill-posed inverse problems, gives rise to regularized learning algorithms. All of these algorithms are consistent kernel methods that can be easily implemented. The intuition behind their derivation is that the same principle allowing for the numerical stabilization of a matrix inversion problem is crucial to avoid overfitting. The various methods have a common derivation but different computational and theoretical properties. We describe examples of such algorithms, analyze their classification performance on several data sets and discuss their applicability to real-world problems.