Feature selection in MLPs and SVMs based on maximum output information

• Published in: IEEE transaction on neural networks and learning systems 2004-07, Vol.15 (4), p.937-948
• Main Authors: Sindhwani, V., Rakshit, S., Deodhare, D., Erdogmus, D., Principe, J.C., Niyogi, P.
• Format: Article
• Language: English
• Subjects: Algorithms; Artificial Intelligence; Breast Neoplasms - classification; Breast Neoplasms - diagnosis; Classifiers; Cluster Analysis; Computer science; Computer Simulation; Computing Methodologies; Decision Support Techniques; Error analysis; Filters; Humans; Information Storage and Retrieval - methods; Information Theory; Likelihood Functions; Mathematical analysis; Mathematical models; Models, Statistical; Multilayer perceptrons; Mutual information; Neural networks; Neural Networks (Computer); Partitioning algorithms; Pattern Recognition, Automated; Probability distribution; Probability Learning; Supervised learning; Support vector machine classification; Support vector machines
• ISSN: 1045-9227; 2162-237X; 1941-0093; 2162-2388
• DOI: 10.1109/TNN.2004.828772

This paper presents feature selection algorithms for multilayer perceptrons (MLPs) and multiclass support vector machines (SVMs), using mutual information between class labels and classifier outputs, as an objective function. This objective function involves inexpensive computation of information measures only on discrete variables; provides immunity to prior class probabilities; and brackets the probability of error of the classifier. The maximum output information (MOI) algorithms employ this function for feature subset selection by greedy elimination and directed search. The output of the MOI algorithms is a feature subset of user-defined size and an associated trained classifier (MLP/SVM). These algorithms compare favorably with a number of other methods in terms of performance on various artificial and real-world data sets.