Deep Least Squares Fisher Discriminant Analysis

• Published in: IEEE transaction on neural networks and learning systems 2020-08, Vol.31 (8), p.2752-2763
• Main Authors: Diaz-Vico, David, Dorronsoro, Jose R.
• Format: Article
• Language: English
• Subjects: Artificial neural networks; Classification; Covariance matrices; Deep neural networks (DNNs); Discriminant analysis; Encoding; Feature extraction; Fisher discriminant analysis (FDA); Kernel; kernel discriminant analysis; Least squares; Linear discriminant analysis; Neural networks; nonlinear classifiers; Proposals; Training
• ISSN: 2162-237X; 2162-2388
• DOI: 10.1109/TNNLS.2019.2906302

While being one of the first and most elegant tools for dimensionality reduction, Fisher linear discriminant analysis (FLDA) is not currently considered among the top methods for feature extraction or classification. In this paper, we will review two recent approaches to FLDA, namely, least squares Fisher discriminant analysis (LSFDA) and regularized kernel FDA (RKFDA) and propose deep FDA (DFDA), a straightforward nonlinear extension of LSFDA that takes advantage of the recent advances on deep neural networks. We will compare the performance of RKFDA and DFDA on a large number of two-class and multiclass problems, many of them involving class-imbalanced data sets and some having quite large sample sizes; we will use, for this, the areas under the receiver operating characteristics (ROCs) curve of the classifiers considered. As we shall see, the classification performance of both methods is often very similar and particularly good on imbalanced problems, but building DFDA models is considerably much faster than doing so for RKFDA, particularly in problems with quite large sample sizes.