Exponential family Fisher vector for image classification

• Published in: Pattern recognition letters 2015-07, Vol.59, p.26-32
• Main Authors: Sánchez, Jorge, Redolfi, Javier
• Format: Article
• Language: English
• Subjects: Exponential family; Fisher kernel; Fisher vectors; Image classification
• ISSN: 0167-8655; 1872-7344
• DOI: 10.1016/j.patrec.2015.03.010

•We generalize the FV to mixtures of non-Gaussian pdf in a unified manner.•We provide a complete derivation of the FV for arbitrary sets.•We derive a block-diagonal normalizer which is general and simple to estimate.•We run experiments using different combinations of local features and mixture models. One of the fundamental problems in image classification is to devise models that allow us to relate the images to higher-level semantic concepts in an efficient and reliable way. A widely used approach consists on extracting local descriptors from the images and to summarize them into an image-level representation. Within this framework, the Fisher vector (FV) is one of the most robust signatures to date. In the FV, local descriptors are modeled as samples drawn from a mixture of Gaussian pdfs. An image is represented by a gradient vector characterizing the distributions of samples w.r.t. the model. Equipped with robust features like SIFT, the FV has shown state-of-the-art performance on different recognition problems. However, it is not clear how it should be applied when the feature space is clearly non-Euclidean, leading to heuristics that ignore the underlying structure of the space. In this paper we generalize the Gaussian FV to a broader family of distributions known as the exponential family. The model, termed exponential family Fisher vectors (eFV), provides a unified framework from which rich and powerful representations can be derived. Experimental results show the generality and flexibility of our approach.