Visual and textual explainability for a biometric verification system based on piecewise facial attribute analysis

• Published in: Image and vision computing 2023-04, Vol.132, p.104645, Article 104645
• Main Authors: Cascone, Lucia, Pero, Chiara, Proença, Hugo
• Format: Article
• Language: English
• Subjects: Explainability; Facial attribute analysis; Feature extraction; Interpretable representation; Semantics
• ISSN: 0262-8856; 1872-8138
• DOI: 10.1016/j.imavis.2023.104645

The decisions behind the mechanics of a biometric verification system based on Machine Learning (ML) are difficult to comprehend. Although there is now well-established research in various fields of application, such as health or justice, the use of ML-based methods is accompanied by a lack of confidence that results in their limited use. The explainability of a ML system and the comprehension of what lies behind its prediction is one of the numerous characteristics that define “trust” in these systems. Over the years, face-based biometric authentication has been the subject of extensive research in both academia and industry. However, existing biometric authentication systems still have problems regarding accuracy, robustness and, explainability. Still lacking in the literature is a comprehensive examination of the use of post-hoc explainability techniques for such systems. Cognitive neuroscience has always been interested in the method by which people perceive faces; local elements such as the nose, eyes, and mouth are critical to the perception and recognition of a face. In this work, starting from this assumption, we propose a framework of visual and textual explainability based on the parts of a face by analyzing them with respect to the facial attributes reported in the CelebA dataset. The primary objective is to be able to explain why two pictures of different subjects are distinct. This is done by sinthesizing pairs of images that illustrate how dissimilar the various parts of the face under investigation are and incisive and direct textual explanations of the distinguishing features are generated. A further study analyzes an interpretable mapping between the semantic space of the text and the space of the image. •Post-hoc techniques for understanding the decision of a biometric verification system•Implementation of a biometric verification system based on parts of the face•A text caption and an image highlighting the distinguishing features of two subjects•Interpretable mapping between text semantic space and image space