Improved likelihood ratios for face recognition in surveillance video by multimodal feature pairing

• Published in: Forensic science international. Synergy 2024-01, Vol.8, p.100458-100458, Article 100458
• Main Authors: Macarulla Rodriguez, Andrea, Geradts, Zeno, Worring, Marcel, Unzueta, Luis
• Format: Article
• Language: English
• Subjects: Face image quality; Face recognition; Likelihood ratio; Multi-modal analysis; Super resolution; Video processing
• ISSN: 2589-871X; 2589-871X
• DOI: 10.1016/j.fsisyn.2024.100458

•Study tackles the problem of accurate likelihood ratios in surveillance video face recognition.•Proposes a frame selection method to enhance forensic face recognition by pairing frames based on quality and face attributes.•Optimal frame selection framework validated across various models and datasets.•As additional result, the study reveals that the use of super-resolution preprocessing via CodeFormer adversely affects the reliability of forensic face recognition. In forensic and security scenarios, accurate facial recognition in surveillance videos, often challenged by variations in pose, illumination, and expression, is essential. Traditional manual comparison methods lack standardization, revealing a critical gap in evidence reliability. We propose an enhanced images-to-video recognition approach, pairing facial images with attributes like pose and quality. Utilizing datasets such as ENFSI 2015, SCFace, XQLFW, ChokePoint, and ForenFace, we assess evidence strength using calibration methods for likelihood ratio estimation. Three models—ArcFace, FaceNet, and QMagFace—undergo validation, with the log-likelihood ratio cost (Cllr) as a key metric. Results indicate that prioritizing high-quality frames and aligning attributes with reference images optimizes recognition, yielding similar Cllr values to the top 25% best frames approach. A combined embedding weighted by frame quality emerges as the second-best method. Upon preprocessing facial images with the super resolution CodeFormer, it unexpectedly increased Cllr, undermining evidence reliability, advising against its use in such forensic applications.