A fuzzy rule based multimodal framework for face sketch-to-photo retrieval

• Published in: Expert systems with applications 2019-11, Vol.134, p.138-152
• Main Authors: Khan, Mohd. Aamir, Jalal, Anand Singh
• Format: Article
• Language: English
• Subjects: Artists; Crime; Face recognition; Forensic sketch; Retrieval; Sketch recognition; Sketches; Uncertainty
• ISSN: 0957-4174; 1873-6793
• DOI: 10.1016/j.eswa.2019.05.040

•We have proposed a multimodal framework for face sketch to photo retrieval.•Semantic information is utilized to improve the retrieval performance.•Ambiguity in forensic sketches are handled using fuzzy rule based inference system.•A Survey is performed to rank the importance of different facial attributes. Facial sketches are widely used by law enforcement agencies to assist in the identification and apprehension of suspects involved in criminal activities. Sketches used in forensic investigations are drawn by forensic artists following the verbal description provided by an eyewitness or the victim. In a forensic sketch, the facial description depends on the memory of the eyewitness, therefore, there is uncertainty in facial attributes. Most of the existing sketch-photo retrieval methods have ignored the uncertainty of facial attributes. In this paper, we have proposed a novel multimodal framework for sketch-photo retrieval. We have fused the facial attributes and the semantic color information using a fuzzy rule based layered classifier. We have also performed a survey to rank the importance of different facial attributes in the identification of a person. We have performed experiments using viewed, semi-forensic and forensic sketches dataset and results are compared with state-of-the-art methods. Experimental results show that the proposed fuzzy rule based method achieves promising performance as compared to state-of-art methods specifically in case of forensic sketches.