Towards reduced dependency and faster unsupervised 3D face reconstruction

• Published in: Journal of real-time image processing 2023-04, Vol.20 (2), p.18, Article 18
• Main Authors: Tiwari, Hitika, Subramanian, Venkatesh K., Chen, Yong-Sheng
• Format: Article
• Language: English
• Subjects: Accuracy; Computer Graphics; Computer Science; Datasets; Exploitation; Image Processing and Computer Vision; Image reconstruction; Methods; Multimedia Information Systems; Original Research Paper; Pattern Recognition; Perceptual errors; Qualitative analysis; Signal,Image and Speech Processing; Supervision; Testing time; Training
• ISSN: 1861-8200; 1861-8219
• DOI: 10.1007/s11554-023-01257-z

Recent monocular 3D face reconstruction methods demonstrate performance improvement regarding 3D face geometry retrieval. However, these methods pose numerous challenges, particularly during testing. One of the significant challenges is the requirement of processed (cropped and aligned) input, which leads to the dependency on the facial landmark coordinates detector. Moreover, input processing time degrades the network’s testing speed, thus increasing the test time. Therefore, we propose a RE duced D ependency F ast U nsupervi SE d 3D Face Reconstruction  ( RED-FUSE ) framework, which exploits unprocessed (uncropped and unaligned) face images to estimate reliable 3D face shape and texture, waiving off the requirement for prior facial landmarks information, and improving the network’s estimation speed. More specifically, we utilize a (1) Multi-pipeline training architecture to reconstruct accurate 3D faces from challenging (transformed) unprocessed test inputs without posing additional requirements and (2) Pose transfer module that ensures reliable training for unprocessed challenging images by attaining the inter-pipeline face pose consistency without requiring the respective facial landmark information. We performed qualitative and quantitative analysis of our model on the unprocessed CelebA-test dataset, LFW-test set, NoW selfie challenge set and various open-source images. Our RED-FUSE outperforms a current method on the unprocessed CelebA-test dataset, e.g., for 3D shape-based, color-based, and 2D perceptual errors, the proposed method shows an improvement of 46.2 % , 15.1 % , and 27.4 % , respectively. Moreover, our approach demonstrates a significant improvement of 29.6 % on NoW selfie challenge. Furthermore, RED-FUSE requires lesser test time (a reduction from 7.30 m.sec. to 1.85 m.sec. per face) and poses minimal test time dependencies, demonstrating the effectiveness of the proposed method.