GHOST-A New Face Swap Approach for Image and Video Domains

Deep fake stands for a face swapping algorithm where the source and target can be an image or a video. Researchers have investigated sophisticated generative adversarial networks (GAN), autoencoders, and other approaches to establish precise and robust algorithms for face swapping. However the achie...

Full description

Saved in:
Bibliographic Details
Published in:IEEE access 2022, Vol.10, p.83452-83462
Main Authors: Groshev, Alexander, Maltseva, Anastasia, Chesakov, Daniil, Kuznetsov, Andrey, Dimitrov, Denis
Format: Article
Language:English
Subjects:
Ablation
AEI-Net
Algorithms
Deep fake
Deepfakes
eye loss
Eye movements
face mask smooth
Face recognition
face swap
Face transfer
Faces
Generative adversarial networks
GHOST
Ghosts
Image reconstruction
Measurement
Pipelines
Retrieval
stabilization
super resolution
Training
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Deep fake stands for a face swapping algorithm where the source and target can be an image or a video. Researchers have investigated sophisticated generative adversarial networks (GAN), autoencoders, and other approaches to establish precise and robust algorithms for face swapping. However the achieved results are far from perfect in terms of human and visual evaluation. In this study, we propose a new one-shot pipeline for image-to-image and image-to-video face swap solutions - GHOST (Generative High-fidelity One Shot Transfer). We take the FaceShifter (image-to-image) architecture as a baseline approach and propose several major architecture improvements which include a new eye-based loss function, face mask smooth algorithm, a new face swap pipeline for image-to-video face transfer, a new stabilization technique to decrease face jittering on adjacent frames and a super-resolution stage. In the experimental stage, we show that our solution outperforms SoTA face swap architectures in terms of ID retrieval (+1.5% improvement), shape (the second best value) and eye gaze preserving (+1% improvement) metrics. We also established an ablation study for our solution to estimate the contribution of pipeline stages to the overall accuracy, which showed that the eye loss leads to 2% improvement in the ID retrieval and 45% improvement in the eye gaze preserving.
ISSN:2169-3536
2169-3536
DOI:10.1109/ACCESS.2022.3196668