Quality assessment of view synthesis based on unsupervised quality-aware pre-training

• Published in: Applied soft computing 2024-03, Vol.154, p.111377, Article 111377
• Main Authors: Shi, Haozhi, Huang, Yipo, Wang, Lizhe, Wang, Lanmei
• Format: Article
• Language: English
• Subjects: Domain adaptive; Quality assessment; Unsupervised pre-training; View synthesis
• ISSN: 1568-4946; 1872-9681
• DOI: 10.1016/j.asoc.2024.111377

The current view synthesis quality metrics mainly rely on hand-crafted features, which have clear physical meanings but fail to comprehensively describe the complex distortion characteristics in synthesized images. With the rapid advancement of deep models, convolutional neural networks hold great promise for learning complex distortion representations. The exceptional performance of deep learning is closely tied to the availability of abundant labeled training data. However, manual annotation of labels for quality assessment tasks is arduous, which impede the application of deep learning in this area. With this motivation, this paper presents a no-reference quality assessment model for view synthesis based on unsuperviseD quality-Aware Pre-Training (DAPT). Specifically, a two-stream network with spatial destruction sensitivity and adaptive heterogeneous awareness branches is firstly designed, and then the two branch networks are pre-trained unsupervised to fully extract the quality-aware feature representations. Finally, a multi-layer perceptron is utilized to generate quality scores based on the spatial domain destruction and structural damage information. Notably, to better align the quality-aware features learned through unsupervised pre-training in the source domain with those of the target domain, we introduce a domain adaptive module in the adaptive heterogeneous awareness branch. Extensive experiments demonstrate that the proposed DAPT model outperforms the state-of-the-arts. •Unsupervised quality-aware pre-training is used to improve performance.•Domain adaptation module helps the encoder to extract domain-invariant features.•Experiments demonstrate the significant advantages of the proposed model.