Viewport-adaptive 360-degree video coding

360-degree videos contain an omnidirectional view with ultra-high resolution, which will lead to the bandwidth-hungry issue in virtual reality (VR) applications. However, only a part of a 360-degree video is displayed on the head-mounted displays (HMDs). Thus, we propose a viewport-adaptive 360-degr...

Full description

Saved in:
Bibliographic Details
Published in:Multimedia tools and applications 2020-05, Vol.79 (17-18), p.12205-12226
Main Authors: Hu, Qiang, Zhou, Jun, Zhang, Xiaoyun, Shi, Zhiru, Gao, Zhiyong
Format: Article
Language:English
Subjects:
Artificial neural networks
Coders
Coding
Computer Communication Networks
Computer Science
Data Structures and Information Theory
Feature extraction
Helmet mounted displays
Lagrange multiplier
Multimedia Information Systems
Optimization
Prediction models
Scaling factors
Special Purpose and Application-Based Systems
Three dimensional models
Virtual reality
Weight
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:360-degree videos contain an omnidirectional view with ultra-high resolution, which will lead to the bandwidth-hungry issue in virtual reality (VR) applications. However, only a part of a 360-degree video is displayed on the head-mounted displays (HMDs). Thus, we propose a viewport-adaptive 360-degree video coding approach based on a novel viewport prediction strategy. Specifically, we firstly introduce a novel viewport prediction model based on deep 3-dimensional convolutional neural networks. In this model, a video saliency encoder and a trajectory encoder are trained to extract the features of video content and the history view path. With the outputs of the two encoders, a video prior analysis network is trained to adaptively determine the best fusion weight to generate the final feature. Moreover, benefiting from the viewport prediction model, a viewport-adaptive rate-distortion optimization (RDO) method is presented to decrease the bitrate and ensure an immersive experience. In addition, we also consider the scaling factor of the area from rectangular plane to spherical surface. Therefore, the Lagrange multiplier and quantization parameter are adaptively adjusted based on the weight of each coding tree unit. The experiments have demonstrated that the proposed RDO method gains considerably better RD performance than the traditional RDO method.
ISSN:1380-7501
1573-7721
DOI:10.1007/s11042-019-08390-7