Subjective Evaluation of Visual Quality and Simulator Sickness of Short 360[Formula Omitted] Videos: ITU-T Rec. P.919

• Published in: IEEE transactions on multimedia 2022-01, Vol.24, p.3087
• Main Authors: Gutierrez, Jesus, Perez, Pablo, Orduna, Marta, Singla, Ashutosh, Cortes, Carlos, Mazumdar, Pramit, Viola, Irene, Brunnstrom, Kjell, Battisti, Federica, Cieplinska, Natalia, Juszka, Dawid, Janowski, Lucjan, Leszczuk, Mikolaj, Adeyemi-Ejeye, Anthony, Hu, Yaosi, Chen, Zhenzhong, Glenn Van Wallendael, Lambert, Peter, Diaz, Cesar, Hedlund, John, Hamsis, Omar, Fremerey, Stephan, Hofmeyer, Frank, Raake, Alexander, Cesar, Pablo, Carli, Marco, Garcia, Narciso
• Format: Article
• Language: English
• Subjects: Augmented reality; Coding; Display devices; Encoding; Helmet mounted displays; Illnesses; ITU; Laboratory tests; Media; Quality assessment; Quality of experience; Resists; Simulation; Video; Virtual reality
• ISSN: 1520-9210; 1941-0077; 1941-0077
• DOI: 10.1109/TMM.2021.3093717

Recently an impressive development in immersive technologies, such as Augmented Reality (AR), Virtual Reality (VR) and 360[Formula Omitted] video, has been witnessed. However, methods for quality assessment have not been keeping up. This paper studies quality assessment of 360[Formula Omitted] video from the cross-lab tests (involving ten laboratories and more than 300 participants) carried out by the Immersive Media Group (IMG) of the Video Quality Experts Group (VQEG). These tests were addressed to assess and validate subjective evaluation methodologies for 360[Formula Omitted] video. Audiovisual quality, simulator sickness symptoms, and exploration behavior were evaluated with short (from 10 seconds to 30 seconds) 360[Formula Omitted] sequences. The following factors’ influences were also analyzed: assessment methodology, sequence duration, Head-Mounted Display (HMD) device, uniform and non-uniform coding degradations, and simulator sickness assessment methods. The obtained results have demonstrated the validity of Absolute Category Rating (ACR) and Degradation Category Rating (DCR) for subjective tests with 360[Formula Omitted] videos, the possibility of using 10-second videos (with or without audio) when addressing quality evaluation of coding artifacts, as well as any commercial HMD (satisfying minimum requirements). Also, more efficient methods than the long Simulator Sickness Questionnaire (SSQ) have been proposed to evaluate related symptoms with 360[Formula Omitted] videos. These results have been instrumental for the development of the ITU-T Recommendation P.919. Finally, the annotated dataset from the tests is made publicly available for the research community.