Assessment of Visual Quality and Simulator Sickness for Omnidirectional Videos

In this book, first, the conventional subjective test methods such as Absolute Category Rating (ACR) (ITU-R BT.500-13. Recommendation ITU-R BT.500-13: Methodology for the subjective assessment of the quality of television pictures, 2012) and Double Stimulus Impairment Scale (DSIS) (ITU-T Rec. P.910. Recommendation ITU-T P.910: Subjective video quality assessment methods for multimedia applications, 2008) were applied for the assessment of visual quality, and a novel test method Modified-ACR (M-ACR) (A. Singla et al. (Comparison of subjective quality evaluation for HEVC encoded omnidirectional videos at different bit-rates for UHD and FHD resolution, in Proceedings of the on Thematic Workshops of ACM Multimedia. Mountain View, California, USA, 2017) is proposed during the book research. A methodological framework for 360\({ }^\circ \) video quality assessment is presented in this book. Second, simulator sickness is assessed to investigate the influence of different factors, such as video resolution, bitrate, session duration, and latency on the scores of simulator sickness. A simplified simulator sickness questionnaire is then proposed to assess the symptoms of simulator sickness and is more directed towards 360\({ }^\circ \) video. Lastly, a correlation between presence, media quality, and simulator sickness is shown.

Measuring and validating QoE of omnidirectional videos using different HMDs have become increasingly important. The resulting user experience plays an important role in making technology successful and acceptable for a wide user group. The respective “QoE” is defined as the “…degree of delight or annoyance of the user of an application or service…” (P. Le Callet et al. (Qualinet white paper on definitions of quality of experience, in European Network on Quality of Experience in Multimedia Systems and Services (COST Action IC 1003) 3.2012, 2012), ITU–T Rec. P.10/G.100, Vocabulary for Performance and Quality of Service. International Telecommunication Union. Geneva, 2017). One component of QoE is the perceptual media quality of the viewing session, resulting from different technical characteristics of the end-to-end chain comprising the recording, post-processing, representation, coding, retrieval, transmission, playback, and display of 360\({ }^\circ \) videos. Furthermore, for 360\({ }^\circ \) video, exploration behavior, presence, and simulator sickness also represent key aspects of QoE. Therefore, besides media quality, these three further constructs need to be assessed for a holistic understanding of QoE in the case of omnidirectional videos.

This chapter describes the pipeline of the 360\({ }^\circ \) video testing system and presents the details of the source sequences and processing used for the subjective tests conducted as part of this book. The source sequences include self-produced videos, in addition to sources obtained from different open as well as proprietary databases. Further, this chapter provides a brief overview of the technical environment such as HMDs, the PC, and the framework for recording behavioral data.

Different test methods have been standardized for assessing video quality of 2D videos, e.g., ITU-T Rec. P.910 (Recommendation ITU-T P.910: subjective video quality assessment methods for multimedia applications, 2008) and ITU-R Rec. BT.500-13 (Recommendation ITU-R BT.500-13: methodology for the subjective assessment of the quality of television pictures, 2012). Due to the lack of a standardized method to assess the visual quality for 360\({ }^\circ \) videos, studies have often used ITU-T Rec. P.910 (Recommendation ITU-T P.910: Subjective video quality assessment methods for multimedia applications, 2008) and ITU-R Rec. BT.500-13 (Recommendation ITU-R BT.500-13: methodology for the subjective assessment of the quality of television pictures, 2012) to evaluate the visual quality of 360\({ }^\circ \) videos. ITU-T P.919 (Subjective test methodologies for 360° video on head-mounted displays. Recommendation P.919, 2020) was the first standardized recommendation on subjective test methods for 360\({ }^\circ \) video – with contributions from the author. In this chapter, two different experiments are described and conducted in two different contexts in a lab setting. In the first experiment, the results of two different subjective test methods, M-ACR (Test 2) and DSIS (Test 3), were compared in two different contexts; namely, the M–ACR test was conducted in winter and the DSIS test in summer. In experiment 2, the results of three different subjective test methods, ACR (Test 4), M-ACR (Test 5), and DSIS (Test 6), were compared. These tests were conducted within a period of three weeks, thus having no difference in context in terms of seasons. Based on the reliability, test time, and general applicability of the procedure across different tests, a methodological framework for 360\({ }^\circ \) video quality assessment is presented in this chapter.

In the last few years, with the advancement in display and rendering technologies, the consumption of VR applications is increasing, with \(360^\circ \) videos being an example. Increasing numbers of higher-quality videos can now be watched on smartphones or HMDs using platforms such as YouTube. The overall QoE of a \(360^\circ \) viewing session depends on the degree of immersion or presence of the user, the degree of simulator sickness he/she experiences, and the media (audio and video) quality, which also interact with each other. Therefore, assessing the quality of \(360^\circ \) videos is an essential step in improving overall QoE. In this chapter, the impact of HMDs, encoding settings (resolution and bitrate), voting methods, and different delay values (for tile-based streaming) are discussed related to the perceived video quality for \(360^\circ \) videos. Furthermore, conclusions are drawn from the analysis, and recommendations are provided on the improvement of the perceived visual quality.

While viewing \(360^\circ \) videos on an HMD, users may experience symptoms of simulator sickness (or cybersickness) such as nausea, dizziness, vertigo, sweating, etc. (Kennedy et al (1993) Int J Aviat Psychol 3(3):203–220). Simulator sickness may occur during and after exposure to a Virtual Environment (VE). Hence, for a holistic picture of omnidirectional video QoE, simulator sickness should be monitored during respective tests. For the self-evaluation of simulator sickness, various long and short simulator sickness questionnaires were discussed in Sect. 2.5.2. In the following, the terms “Cybersickness” and “Simulator Sickness” will be used interchangeably. In this chapter, the main aim is to investigate the impact of different influencing factors such as video resolution, bitrate, session duration, and latency on cybersickness. At present, practically no solution exists that can efficiently eradicate the symptoms of simulator sickness from virtual environments. Therefore, in the absence of a solution, it is required to quantify and predict the amount of sickness associated with a given VE, or as investigated in this book, \(360^\circ \) video viewing. Hence, another objective of this chapter is to present initial work on a Simulator Sickness Model SiSiMo, including a first component to predict simulator sickness scores over time. Using linear regression of short-term scores already shows promising performance for predicting the scores collected from a number of user tests.

\(360^\circ \) videos provide an immersive experience to users. Besides this, \(360^\circ \) videos may lead to an undesirable effect when consumed with HMDs, referred to as simulator sickness/cybersickness. The SSQ is the most widely used questionnaire for the assessment of simulator sickness. Since the SSQ with its 16 questions was not designed for \(360^\circ \) video related studies, the research hypothesis was that it may be simplified to enable more efficient testing for \(360^\circ \) video. Hence, the SSQ was evaluated to reduce the number of questions asked from subjects, based on six different previously conducted studies. The reduced set of questions from the SSQ are derived using Principal Component Analysis (PCA) for each test. Pearson Correlation is analyzed to compare the relation of all obtained reduced questionnaires as well as two further variants of SSQ reported in the literature, namely VRSQ (Virtual Reality Sickness Questionnaire) and CSQ (Cybersickness Questionnaire). The analysis suggests that a reduced questionnaire with 9 out of 16 questions yields the best agreement with the initial SSQ, with less than 44% of the initial questions. Exploratory Factor Analysis (EFA) shows that the nine symptom-related attributes determined as relevant by PCA also appear to be sufficient to represent the three dimensions resulting from EFA, namely, Uneasiness, Visual Discomfort, and Loss of Balance. The simplified version of the SSQ has the potential to be more efficiently used than the initial SSQ for \(360^\circ \) video by focusing on the questions that are most relevant for individuals, shortening the required testing time.

QoE for omnidirectional videos comprises additional components such as simulator sickness and presence. Besides media quality, also presence and simulator sickness need to be assessed for a holistic understanding of QoE. In this chapter, a series of tests is presented comparing different test protocols to assess integral quality, simulator sickness, and presence for omnidirectional videos in one test run, using the HTC Vive Pro as the head-mounted display. For quality ratings, the five-point ACR scale was used. In addition, the well-established SSQ and Presence Questionnaire (PQ) methods were used, once in a full version, and once with only one single integral scale, to analyze how well presence and simulator sickness can be captured using only a single scale. Furthermore, the inter-relation between the ratings for quality, presence, and simulator sickness is investigated.

This book presents extensive research on the assessment of the quality of \(360^{\circ }\) video perceived by users using HMDs. Secondly, simulator sickness is analyzed to investigate the impact of different influencing factors that may affect comfort among users. Additionally, a simplified Simulator Sickness Questionnaire (SSQ) with nine questions instead of the original 16 questions is proposed for the self-reporting of symptoms. Finally, the suitability of short, single-scale versions of the simulator sickness and presence questionnaires was investigated in comparison to the long versions. Also, the relationship between integral quality, simulator sickness, and presence was studied. The following sections list the contribution of the book and provide recommendations for future research.