A Subjective Study on User Perception Aspects in Virtual Reality

Currently, we witness dramatically increasing interest in immersive media technologies like Virtual Reality (VR), particularly in omnidirectional video (OV) streaming. Omnidirectional (also called 360-degree) videos are panoramic spherical videos in which the user can look around during playback and which therefore can be understood as hybrids between traditional movie streaming and interactive VR worlds. Unfortunately, streaming this kind of content is extremely bandwidth intensive (compared to traditional 2D video) and therefore, Quality of Experience (QoE) tends to deteriorate significantly in absence of continuous optimal bandwidth conditions. In this paper, we present a first approach towards subjective QoE assessment for omnidirectional video (OV) streaming. We present the results of a lab study on the QoE impact of stalling in the context of OV streaming using head-mounted displays (HMDs). Our findings show that subjective testing for immersive media like OV is not trivial, with even simple cases like stalling leading to unexpected results. After a discussion of characteristic pitfalls and lessons learned, we provide a a set of recommendations for upcoming OV assessment studies.

In this study, the perceptual video quality (PVQ) for the vogue 360-degree video with compression artefacts viewed on the visual reality (VR) head-mounted display (HMD) is obtained for the first time by an elaborately designed subjective experiment. The characteristic of the PVQ of omnidirectional (360-degree) video on the VR HMD and PC monitor is then investigated. The PVQ on the HMD is found to be linearly related to the video coding quality (VCQ) on the PC monitor. A quality evaluation model is then proposed based on the mapping formula and a new assessment parameter, where the impact of the video resolution and display device is involved. In this way, the traditional video quality assessment metrics for two-dimensional video can be extended to assess the PVQ of the 360-degree video viewed on the HMD. At last, the video structural similarity metric is taken as an example to predict the input of the proposed model, i.e. the VCQ of the 360-degree video. Experimental results demonstrate that the proposed model can effectively and conveniently solve the challenge of PVQ assessment for the 360-degree video with compression distortions on the HMD.

The user visual attention has been playing an imperative role in visual Quality of Experience (QoE) modeling. In recent years, researchers have developed numerous visual attention models for images and videos. However, existing studies on visual attention generally perform on spatial domain (i.e. pictures) without consideration on time-variant user attention. Besides, the user attention on immersive video playback has not been well investigated in Virtual Reality (VR) environment. In this paper, we design subjective experiments to acquire the user attention during 360-degree video playback, in order to characterize the visual attention changes on time axis. We compare and model the temporal attentions in two scenarios: 360-degree video and conventional 2D video display. It is observed in video display, the user attention needs a short period (~ 30 seconds) to concentrate to a relatively stable level. Within this period, the user attention to 360-degree video is significantly lower than that to traditional 2D videos, which shows that the users are relatively difficult to adapt to VR display. Therefore, short videos less than 30 seconds, especially short 360-degree videos, are not preferable in subjective video quality assessment, although they have been tested in some scenarios. We hope this conclusion would conduce to QoE modeling during video display of VR environment.

Recently, virtual reality (VR) has become popular in many fields. Users can view a 360-degree video in all directions and have an immersive experience with a VR device. However, the transmission of a 360-degree video, which is compared to a traditional video, is a significant burden on wireless networks. In other words, users may suffer poor quality of experience (QoE) under constrained power. In order to improve transmission efficiency and ensure the QoE of users, in this paper, we propose an adaptive transmission strategy (ATS) for resource allocation in 360-degree videos to multiple users using a non-orthogonal multiple access (NOMA) scheme. In ATS, the delivery mode (multicast or unicast) of tiles is based on the users' field of views (FoVs) and their channel conditions, and the quality of tiles is dynamically determined by their priorities. The simulation results show that ATS significantly improves the QoE of users under constrained power.

Recently, we have seen an emergence of affordable Head Mounted Displays (HMD) such as the Oculus Rift, HTC Vive, and the PS4 Project Morpheus which allow users to experience 3D virtual reality (VR). These types of hardware aim to facilitate new and novel experiences for users above and beyond what is possible with traditional audiovisual displays. However, a very limited number of studies exist in the literature to determine the influence of these technologies on user Quality of Experience (QoE). In order to evaluate QoE as users consume VR content, this paper proposes the use of affordable consumer electronics to capture objective physiological metrics: Heart Rate (HR) and ElectroDermal Activity (EDA). Our findings indicate different HR and EDA dependent on VR and non-VR environments. Additionally, we examine the relationship between these objective metrics and user QoE captured via a post-test questionnaire. To the best of the authors knowledge, this is the first work which demonstrates a tangible relationship between the EDA/HR combination and user QoE of immersive VR environments.

Virtual Reality (VR) has attracted widespread attention in recent years due to its capability to create immersive experiences by presenting multi-modal information to users. Omnidirectional videos (ODVs), as a prominent component of VR content, are essential across diverse applications. This necessitates service providers to monitor and optimize the quality of ODVs throughout the filming, encoding, decoding, and transmission stages to ensure a high-quality viewing experience. However, most existing Quality of Experience (QoE) studies for ODVs only focus on the visual quality, while overlooking the impact of the audio modality on perceptual quality. This paper presents a comprehensive study of omnidirectional audio-visual quality assessment (OD-AVQA) from both subjective and objective perspectives. Specifically, we first establish a large-scale audio-visual quality assessment database for ODVs named OAVQAD+, which includes 625 distorted omnidirectional audio-visual sequences derived from 25 pristine ODVs, and the corresponding collected mean opinion scores (MOSs) for the QoE of these ODVs. This contributes to the largest database for assessing the audio-visual quality of ODVs. To advance the fields of objective OD-AVQA, we construct a benchmark that includes three types of benchmark models. Type I and Type II models integrate well-known video quality assessment (VQA) and audio quality assessment (AQA) methods using support vector regression (SVR) and multi-layer perceptron (MLP), respectively, while Type III consists of AVQA models specifically designed for traditional 2D audio-visual sequences. We also propose a novel Omnidirectional Audio-Visual quality assessment Network (OmniAVNet) that integrates quality-aware audio, visual, and motion features to predict overall audio-visual quality for ODVs effectively, which supports both full-reference (FR) and no-reference (NR) assessment. Extensive experimental results demonstrate that OmniAVNet outperforms the aforementioned benchmark OD-AVQA models on two OD-AVQA databases, and shows great performance on one omnidirectional VQA database. The database and code are available at https://github.com/IntMeGroup/OmniAVNet.

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Recent advancements in extended reality (XR) showcase their potential ability to enhance the user experience of traditional displays such as liquid crystal displays and organic light‐emitting diode screens. To achieve this goal, the upcoming generation of head‐mounted displays (HMDs) necessitates a seamless transition between different XR modes, including augmented reality (AR) and virtual reality (VR) modes. Herein, an innovative cholesteric liquid crystal‐ (CLC‐) based optical combiner for HMDs is introduced. This approach enables the display device to switch between AR, VR, and transparent modes via temperature modulation. Since CLC materials demonstrate varying optical properties at different temperatures, a real‐time temperature monitoring system that is paired with an external controller is integrated. This allows for dynamic temperature adjustments of the optical combiner and enables smooth transitions between display modes. Furthermore, the Berreman 4 × 4 matrix method is used to simulate the optical properties of the optical combiner, finding strong agreement with the experimental results. The performance of the tunable optical combiners using 450, 532, and 635 nm laser sources is explored. The findings demonstrate that CLC‐based optical combiners can effectively switch between the three distinct modes at corresponding temperatures, paving the way for versatile applications in future HMDs.

Counteracting economic challenges in the network industry partially resulting from enormous traffic growth rates and the absence of successful Quality of Service (QoS) mechanisms in practice, we propose a novel concept centering price and quality differentiation on customers' quality perceptions, i.e., Quality of Experience (QoE). By fundamentally building on the known Paris Metro Pricing (PMP) concept we iteratively integrate pricing mechanisms with QoE aspects. Our concept also cardinally incorporates the interests of users, Internet Service Providers (ISPs) offering Internet access, and regulatory bodies-e.g., controlling the fraction of premium service class capacity. Our findings demonstrate the economic desirability for ISPs to apply price discrimination based on QoE considerations. The inclusion of QoE aspects in the traffic classification process, moreover, yields a significant increase of user utilities, while further stimulating the overall demand. This may also prepare the scene for involvements in the Net Neutrality (NN) debate.

With the development and popularity of virtual reality (VR) and 360 panoramic videos with head mounted display (HMD), user satisfaction is very significant as compared to other important end to end quality of service. To improve Quality of Experience (QoE) of 360 panoramic video, user profile needs to be analyzed. The user profile analysis can provide vital information to service providers which can help them to deal with their resources capably, by evaluating users behaviour and expectation. In this paper, we conduct a subjective test and propose a method to investigate QoE in VR by analyzing user profile. To do this, we first store the user profile data (e.g. age, gender, familiarity with 360 panoramic VR video, interest in the content) then we investigate the impact of different factors that influence QoE (encoding parameters, content type and device type) on user. The results of our analytical study provide another opportunity for content providers to offer best level service that meets customers expectations and desired experience to obtain high user satisfaction. To the best of our knowledge, this is the first attempt that analyzes the user profile for 360 panoramic videos in VR environment.

The Development of Virtual Reality as a Tool to Investigate Eating Behavior

The head-mounted display (HMD) for 360-degree videos cannot be shared by multiple users, which significantly increases the bandwidth consumption when multiple users request the same video content simultaneously. This paper proposes a server-side hybrid multicast-unicast strategy for multi-user adaptive 360-degree video streaming, aiming to ensure the overall quality of experience (QoE) of the users in a bandwidth-constrained environment. A framework is established to realise this strategy, through which the delivery mode (multicast or unicast) and the corresponding bitrate for each tile can be jointly adapted to the dynamics of users’ throughputs and field of views (FoVs). To maximize the overall QoE, a user clustering method is proposed to classify the users into several multicast clusters. Based on these clusters, we then formulate the joint delivery mode selection and rate adaptation problem as a non-linear integer programming problem which can be solved using the steepest ascent gradient algorithm. Finally, experiments are carried out to verify the performance of the proposed strategy.

There is a growing interest nowadays of using immersive technologies to promote Cultural Heritage (CH), engage and educate visitors, tourists and citizens. Such examples refer mainly to the use of Virtual Reality (VR) technology or focus on the enhancement of the real world by superimposing digital artefacts, so called Augmented Reality (AR) applications. A new medium that has been introduced lately as an innovative form of experiencing immersion is the 360-degree video, imposing further research challenges. This paper presents a study on the creation of an immersive application that is based on 360-degree video and targets the users’ education in the importance of CH. The user is engaged in a journey where they learn about preserved artefacts in the city of Rethymno, Greece and their historical value in a playful way. The integrated architecture and development steps for creating such applications that are based on 360- egree video are demonstrated. The overall design process is driven by the goal to increase the level of immersion offered by incorporating game elements in a realistic storytelling experience. The final application can be experienced with the use of a VR low-cost device such as the Google cardboard and a compatible mobile phone. The study concludes with a set of recommendations and future directions for offering more immersive experiences in the CH sector.

In this paper, we describe a new telepresence system which enables a user to look around a virtualized real world easily in network environments. The proposed system includes omni-directional video viewers on web browsers and allows the user to look around the omni-directional video contents on the web browsers. The omni-directional video viewer is implemented as an Active-X program so that the user can install the viewer automatically only by opening the web site which contains the omni-directional video contents. The system allows many users at different sites to look around the scene just like an interactive TV using a multi-cast protocol without increasing the network traffic. This paper describes the implemented system and the experiments using live and stored video streams. In the experiment with stored video streams, the system uses an omni-directional multi-camera system for video capturing. We can look around high resolution and high quality video contents. In the experiment with live video streams, a car-mounted omni-directional camera acquires omni-directional video streams surrounding the car, running in an outdoor environment. The acquired video streams are transferred to the remote site through the wireless and wired network using multi-cast protocol. We can see the live video contents freely in arbitrary direction. In the both experiments, we have implemented a view-dependent presentation with a head-mounted display (HMD) and a gyro sensor for realizing more rich presence.

Virtual Reality (VR) has expanded from the training and high-end entertainment world into the homes of everyday users through mobile head mounted displays (HMDs). Currently, several different interaction methods for mobile VR HMDs exist, including remotes, external controllers, motion controllers, touch, and free form hand tracking. The goal of this study was to evaluate the impact that these interaction methods have on specifically mobile VR HMD user experience, including performance, user preference, and enjoyment. It was hypothesized that interaction methods would vary significantly in enjoyment, preference, and performance as many factors such as ease of use and familiarity can influence these factors. To evaluate the interaction methods, participants performed trials of a matching task utilizing menus in a VR environment in which the participants selected the features of an object (e.g., shape, color, pattern) from a menu to the right of the object. The Leap Motion had usability issues that impacted it’s performance and enjoyment scores resulting in it being the poorest performer of the group. Overall the gamepad was most preferred and performed the best due to its ease of use and familiarity, but the touchpad and leap motion offered immersive aspects. These results highlight the importance of ease of use in interaction methods and the impact it has on overall user experience. This paper will discuss these results and implications for design of interaction methods and associated VR HMD controllers.