SVP: Sinusoidal Viewport Prediction for 360-Degree Video Streaming

Virtual reality and 360-degree video streaming are growing rapidly; however, streaming 360-degree video is very challenging due to high bandwidth requirements. To address this problem, the video quality is adjusted according to the user viewport prediction. High quality video is only streamed for the user viewport, reducing the overall bandwidth consumption. Existing solutions use shallow buffers limited by the accuracy of viewport prediction. Therefore, playback is prone to video freezes which are very destructive for the Quality of Experience (QoE). We propose using layered encoding for 360-degree video to improve QoE by reducing the probability of video freezes and the latency of response to the user head movements. Moreover, this scheme reduces the storage requirements significantly and improves in-network cache performance.

The 360-degree video streaming has higher bandwidth requirements compared with traditional video to achieve the same user-perceived playback quality. Since users only view part of the entire videos, viewport-adaptive streaming is an effective approach to guarantee video quality. However, the performance of viewport-adaptive schemes is highly dependent on the bandwidth estimation and viewport prediction. To overcome these issues, we propose a novel reinforcement learning (RL) based viewport-adaptive streaming framework called RLVA, which optimizes the 360-degree video streaming in viewport prediction, prefetch scheduling and rate adaptation. Firstly, RLVA adopts t location-scale distribution rather than Gaussian distribution to describe the viewport prediction error characteristic more accurately and achieve the tile viewing probability based on the distribution. Besides, a tile prefetch scheduling algorithm is proposed to update the tiles according to the latest prediction results, which further reduces the adverse effect of prediction error. Furthermore, the tile viewing probabilities are treated as input status of RL algorithm. In this way, RL can adjust its policy to adapt to both of the network conditions and viewport prediction error. Through extensive evaluations, the simulation results show that the proposed RLVA outperforms other viewport-adaptive methods by about 4.8%-66.8% improvement of Quality of Experience (QoE) and effectively reduces the impact of viewport prediction errors.

The increasing popularity of virtual reality and 360-degree video streaming has created a demand for streaming high-quality video content over cellular networks. However, streaming 360-degree videos over cellular networks poses a challenge due to the need for ultra-low latency and high bandwidth. Although viewport adaptive streaming presents a promising solution to reduce the bandwidth requirement of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> video, it remains challenging to predict the Field of View (FoV) due to the adverse impact of prediction errors on the user's Quality of Experience (QoE). Moreover, bitrate adaptation gets complicated due to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> videos and the dynamic nature of the cellular network. Therefore, in this paper, we propose an intelligent framework, termed Seer, to fully realize <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> video streaming over cellular networks by leveraging Multi-access Edge Computing (MEC). It incorporates an FoV prediction scheme and a bitrate adaptation policy into the MEC platform. In particular, the FoV prediction scheme uses perceived video features coupled with time-series based viewport prediction to achieve higher prediction accuracy. For an effective bitrate selection in cellular network conditions, a Reinforcement Learning (RL) based bitrate adaptation policy is further employed. The results demonstrate that Seer's FoV prediction scheme delivers higher prediction accuracy, while the streaming framework significantly reduces the bandwidth requirements and improves the user's QoE.

Recently, the demand for immersive VR (360-degree) video delivery is increasing, and an efficient (high quality and low traffic) 360-degree video streaming methodology is mandatory. To address this fact, in this paper, we introduce a tile-based 360-degree DASH streaming with viewport prediction. We evaluate the prediction accuracy of future viewport movement patterns by using clustering and image and audio information. In addition, we perform rate adaptation based on viewport prediction results. Through evaluations, we confirm the tile-based 360-degree DASH streaming achieves higher objective image quality and lower total video traffic volume.

Smartphones have recently become a popular platform for deploying the computation-intensive virtual reality (VR) applications, such as immersive video streaming (a.k.a., 360-degree video streaming). One specific challenge involving the smartphone-based head mounted display (HMD) is to reduce the potentially huge power consumption caused by the immersive video. To address this challenge, we first conduct an empirical power measurement study on a typical smartphone immersive streaming system, which identifies the major power consumption sources. Then, we develop QuRate, a quality-aware and user-centric frame rate adaptation mechanism to tackle the power consumption issue in immersive video streaming. QuRate optimizes the immersive video power consumption by modeling the correlation between the perceivable video quality and the user behavior. Specifically, QuRate builds on top of the user's reduced level of concentration on the video frames during view switching and dynamically adjusts the frame rate without impacting the perceivable video quality. We evaluate QuRate with a comprehensive set of experiments involving 5 smartphones, 21 users, and 6 immersive videos using empirical user head movement traces. Our experimental results demonstrate that QuRate is capable of extending the smartphone battery life by up to 1.24X while maintaining the perceivable video quality during immersive video streaming. Also, we conduct an Institutional Review Board (IRB)-approved subjective user study to further validate the minimum video quality impact caused by QuRate.

In recent years, with the rapid development of virtual reality (VR), 360-degree panoramic video has grown up to become a popular Internet service. Streaming 360-degree video on today's Internet is challenging due to its ultra-high bandwidth and ultra-low motion-to-photon latency requirements. Although some VR solutions use tiling technology and Scalable Video Coding(SVC) encoding for 360-degree video transmission to reduce bandwidth consumption, it is still being challenging to transmit 360-degree video streams from remote content servers due to network latency. Caching popular video content at the edge of the network can decrease network latency and bandwidth consumption by reducing the number of future requests that have to be sent all the way to remote content servers. Because of the limited cache size, it is essential to decide which content to cache and how to replace the cache when the cache size is insufficient. In this paper, we propose an edge cache replacement strategy named Size-Popularity-Layer-FoV strategy(SPLF) for tile-based 360-degree video streaming using SVC. To improve caching performance, every video chunks' cache value is estimated according to its popularity, size, SVC layer and whether it is in FoV. Simulation results manifest that the proposed strategy can not only improve the hit rate and video quality compared to various baseline algorithms, but also decrease the playback stalls and the duration of rebuffering.

In the tile-based 360-degree video streaming, predicting user's future viewpoints and developing adaptive bitrate (ABR) algorithms are essential for optimizing user's quality of experience (QoE). Traditional single-user based viewpoint prediction methods fail to achieve good performance in long-term prediction, and the recently proposed reinforcement learning (RL) based ABR schemes applied in traditional video streaming can not be directly applied in the tile-based 360-degree video streaming due to the exponential action space. Therefore, we propose a sequential reinforced 360-degree video streaming scheme with cross-user attentive network. Firstly, considering different users may have the similar viewing preference on the same video, we propose a cross-user attentive network (CUAN), boosting the performance of long-term viewpoint prediction by selectively utilizing cross-user information. Secondly, we propose a sequential RL-based (360SRL) ABR approach, transforming action space size of each decision step from exponential to linear via introducing a sequential decision structure. We evaluate the proposed CUAN and 360SRL using trace-driven experiments and experimental results demonstrate that CUAN and 360SRL outperform existing viewpoint prediction and ABR approaches with a noticeable margin.

360-degree video streaming is an important component of virtual reality. A video streaming server stores video streams with various quality levels. These streams are fragmented into several video segments. In 360-degree video streaming, these segments are tiled into video tiles. In general, the client changes the viewport quality level based on predictions of the user’s viewport and network throughput. In named data networking (NDN), the throughput measured at the client is not always stable due to network-inherent caching. Therefore, the viewport video quality changes (referred to as quality switching) frequently in 360-degree video streaming over NDN. In this paper, we propose buffer-based adaptation using scalable video coding for 360-degree video streaming over NDN. In the proposed method, a buffer-based adaptation algorithm and the prefetching of video tiles are used to avoid video quality switching. A simple performance evaluation using computer simulations shows that the proposed method reduces the quality switching compared with that for a rate-based adaptation algorithm.

Virtual reality (VR) video streaming and 360◦ panoramic video have received extensive attention in recent years, which can bring users an immersive experience. However, the ultra-high bandwidth and ultra-low latency requirements of virtual reality video or 360◦ panoramic video also put tremendous pressure on the carrying capacity of the current network. In fact, since the user’s field of view (a.k.a viewport) is limited when watching a panoramic video and users can only watch about 20%∼30% of the video content, it is not necessary to directly transmit all high-resolution content to the user. Therefore, predicting the user’s future viewing viewport can be crucial for selective streaming and further bitrate decisions. Combined with the tile-based adaptive bitrate (ABR) algorithm for panoramic video, video content within the user’s viewport can be transmitted at a higher resolution, while areas outside the viewport can be transmitted at a lower resolution. This paper mainly proposes a viewport-driven adaptive 360◦ live streaming optimization framework, which combines viewport prediction and ABR algorithm to optimize the transmission of live 360◦ panoramic video. However, existing viewport prediction always suffers from low prediction accuracy and does not support real-time performance. With the advantage of convolutional network (CNN) in image processing and long short-term memory (LSTM) in temporal series processing, we propose an online-updated viewport prediction model called LiveCL which mainly utilizes CNN to extract the spatial characteristics of video frames and LSTM to learn the temporal characteristics of the user’s viewport trajectories. With the help of the viewport prediction and ABR algorithm, unnecessary bandwidth consumption can be effectively reduced. The main contributions of this work include: (1) a framework for 360◦ video transmission is proposed; (2) an online real-time viewport prediction model called LiveCL is proposed to optimize 360◦ video transmission combined with a novel ABR algorithm, which outperforms the existing model. Based on the public 360◦ video dataset, the tile accuracy, recall, precision, and frame accuracy of LiveCL are better than those of the latest model. Combined with related adaptive bitrate algorithms, the proposed viewport prediction model can reduce the transmission bandwidth by about 50%.

Viewport-adaptive streaming is now a widely embraced solution for transmission of 360-degree video, where users perceptual quality after their head movement suffers from the temporary exposure of a low-quality area, as well as the related delay in quality refinement. In this article, subjective experiments were conducted to investigate the humans perception of quality refinement in viewport-adaptive 360-degree video streaming, considering the saccadic suppression, foveal vision, and contrast sensitivity of the human visual system. According to the experimental results, a measurement of the Just Noticeable Refinement Delay is proposed to describe the delay when quality refinement is just noticeable. Based on the findings in perception analysis, a real-time adaptive 360-degree video streaming system using the Random Access Reference Frame is proposed, which can dramatically reduce the refinement delay to enhance users quality of experience. Experimental results demonstrated that the proposed system provide better QoE to users compared to the traditional viewport-adaptive 360-degree video streaming system.

Tile-based 360-degree video (360 video) streaming, employed with adaptive bitrate (ABR) algorithms, is a promising approach to offer high video quality of experience (QoE) within limited network bandwidth. Existing ABR algorithms, however, fail to achieve optimal performance in real-world fluctuated network conditions as they heavily rely on unbiased bandwidth predictions. Recently, reinforcement learning (RL) has shown promising potential in generating better ABR algorithms in 2D video streaming. However, unlike existed work in 2D video streaming, directly applying RL in the tile-based 360 video streaming is infeasible due to the resulting exponential decision space. To overcome these limitations, we propose in this paper 360SRL, an improved ABR algorithm employing Sequential RL (360SRL). Firstly, we reduce the decision space of 360SRL from exponential to linear by introducing a sequential ABR decision structure, thus making it feasible to be employed with RL. Secondly, instead of relying on accurate bandwidth predictions, 360SRL learns to make ABR decisions solely through observations of the resulting QoE performance of past decisions. Finally, we compare 360SRL to state-of-the-art ABR algorithms using trace-driven experiments. The experiment results demonstrate that 360SRL outperforms state-of-the-art algorithms with around 12% improvement in average QoE.

With the continuous advancement of computer vision, image processing technologies, volumetric video, represented by point cloud videos, holds the potential for extensive applications in areas such as Virtual Reality (VR) and Augmented Reality (AR). Viewport prediction, also referred to as Field of View (FoV) prediction, is a crucial component in emerging VR and AR applications, playing a vital role in the transmission of point cloud videos. Currently, models for viewpoint prediction that integrate feature extraction and FoV information heavily rely on the spatial-temporal features extracted by convolutional neural networks. However, the drawback of 3D convolution lies in its inability to effectively capture long-term spatial-temporal dependencies within videos. Moreover, the temporal contrast layer used for time feature extraction only compares features within each block, leading to matching errors and inaccurate temporal feature extraction, consequently diminishing predictive performance. To address these limitations, we propose a Transformer-based Volumetric Point Cloud Video Viewport Prediction Network (VPFormer) that can efficiently extract spatial-temporal features from point cloud videos. VPFormer constitutes a viewport prediction framework that combines the spatial-temporal features of point cloud videos with user trajectory information. Specifically, we introduce a novel sampling method that effectively preserves spatial-temporal information while reducing computational complexity. Additionally, we incorporate context-aware dynamic positional encoding to capture inter-frame spatial-temporal context information. Subsequently, we introduce a voxel-based temporal contrast layer and partition the point cloud into smaller voxel blocks during feature matching, significantly reducing matching errors and enhancing the analysis and extraction of temporal features. Finally, by combining the spatial-temporal features of point cloud videos with user head trajectory information, we successfully predict future user viewpoints. Experimental results demonstrate that this approach outperforms other solutions in terms of performance.

Wireless 360-degree video delivery provides virtual reality (VR) immersive experience where each user can freely switch his/her viewing orientation. The existing schemes of the wireless 360-degree video streaming use digital-based compression and transmission. However, they have many disadvantages in terms of video traffic and quality: cliff effect due to unstable wireless channels, large video traffic due to the extremely high resolution of the 360-degree video, and the perceptual redundancy within the transmitted video. To solve the above problems, this paper proposes a novel wireless 360-degree video transmission scheme called 360Cast. 360Cast adopts the analog-based transmission, including power allocation and analog modulation, to achieve graceful video quality improvement even in time-varying wireless channels. Here, the power allocation considers the distortion of human perception and sphere-to-plane projection to maximize the human perceptual quality at the 360-degree video playback. In addition, 360Cast predicts and transmits the user's viewport based on the recent HMD user's orientation by using dynamic linear regression (DLR) and only transmits the viewport for traffic reduction. Evaluation results show that the proposed 360Cast provides better human perceptual quality via HMD compared with the existing analog transmission schemes, i.e., SoftCast and FoveaCast, by using the integration of viewport prediction, power allocation, and analog modulation.

360-degree video provides the users with immersive experience of the designated scene. However, the challenge is that delivery of the full high-quality 360-degree videos may exhaust the bandwidth leading to degraded user experience. Moreover, unreasonable number of tiles results in uneven distribution of resources during transmitting. In this paper, we propose a Two-Stage deep reinforcement learning system for 360-degree video streaming, named TS360, which determines the division of tiles and the quality allocation in two successive stages. In the tile division stage, the tiles are divided according to the viewing region weights so that the bandwidth can be utilized more efficiently in the quality allocation stage. On the other hand, we employ the Asynchronous Advantage Actor-Critic (A3C) with the aid of ResNet for optimizing the Quality of Experience (QoE) and the bandwidth efficiency. Simulation results compared with the state-of-the-art benchmarkers indicate that TS360 increases the bandwidth efficiency by 56% while improving the user QoE by 10% on average.

Interactive applications that come along with 360- degree (or 360) videos have brought immersive experiences to users thanks to the elevated machine computability. In fact, the provision of such high quality of experience (QoE) hinges on the persistent delivery of 360 videos, potentially consuming an excessive need of network bandwidth. To prevent the delivery of entire 360 videos from adversely affecting QoE, tile-based viewport adaptive streaming that divides 360 video chunks into tiles and conveys streams with differentiated quality levels to viewport and non-viewport areas has been regarded as a promising solution. Existing works have been devoted to the design of 1) viewport prediction (VPP) to predict users' viewport orientation due to head movements, and 2) tile bitrate selection (TBS) to determine tile-based bitrates for viewport and non-viewport areas. Despite the heuristic solutions proposed by the existing works, there is lack of knowledge of whether QoE can be enhanced by learning from historical data. In this paper, we propose the system-Plato, to leverage machine learning to tile-based viewport adaptive streaming for 360 videos. In particular, Plato applies long short term memory (LSTM) model to VPP, and uses part of non-viewport areas to help resist prediction errors. In addition, Plato uses real-world traces to train a TBS agent based on reinforcement learning to determine tile bitrates for both viewport and non-viewport areas. Our simulation results show that Plato outperforms existing schemes in various QoE metrics.