Compression techniques for active video content

Detecting actions in videos have been widely applied in on-device applications, such as cars, robots, etc. Practical on-device videos are always untrimmed with both action and background. It is desirable for a model to both recognize the class of action and localize the temporal position where the action happens. Such a task is called temporal action location (TAL), which is always trained on the cloud where multiple untrimmed videos are collected and labeled. It is desirable for a TAL model to continuously and locally learn from new data, which can directly improve the action detection precision while protecting customers’ privacy. However, directly training a TAL model on the device is nontrivial. To train a TAL model which can precisely recognize and localize each action, tremendous video samples with temporal annotations are required. However, annotating videos frame by frame is exorbitantly time consuming and expensive. Although weakly supervised temporal action localization (W-TAL) has been proposed to learn from untrimmed videos with only video-level labels, such an approach is also not suitable for on-device learning scenarios. In practical on-device learning applications, data are collected in streaming. For example, the camera on the device keeps collecting video frames for hours or days, and the actions of nearly all classes are included in a single long video stream. Dividing such a long video stream into multiple video segments requires lots of human effort, which hinders the exploration of applying the TAL tasks to realistic on-device learning applications. To enable W-TAL models to learn from a long, untrimmed streaming video, we propose an efficient video learning approach that can directly adapt to new environments. We first propose a self-adaptive video dividing approach with a contrast score-based segment merging approach to convert the video stream into multiple segments. Then, we explore different sampling strategies on the TAL tasks to request as few labels as possible. To the best of our knowledge, we are the first attempt to directly learn from the on-device, long video stream. Experimental results on the THUMOS’14 dataset show that the performance of our approach is comparable to the current W-TAL state-of-the-art (SOTA) work without any laborious manual video splitting.

An innovative Internet streaming video player, we call ePlayer, that supports personalised camera switching for live events has been researched, developed and evaluated. The main novelty of the system is that in dispensing the available bandwidth amongst multiple input video streams it can automatically coordinate and preserve the video quality across multiple live video streams. An additional novelty of the system is that it supports automatic camera switching based upon an individual user's preferences. The experimental results indicate that the system is able to effectively allocate a contested bandwidth resource amongst multiple streams and is able to infer a user's camera switching preferences via dynamically predicting a user's switching intervals amongst multiple cameras.

The problem of human activity recognition (HAR) has been increasingly attracting the efforts of the research community, having several applications. It consists of recognizing human motion and/or behavior within a given image or a video sequence, using as input raw sensor measurements. In this paper, a multimodal approach addressing the task of video-based HAR is proposed. It is based on 3D visual data that are collected using an RGB + depth camera, resulting to both raw video and 3D skeletal sequences. These data are transformed into six different 2D image representations; four of them are in the spectral domain, another is a pseudo-colored image. The aforementioned representations are based on skeletal data. The last representation is a "dynamic" image which is actually an artificially created image that summarizes RGB data of the whole video sequence, in a visually comprehensible way. In order to classify a given activity video, first, all the aforementioned 2D images are extracted and then six trained convolutional neural networks are used so as to extract visual features. The latter are fused so as to form a single feature vector and are fed into a support vector machine for classification into human activities. For evaluation purposes, a challenging motion activity recognition dataset is used, while single-view, cross-view and cross-subject experiments are performed. Moreover, the proposed approach is compared to three other state-of-the-art methods, demonstrating superior performance in most experiments.

Video-based free-viewpoint rendering from multiple view video capture has achieved video-realistic performance replay. Existing free-viewpoint rendering approaches require storage, streaming and re-sampling of multiple videos, which requires high bandwidth and computational resources limiting applications to local replay on high-performance computers. This paper introduces a layered texture representation for efficient storage and view-dependent rendering from multiple view video capture whilst maintaining the video-realism. Layered textures re-sample the captured video according to the surface visibility. Prioritisation of layers according to surface visibility allows the N-best views for all surface elements to be pre-computed significantly reducing both storage and rendering cost. Typically 3 texture map layers are required for free-viewpoint rendering with an equivalent visual quality to the multiple view video giving a significant reduction in storage cost. Quantitative evaluation demonstrates that the layered representation achieves a 90% reduction in storage cost and 50% reduction in rendering cost without loss of visual quality compared to storing only the foreground of the original multiple view video. This reduces the storage and transmission cost for free-viewpoint video rendering from eight cameras to be similar to the requirements for a single video. Streaming the layered representation enables, for the first time, demonstration of free-viewpoint video rendering on mobile devices and web platforms.

Depth-image-based rendering (DIBR) is a key method for synthesizing virtual views from multiple RGB-D video streams. A challenging issue inherent in this approach is the disocclusion that occurs as the virtual viewpoint moves away from a reference view. In this work, we present a technique for extracting 3D geometric data, called the disocclusion-reducing geometry, from the input video streams. This auxiliary information, represented as a 3D point cloud, can then be combined easily with a conventional DIBR pipeline to reduce the disoccluded region as much as possible during the view warping process, eventually decreasing the visual artifacts by the subsequent hole-filling process.

Video retrieval continues to be one of the most exciting research areas in the field of multimedia technology. With the advancement of sensing and tracking technologies it is possible to generate multiple personal video streams during events from different perspectives. This paper presents the first prototype of the MyView system that provides on-demand personalized video streams with multiple cameras and additional sensors. This system captures and stores video streams in an indoor office, extracts high-level events from local positioning system (LPS) tracking information, and provides on-demand video segments for high-level queries by mixing the multi-camera video streams for a stream of the best view.

Goal of the reported experiment was to determine in which way wirelessly transmitted video streams influence each other. Former experiments gave the impression that the behavior of video streams is unpredictable, e.g. due to unstable play-back and uncertain starts. To transport two video streams simultaneously over a wireless/wired network, the video servers use different network protocols (TCP and RTP) and variants of video bandwidth adaptation algorithms; none, random drop and I-frame Delay (IFD), a special scheduling algorithm for MPEG streams. The access point was operating in congestion mode. Even under these extremely bad network conditions, IFD still offers the highest flexibility in combination with TCP. In contrast, the performance of the server using RTP was topology dependent. The servers with no adaptation showed the infeasibility of complete TCP transmission for real-time media. The TCP/random drop server opened the possibility of a non-complete TCP transmission which was exploited in the TCP/IFD server.

Managing the Quality-of-Experience (QoE) of video streaming for wireless clients is becoming increasingly important due to the rapid growth of video traffic on wireless networks. The inherent variability of the wireless channel as well as the Variable Bit Rate (VBR) of the compressed video streams make QoE management a challenging problem. Prior work has studied this problem in the context of transmitting a single video stream. In this paper, we investigate multiplexing schemes to transmit multiple video streams from a base station to mobile clients that use number of playout stalls as a performance metric. In this context, we present an epoch-by-epoch framework to fairly allocate wireless transmission slots to streaming videos. In each epoch our scheme essentially reduces the vulnerability to stalling by allocating slots to videos in a way that maximizes the minimum `playout lead' across all videos. Next, we show that the problem of allocating slots fairly is NP-complete even for a constant number of videos. We then present a fast lead-aware greedy algorithm for the problem. Our choice of greedy algorithm is motivated by the fact that this algorithm is optimal when the channel quality of a user remains unchanged within an epoch (but different users may experience different channel quality). Moreover, our experimental results based on public MPEG-4 video traces and wireless channel traces that we collected from a WiMAX test-bed show that the lead-aware greedy approach performs a fair distribution of stalls across the clients when compared to other algorithms, while still maintaining similar or lower average number of stalls per client.

We study the problem of broadcasting multiple variable bit rate scalable video streams from a base station to a large number of mobile receivers. Our objective is to maximize bandwidth utilization, energy saving and perceived quality of the transmitted videos. In practice, the aggregate bitrate of the video streams can be greater than the available bandwidth and then the receivers may experience playout glitches. We propose an algorithm to take advantage of opportunities provided by scalable video coding to improve performance when the bandwidth is limited. To achieve this, we provide a model for prioritizing video packets coded in medium grain scalability and a decision mechanism to drop quality layers of video frames when the bandwidth is limited. The result is increased bandwidth utilization and quality of the transmitted videos. Our experiments, performed on a real mobile TV testbed, show that our algorithm significantly out-performs current methods in terms of bandwidth utilization and also achieves near optimal energy saving and quality of the transmitted video streams.

This paper introduces a decentralized architecture designed for the sharing and distribution of user-generated video streams. The proposed system employs HTTP Live Streaming (HLS) as the delivery method for these video streams. In the architecture, a creator who captures a video stream using a smartphone camera subsequently transcodes it into a sequence of video chunks called HLS segments. These chunks are then stored in a distributed manner across the worker network, forming the core of the proposed architecture. Despite the presence of a coordinator for bootstrapping within the worker network, the selection of worker nodes for storing generated video chunks and autonomous load balancing among worker nodes are conducted in a decentralized fashion, eliminating the need for central servers. The worker network is implemented using the Golang-based IPFS (InterPlanetary File System) client, called kubo, leveraging essential IPFS functionalities such as node identification through Kademlia-DHT and message exchange using Bitswap. Beyond merely delivering stored video streams, the worker network can also amalgamate multiple streams to create a new composite stream. This bundling of multiple video streams into a unified video stream is executed on the worker nodes, making effective use of the FFmpeg library. To enhance download efficiency, parallel downloading with multiple threads is employed for retrieving the video stream from the worker network to the requester, thereby reducing download time. The result of the experiments conducted on the prototype system indicates that those concerned with the transmission time of the requested video streams compared with a server-based system using AWS exhibit a significant advantage, particularly evident in the case of low-resolution video streams, and this advantage becomes more pronounced as the stream length increases. Furthermore, it demonstrates a clear advantage in scenarios characterized by a substantial volume of viewing requests.

Storage, browsing and analysis of human activity videos can be significantly facilitated by automated video summarization. Unsupervised key-frame extraction remains the most widely applicable technique for summarizing activity videos. However, their specific properties make the problem difficult to solve. Typical relevant algorithms fall under the video frame clustering or the dictionary-of-representatives families, with salient dictionary learning having been recently proposed. Under this formulation, the video frames selected as key-frames are the ones which simultaneously best reconstruct the entire video and are salient compared to the rest. This paper improves upon such a method by replacing the video frame saliency estimation term with one based on Regularized SVD-based Low Rank Approximation, taking advantage of the well-established correlation between midrange matrix singular values and salient regions. Extensive empirical evaluation showcases the high performance of both the salient dictionary learning framework and the specific proposed method.

In a typical security and monitoring system a large number of networked cameras are installed at fixed positions around a site under surveillance. There is generally no global view or map that shows the guard how the views of different cameras relate to one another. Individual cameras may be equipped with pan, tilt and zoom capabilities, and the guard may be able to follow an intruder with one camera, then pick him up with another. But such tracking can be difficult, and hand off between cameras disorienting. The guard does not have the ability to continually shift his viewpoint. More over current systems do not scale up with the number of cameras. The system becomes more unwieldy as cameras are added to the system. In this paper, we will present the system and key algorithms for remote immersive monitoring of an urban site using a blanket of video cameras. The guard monitors the world using a live 3D model, which is constantly being updated from different directions using the multiple video streams. The world can be monitored remotely from any virtual viewpoint. The observer can see the entire scene from far and get a bird's eye view or can fly/zoom in and see activity of interest up close. A 3D-site model is constructed of the urban site and used as glue for combining the multiple video streams. Moreover each of the video cameras has smart image processing associated with it, which allows it to detect moving and new objects in the scene and recover their 3D geometry and pose of the camera with respect to the world model. Each video stream is overlaid on top of the video model using the recovered pose. Virtual views of the scene are generated by combining the various video streams, the background 3D model and the recovered 3D geometry of foreground objects. These moving objects are highlighted on the 3D model and used as a cue by the operator to direct his viewpoint.

Tracking people and cars using 3D modeling and CCTV

Congestion control of real-time streaming a video clip or film across the Internet is vital, as network traffic volatility requires constant adjustment of the bit rate in order to reduce packet loss. Traditional solutions to congestion control are prone to delivery rate fluctuations and may respond only when packet loss has already occurred, while both fluctuations and packet loss seriously affect the end user’s appreciation of the delivered video. In this chapter, fuzzy logic control (FLC) is newly applied to control of video streaming in fixed and wireless networks. In a fixed network, by way of congestion control the encoded video bitstream’s rate is adjusted according to the available bandwidth. Compared to existing controllers, FLC’s sending rate is significantly smoother, allowing it to closely track available bandwidth at a bottleneck on the video stream’s path across a network. The chapter also shows that when multiple video streams are congestion controlled through FLC, the result is a fairer and more efficient sharing of the bandwidth capacity. Also considered is a pioneering application of FLC to wireless networks, where other resources, apart from available bandwidth, come into play. An FLC system has been designed that provides a modular solution to control of latency and energy consumption, which is important for battery-powered devices, but must be balanced against the quality of delivered video. The chapter concludes by presenting the potential of emerging type-2 fuzzy logic as a way of significantly improving the robustness of classical type-1 fuzzy logic.

Action recognition is an important precursor for understanding human activities in videos. The current paradigm of action recognition is to classify a video sequence as a whole. However, actions usually occur only in part of a video sequence, rendering the rest of the video irrelevant for action recognition. In this paper, we propose a method for learning a subsequence classifier which can detect and classify part of a video that corresponds to the action. The subsequence classifier is trained from weakly labeled training videos whose subsequence labels are not provided, but need to be inferred during learning. We use the framework of multiple instance learning to solve two problems jointly: i) find the action subsequences in training videos, ii) train the subsequence classifier using the inferred action subsequences. To obtain a robust solution to the MIL problem, we propose a sequential algorithm that consecutively decreases the number of inferred action subsequences per video and trims their length until only one short subsequence is used as the action representative in each video. We evaluate the combination of the automatically trained subsequence classifier and the full sequence classifier on the very challenging Hollywood2 benchmark set and observe a significant gain in the performance over the baseline full sequence classifier. Moreover, a favorable performance of the subsequence classifier for temporal localization of actions in videos is evidenced on two categories of the Hollywood2 dataset.