Compressed-domain Video Processing

CHAPTER 14 - Video Coding Technology

There is tremendous growth in the amount of digital video creation and communication with the proliferation of mobile consumer electronic devices such as cellular phones with video capture and display capabilities. This trend will continue with the demand for higher spatial and temporal video resolutions accompanied by 3D and multiview video. Since it is not practical to store or transmit the raw digital video data, it is essential to reduce it with digital video compression. To cope with the vast amount of raw digital video data, digital video compression standards use increasingly more complex algorithms. These algorithms achieve higher compression at the same quality at the expense of significant computational complexity. Therefore, it is essential to design low power digital video compression hardware especially for mobile devices. In this talk, I will present two sets of power reduction techniques for digital video compression hardware. The ones in the first set reduce the power consumption with no impact on the video quality. The ones in the second set trade off video quality for power consumption which also enables adjusting video quality based on available energy in the battery of a mobile device for graceful degradation. I will show the impact of these techniques on the power consumption of digital video compression hardware implementing H.264/AVC and H.265/HEVC standards.

Frame interpolation, which generates an intermediate frame given adjacent ones, finds various applications such as frame rate up-conversion, video compression, and video streaming. Instead of using complex network models and additional data involved in the state-of-the-art frame interpolation methods, this paper proposes an approach based on an end-to-end generative adversarial network. A combined loss function is employed, which jointly considers the adversarial loss (difference between data models), reconstruction loss, and motion blur degradation. The objective image quality metric values reach a PSNR of 29.22 dB and SSIM of 0.835 on the UCF101 dataset, similar to those of the state-of-the-art approach. The good visual quality is notably achieved by approximately one-fifth computational time, which entails possible real-time frame rate up-conversion. The interpolated output can be further improved by a GAN based refinement network that better maintains motion and color by image-to-image translation.

Chapter two - Digital Image and Video Compression and Processing

Digital video playback on mobile devices is fast becoming widespread and popular. Since mobile devices are typically resource constrained in terms of network bandwidth, battery power and available screen resolution, it is often necessary to formulate special encoding techniques in order to optimize power consumption during video streaming and playback. The existing H.264 standard is popular for video encoding on mobile devices, since it results in a low-bitrate video with visual clarity that is adequate for video playback on mobile devices. However, due to the complexity of the H.264 representation, the video decoding procedure is typically computationally intensive. In this paper, we propose a novel lossy video representation termed as Ligne-Claire (LC) video. LC videos are obtained via graphics overlay of outlines or silhouettes of objects in the video over an approximated texture video. Since the playback of LC video is typically meant for mobile devices, the visual quality of video is adequate for most mobile applications wherein the semantic content of the video can be characterized by object shapes and approximate texture information. Experimental results presented in the paper demonstrate that the proposed lossy LC video encoding scheme results in power savings of 50% or more during video playback compared to standard H.264-encoded videos, of similar video file size. In order to evaluate the visual quality of the LC video, we compare the performance of LC videos with H.264-encoded videos in the context of some typical computer vision tasks. Our results indicate that the performance of the computer vision algorithms on these videos is similar. This fact, coupled with subjective evaluation, and the resulting significant power savings, indicates that the proposed LC representation can be used effectively to encode video for power-constrained mobile devices.

The purpose of this CRADA was to establish a framework for LLNL and NOVA Management, Inc. to Cooperatively develop video compression routines to be used to condense maintenance documentation provided on computers issued to Army mechanics to help them diagnose and repair equipment faults. In the past, paper technical manuals were the only means of communicating fault isolation, diagnostics and repair action information to maintenance personnel. The need for this information was at the weapon system itself as well as at the maintenance depot. This project addressed the need for means of efficient storage and retrieval of video (bandwidth issue) by maintenance personnel as they moved through the diagnostic procedures. Several research and development projects have employed interactive stored video as a means of providing maintenance information to the soldier. The problem with this approach was the massive data storage requirement. The overall objective of the project was to develop fast decompression and playback routines for full motion, coIor video. A survey was done of existing standards and potential new routines and techniques for video compression and playback to determine which were suitable for use on the CTS. Playback speeds were preferred to beat 30 frames per second on the target architecture. Compression ratios were expected to store several minutes of synchronized video in 1 MB. Playback routines were Windows 3.1 compatible. Identification of any hardware upgrades associated with playback were made when necessary. A demonstration of existing LLNL algorithms for still frame decompression which represented a current benchmark of data compression performance was completed. NOVA Management and LLNL used some of the compression technologies developed or currently being developed at LLNL to address the requirements of this SBIR. LLNL started by evaluating current compression technologies or combinations thereof to estimate the best compression ratio LLNL could achieve at this stage. LLNL ported some of our software from the SUN workstation environment to the PC or equivalent environment (e.g., MAC) so as to closely match the computer environment suggested by the Army. Finally, LLNL designed and outlined new algorithms to meet the desired video compression of CTS.

Advances in fiber optics, video compression and transmission technologies are driving the development and deployment of a new generation of CATV equipment. Optical fiber super- trunks are now expanding program delivery capabilities to over 150 channels and digital video compression will soon increase the capacity of existing satellite, fiber and cable systems by a factor of 5 to 10 or more. These next generation CATV systems will support Video on Demand (VOD) and advanced communications services such as broad band LAN/WAN networking and Personal Communications Services (PCS).

Motion estimation has a critical role to play in the transmission and compression of digital video. As it has a complexity in terms of computation that is inherent it is a big challenge for the implementation of real time codes. Recently there have been studies on block matching algorithms (BMA) for bringing down the complexity in the computation and has also got better attention in many algorithms that are very effective for very large scale integration (VLSI) systems for bringing down the complexity. These BMA algorithms are relatively easy to implement which can give solutions that are suboptimal even if the entire search has not been performed. The entire search is high in terms of cost of computation that prevents it from being applied in the systems of VLSI. A novel and fast motion estimation based algorithm known as charged system search (CSS) and full search algorithm (FSA) for multipliers of low power and for H.264 encoders is proposed. The CSS is used in all fields of optimization and is well suited for most domains and does not require any gradient information. The results have proved that proposed FSA has better performance.

Several computer vision applications such as e-learning, video editing, video compression, video-on-demand and surveillance etc. are popular in recent days. Most of the applications need videos to be retrieved and processed regularly. First and foremost step towards video retrieval and management is keyframe extraction. The perfect identification of shot transition boundaries is trivial in extracting keyframes. In present article, a framework for shot transition detection and keyframe extraction have been proposed. The proposed method is efficient, simple and does not require supervision which makes it attractive. The proposed method establishes the shot transition boundaries by estimating feature similarity (FSIM) between gradient magnitudes of consecutive frames. Then the frame with the highest mean and standard deviation is chosen as keyframe to that shot. In any situation if one feature fails to establish shot transition boundary another feature may succeed in establishment of shot transition boundary at proper frame locations of video. The proposed algorithm is tested on four different datasets, among them one is developed by us, two are well known standard datasets to evaluate keyframe extraction algorithm and the other one is standard surveillance video dataset. All the datasets are publicly available. Performance evaluation of the method is done in terms of Figure of merit, Detection percentage, Accuracy and Missing factor. The experimental results prove that the proposed method outperforms other state-of-art methods.

Object tracking in video is a challenging problem in several applications such as video surveillance, video compression, video retrieval, and video editing. Tracking an object in a video is not easy due to loss of information caused by illumination changing in a scene, occlusions with other objects, similar target appearances, and inaccurate tracker responses. In this paper, we present a novel object detection and tracking algorithm via structured output prediction classifier. Given an initial bounding box with its position, we first divide it into sub-blocks with a predefined size. Next, we extract the features from each sub-blocks with Haar-like features method. And then we learn those features with a structured output prediction classifier. We treat the sub-blocks obtained from the initial bounding box as positive samples and then randomly choose negative samples from search windows defined by the specific area around the bounding box. After that, we obtain prediction scores for each sub-blocks both from positive and negative samples. We construct a region-graph with sub-blocks as nodes and classifier’s score as weight to detect the target object in each frame. Our experimental results show that the proposed method outperforms state-of-the-art object tracking algorithms.KeywordsObject DetectionObject TrackingSupport Vector Machine

Objectives. The widespread use of systems for capturing light fields is due to the high quality of the reproduced image. This type of capture, although qualitatively superior to traditional methods to capturing volumetric images, generates a huge amount of data needed to reconstruct the original captured 4D light field. The purpose of the work is to consider traditional and extended to four-dimensional image compression algorithms, to perform a comparative analysis and determine the most suitable.Methods. Mathematical methods of signal processing and methods of statistical analysis are used.Results. Algorithms are compared and analyzed in relation to the compression of four-dimensional light fields using the PSNR metric. The selected evaluation criterion is affected not only by the dimension of the compression algorithm, but also by the distance of the baseline of the capture setting, since the difference between images increases with the distance between the optical centers of each camera matrix. Thus, for installations consisting of an array of machine vision cameras located on racks and placed in a room, the obvious choice would be to use conventional image compression methods. Furthermore, based on the assessment of the arbitrariness of video compression methods, it should be noted that the XVC algorithm remains undervalued, although its results are higher. Algorithm AV1 can be considered the next in order of importance. It has been established that the latest compression algorithms show higher performance if compared to their predecessors. It has also been shown that with a small distance between the optical centers of the captured images, the use of video compression algorithms is preferable to the use of image compression algorithms, since they show better results in both three-dimensional and four-dimensional versions.Conclusions. A comparison of the results obtained shows the need to use algorithms from the video compression family (XVC, AV1) on installations with a long baseline (mounted on camera stands). When working with integrated light field cameras (Lytro) and setting the capture with a short baseline, it is recommended to use image compression algorithms (JPEG). In general, video compression algorithms are recommended, in particular XVC, since on average it shows an acceptable level of PSNR in both the case of a short and long installation baseline.

Frame Rate Up Conversion (FRUC) makes video playback smoother and more natural to the human eye by inserting interleave frames between existing frames. Quality of existing FRUC has had issues of ghosting and blurring effects caused by interpolation based on blending function. Existing motion estimation method produces inaccurate motion vectors, which are the culprit of low quality frames when motion-compensated interpolation is applied. Unidirectional FRUC needs to address hole area; most existing methods handle hole areas while considering spatial relation. In this approach, motions appear unnatural when hole areas are too large due to large motions or when a hole appears on an object border. Addressing these issues, we propose in this work a path-based interpolation framework that reduces ghosting and blurring effects and minimizes hole areas. To reduce occurrence of hole regions, forward/backward motion estimation is first performed. During interpolation, more natural-looking trajectory path selected; pixels on the path are copied to the interpolated frame. This is different from the approach MCI takes, which blends pixels of reference and source frames. As a guide to selecting a more appropriate frame, a rule is proposed considering temporal relation. To select an appropriate frame, we present a rule that considers temporal relations. With our method, improvement in PSNR was insignificant when compared to conventional methods, but frame quality is much improved with significantly reduced ghosting and blurring effects.

Data mining is a technique to discover useful patterns or patterns of special interest as explicit knowledge from a vast quantity of data. In video editing, there are a lot of editing patterns. According to the editor's preferences, different editing patterns give the opportunity to achieve a variety of effects. Discovering the editing patterns is required, because it is useful to find each editor's skills and to use them for editing new video material. In this paper, we propose the methods of extracting editing rules from video stream by introducing data mining technique. We can edit a video material by applying the extracted rules. The edited video may produce the same quality as the video from which we extracted the patterns.

We present several compressed-domain methods for reverse- play transcoding of MPEG video streams. A reverse-play transcoder takes any original MPEG IPB bitstream as input and creates an output MPEG IPB bitstream which, when decoded by a generic MPEG decoder, displays the original video games in reverse order. A baseline spatial-domain method requires decoding the MPEG bitstream, storing and reordering the decoded video frames, and re-encoding the reordered video. The proposed compressed-domain transcoding methods achieve an order of magnitude reduction in computational complexity over the baseline spatial-domain approach. Much of the savings are achieved by using the forward motion vector fields available in the forward-play MPEG bitstream to efficiently generate the reverse motion vector fields used in the reverse-play MPEG bitstream. Furthermore, the storage requirements of the compressed-domain methods are reduced and the resulting image quality is within 0.6 dB of the baseline spatial-domain approach for a difficult highly detailed computer-generated video sequence. For more typical video sequences, the resulting image quality is even closer to the baseline spatial-domain approach.

The digital world demands the transmission and storage of high quality video for streaming and broadcasting applications, the constraints are the network bandwidth and the memory of devices for the various multimedia and scientific applications, the video consists of spatial and temporal redundancies. The objective of any video compression algorithm is to eliminate the redundant information from the video signal during compression for effective transmission and storage. The correlation between the successive frames has not been exploited enough by the current compression algorithms. In this paper, a novel method for video compression is presented. The proposed model, applies the transformation on set of group of pictures (GOP). The high spatial correlation is achieved from the spatial and temporal redundancy of GOP by accordion representation and this helps to bypass the computationally demanding motion compensation step. The core idea of the proposed technique is to apply Tucker Decomposition (TD) on the Discrete Wavelet Transform (DWT) coefficients of the Accordion model of the GOP. We use DWT to separate the video in to different sub-images and TD to efficiently compact the energy of sub-images. The blocking artifacts will be considerably eliminated as the block size is huge. The proposed method attempts to reduce the spatial and temporal redundancies of the video signal to improve the compression ratio, computation time, and PSNR. The experimental results prove that the proposed method is efficient especially in high bit rate and with slow motion videos.