Video stabilization using image alignment approach

Video stabilization is an instrumental and a very commonly employed image processing operations, yet still a very active area of research. Although lots of work have been done in the field of Video Stabilization, however still not much work has been done in the field of Real time Video Stabilization, especially works dealing with implementation of Technology on prototyping Boards. This work discusses the implementation of a Real time implementation of a Digital Video Stabilization system based on Upright Speeded up Robust Features mechanism. The U-SURF used in the algorithm has been found to be computationally much lighter as compared to the existing proposed mechanisms and has been implemented using Verilog as the preferred HDL Language. The developed architecture synthesized for Xilinx Zynq XC7030 has the significant benefits of being a fully pipelined architecture, making it much faster than the existing systems and at the same time exhibits better PSNR and MSE values. The system has been meticulously tested on 3 different standard Datasets namely StableNet Dataset, StabNet Dataset and the MATLAB Computer Vision Toolbox Dataset.

In recent years, a digital video stabilization improving the results of hand-held shooting or shooting from mobile platforms is the most popular approach. In this chapter, the task of digital video stabilization in static scenes is investigated. The unwanted motion caused by camera jitters or vibrations ought to be separated from the objects motion in a scene. Our contribution connects with the development of deblurring method to find and improve the blurred frames, which have strong negative influence on the following processing results. The use of fuzzy Takagi-Sugeno-Kang model for detection the best local and global motion vectors is the novelty of our approach. The quality of test videos stabilization was estimated by Peak Signal to Noise Ratio (PSNR) and Interframe Transformation Fidelity (ITF) metrics. Experimental data confirmed that the ITF average estimations increase up on 3–4 dB or 15–20 % relative to the original video sequences.

Assessing the quality of the motion stability is important to evaluating the performance of video stabilization algorithms. This paper presents a novel quality assessment scheme for the video motion stability in a full-reference (FR) manner. Given ideally stable videos and their corresponding shaky videos, our method measures the geodesic distance between motion paths of the stable and the stabilized videos. Due to the use of the Riemannian metric defined on the manifold of spatial transformations, our method enables the intrinsic and faithful measurement on pairwise motion disparities. To facilitate the FR assessment, a data set of stable and shaky videos is constructed by directly capturing realistic stable/shaky videos with a customized device. Then, digital video stabilization algorithms can be run on shaky videos to obtain the stabilized sequence of frames, whereupon their performances are evaluated by using our stability assessment. The experiments demonstrate that our stability assessment gains good concordance with the subjective assessment.

The problem of error control and concealment in video communication is becoming increasingly important because of the growing interest in video delivery over unreliable channels such as wireless networks and the Internet. This paper reviews the techniques that have been developed for error control and concealment. These techniques are described in three categories according to the roles that the encoder and decoder play in the underlying approaches. Forward error concealment includes methods that add redundancy at the source end to enhance error resilience of the coded bit streams. Error concealment by postprocessing refers to operations at the decoder to recover the damaged areas based on characteristics of image and video signals. Last, interactive error concealment covers techniques that are dependent on a dialogue between the source and destination. Both current research activities and practice in international standards are covered.

A medical video analysis provides a forum for the dissemination of diverse results in the area of medical field. The swiftly sprouting changes in the health care entail the modern advances with imaging. The effect of camera motion from a video stream leads to improper diagnosis in case of medical video processing. Hence, video stabilization technique enables the video stream to generate a stabilized video. Compression also plays a very vital role in the field of clinical and image analysis, since it carries massive information thereby reducing the file size. To further ensure effectiveness in diagnosis, a video stabilization is performed before compression. The results proved that the proposed methodology offered a substantial increase in the quality of video by 1.25 dB peak-signal-to-noise ratio and 0.06 structural similarity index.

Undesired vibration in videos is more and more common due to the rise of hand-held or vehicle mounted cameras. Non-indented motion of a video capturing device causes an unpleasant vibrating effect for the consumer. Therefore, video stabilization has been an extremely active field of research in the past years. Motion estimation is the most computationally expensive step of the video stabilization process. Our goal is to circumvent this expensive step in order to achieve real-time performance. We do so by using the precomputed motion vectors from the encoded video streams and thus operate in the compressed domain. These vectors already contain an approximation of the needed motion information. A low-level motion model is used for mitigating complexity, and a low-pass filter performs motion smoothing before the final motion compensation step which is used for correcting the video. In many real-time applications where the video vibration is moderate, the proposed framework can reach online video stabilization at 30 frames per second for high definition video and 60 frames per second for lower resolutions, while retaining satisfactory performance in video correction, comparable to pixel-based equivalent algorithms.

Distributed Video Coding (DVC) is a new paradigm for video compression based on the information theoretical results of Slepian and Wolf (SW), and Wyner and Ziv (WZ). While conventional coding has a rigid complexity allocation as most of the complex tasks are performed at the encoder side, DVC enables a flexible complexity allocation between the encoder and the decoder. The most novel and interesting case is low complexity encoding and complex decoding, which is the opposite of conventional coding. While the latter is suitable for applications where the cost of the decoder is more critical than the encoder's one, DVC opens the door for a new range of applications where low complexity encoding is required and the decoder's complexity is not critical. This is interesting with the deployment of small and battery-powered multimedia mobile devices all around in our daily life. Further, since DVC operates as a reversed-complexity scheme when compared to conventional coding, DVC also enables the interesting scenario of low complexity encoding and decoding between two ends by transcoding between DVC and conventional coding. More specifically, low complexity encoding is possible by DVC at one end. Then, the resulting stream is decoded and conventionally re-encoded to enable low complexity decoding at the other end. Multiview video is attractive for a wide range of applications such as free viewpoint television, which is a system that allows viewing the scene from a viewpoint chosen by the viewer. Moreover, multiview can be beneficial for monitoring purposes in video surveillance. The increased use of multiview video systems is mainly due to the improvements in video technology and the reduced cost of cameras. While a multiview conventional codec will try to exploit the correlation among the different cameras at the encoder side, DVC allows for separate encoding of correlated video sources. Therefore, DVC requires no communication between the cameras in a multiview scenario. This is an advantage since communication is time consuming (i.e. more delay) and requires complex networking. Another appealing feature of DVC is the fact that it is based on a statistical framework. Moreover, DVC behaves as a natural joint source-channel coding solution. This results in an improved error resilience performance when compared to conventional coding. Further, DVC-based scalable codecs do not require a deterministic knowledge of the lower layers. In other words, the enhancement layers are completely independent from the base layer codec. This is called the codec-independent scalability feature, which offers a high flexibility in the way the various layers are distributed in a network. This thesis addresses the following topics: First, the theoretical foundations of DVC as well as the practical DVC scheme used in this research are presented. The potential applications for DVC are also outlined. DVC-based schemes use conventional coding to compress parts of the data, while the rest is compressed in a distributed fashion. Thus, different conventional codecs are studied in this research as they are compared in terms of compression efficiency for a rich set of sequences. This includes fine tuning the compression parameters such that the best performance is achieved for each codec. Further, DVC tools for improved Side Information (SI) and Error Concealment (EC) are introduced for monoview DVC using a partially decoded frame. The improved SI results in a significant gain in reconstruction quality for video with high activity and motion. This is done by re-estimating the erroneous motion vectors using the partially decoded frame to improve the SI quality. The latter is then used to enhance the reconstruction of the finally decoded frame. Further, the introduced spatio-temporal EC improves the quality of decoded video in the case of erroneously received packets, outperforming both spatial and temporal EC. Moreover, it also outperforms error-concealed conventional coding in different modes. Then, multiview DVC is studied in terms of SI generation, which differentiates it from the monoview case. More specifically, different multiview prediction techniques for SI generation are described and compared in terms of prediction quality, complexity and compression efficiency. Further, a technique for iterative multiview SI is introduced, where the final SI is used in an enhanced reconstruction process. The iterative SI outperforms the other SI generation techniques, especially for high motion video content. Finally, fusion techniques of temporal and inter-view side informations are introduced as well, which improves the performance of multiview DVC over monoview coding. DVC is also used to enable scalability for image and video coding. Since DVC is based on a statistical framework, the base and enhancement layers are completely independent, which is an interesting property called codec-independent scalability. Moreover, the introduced DVC scalable schemes show a good robustness to errors as the quality of decoded video steadily decreases with error rate increase. On the other hand, conventional coding exhibits a cliff effect as the performance drops dramatically after a certain error rate value. Further, the issue of privacy protection is addressed for DVC by transform domain scrambling, which is used to alter regions of interest in video such that the scene is still understood and privacy is preserved as well. The proposed scrambling techniques are shown to provide a good level of security without impairing the performance of the DVC scheme when compared to the one without scrambling. This is particularly attractive for video surveillance scenarios, which is one of the most promising applications for DVC. Finally, a practical DVC demonstrator built during this research is described, where the main requirements as well as the observed limitations are presented. Furthermore, it is defined in a setup being as close as possible to a complete real application scenario. This shows that it is actually possible to implement a complete end-to-end practical DVC system relying only on realistic assumptions. Even though DVC is inferior in terms of compression efficiency to the state of the art conventional coding for the moment, strengths of DVC reside in its good error resilience properties and the codec-independent scalability feature. Therefore, DVC offers promising possibilities for video compression with transmission over error-prone environments requirement as it significantly outperforms conventional coding in this case.

Video transmission in error-prone networks suffers bit errors and packet losses that degrade the decoded video quality. In this paper, a video error concealment scheme based on H.264/AVC is proposed to enhance the error robustness using reversible data hiding. At the encoder, motion vectors of macro blocks belonging to the region of interest (ROI) are embedded into the background region within the same frame. The histogram modification technique based on quantized DCT coefficients of coded macro blocks is used to achieve loss less recovery after data extraction. If a macro block in the ROI is found missing at the decoder, the embedded motion vector can be extracted from the corresponding host macro block for the recovery of the missing macro block. This proposed scheme makes a tradeoff between error concealment performance and compression efficiency. Experimental results show the better reconstructed video quality can be achieved compared with the related error concealment methods.

Smartphone cameras are often used to take pictures and videos because so many people now carry their smartphones everywhere. Therefore, smartphone camera quality plays an important role in overall smartphone evaluation and usage. Although smartphone camera quality has been continually improving, mobile media processing can provide significant enhancements. The authors discuss mobile media processing techniques, including color and contrast enhancement, backlight image compensation, deblurring, denoising, and video stabilization, within the framework of smartphone camera quality improvement.

We introduce model-based schemes for error concealment of networked video. We build appearance models for specific objects in the scene and use these models to replenish any lost information. Due to the models being designed specifically for the object, they are able to capture the statistical variations in the object appearance more effectively, thereby leading to better error-concealment performance. We examine statistical modeling techniques from the literature and introduce a new efficient and accurate linear model for data representation, called the mixture of principal components (MPC), and use these models for error concealment. We simulate lossy network conditions and show that these model-based concealment schemes outperform traditional concealment schemes across a variety of loss probabilities and bit rates for the coded video.

This thesis covers a broad part of the field of computational photography, including video stabilization and image warping techniques, introductions to light field photography and the conversion of monocular images and videos into stereoscopic 3D content. We present a user assisted technique for stereoscopic 3D conversion from 2D images. Our approach exploits the geometric structure of perspective images including vanishing points. We allow a user to indicate lines, planes, and vanishing points in the input image, and directly employ these as guides of an image warp that produces a stereo image pair. Our method is most suitable for scenes with large scale structures such as buildings and is able to skip the step of constructing a depth map. Further, we propose a method to acquire 3D light fields using a hand-held camera, and describe several computational photography applications facilitated by our approach. As the input we take an image sequence from a camera translating along an approximately linear path with limited camera rotations. Users can acquire such data easily in a few seconds by moving a hand-held camera. We convert the input into a regularly sampled 3D light field by resampling and aligning them in the spatio-temporal domain. We also present a novel technique for high-quality disparity estimation from light fields. Finally, we show applications including digital refocusing and synthetic aperture blur, foreground removal, selective colorization, and others.

Error resilient coding and error concealment techniques have been introduced in scalable video coding to reduce the quality degradation impact of transmission errors. Redundant picture is one of the error resilient tools, appropriate for delay sensitive applications where re-transmission is not possible and channel loss rates such as in packet-based wireless networks, are not a priori known. In this paper, combination of redundant-key-picture error resilient coding with motion-copy error concealment at the decoder is proposed based on experimental evaluation. Experimental results demonstrate that significant PSNR gain can be achieved using key picture error concealment with motion-copy for various tested sequences under different packet loss rates compared to JVT-W049.

Video transmission through wireless channels is still a challenging problem on the limitations in bandwidth and the presence of channel errors. Video transcoding, due to its high practical values for a wide range of networked video applications, has become an active research topic. It is typically used to reduce the rate and change the format of the originally encoded video source to match network conditions and terminal capabilities. Error resilient video transcoding can insert error resilient tools in the compressed video to enhance error resilience of the video over wireless channels by increasing bit rate. In this paper, firstly we outline the technical issues and research results related to video transcoding, Then we discuss techniques for reducing the complexity and improving the video quality, such as intra marcoblock refresh, synchronization marker insertion, error concealment, joint optimization, redundant slices, unequal error protection, and so on. To achieve good resilient performance, a experimental video transcoding is designed and established. The results show its improved video quality at the reducing the complexity and the cost of bit rate.

This paper presents an MPEG-4 video codec, called MoVa, for video coding applications that adopts 3G-324M. We designed MoVa to be optimal by embedding a cost-effective ARM7TDMI core and partitioning it into hardwired blocks and firmware blocks to provide a reasonable tradeoff between computational requirements, power consumption, and programmability. Typical hardwired blocks are motion estimation and motion compensation, discrete cosine transform and quantization, and variable length coding and decoding, while intra refresh, rate control, error resilience, error concealment, etc. are implemented by software. MoVa has a pipeline structure and its operation is performed in four stages at encoding and in three stages at decoding. It meets the requirements of MPEG-4 SP@L2 and can perform either 30 frames/s (fps) of QCIF or SQCIF, or 7.5 fps (in codec mode) to 15 fps (in encode/decode mode) of CIF at a maximum clock rate of 27 MHz for 128 kbps or 144 kbps. MoVa can be applied to many video systems requiring a high bit rate and various video formats, such as videophone, videoconferencing, surveillance, news, and entertainment.

Error concealment for robust arithmetic decoding in mobile radio environments