Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns

A control approach is proposed in this study that controls the growth of a spatially growing mixing layer in an open loop. Due to the similarity of the mixing layer dynamics in the temporal and spatial domains, the control is exercised in the temporal domain that leads to optimization of the layer in the spatial domain. The proportional-integral-derivative (PID) control in the temporal domain uses Navier–Stokes equations as the plant model. Apart from being prohibitively expensive, control in the spatial domain presents convective time delay problems. These barriers are circumvented by solving the control problem in the temporal domain very rapidly, based on a given set point in the linear regime. Once the set point in the temporal domain is reached, the corresponding initial conditions in the temporal domain are mapped to inflow boundary conditions in the spatial domain. With these mapped inflow boundary conditions in the spatial domain, the spatial development of the mixing layer is followed, which confirms the efficacy of control in the spatial domain. The present control methodology offers a new paradigm for control of the spatially growing mixing layer.

A pose-wise linear illumination manifold model for face recognition using video

This paper introduces an iterative regularized approach to increase the resolution of a video sequence. A multiple input smoothing convex functional is defined and used to obtain a globally optimal high resolution video sequence. A mathematical model of multiple inputs is described by using the point spread function between the original and bilinearly interpolated images in the spatial domain, and motion estimation between frames in the temporal domain. An iterative algorithm is utilized for obtaining the solution. The regularization parameter is updated at each iteration step from the partially restored video sequence. Experimental results demonstrate the capability of the proposed approach.

We address the problem of learning dynamic patterns from unlabeled video sequences, either in the form of generating new video sequences, or recovering incomplete video sequences. This problem is challenging because the appearances and motions in the video sequences can be very complex. We propose to use the alternating back-propagation algorithm to learn the generator network with the spatial-temporal convolutional architecture. The proposed method is efficient and flexible. It can not only generate realistic video sequences, but can also recover the incomplete video sequences in the testing stage or even in the learning stage. The proposed algorithm can be further improved by using learned initialization which is useful for the recovery tasks. Further, the proposed algorithm can naturally help to learn the shared representation between different modalities. Our experiments show that our method is competitive with the existing state of the art methods both qualitatively and quantitatively.

The neural pathways for generating willed actions have been increasingly investigated since the famous pioneering work by Benjamin Libet on the nature of free will. To better understand what differentiates the brain states underlying willed and forced behaviours, we performed a study of chosen and forced actions over a binary choice scenario. Magnetoencephalography recordings were obtained from six subjects during a simple task in which the subject presses a button with the left or right finger in response to a cue that either (1) specifies the finger with which the button should be pressed or (2) instructs the subject to press a button with a finger of their own choosing. Three independent analyses were performed to investigate the dynamical patterns of neural activity supporting willed and forced behaviours during the preparatory period preceding a button press. Each analysis offered similar findings in the temporal and spatial domains and in particular, a high accuracy in the classification of single trials was obtained around 200ms after cue presentation with an overall average of 82%. During this period, the majority of the discriminatory power comes from differential neural processes observed bilaterally in the parietal lobes, as well as some differences in occipital and temporal lobes, suggesting a contribution of these regions to willed and forced behaviours.

We develop a new no-reference (NR) stereoscopic video quality assessment model that adopts oriented local gravitational force (OLGF) statistics in the space-time domain. The OLGF is a novel extension of an existing local gravitational force descriptor and includes two new components: relative local gravitational force magnitude and relative local gravitational force orientation. The resulting algorithm, called Stereoscopic Video Integrity Predictor using OLGF Statistics (SVIPOS), first uses our previous work to generate a cyclopean image and a product image in spatial domain from a stereoscopic video sequence pair, and calculates a frame difference image in temporal domain from a left video sequence. Then the OLGF model is deployed to compute various local gravitational force responses on the cyclopean image, the product image and the frame difference image. Finally, a support vector regression is used to map statistical features extracted from the computed gravitational force responses to stereoscopic video quality predications. SVIPOS is thoroughly tested on three publicly stereoscopic video quality databases. The experimental results show that SVIPOS outperforms state-of-the-art stereoscopic video quality methods.

We propose a video denoising method based on a spatiotemporal Kalman-bilateral mixture model to reduce the noise in video sequences that are captured with low light. To take full advantage of the strong spatiotemporal correlations of neighboring frames, motion estimation is first performed on video frames consisting of previously denoised frames and the current noisy frame by using block-matching method. Then, current noisy frame is processed in temporal domain and spatial domain by using Kalman filter and bilateral filter, respectively. Finally, by weighting the denoised frames from Kalman filtering and bilateral filtering, we can obtain a satisfactory result. Experimental results show that the performance of our proposed method is competitive when compared with state-of-the-art video denoising algorithms based on both peak signal-to-noise-ratio and structural similarity evaluations.

Video compression has great significance in the communication of motion pictures. Video compression techniques try to remove the different types of redundancy within or between video sequences. In the temporal domain, the video compression techniques remove the redundancies between the highly correlated consequence frames of the video. In the spatial domain, the video compression techniques remove the redundancies between the highly correlated consequence pixels (samples) in the same frame. Evolving neural-networks based video coding research efforts are focused on improving existing video codecs by performing better predictions that are incorporated within the same codec framework or holistic methods of end-to-end video compression schemes. Current neural network-based video compression adapts static codebook to achieve compression that leads to learning inability from new samples. This paper proposes a modified video compression model that adapts the genetic algorithm to build an optimal codebook for adaptive vector quantization that is used as an activation function inside the neural network’s hidden layer. Background subtraction algorithm is employed to extract motion objects within frames to generate the context-based initial codebook. Furthermore, Differential Pulse Code Modulation (DPCM) is utilized for lossless compression of significant wavelet coefficients; whereas low energy coefficients are lossy compressed using Learning Vector Quantization (LVQ) neural networks. Finally, Run Length Encoding (RLE) is engaged to encode the quantized coefficients to achieve a higher compression ratio. Experiments have proven the system’s ability to achieve higher compression ratio with acceptable efficiency measured by PSNR.

In this paper, we will propose a novel pixelwise adaptive prediction method for lossless video compression. The proposed method adequately takes use of the temporal and spatial redundancy in the video sequences. Firstly, the pixel to be encoded is predicted in temporal domain and spatial domain respectively, and then the final prediction values are chosen by an adaptive mode selection. In addition, we introduce a forgetting factor α in the temporal prediction and the adaptive mode selection to improve the prediction accuracy. Experimental results show that our proposed method gains the minimum zero-entropy compared to other prediction methods.

This paper introduces an iterative regularized approach to obtain a high resolution video sequence. A multiple input smoothing convex functional is defined and used to obtain a globally optimal high resolution video sequence. A mathematical model of multiple inputs is described by using the point spread function between the original and bilinearly interpolated images in the spatial domain, and motion estimation between frames in the temporal domain. Properties of the proposed smoothing convex functional are analyzed. An iterative algorithm is utilized for obtaining a solution. The regularization parameter is updated at each iteration step from the partially restored video sequence. Experimental results demonstrate the capability of the proposed approach.

Video synopsis is an effective solution for fast browsing and retrieval of long surveillance videos. It aims to shorten long video sequences into its equivalent compact video representation by rearranging the video events in the temporal domain and/or spatial domain. Conventional video synopsis methods focus on reducing the collisions between tubes and maintaining their chronological order, which may alter the original interactions between tubes due to improper tube rearrangement. In this paper, we present an approach to preserve the relationships among tubes (tracks of moving objects) of the original video in the synopsis video. First, a recursive tube-grouping algorithm is proposed to determine the behavior interactions among tubes in a video and group the related tubes together to form tube sets. Second, to preserve the discovered relationships, a spatio-temporal cube voting algorithm is proposed. This cube voting method optimally rearranges the tube sets in the synopsis video, minimizing false collisions between tubes. Third, a method to estimate the duration of the synopsis video is proposed based on an entropy measure of tube collisions. The extensive experimental results demonstrate that the proposed video synopsis framework condenses videos by preserving the original tube interactions and reducing false tube collisions.

Ensemble methods offer an elegant way of training an ensemble of weak classifiers into a strong classifier through the use of the AdaBoost algorithm. In this abstract we discuss two extensions of AdaBoost and demonstrate them on two problems in the field of Computer Vision. The first, termed Ensemble Tracking, extends AdaBoost in the temporal domain and adapts it to the problem of tracking an object in a video sequence. The second, termed SpatialBoost, extends AdaBoost in the spatial domain and adapts it to the problem of interactive image segmentation.

Rhythmic and sequential subdivision of the elongating vertebrate embryonic body axis into morphological somites is controlled by an oscillating multicellular genetic network termed the segmentation clock. This clock operates in the presomitic mesoderm (PSM), generating dynamic stripe patterns of oscillatory gene-expression across the field of PSM cells. How these spatial patterns, the clock's collective period, and the underlying cellular-level interactions are related is not understood. A theory encompassing temporal and spatial domains of local and collective aspects of the system is essential to tackle these questions. Our delayed coupling theory achieves this by representing the PSM as an array of phase oscillators, combining four key elements: a frequency profile of oscillators slowing across the PSM; coupling between neighboring oscillators; delay in coupling; and a moving boundary describing embryonic axis elongation. This theory predicts that the segmentation clock's collective period depends on delayed coupling. We derive an expression for pattern wavelength across the PSM and show how this can be used to fit dynamic wildtype gene-expression patterns, revealing the quantitative values of parameters controlling spatial and temporal organization of the oscillators in the system. Our theory can be used to analyze experimental perturbations, thereby identifying roles of genes involved in segmentation.

A systematic and robust laser speckle contrast imaging (LSCI) method and procedure is presented covering the LSCI system calibration, static scattering removal, and measurement noise estimation and correction to obtain a true flow speckle contrast and the flow speed from single-exposure LSCI measurements. We advocate the use of as the speckle contrast instead of the conventional contrast K, as the former relates simply to the flow velocity and is with additive noise alone. We demonstrate the efficacy of the proposed true flow speckle contrast by imaging phantom flow at varying speeds, showing that (1) the proposed recipe greatly enhances the linear sensitivity of the flow index (inverse decorrelation time) and the linearity covers the full span of flow speeds from 0 to 40 mm/s; and (2) the true flow speed can be recovered regardless of the overlying static scattering layers and the type of speckle statistics (temporal or spatial). The fundamental difference between the apparent temporal and spatial speckle contrasts is further revealed. The flow index recovered in the spatial domain is much more susceptible to static scattering and exhibit a shorter linearity range than that obtained in the temporal domain. The proposed LSCI analysis framework paves the way to estimate the true flow speed in the wide array of laser speckle contrast imaging applications.

In this paper we present an algorithm for depth estimation from a monocular video sequence containing moving and deformable objects. The method is based on a coded aperture system (i.e., a conventional camera with a mask placed on the main lens) and it takes a coded video as input to provide a sequence of dense depth maps as output. To deal with nonrigid deformations, our work builds on the state-of-the-art single-image depth estimation algorithm. Since single-image depth estimation is very ill-posed, we cast the reconstruction task as a regularized algorithm based on nonlocal-means filtering applied to both the spatial and temporal domain. Our assumption is that regions with similar texture in the same frame and in neighbouring frames are likely to belong to the same surface. Moreover, we show how to increase the computational efficiency of the method. The proposed algorithm has been successfully tested on challenging real scenarios. (6 pages)