Rapid Disparity Prediction for Dynamic Scenes

Disparity prediction between adjacent frames for dynamic scenes

In this paper, a new stereo object tracking system using the disparity motion vector(DMV) is proposed. Here, the DMV is defined as a disparity difference between two time-sequential disparity maps of T-1 and T frames and it can be sequentially estimated from the disparity maps which are extracted from the input stereo sequence. Here, the disparity maps are obtained by using the feature-based matching algorithm, in which the block size and the search range for the feature-based matching are given by N x × N y = 1 × 1 and S x = ±32, respectively. And then, the DMV can be estimated by calculating the motion difference between two consecutive disparity maps. Basically, the DMV has a relatively large change of disparity values in the target region because of the target's movement, whereas it has very low difference values in the background region; there are almost no changes by comparing with the target region. As a result, the DMV can provide us with the motional information of a moving target by showing a large disparity difference in the target area. That is, the dynamic relationship between these disparity vectors of T-1 and T frames seems to be very similar to that between the motion vectors of T-1 and T frames in the sequence of the conventional 2D video images. Accordingly, the target area and its location coordinates can be detected by using these DMV maps. Basing on these locational data of a moving target, the pan/tilt embedded in the stereo camera system can be controlled and as a result, real-time stereo tracking of a moving target can be achieved. From some experiments with 9 frames of the stereo sequence having 256×256 pixels, it is shown that the proposed DMV-based stereo object tracking system can track the moving target with a relatively low error ratio of about 3.02 % on average. This good experimental result finally suggests a possibility of implementing the DMV-based stereo object tracking system.

This paper addresses the problem of matching stereo images acquired by a stereo system mounted aboard an intelligent vehicle. The main idea behind the new method consists in involving temporal matching between a current stereo pair and its preceding one to achieve the spatial matching of the former stereo by involving the matching results obtained at the last frame. The proposed method is achieved in three main steps. First, an edge based disparity map is derived from the disparity map of the preceding frame, we call it the assisting disparity map (ADM). Second, for each scan-line, a set of local ranges and global ranges are deduced from the ADM to keep only potential matching candidates. Third, the matching is done on the basis of dynamic programming algorithm by involving in the resulting local and global ranges we get from the last step. The proposed approach has been tested on both real and synthetic stereo sequences and the results demonstrate its effectiveness.

Innovative applications in rapidly evolving domains such as robotic navigation and autonomous (driverless) vehicles rely on binocular computer vision systems that meet stringent response time and accuracy requirements. A key problem in these vision systems is stereo matching, which involves matching pixels from two input images in order to construct the output, a 3D map. Building upon the existing local stereo matching algorithms, this paper proposes a novel stereo matching algorithm that is based on a weighted guided filtering foundation. The proposed algorithm consists of three main steps; each step is designed with the goal of improving accuracy. First, the matching costs are computed using a unique combination of complementary methods (absolute difference, Census, and gradient algorithms) to reduce errors. Second, the costs are aggregated using an adaptive weighted guided image filtering method. Here, the regularization parameters are adjusted adaptively using the Canny method, further reducing errors. Third, a disparity map is generated using the winner-take-all strategy; this map is subsequently refined using a densification method to reduce errors. Our experimental results indicate that the proposed algorithm provides a higher level of accuracy in comparison to a collection of the existing state-of-the-art local algorithms.

Stereo vision finds a wide range of applications in automotive, object detection, robot navigation, agriculture mapping, and others. Stereo matching is a stereo algorithm targeted to identify the corresponding pixels from two or more images. This study shows the implementation of stereo matching using the Sum of Absolute Difference (SAD) algorithm to extract the object's depth or disparity from stereo images. Our key objective revolves around the implementation of the stereo matching algorithm for a small field-programmable gate array (FPGA)—requiring relatively few resources while maintaining the processing speed as well as disparity map. For meeting this requirement, we used small window buffers to compute the stereo matching. The occluded pixels were reduced by introducing secondary consistency checking implementation. From the results of experiments performed using the Zynq UltraScale+ ZCU102 FPGA board with SDSoC compiler, the processing speed for computing the stereo matching algorithm with a 4×4 window buffer was 0.038 s for an image size of 486×720 pixels and 0.051 s for 375×1242 pixels resolution. The proposed design needed 1% each of BRAM and FF and 7% of LUT. An 18% reduction in the pixel errors has been observed when employing the secondary consistency matching on the post-processing.

Depth estimation has vital importance in various applications like Advance driver assistance systems, Robot navigation, 3D movies, etc. Depth information is represented using disparity maps which are generated from various stereo correspondence algorithms. This paper proposes an implementation of Semi global block matching algorithm on raspberry pi using python. The proposed algorithm generates a disparity map by block wise matching and smoothness constraint. The novel algorithm is compared with other algorithms on the standard dataset. It provides an efficient accuracy over local methods. The real time implementation of SGM algorithm on the system uses a block size of 21 × 21 for images having resolution 1280×720P. The system provides faster processing time for the proposed algorithm.

Target tracking is an important issue in computer vision for its widely applications in video surveillance, human-computer interaction, robotic navigation, image compression and so on. There exist challenging problems such as occlusion, pose change, illumination change etc. in real videos. The two-stage sparse coding method for target tracking is proposed in this paper. The proposed method is based on Bayesian network and sparse coding with dynamic dictionary. In order to overcome the visual drift, the two-stage method is applied with target and dictionary updating to realize tracking accurately and quickly. Some public and popular challenging sequences are used to demonstrate the effectiveness of the proposed method. The experiment results show that the proposed method has better performance compared with other state-of-the-art methods.

Stereo vision system is a practical method for depth gathering of objects and features in an environment. This paper presents the depth evaluation in selected region of disparity mapping for stereo vision mobile robot using block matching algorithm. This region is a reference sight of the stereo camera and stereo vision baseline is based on horizontal configuration. The block matching technique is briefly described with the performance of its output. The disparity mapping is generated by the algorithm with the reference to the left image coordinate. The algorithm uses Sum of Absolute Differences (SAD) which is developed using Matlab software.

Rover navigation using stereo ego-motion

We propose a novel motion segmentation formulation over spatio-temporal depth images obtained from stereo sequences that segments multiple motion models in the scene in an unsupervised manner. The motion segmentation is obtained at frame rates that compete with the speed of the stereo depth computation. This is possible due to a decoupling framework that first delineates spatial clusters and subsequently assigns motion labels to each of these cluster with analysis of a novel motion graph model. A principled computation of the weights of the motion graph that signifies the relative shear and stretch between possible clusters lends itself to a high fidelity segmentation of the motion models in the scene.

In the past decades, the insertion of cameras for the aim of surveillance is increased. Hence, a huge amount of data is produced by cameras. It is impossible to categorize and store all data. Therefore, algorithms that automatically process big data and track objects of interest are needed. Many methods are based on the opinion that the movement of objects causes differences in frames of a video, and the background would remain motionless during the video. Continuous dynamic behavior in the background deteriorates object detection performance.On the other hand, an excellent background extraction model can help to gain beneficent foreground detection results. The target of this paper is to model an algorithm that provides the pure background from video sequences. The idea of the proposed approach is to extract the background of complex and crowded scenes by using the differences of two consecutive frames’ pixels. Our experimental results show that the proposed approach provides significant performance in comparison with some previous techniques.

Video analysis is an important branch of computer vision due to its wide applications, ranging from video surveillance, video indexing, and retrieval to human computer interaction. All of the applications are based on a good video representation, which encodes video content into a feature vector with fixed length. Most existing methods treat video as a flat image sequence, but from our observations we argue that video is an information-intensive media with intrinsic hierarchical structure, which is largely ignored by previous approaches. Therefore, in this work, we represent the hierarchical structure of video with multiple granularities including, from short to long, single frame, consecutive frames (motion), short clip, and the entire video. Furthermore, we propose a novel deep learning framework to model each granularity individually. Specifically, we model the frame and motion granularities with 2D convolutional neural networks and model the clip and video granularities with 3D convolutional neural networks. Long Short-Term Memory networks are applied on the frame, motion, and clip to further exploit the long-term temporal clues. Consequently, the whole framework utilizes multi-stream CNNs to learn a hierarchical representation that captures spatial and temporal information of video. To validate its effectiveness in video analysis, we apply this video representation to action recognition task. We adopt a distribution-based fusion strategy to combine the decision scores from all the granularities, which are obtained by using a softmax layer on the top of each stream. We conduct extensive experiments on three action benchmarks (UCF101, HMDB51, and CCV) and achieve competitive performance against several state-of-the-art methods.

Accurate optical and synthetic aperture radar (SAR) image registration is crucial to multisensor remote sensing applications. Though several algorithms have been proposed, the practical implementation that directly matches large and high-resolution optical and SAR images remains underexplored by the community. Equipped with the satellite positioning parameters, optical and SAR images can be roughly registered based on geographic coordinates. However, the relative positioning accuracy is still dozens, even hundreds of pixels due to the inaccuracies of sensor parameters and elevation. Consequently, we propose a robust registration algorithm, which consists of two stages, where the horizontal positioning errors can be reduced in the first stage, and then, we fine-tune the correspondences in the second stage. Specifically, we propose a novel template matching method based on the dilated convolutional feature (DCF) and epipolar-oriented phase correlation. DCF is constructed by a depthwise-separable dilated convolution with multichannel gradients, which are generated by the Sobel operator for optical images and a ratio of exponentially weighted averages (ROEWA) operator for SAR images. Due to the large reception field of dilated convolution, DCF can retain invariance even for large relative positioning errors. The epipolar-oriented rectangle template, which stretches along the epipolar line, is then proposed to capture more overlapping areas compared to square templates. Furthermore, the outlier removal is implemented in the coordinate system of optical images to avoid the effect of range compression in SAR images. Inliers are finally used to refine the rational polynomial coefficients (RPCs) based on the bundle adjustment technique. Experimental results on high-resolution optical and SAR image products of various scenarios demonstrate the effectiveness of the proposed registration framework. The relative positioning errors of the refined RPCs can be reduced from hundreds of pixels to the subpixel level.

3D reconstruction is the base of mobile robot navigation. And it is necessary to provide the environment information for robots effectively and rapidly. The traditional 3D reconstruction method relies on the dense disparity map which requires high computing ability, and there is large redundant data in the reconstructed 3D data. Unlike traditional methods, the proposed method needn't obtain the disparity map. We partitioned the scene scope uniformly into grids, and introduced virtual height lines(VHLs) which do not exist in the scene. Each VHL is projected in left image and right image. The intersection of VHL and the scene is only existent on projection line. So the point has maximum similarity metric. We used dynamic programming(DP) to find the matched lines in left image corresponding to the projections of VHLs(PVHLs) in right image. The intersection point of the matched line and the PVHL in left image is the answer. The experiments to reconstruct 3D scene are performed and the results show the proposed method is reliable and effective.

The inverse synthetic aperture lidar (ISAL) have attracted increasing attention for its merits including small visual tracking which is considered as one of the important research topics in the field of computer vision due to its key role in versatile applications, such as precision guidance, intelligent video surveillance, human-computer interaction, robot navigation and public safety. The basic idea for implementing visual tracking is composed of finding the target object in a video or sequence of images, then determining its exact position in the next successive frames and finally generating the corresponding trajectory of this object. Visual tracking, however, is still a challenging problem in practice while taking into account the abrupt appearance changes of the target objects induced by their non-rigid transformation, the sophisticated lighting variation, the obstruction by the block or similar objects in the background and the camera jitter. Motivated by the successful applications in target detection and recognition in recent years, plenty of deep learning models have been integrated in the visual tracking and better performance over traditional methods was achieved in a series of data evaluations, which opens a new door in the field of visual tracking. In this paper, the overview and progress on visual tracking were summarized. The current challenges and corresponding solving approaches in this field are introduced firstly and in particular, several novel and mainstream visual tracking algorithms based on the deep learning are specially described and analyzed in details, including their basic ideas, advantages and disadvantages and future prospect.