Scene-adaptive accurate and fast vertical crowd counting via joint using depth and color information

This paper presents our approach to using a stereo camera to obtain 3-D image data to be used to improve existing lip boundary detection techniques. We show that depth information as provided by our approach can be used to significantly improve boundary detection systems. Our system detects the face and mouth area in the image by using color, geometric location, and additional depth information for the face. Initially, color and depth information can be used to localize the face. Then we can determine the lip region from the intensity information and the detected eye locations. The system has successfully been used to extract approximate lip regions using RGB color information of the mouth area. Merely using color information is not robust because the quality of the results may vary depending on light conditions, background, and the human race. To overcome this problem, we used a stereo camera to obtain 3-D facial images. 3-D data constructed from the depth information along with color information can provide more accurate lip boundary detection results as compared to color only based techniques.

The effectiveness of depth information in saliency detection has been fully proved. However, it is still worth exploring how to utilize the depth information more efficiently. Erroneous depth information may cause detection failure, while non-salient objects may be closer to the camera which also leads to erroneously emphasis on non-salient regions. Moreover, most of the existing RGB-D saliency detection models have poor robustness when the salient object touches the image boundaries. To mitigate these problems, we propose a multi-stage saliency detection model with the bilateral absorbing Markov chain guided by depth information. The proposed model progressively extracts the saliency cues with three level (low-, mid-, and high-level) stages. First, we generate low-level saliency cues by explicitly combining color and depth information. Then, we design a bilateral absorbing Markov chain to calculate mid-level saliency maps. In mid-level, to suppress boundary touch problem, we present the background seed screening mechanism (BSSM) for improving the construction of the two-layer sparse graph and better selecting background-based absorbing nodes. Furthermore, the cross-modal multi-graph learning model (CMLM) is designed to fully explore the intrinsic complementary relationship between color and depth information. Finally, to obtain a more highlighted and homogeneous saliency map in high-level, we structure a depth-guided optimization module which combines cellular automata and suppression-enhancement function pair. This optimization module refines the saliency map in color space and depth space, respectively. Comprehensive experiments on three challenging benchmark datasets demonstrate the effectiveness of our proposed method both qualitatively and quantitatively.

Color and Depth-Based Superpixels for Background and Object Segmentation

This paper presents a new approach to real-time human detection and tracking in cluttered and dynamic environments by integration of RGB and depth data. We introduce the notion of Point Ensemble Image, which fully encodes both RGB and depth information from a virtual plan-view perspective, and we reveal that human detection and tracking in 3D space can be performed very effectively based on this new representation. Our human detector is able to take advantage of depth data by effectively locate physically plausible candidates as a first step, and then both depth and color information is made full use of in a supervised learning manner at the second stage. 3D trajectories of humans are finally generated by data association in which joint statistics of color and height are computed and compared. Experimental results show that the system is able to work satisfactorily in complex real-world situations.

In this paper, we propose a background subtraction (BGS) method based on the Gaussian mixture models using color and depth information. For combining color and depth information, we used the probabilistic model based on Gaussian distribution. In particular, we focused on solving color camouflage problem and depth denoising. For evaluating our method, we built a new dataset containing normal, color camouflage and depth camouflage situations. The dataset files consist of color, depth and ground truth image sequences. With these files, we compared the proposed algorithm with the conventional color-based BGS techniques in terms of precision, recall and F-measure. As a result, our method showed the best performance. Thus, this technique will help to robustly detect regions of interest as pre-processing in high-level image processing stages.

Recent inpainting techniques usually require human interactions which are labor intensive and dependent on the user experiences. In this paper, we introduce an automatic inpainting technique to remove undesired fence-like structures from images. Specifically, the proposed technique works on the RGBD images which have recently become cheaper and easier to obtain using the Microsoft Kinect. The basic idea is to segment and remove the undesired fence-like structures by using both depth and color information, and then adapt an existing inpainting algorithm to fill the holes resulting from the structure removal. We found that it is difficult to achieve a satisfactory segmentation of such structures by only using the depth channel. In this paper, we use the depth information to help identify a set of foreground and background strokes, with which we apply a graph-cut algorithm on the color channels to obtain a more accurate segmentation for inpainting. We demonstrate the effectiveness of the proposed technique by experiments on a set of Kinect images.

A multiple object tracking approach that combines colour and depth information using a confidence measure

Object extraction is one of the important and chal-lenging tasks in the computer vision and/or robotics ? elds.This task is to extract the object from the scene using anypossible cues. The scenario discussed in this study was the objectextraction which considering the Space of Interest (SOI), i.e.,the three dimensional area where the object probably existed.To complete such task, the object extraction method based onthe probabilistic maps of multiple cues was proposed. Thanksto the Kinect V2 sensor, multiple cues such as color, depth, andnear-infrared information can be acquired simultaneously. TheSOI was modeled by a simple probabilistic model by consideringthe geometry of the possible objects and the reachability of thesystem acquired from depth information. To model the color andnear-infrared information, a Gaussian mixture models (GMM)was used. All of the models were combined to generate theprobabilistic maps that were used to extract the object fromthe scene. To validate the proposed object extraction, severalexperiments were conducted to investigate the best combinationof the cues used in this study.

Crowd counting with heavy occlusions is highly desired for public security. However, few works have been studied towards this goal. Most previous systems only count passing people robustly without heavy occlusions, else they have to estimate the crowds to a certain extent. To solve this difficult problem, this paper proposes an effective algorithm by combining head detection and motion flow estimation together. First, we detect each head in the crowd by using our proposed scene-adaptive scheme on depth data. Then, we estimate the motion flow based on the interest points in each head region on color data. We can ultimately achieve multi-direction crowd counting results with the loop of above steps. Based on this approach, we have built a practical system for reliable crowd counting. Extensive experimental results show that our system is effective.

This paper describes a new method for indoor environment mapping and localization with stereo camera. For environmental modeling, we directly use the depth and color information in image pixels as visual features. Furthermore, only the depth and color information at horizontal centerline in image is used, where optical axis passes through. The usefulness of this method is that we can easily build a measure between modeling and sensing data only on the horizontal centerline. That is because vertical working volume between model and sensing data can be changed according to robot motion. Therefore, we can build a map about indoor environment as compact and efficient representation. Also, based on such nodes and sensing data, we suggest a method for estimating mobile robot positioning with random sampling stochastic algorithm. With basic real experiments, we show that the proposed method can be an effective visual navigation algorithm.

Due to the sparsity of point clouds obtained by LIDAR, the depth information is usually not complete and dense. The depth completion task is to recover dense depth information from sparse depth information. However, most of the current deep completion networks use RGB images as guidance, which are more like a processing method of information fusion. They are not valid when there is only sparse depth data and no other color information. Therefore, this paper proposes an information-reinforced completion network for a single sparse depth input. We use a multi-resolution dense progressive fusion structure to maximize the multi-scale information and optimize the global situation by point folding. At the same time, we re-aggregate the confidence and impose another depth constraint on the pixel depth to make the depth estimation closer to the ground trues. Our experimental results on KITTI and NYU Depth v2 datasets show that the proposed network achieves better results than other unguided deep completion methods. And it is excellent in both accuracy and real-time performance.

Moving target detection and tracking, recognition, behaviors analysis are the key issues in the intelligent visual surveillance system (IVSS). The challenge is how to process the real-time video stream in an effective way in case that we could find the interested objects for analysis. However, the traditional video surveillance technology often does not meet the needs of real-time key frame recognition for the on-line intelligent video monitoring system. In our paper, we adopt the state-of-the-art Faster R-CNN [7] that takes advantages of convolutional neural networks into our real-time target recognition system - Deep Intelligent Visual Surveillance (DIVS). The key aspects of our DIVS are consisted of four parts: (i) Getting the real-time video images from remote cameras; (ii) Processing the data with the deep learning framework caffe [23] built for Faster R-CNN; (iii) Storing the valuable data with MySQL; (iv) Data presentation on the website. Experiments based on our system validated the effectiveness, stability and accuracy of our proposed solutions.

What is most needed in the development of intelligent robots is visual information. Especially, Domestic Service Robots (DSR) in carrying out their duties in a complex and dynamic environment such as a household. To accomplish the task given in such environment, various kinds of recognition including household object recognition are necessary. The first step in object recognition is object learning, which one of the processes needs visual information. This information is provided by a visual sensor such as a second version of Microsoft Kinect (Kinect V2). Kinect V2 provides data such as, color information, depth information, and near infrared information. To make the object learning process simple and trackable, the development of object data acquisition systems is needed. In general, the captured data will be incorporated with labels such as the name of household object and the corresponding pose. To facilitate the object labeling, we develop smartphone applications to enable simple user interaction. To capture visual information in several poses, we make a turntable that can rotate synchronously during the data acquisition process. To extract the visual information of the object autonomously, an object extraction is inevitable. To this end, we propose an object extraction based on combination of information captured from Kinect V2. Our proposed method using a probabilistic method which integrates several Gaussian Mixture Models (GMM). Evaluation of proposed systems has been done through several experiments. Based on conducted experiments, our systems can capture the household object with size specifications greater than 3 cm × 3 cm × 2 cm, smaller than 28 cm × 28 cm × 30 cm, and with a weight not exceeding 800 g. In addition, the proposed systems can extract 40 objects (each of which covered 40 poses) with the F1-score of 76.43%.

To enable high-level understanding of a scene, it is important to understand the occlusion and connected boundaries of objects in the image. In this paper, we propose a new framework for inferring boundaries from color and depth information. Even with depth information, it is not a trivial task to find and classify boundaries. Real-world depth images are noisy, especially at object boundaries, where our task is focused. Our approach uses features from both the color (which are sharp at object boundaries) and depth images (for providing geometric cues) to detect boundaries and classify them as occlusion or connected boundaries. We propose depth features based on surface fitting from sparse point clouds, and perform inference with a Conditional Random Field. One advantage of our approach is that occlusion and connected boundaries are identified with a single, common model. Experiments show that our mid-level color and depth features outperform using either depth or color alone, and our method surpasses the performance of baseline boundary detection methods.

Improved face detection method based on information of skin colour and depth information is presented which is also able to differentiate between real face and picture face. This method can be apply to a service robot equipped with binocular camera system for real-time face detection and with this algorithm the service robot will able detect the real face instead of fake one. Hence the proposed algorithm helps the robot, security system and military application to distinguish between the real face and fake face. The main objective of this work is to design a system which is able to detect face in the image and obtain depth information by using Binocular vision system and also get the depth map of an input image.