Depth Error Elimination for RGB-D Cameras

We present a sensor-fusion method that exploits a depth camera and a gyroscope to track the articulation of a hand in the presence of excessive motion blur. In case of slow and smooth hand motions, the existing methods estimate the hand pose fairly accurately and robustly, despite challenges due to the high dimensionality of the problem, self-occlusions, uniform appearance of hand parts, etc. However, the accuracy of hand pose estimation drops considerably for fast-moving hands because the depth image is severely distorted due to motion blur. Moreover, when hands move fast, the actual hand pose is far from the one estimated in the previous frame, therefore the assumption of temporal continuity on which tracking methods rely, is not valid. In this paper, we track fast-moving hands with the combination of a gyroscope and a depth camera. As a first step, we calibrate a depth camera and a gyroscope attached to a hand so as to identify their time and pose offsets. Following that, we fuse the rotation information of the calibrated gyroscope with model-based hierarchical particle filter tracking. A series of quantitative and qualitative experiments demonstrate that the proposed method performs more accurately and robustly in the presence of motion blur, when compared to state of the art algorithms, especially in the case of very fast hand rotations.

Structured light range sensors (SLRS) like the Microsoft Kinect have electronic rolling shutters (ERS). The output of such a sensor while in motion is subject to significant motion blur (MB) and rolling shutter (RS) distortion. Most robotic literature still does not explicitly model this distortion, resulting in inaccurate camera motion estimation. In RGBD cameras, we show via experimentation that the distortion undergone by depth images is different from that of color images and provide a mathematical model for it. We propose an algorithm that rectifies for these RS and MB distortions. To assess the performance of the algorithm we conduct an extensive set of experiments for each step of the pipeline. We assess the performance of our algorithm by comparing the performance of the rectified images on scene-flow and camera pose estimation, and show that with our proposed rectification, the performance improvement is significant.

Consumer depth cameras, such as the Microsoft Kinect, are capable of providing frames of dense depth values at real time. One fundamental question in utilizing depth cameras is how to best extract features from depth frames. Motivated by local descriptors on images, in particular kernel descriptors, we develop a set of kernel features on depth images that model size, 3D shape, and depth edges in a single framework. Through extensive experiments on object recognition, we show that (1) our local features capture different aspects of cues from a depth frame/view that complement one another; (2) our kernel features significantly outperform traditional 3D features (e.g. Spin images); and (3) we significantly improve the capabilities of depth and RGB-D (color+depth) recognition, achieving 10–15% improvement in accuracy over the state of the art.

This paper is a selective review of the development of the threshold model in time series analysis over the past 30 years or so. First, it re-visits the motivation of the model. Next, it describes the various expressions of the model, highlighting the principle underlying them and the main probabilistic and statistical properties. Finally, after listing some of the recent offsprings of the threshold model, it finishes with some on-going research in the context of threshold volatility.

A view synthesis problem is to generate a virtual view based on the given one or multiple views and their associated depth maps. We adopt the depth image based rendering (DIBR) approach in this paper for synthesizing the new views. No explicit 3D modeling is involved. Another component of this study is the popular commodity RGB-D (color plus depth) cameras. The color and depth images captured by a pair of RGB-D cameras (Microsoft Kinect for Windows v2) are our inputs to synthesize intermediate virtual views between these two cameras. Several methods include depth to color warping, disocclusion filling, and color to color warping are adopted and designed to achieve this target. One of our major contributions is a new disocclusion detection algorithm proposed to improve the disocclusion filling result. Furthermore, an improved camera calibration method is proposed to make use of the additional depth information. Good quality synthesized views are shown at the end.

Depth cameras started to gain popularity in agricultural applications during the last years. This type of cameras has been implemented mainly for three-dimensional (3D) reconstruction of objects in indoor or outdoor sceneries. The use of such cameras in the construction of 3D models for simulation purposes of complex structures that are usually met in nature, such as trees and other plants, is a great challenge. Remarkably, agricultural environments are extremely complex. Thus, the proper setup and implementation of such technologies is particularly important in order to attain useable data. So far, the depth information collected using these cameras varies among different objects’ structure and sensing conditions due to the uncertainty of the outdoor environment. The use of a specific methodology using color and depth images gives the opportunity to extract geometrical characteristics information about point clouds of the targeted objects. This chapter explores the different technologies used by depth cameras and presents several applications concerning indoor and outdoor environments by presenting indicative scenarios for agricultural applications. Towards that direction, a 3D reconstruction of trees was established producing point clouds from Red Green Blue Depth (RGB-D) images acquired in real field conditions. The point cloud samples of trees were collected using an unmanned ground vehicle (UGV) and imported in Gazebo in order to visualize a simulation of the environment. This simulation technique can be used for testing and evaluating the navigation of robotic systems. By further analyzing the resulted 3D point clouds, various geometrical measurements of the simulated samples, such as the volume or the height of tree canopies, can be calculated. Possible weaknesses of this procedure are mainly attributed to the camera’s limitations and the sampling parameters. However, results show that it is possible to establish a suitable simulation environment to implement it in several agricultural applications by utilizing automated unmanned robotic platforms.KeywordsDepth camerasUnmanned ground vehiclesScene reconstruction3D point cloudAgricultural robotsSituation awareness

Fruit classification is a computer vision task that aims to classify fruit classes correctly, given an image. Nearly all fruit classification studies have used RGB color images as inputs, a few have used costly hyperspectral images, and a few classical ML-based have used colorized depth images. Depth images have apparent benefits such as invariance to lighting, less storage requirement, better foreground-background separation, and more pronounced curvature details and object edge discontinuities. However, the use of depth images in CNN-based fruit classification remains unexplored. The purpose of this study is to investigate the use of colorized depth images in fruit classification with four CNN models, namely, AlexNet, GoogleNet, ResNet101, and VGG16, and compare their performance and computational efficiency, as well as the impact of transfer learning. Depth images of apple, orange, mango, banana and rambutan (Nephelium Lappaceum) were manually collected using a depth sensor with sub-millimeter accuracy and subjected to jet, uniform, and inverse colorization to produce three sets of dataset. Results show that depth images can be used to train CNN models for fruit classification with ResNet101 achieving the best accuracy of 96%on the inverse dataset. It achieved 100% accuracy after transfer learning. GoogleNet showed the most significant improvement after transfer learning on the uniform dataset, at 12.27%. It also exhibited the lowest training and inference times. The results show the potential use of depth images for fruit classification and similar computer vision tasks.

In this work, we address the problem of 3D object recognition with depth sensors such as Kinect or Structure sensor. Compared with traditional approaches based on local descriptors, which depends on local information around the object keypoints, we propose a novel global features based descriptor. Proposed descriptor, which we name as Differential Histogram of Normal Vectors (DHONV), is designed particularly to capture the surface geometric characteristics of the 3D objects represented by depth images. We describe the 3D surface of an object in each depth image using a 1D histogram capturing the distribution of deferential angles of the surface normal vectors. The object recognition experiments on the benchmark RGB-D object dataset and a self-collected dataset show that our proposed descriptor outperforms two others descriptors based on spin-images and histogram of normal vectors with linear-SVM classifier.

Obtaining accurate body weight (BW) is crucial for management decisions yet can be a challenge for cow-calf producers. Fast-evolving technologies such as depth sensing have been identified as low-cost sensors for agricultural applications but have not been widely validated for U.S. beef cattle. This study aimed to (1) estimate the body volume of mature beef cows from depth images, (2) quantify BW and metabolic weight (MBW) from image-projected body volume, and (3) classify body condition scores (BCS) from image-obtained measurements using a machine-learning-based approach. Fifty-eight crossbred cows with a mean BW of 410.0 ± 60.3 kg and were between 4 and 6 yr of age were used for data collection between May and December 2021. A low-cost, commercially available depth sensor was used to collect top-view depth images. Images were processed to obtain cattle biometric measurements, including MBW, body length, average height, maximum body width, dorsal area, and projected body volume. The dataset was partitioned into training and testing datasets using an 80%:20% ratio. Using the training dataset, linear regression models were developed between image-projected body volume and BW measurements. Results were used to test BW predictions for the testing dataset. A machine-learning-based multivariate analysis was performed with 29 algorithms from eight classifiers to classify BCS using multiple inputs conveniently obtained from the cows and the depth images. A feature selection algorithm was performed to rank the relevance of each input to the BCS. Results demonstrated a strong positive correlation between the image-projected cow body volume and the measured BW (r = 0.9166). The regression between the cow body volume and the measured BW had a co-efficient of determination (R2) of 0.83 and a 19.2 ± 13.50 kg mean absolute error (MAE) of prediction. When applying the regression to the testing dataset, an increase in the MAE of the predicted BW (22.7 ± 13.44 kg) but a slightly improved R2 (0.8661) was noted. Among all algorithms, the Bagged Tree model in the Ensemble class had the best performance and was used to classify BCS. Classification results demonstrate the model failed to predict any BCS lower than 4.5, while it accurately classified the BCS with a true prediction rate of 60%, 63.6%, and 50% for BCS between 4.75 and 5, 5.25 and 5.5, and 5.75 and 6, respectively. This study validated using depth imaging to accurately predict BW and classify BCS of U.S. beef cow herds.

Bridging the simulation-to-real gap of depth images for deep reinforcement learning

Large intra-class variations of action features lead to low classification accuracy of action recognition,on the other hand,current algorithms exist drawbacks in computational complexity and extension of recognizable action classes. A method based on locality-constrained linear coding( LLC) for action recognition from depth images was proposed. In order to reduce the intra-class variations and increase classification accuracy,joints' positions,velocities and acceleration features were concatenated to form local action features,then LLC was used to calculate sparse representations of local action features. Analytical solution of LLC ensures computational speed of our method is up to 760 frames per second. Dictionary is composed by sub-dictionaries learned by K-means from features of each class separately,so global optimization is avoided during extending recognizable action classes. Moreover,to avoid classifier to be over-fitting,a dimensionality reduction method based on labels of dictionary items was proposed. The proposed method was evaluated on MSRAction3 D dataset captured by depth cameras. The experimental results show that the proposed approach achieves classification accuracy of 85. 7%.

This paper outlines a method and applications for detection and tracking of people in depth images, acquired with a low-resolution Time-of-Flight (ToF) camera. This depth sensor is placed perpendicular to the ground in order to provide distance information from a top-view position. Usage of intrinsic and extrinsic camera parameters allows estimation of a ground plane and comparison to the measured distances of the ToF sensor in every pixel. Differences to the expected ground plane define foreground information, that is subsequently combined to associated regions. These regions of interest (ROI) are analyzed to distinguish persons from other objects by using a matched filter that is applied the height segmented depth information of each of these regions. The proposed method separates crowds into individuals and facilitates a multi-object tracking system based on Kalman filtering. Furthermore, we present several applications for the proposed method. Experiments with different crowding situations - from very low to very high density - and different heights of camera placements have proven the applicability and practicability of the system.

Time-of-flight (ToF) and structured light depth cameras have got increasing interest for their ability of direct 3D geometry acquisition. Robotics, computer vision and graphics researchers have tried to employ depth camera for their applications such as robot navigation, gesture recognition, 3d reconstruction, etc. Depth cameras, however, suffer from lots of unique noises such as range ambiguity, scattering and motion blur. For instance, the depth motion blur cannot be correctly removed by any conventional deblurring method yielding serious distortions in 3D reconstruction. In this paper, we introduce various systematic and non-systematic depth errors and state of the art enhancement methods.

Length Measurement of Potato Leaf using Depth Camera

A depth camera is widely used in various applications because it provides a depth image of the scene in real time. However, due to the limited power consumption, the depth camera presents severe noises, incapable of providing the high quality 3D data. Although the smoothness prior is often employed to subside the depth noise, it discards the geometric details so to degrade the distance resolution and hinder achieving the realism in 3D contents. In this paper, we propose a perceptual-based depth image enhancement technique that automatically recovers the depth details of various textures, using a statistical framework inspired by human mechanism of perceiving surface details by texture priors. We construct the database composed of the high quality normals. Based on the recent studies in human visual perception (HVP), we select the pattern density as a primary feature to classify textures. Upon the classification results, we match and substitute the noisy input normals with high quality normals in the database. As a result, our method provides the high quality depth image preserving the surface details. We expect that our work is effective to enhance the details of depth image from 3D sensors and to provide a high-fidelity virtual reality experience.