Accurate body-part reconstruction from a single depth image

We tackle the practical problem of hand pose estimation from a single noisy depth image. A dedicated three-step pipeline is proposed: Initial estimation step provides an initial estimation of the hand in-plane orientation and 3D location, Candidate generation step produces a set of 3D pose candidate from the Hough voting space with the help of the rotational invariant depth features, Verification step delivers the final 3D hand pose as the solution to an optimization problem. We analyze the depth noises, and suggest tips to minimize their negative impacts on the overall performance. Our approach is able to work with Kinect-type noisy depth images, and reliably produces pose estimations of general motions efficiently (12 frames per second). Extensive experiments are conducted to qualitatively and quantitatively evaluate the performance with respect to the state-of-the-art methods that have access to additional RGB images. Our approach is shown to deliver on par or even better results.

Content-based 3D shape retrieval is an important problem in computer vision. Traditional retrieval interfaces require a 2D sketch or a manually designed 3D model as the query, which is difficult to specify and thus not practical in real applications. With the recent advance in low-cost 3D sensors such as Microsoft Kinect and Intel Realsense, capturing depth images that carry 3D information is fairly simple, making shape retrieval more practical and user-friendly. In this paper, we study the problem of cross-domain 3D shape retrieval using a single depth image from low-cost sensors as the query to search for similar human designed CAD models. We propose a novel method using an ensemble of autoencoders in which each autoencoder is trained to learn a compressed representation of depth views synthesize d from each database object. By viewing each autoencoder as a probabilistic model, a likelihood score can be derived as a similarity measure. A domain adaptation layer is built on top of autoencoder outputs to explicitly address the cross-domain issue (between noisy sensory data and clean 3D models) by incorporating training data of sensor depth images and their category labels in a weakly supennsed learning formulation. Experiments using real-world depth images and a large-scale CAD dataset demonstrate the effectiveness of our approach, which offers significant improvements over state-of-the-art 3D shape retrieval methods.

Depth images are widely used in 3D head pose estimation and face reconstruction. The device-specific noise and the lack of textual constraints pose a major problem for estimating a nonrigid deformable face from a single noisy depth image. In this article, we present a deep neural network-based framework to infer a 3D face consistent with a single depth image captured by a consumer depth camera Kinect. Confronted with a lack of annotated depth images with facial parameters, we utilize the bidirectional CycleGAN-based generator for denoising and noisy image simulation, which helps to generalize the model learned from synthetic depth images to real noisy ones. We generate the code regressors in the source (synthetic) and the target (noisy) depth image domains and present a fusion scheme in the parametric space for 3D face inference. The proposed multi-level shape consistency constraint, concerning the embedded features, depth maps, and 3D surfaces, couples the code regressor and the domain adaptation, avoiding shape distortions in the CycleGAN-based generators. Experiments demonstrate that the proposed method is effective in depth-based 3D head pose estimation and expressive face reconstruction compared with the state-of-the-art.

Semantic Scene Completion (SSC) is a task that simultaneously predicts the occupancy and semantic labels of the environment. Compared with separate processing, SSC leverages the coupled nature of scene completion and semantic segmentation. Although this multitask integration can utilize complementarity and correlation between tasks, it also increases the training difficulty. To address this, in this paper, we propose a Semantic Decoupling based Semantic Scene Completion (SD-SSC) network from a single depth image. The semantic segmentation task is decoupled from the semantic scene completion task, and we use 2D and 3D semantic supervision to simplify the scene completion task and improve SSC performance. Specifically, our network first performs 2D semantic segmentation on the depth image and transforms features into 3D voxel space as semantic priors. Then, the 3D SSC is performed based on the voxel features and the flipped Truncated Signed Distance Field (f-TSDF). We use multi-scale 3D semantic supervision to further enhance the semantic information and fuse semantic and geometric features through the Planar Attention Fusion Module (PAFM) to obtain accurate SSC results. The proposed SD-SSC network achieves state-of-the-art performance on the NYU dataset (51.1% mIoU) and the NYUCAD dataset (61.9% mIoU) among all single depth-image based methods. It is even better than most RGB-D fusion-based SSC methods. The code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Saturday-xql/SD-SSC</uri>.

Simple and effective deep hand shape and pose regression from a single depth image

Depth images play an important role and are popularly used in many computer vision tasks recently. However, the limited resolution of the depth image has been hindering its further applications. To address this problem, we propose a novel dual sparsity model based single depth image super resolution algorithm, with a single low-resolution depth image as input. We formulate this problem by combining the recently developed analysis model and synthesis model exploiting the sparsity of analyzed vectors and the sparse coefficients respectively. The analysis operator and dictionaries are trained over extensive samples separately. We show that our model clearly outperforms state-of-the-art methods on the widely used Middlebury 2007 datasets both quantitatively and visually.

In recent years, numerous consumer devices have emerged that are capable of capturing 3D point data originating from depth images. Many computer vision tasks, such as object recognition, environment mapping, augmented reality, and more, rely on accurately registering 3D point sets. One method to compute this registration is to use 3D local feature descriptors for a coarse alignment, and to further refine the alignment with a variant of the Iterative Closest Point algorithm. While robust feature descriptors work well for this approach, online computation for all points in a single depth image is typically intractable. In this work, a method to facilitate real-time 3D registration by performing descriptor extraction on depth images using a Fully Convolutional Network (FCN) is presented. The input to this method is a raw depth image and results in a 33-bin descriptor for each pixel, enabling a general-purpose 3D registration process that doesn't require future network retraining and refinement. Experimental results on consumer hardware demonstrate that the proposed method significantly outperforms the state-of-the-art in terms of computation time and approaches depth sensor frame capture times, with only a slight reduction in descriptiveness.

Dormant pruning is one of the most costly and labor-intensive operations in specialty crop production. During winter, a large crew of trained seasonal workers has to carefully remove the branches from hundreds of trees using a set of pre-defined rules. The goal of automatic pruning is to reduce this dependence on a large work-force that is currently needed for the job. Automatically applying the pruning rules entails construction of 3D models of the trees in their dormant condition (that is, without foliage) and accurate estimation of the pruning points on the branches. This paper investigates the contribution of modeling the trunk and primary branches of a tree using semi-circles in a 3D reconstruction scheme. It involves three main steps; estimating the diameter-error, filtering the depth images, and semi-circle-based modeling which estimates the required pruning data. The obtained results demonstrate the effectiveness of the proposed framework for automatic reconstruction using only single depth image. We further propose a novel empirical model that estimates the diameter of the primary branches under varying depth values. Both qualitative and quantitative evaluations are performed on a new challenging apple trees dataset under various indoor and outdoor environments. Our results show that the proposed scheme provides a performance of 89% for correctly estimating the diameter of the primary branches.

How to determine the camera’s next best view is a challenging problem in vision field. A next best view approach is proposed based on occlusion information in a single depth image. First, the occlusion detection is accomplished for the depth image of visual object in current view to obtain the occlusion boundary and the nether adjacent boundary. Second, the external surface of occluded region is constructed and modeled according to the occlusion boundary and the nether adjacent boundary. Third, the observation direction, observation center point, and area information of external surface of occluded region are solved. And then, the set of candidate observation directions and the visual space of each candidate direction are determined. Finally, the next best view is achieved by solving the next best observation direction and camera’s observation position. The proposed approach does not need the prior knowledge of visual object or limit the camera position on a specially appointed surface. Experimental results demonstrate that the approach is feasible and effective.

Deep hand pose estimation from single depth image plays a significant role in human-computer interaction. This paper proposes a novel method based on multiple transfer net to estimate hand pose utilizing only single-channel depth photos. A channel extending process for original single channel depth image is implemented to extend hand and hand palm regions and match the input format of a pre-trained network and fully utilize the parameters. A multiple transfer network refinement for the previous convolutional neural network is made to obtain various different feature maps. Also, a region ensemble is used to merge all output feature maps and integrate the results. The experimental results demonstrate that the proposed method outperforms state-of-art results with considerable accuracy on the NYU [1] and ICVL [2] datasets.

We propose a viewpoint invariant model for 3D human pose estimation from a single depth image. To achieve this, our discriminative model embeds local regions into a learned viewpoint invariant feature space. Formulated as a multi-task learning problem, our model is able to selectively predict partial poses in the presence of noise and occlusion. Our approach leverages a convolutional and recurrent network architecture with a top-down error feedback mechanism to self-correct previous pose estimates in an end-to-end manner. We evaluate our model on a previously published depth dataset and a newly collected human pose dataset containing 100 K annotated depth images from extreme viewpoints. Experiments show that our model achieves competitive performance on frontal views while achieving state-of-the-art performance on alternate viewpoints.

Generation of human depth images with body part labels for complex human pose recognition

3D hand pose estimation can provide basic information about gestures, which has an important significance in the fields of Human-Machine Interaction (HMI) and Virtual Reality (VR). In recent years, 3D hand pose estimation from a single depth image has made great research achievements due to the development of depth cameras. However, 3D hand pose estimation from a single RGB image is still a highly challenging problem. In this work, we propose a novel four-stage cascaded hierarchical CNN (4CHNet), which leverages hierarchical network to decompose hand pose estimation into finger pose estimation and palm pose estimation, extracts separately finger features and palm features, and finally fuses them to estimate 3D hand pose. Compared with direct estimation methods, the hand feature information extracted by the hierarchical network is more representative. Furthermore, concatenating various stages of the network for end-to-end training can make each stage mutually beneficial and progress. The experimental results on two public datasets demonstrate that our 4CHNet can significantly improve the accuracy of 3D hand pose estimation from a single RGB image.

Fall is the biggest threat to seniors, with significant emotional, physical and financial implications. It is the major cause of serious injuries, disabilities, hospitalizations and even death especially for elderly people living alone. Timely detection could provide immediate medical service to the injured and avoid its harmful consequences. Great number of vision-based techniques has been proposed by installing cameras in several everyday environments. Recently, deep learning has revolutionized these techniques, mostly using convolutional neural networks (CNNs). In this paper, we propose weighted multi-stream deep convolutional neural networks that exploit the rich multimodal data provided by RGB-D cameras. Our method detects automatically fall events and sends a help request to the caregivers. Our contribution is three-fold. We build a new architecture composed of four separate CNN streams, one for each modality. The first modality is based on a single combined RGB and depth image to encode static appearance information. RGB image is used to capture color and texture and depth image deals with illumination variations. In contrast of the first feature that lacks the contextual information about previous and next frames, the second modality characterizes the human shape variations. After background subtraction and person recognition, human silhouette is extracted and stacked to define history of binary motion HBMI. The last two modalities are used to more discriminate the motion information. Stacked amplitude and oriented flow are used in addition to stacked optical flow field to describe respectively the velocity, the direction and the motion displacements. The main motivation behind the use of these multimodal data is to combine complementary information such as motion, shape, RGB and depth appearance to achieve more accurate detection than using only one modality. Our second contribution is the combination of the four streams to generate the final decision for fall detection. We evaluate early and late fusion strategies and we have defined the weight of each modality based on its overall system performance. Weighted score fusion is finally adopted based on our experiments. In the third contribution, transfer learning and data augmentation are applied to increase the amount of training data, avoid over fitting and improve the accuracy. Experiments have been conducted on publicly available standard datasets and demonstrate the effectiveness of the proposed method compared to existing methods.

Automatic estimation of face pose from images has wide applications in human computer interaction, biometrics, etc. This paper proposes a novel method for accurate face pose estimation from single depth images. Integral slice features are developed to characterize the distinctive and robust patterns of different face poses. Firstly the sampling points on the 3D facial surfaces are divided into multiple slices along the depth axis. And then a segment of facial slices is averaged to get an integral center. The vector from the integral center to a reference point is defined as a pose sensitive feature. Two kinds of reference points are tried in our method, i.e. nose tip and the geometric center of any slice. Therefore a large number of feature vectors can be obtained to describe the distinguishing pattern of each face pose by changing the starting and ending points of the integral segment of facial slices. Finally all these integral slice features are sent to a neural network to train a face pose estimator. Experimental results on the ETH database demonstrate that the proposed method is more accurate than state-of-the-art methods for face pose estimation.