Fruit Classification using Colorized Depth Images

Gesture recognition is a key research field in the human-computer interaction. At present, most of researchers focus on one-handed gesture recognition, but do not pay much attention to bimanual (two hands) gesture recognition. This paper presents a deep learning-based solution to tackle the self-occlusion and self-similarity. To solve this problem, this paper uses Kinect to collect many colour and depth images of different gestures, and each gesture contains multiple sample individuals. Colour images and depth images are used to train the recognition model of bimanual gesture respectively, and then the colour image and depth image are fused, and the bimanual gesture recognition model is trained based on colour image and depth image fusion. Then, the bimanual recognition effects of the three models are compared. The experimental results show that, regardless of the single gesture precision or the mean average precision, the bimanual gesture recognition effect of the fused model is better than the gesture recognition models based on either colour image or depth image.

Abstract. Size is an essential metric for the postharvest grading of Vidalia sweet onions. Currently, the size of the Vidalia onion is mainly measured by machine vision systems using 2-D imaging or mechanical sizers. This work investigated the potential of using the RGB-depth sensor to improve the accuracy and efficiency of quantifying the size features of onions (the maximum diameter and volume). In the study, the color and depth images of onions in various postures were collected from different viewpoints. The maximum diameter of the onion was calculated in 2-D and 3-D Euclidean space using its color and depth image, respectively. The depth image of the onion was converted to the voxel image for calculating the volume. Linear regression model was developed to predict the onion volume based on the total volume of the onion voxel image. The results showed that the proposed approaches accurately measured the maximum diameter using either the color image (RMSE=3.1 mm) or the depth image (RMSE=1.5 mm). The onion diameter estimation using the depth image showed a higher accuracy and robustness than the method using the color image. The proposed method of measuring the onion volume showed a RMSEP of 16 cm 3 and an accuracy of 96.7%. Results showed that both the onion maximum diameter and volume can be estimated by using a single depth image of the onion. The proposed methods of measuring the onion diameter and volume based on depth image were quite robust to the change of the onion orientation.

In this paper, a landmark based localization system using a Kinect sensor is proposed and evaluated with the implemented system for precise and autonomous navigation of low cost robots. The proposed localization method finds the positions of landmark on the image plane and the depth value using color and depth images. The coordinates transforms are defined using the depth value. Using coordinate transformation, the position in the image plane is transformed to the position in the body frame. The ranges between the landmarks and the Kinect sensor are the norm of the landmark positions in body frame. The Kinect sensor position is computed using the tri-lateral whose inputs are the ranges and the known landmark positions. In addition, a new matching method using the pin hole model is proposed to reduce the mismatch between depth and color images. Furthermore, a height error compensation method using the relationship between the body frame and real world coordinates is proposed to reduce the effect of wrong leveling. The error analysis are also given to find out the effect of focal length, principal point and depth value to the range. The experiments using 2D bar code with the implemented system show that the position with less than 3cm error is obtained in enclosed space(3,500mm×3,000mm×2,500mm)

Depth cameras started to gain popularity in agricultural applications during the last years. This type of cameras has been implemented mainly for three-dimensional (3D) reconstruction of objects in indoor or outdoor sceneries. The use of such cameras in the construction of 3D models for simulation purposes of complex structures that are usually met in nature, such as trees and other plants, is a great challenge. Remarkably, agricultural environments are extremely complex. Thus, the proper setup and implementation of such technologies is particularly important in order to attain useable data. So far, the depth information collected using these cameras varies among different objects’ structure and sensing conditions due to the uncertainty of the outdoor environment. The use of a specific methodology using color and depth images gives the opportunity to extract geometrical characteristics information about point clouds of the targeted objects. This chapter explores the different technologies used by depth cameras and presents several applications concerning indoor and outdoor environments by presenting indicative scenarios for agricultural applications. Towards that direction, a 3D reconstruction of trees was established producing point clouds from Red Green Blue Depth (RGB-D) images acquired in real field conditions. The point cloud samples of trees were collected using an unmanned ground vehicle (UGV) and imported in Gazebo in order to visualize a simulation of the environment. This simulation technique can be used for testing and evaluating the navigation of robotic systems. By further analyzing the resulted 3D point clouds, various geometrical measurements of the simulated samples, such as the volume or the height of tree canopies, can be calculated. Possible weaknesses of this procedure are mainly attributed to the camera’s limitations and the sampling parameters. However, results show that it is possible to establish a suitable simulation environment to implement it in several agricultural applications by utilizing automated unmanned robotic platforms.KeywordsDepth camerasUnmanned ground vehiclesScene reconstruction3D point cloudAgricultural robotsSituation awareness

In the state of the art, grayscale image enhancement algorithms are typically adopted for enhancement of RGB color images captured with low or non-uniform illumination. As these methods are applied to each RGB channel independently, imbalanced inter-channel enhancements (color distortion) can often be observed in the resulting images. On the other hand, images with non-uniform illumination enhanced by the retinex algorithm are prone to artifacts such as local blurring, halos, and over-enhancement. To address these problems, an improved RGB color image enhancement method is proposed for images captured under non-uniform illumination or in poor visibility, based on weighted guided image filtering (WGIF). Unlike the conventional retinex algorithm and its variants, WGIF uses a surround function instead of a Gaussian filter to estimate the illumination component; it avoids local blurring and halo artifacts due to its anisotropy and adaptive local regularization. To limit color distortion, RGB images are first converted to HSI (hue, saturation, intensity) color space, where only the intensity channel is enhanced, before being converted back to RGB space by a linear color restoration algorithm. Experimental results show that the proposed method is effective for both RGB color and grayscale images captured under low exposure and non-uniform illumination, with better visual quality and objective evaluation scores than from comparator algorithms. It is also efficient due to use of a linear color restoration algorithm.

This paper proposes recognition and grasping objects from 3D environment by combining depth and color stereo image in the mobile picking robot system. To do this task, the followings are done. Firstly, an image processing system including Kinect camera sensor is described. Secondly, RGB color map and new depth map for image inpainting are obtained using Kinect SDK mapping function to align RGB image with depth image. Thirdly, the new depth map are segmented to distinguish between the image background and the objects that should be recognized. The feature colours are generated based on colour histograms. Euclidean distance is used to measure the similarity between the feature vectors computed from the colour image and the feature vectors stored in a database. Fourthly, by converting RGB map and new depth map into 3D point clouds, an algorithm for localizing handle-like grasp affordances is proposed. The main idea is to search the point cloud for neighborhoods that satisfy handle-like grasp affordances and can be grasped by the end-effector of the manipulator. Finally, the effectiveness of the proposed algorithms is verified by using experiment. The experimental results show that the mobile picking robot successfully detects an object and finds its grasping points with an acceptable small error.

In order to ensure the safety of driving, this paper studies the vehicle detection algorithm based on convolutional neural network which integrates color and depth images. This paper mainly discusses the subject through convolutional network multi-scale forward looking depth imaging positioning model, forward looking variable scale vehicle detection pre-positioning algorithm, and typical model identification algorithm based on transfer learning, and verifies the algorithm performance and model identification algorithm accuracy by combining with simulation test. The results show that the accuracy of the algorithm is 96.89% under different weather conditions, indicating its effectiveness in application. Compared with previous vehicle detection algorithms, the detection accuracy of this algorithm is effectively improved by 3.96%, indicating that this algorithm has innovative significance.

Person re-identification is an important topic in retail, scene monitoring, human-computer interaction, people counting, ambient assisted living and many other application fields. A dataset for person re-identification TVPR (Top View Person Re-Identification) based on a number of significant features derived from both depth and color images has been previously built. This dataset uses an RGB-D camera in a top-view configuration to extract anthropometric features for the recognition of people in view of the camera, reducing the problem of occlusions while being privacy preserving. In this paper, we introduce a machine learning method for person re-identification using the TVPR dataset. In particular, we propose the combination of multiple k-nearest neighbor classifiers based on different distance functions and feature subsets derived from depth and color images. Moreover, the neighborhood component feature selection is used to learn the depth features’ weighting vector by minimizing the leave-one-out regularized training error. The classification process is performed by selecting the first passage under the camera for training and using the others as the testing set. Experimental results show that the proposed methodology outperforms standard supervised classifiers widely used for the re-identification task. This improvement encourages the application of this approach in the retail context in order to improve retail analytics, customer service and shopping space management.

Somatosensory interaction is an important part of human-computer interaction. The core of somatosensory interaction must accurately obtain the 3D spatial information of the human body. This article uses an RGBD camera to acquire both color and depth images. We Perform 2D human pose estimation in color images using hourglass network with residual structure. At the same time, we use the sparse feature points to align the color map with the depth map in order to accurately map the detection results in color images to the corresponding depth images. The experimental results show that this method can accurately and quickly obtain the 3-dimensional posture information of the human body, which is an important guarantee for somatosensory interaction.

In order to use a depth camera to simplify the acquisition process of integral imaging, this paper proposes a depth map restoration algorithm combining pixel filling and improved joint bilateral filtering(DIJBF). The depth image collected by the depth camera has the problems of background noise and holes in the foreground and background of the object. Firstly, the background filling method or the neighborhood value filling method is adopted to complete the preliminary restoration of the large edge void area according to the situation, and then combined with the improved joint bilateral filtering algorithm that adds the depth image depth value similarity factor to optimize the secondary restoration of the preliminary restoration depth map. After the depth image is restored and optimized, the contour of the three-dimensional object is clear and the edge is smooth. Combined with the color image, a high-quality elemental image array can be subsequently generated for the integral imaging display system.

In order to obtain better super-resolution reconstruction results of depth images, this paper constructs a multi-scale color image guidance depth image super-resolution reconstruction convolutional neural network. In this paper, the multi-scale fusion method is used to realize the guidance of high resolution (HR) color image features to low resolution (LR) depth image features, which is beneficial to the restoration of image details. In the process of extracting features from LR depth images, a multiple receptive field residual block (MRFRB) is constructed to extract and fuse the features of different receptive fields, and then connect and fuse the features of each MRFRB output to obtain global fusion features. Finally, the HR depth image is obtained through sub-pixel convolution layer and global fusion features. The experimental results show that the super-resolution image obtained by this method alleviates the edge distortion and artifact problems, and has better visual effects.

A view synthesis problem is to generate a virtual view based on the given one or multiple views and their associated depth maps. We adopt the depth image based rendering (DIBR) approach in this paper for synthesizing the new views. No explicit 3D modeling is involved. Another component of this study is the popular commodity RGB-D (color plus depth) cameras. The color and depth images captured by a pair of RGB-D cameras (Microsoft Kinect for Windows v2) are our inputs to synthesize intermediate virtual views between these two cameras. Several methods include depth to color warping, disocclusion filling, and color to color warping are adopted and designed to achieve this target. One of our major contributions is a new disocclusion detection algorithm proposed to improve the disocclusion filling result. Furthermore, an improved camera calibration method is proposed to make use of the additional depth information. Good quality synthesized views are shown at the end.

Convolutional neural networks have demonstrated state-of-the-art performance in image classification and various other computer vision tasks. Plant disease detection is an important area of deep learning which has been addressed by many recent methods. However, there is a dire need to optimize these solutions for resource-constrained portable devices such as smartphones. This is a challenging problem because deep learning models are resource extensive in nature. This paper proposes an efficient method to systematically classify plant disease symptoms using convolutional neural networks. These networks are memory efficient and when coupled with the proposed training configuration it enables rapid development of industrial applications by reducing the training times. Another critical problem arises with the improper distribution of samples among classes known as the class imbalance problem, which is addressed by employing a simple statistical methodology. Transfer learning is a known technique for training small datasets which transfers pre-trained weights learned on a large dataset. However, during transfer learning, negative transfer learning is a common problem. Therefore, a stepwise transfer learning approach is proposed which can help in fast convergence, while reducing overfitting and preventing negative transfer learning during knowledge transfer across domains. The system is trained and evaluated on two plant disease datasets i.e., PlantVillage (a publicly available dataset) and pepper disease dataset provided by the National Institute of Horticultural and Herbal Science, Republic of Korea. The pepper dataset is particularly challenging as it contains images from different parts of the plant such as the leaf, pulp, and stem. The proposed system has dominated the previous works on the PlantVillage dataset and achieved 99% and 99.69% accuracy on the Pepper dataset and PlantVillage datasets, respectively.

The rapid spreading of RGB-D cameras has led to wide applications of 3D videos in both academia and industry, such as 3D entertainment and 3D visual understanding. Under these circumstances, extensive research efforts have been dedicated to RGB-D camera--oriented topics. In these topics, quality promotion of depth videos with the temporal characteristic is emerging and important. Due to the limited exposure time of RGB-D cameras, object movement can easily lead to motion blurs in intensive images, which can further result in obvious artifacts (holes or fake boundaries) in the corresponding depth frames. With regard to this problem, we propose a depth error elimination method based on time series analysis to remove the artifacts in depth images. In this method, we first locate the regions with erroneous depths in intensive images by using motion blur detection based on a time series analysis model. This is based on the fact that the depth image is calculated by intensive color images that are captured synchronously by RGB-D cameras. Then, the artifacts, such as holes or fake boundaries, are fixed by a depth error elimination method. To evaluate the performance of the proposed method, we conducted experiments on 250 images. Experimental results demonstrate that the proposed method can locate the error regions correctly and eliminate these artifacts effectively. The quality of depth video can be improved significantly by using the proposed method.

The intent of this study was to develop a technique for weight and ripeness estimation of oil palm (Elaeis guieensis Jacq. var. tenera) bunches from hyperspectral and RGB color images. In the experiments, color and hyperspectral images of the bunch were acquired from four different angles, each differing by 90 degrees. Acquired RGB color images were converted to HSI and L*a*b color space. Gray-scale thresholds were used to identify the area of the bunch and the area of space between the fruits. The total number of pixels in the bunch and the space were counted, respectively. In the hyperspectral images, the total number of pixels in the bunch was also counted from an image composed of three wavelengths (560 nm, 680 nm, and 740 nm), while the total number of pixels of space between fruits was obtained at a wavelength of 910 nm. From these sets of data, weight-estimation equations were determined by linear regression (LR) or multiple linear regression (MLR). As a result, the coefficient of determination (R2) of actual weight and estimated weight were at a level of 0.989 and 0.992 for color and hyperspectral images, respectively. Estimation of oil palm bunch ripeness was also tested. Bunches belonging to 4 classes of ripeness (overripe, ripe, underripe, and unripe) were used for this study. Since ripeness estimation from overall data from a bunch was quite difficult, we focused on the difference in colors or reflectivity of the portion concealed and not-concealed with fronds. Euclidean distances between the test sample and the standard 4 classes of ripeness were calculated, and the test sample was classified into the ripeness class that had the shortest distance from the sample. In the classification based on color image, average RGB values of concealed and not-concealed areas were used, while in hyperspectral images the average intensity values of fruits pixels from the concealed area were used. The results of validation experiments with the developed estimation methods indicated acceptable estimation accuracy, and a possibility for practical use to estimate the ripeness of oil palm bunches.