Human Detection from RGB Depth Image using Active contour and Grow-cut Segmentation

From object tracking to 3D reconstruction, RGB-Depth (RGB-D) camera networks play an increasingly important role in many vision and graphics applications. Practical applications often use sparsely-placed cameras to maximize visibility, while using as few cameras as possible to minimize cost. In general, it is challenging to calibrate sparse camera networks due to the lack of shared scene features across different camera views. In this paper, we propose a novel algorithm that can accurately and rapidly calibrate the geometric relationships across an arbitrary number of RGB-D cameras on a network. Our work has a number of novel features. First, to cope with the wide separation between different cameras, we establish view correspondences by using a spherical calibration object. We show that this approach outperforms other techniques based on planar calibration objects. Second, instead of modeling camera extrinsic calibration using rigid transformation, which is optimal only for pinhole cameras, we systematically test different view transformation functions including rigid transformation, polynomial transformation and manifold regression to determine the most robust mapping that generalizes well to unseen data. Third, we reformulate the celebrated bundle adjustment procedure to minimize the global 3D reprojection error so as to fine-tune the initial estimates. Finally, our scalable client-server architecture is computationally efficient: the calibration of a five-camera system, including data capture, can be done in minutes using only commodity PCs. Our proposed framework is compared with other state-of-the-arts systems using both quantitative measurements and visual alignment results of the merged point clouds.

Background: Detection and tracking of humans in real time is a challenging and important field of research. Human detection has many applications in detection of human and video surveillance. Nowadays, quadcopters are used in different fields like research, military and law enforcement, commercial fields for navigation, searching, and areal imagery. Methods: In this paper, the primary objective of the system is to detect and locate people who got isolated in some areas due to natural disasters like floods and earthquake. It is carried out using a single-board computer Raspberry Pi (RPI), and the image processing part is carried out with the help of OpenCV. The detection and tracking of humans is done using quadcopter. Quadcopter is equipped with camera, the video capture is processed using RPI, and live streaming along with human detection is sent to the base station over the wireless network. Applications: It is used in effective human detection in real time video streaming. It helps the rescue team to locate people and to know their current situation and thus to take necessary actions including medication and evacuation. Development/future work: A prototype model for human detection and tracking using quadcopter is designed and developed. Better battery pack for longer flying time and use of 3G USB dongle for large area surveillance.

This paper presents a new approach to real-time human detection and tracking in cluttered and dynamic environments by integration of RGB and depth data. We introduce the notion of Point Ensemble Image, which fully encodes both RGB and depth information from a virtual plan-view perspective, and we reveal that human detection and tracking in 3D space can be performed very effectively based on this new representation. Our human detector is able to take advantage of depth data by effectively locate physically plausible candidates as a first step, and then both depth and color information is made full use of in a supervised learning manner at the second stage. 3D trajectories of humans are finally generated by data association in which joint statistics of color and height are computed and compared. Experimental results show that the system is able to work satisfactorily in complex real-world situations.

Early diagnosis of cerebral palsy in infants has produced promising results using tools like the General Movement Assessment (GMA). Pose estimation of infants lying supine is an important step towards an automated system for GMA. Developing methods for accurate, reliable, fast estimation of the three-dimensional (3D) position of human limbs and proposing motion features suitable for 3D models to classify typical and atypical movements are attracting increasing research interest lately. In this study, we propose a 3D pose estimation method with low training cost suitable for infants in lying positions. The method uses an existing two-dimensional human body keypoint detection method combined with the in-depth information in Red-Green-Blue Depth (RGB-D) data from a Kinect sensor. The method is evaluated using the Moving INfants In RGB-D (MINI-RGBD) open dataset. The results show that the average error of the estimated body part length is 13.76 mm, while the accuracy of the Percentage of Correctly-localized Parts (PCP) and Percentage of Correct Keypoint (PCK) is 80.7 and 86.1%, respectively. The results are comparable to those achieved in the baseline study performed by the researchers who generated this open dataset. The advantage of our method is its low training cost.

With the increasing popularity of RGB-depth (RGB-D) sensor, research on the use of RGB-D sensors to reconstruct three-dimensional (3D) indoor scenes has gained more and more attention. In this paper, an automatic point cloud registration algorithm is proposed to efficiently handle the task of 3D indoor scene reconstruction using pan-tilt platforms on a fixed position. The proposed algorithm aims to align multiple point clouds using extrinsic parameters of the RGB-D camera obtained from every preset pan-tilt control point. A computationally efficient global registration method is proposed based on transformation matrices formed by the offline calibrated extrinsic parameters. Then, a local registration method, which is an optional operation in the proposed algorithm, is employed to refine the preliminary alignment result. Experimental results validate the quality and computational efficiency of the proposed point cloud alignment algorithm by comparing it with two state-of-the-art methods.

Measuring characteristics of animals in the wild is not always possible, due to their demeanour and lack of human contact. Remote capture and processing methods, including the segmentation of animal data into relevant body parts, are required. Existing solutions are either costly or too cumbersome to use in the wild. This study explores the use of RGB depth (RGB-D) cameras for data capture of a target animal from a distance. In addition, this study explores the extraction and segmentation of the resulting animal data into point clouds, and the creation of machine learning models for the automated segmentation of this data. Results of this study, including an experimental evaluation, demonstrate the feasibility of utilizing RGB-D cameras for animal data capture, and that classification outperformed clustering for automated animal data segmentation.

Unexpected fall remains the major threat for elderly since it can cause severe consequences. In this paper, an automatic human fall detection approach is proposed using RGB-Depth (RGBD) cameras. Based on the skeleton and joints data, three velocity features and a head-hip height difference feature are extracted within each sliding window. The features are then gradually analyzed and compared to adaptive thresholds by the proposed multi-step fall detection method. We also conducted a series evaluation on the determination of thresholds and the whole approach. The experiments show a good result with a high F1-measure of 0.944.

Superpixel segmentation is a widely used preprocessing method in computer vision, but its performance is unsatisfactory for color images in cluttered indoor environments. In this work, a superpixel method named weighted coplanar feature clustering (WCFC) is proposed, which produces full coverage of superpixels in RGB-depth (RGBD) images of indoor scenes. Basically, a linear iterative clustering is adopted based on a cluster criterion that measures the color similarity, space proximity and geometric resemblance between pixels. However, to avoid the adverse impact of RGBD image flaws and to make full use of the depth information, WCFC first preprocesses the raw depth maps with an inpainting algorithm called a Cross-Bilateral Filter. Second, a coplanar feature is extracted from the refined RGBD image to represent the geometric similarities between pixels. Third, combined with the colors and positions of the pixels, the coplanar feature constructs the feature vector of the clustering method; thus, the distance measure, as the cluster criterion, is computed by normalizing the feature vectors. Finally, in order to extract the features of the RGBD image dynamically, a content-adaptive weight is introduced as a coefficient of the coplanar feature, which strikes a balance between the coplanar feature and other features. Experiments performed on the New York University (NYU) Depth V2 dataset demonstrate that WCFC outperforms the available state-of-the-art methods in terms of accuracy of superpixel segmentation, while maintaining a high speed.

Viewpoint variation is one of the main challenges for object-detection frameworks. In this work we describe strategies to improve object detection pipelines by introducing viewpoint based mixture components. We learn accurate mixtures of object detectors for RGB-Depth (RGBD) data using the latent SVM framework. Our contributions are threefold. First, we use surface-based object representations (3D mesh models) from available 3D object model repositories to learn strongly supervised viewpoint classifiers. These are used to guide the first stages of model learning, and help avoid inaccurate local minima of latent SVM training. Second, we develop a geometric dataset augmentation scheme that uses scene geometry to ‘take another look’ at the training data, simulating the effect of camera viewpoint changes. Third, to better exploit depth information, we develop a novel depth-based dense feature extraction method that provides a robust statistical description of scene geometry. We evaluate our learned detectors on the NYU dataset, and demonstrate that each of our advances results in systematic performance improvements over the traditional HOG-based detection pipeline.

Accurate extrinsic calibration of inertial sensors, such as Inertial Measurement Units (IMUs) and cameras is crucial for trajectory estimation of Uncrewed Aerial Vehicles (UAVs). While numerous calibration methods have been proposed, these techniques often rely on specialized equipment, planar targets, and an initial estimate of the calibration parameters. In this research, we propose a targetless calibration method designed for UAVs equipped with IMUs and RGBDepth (RGB-D) cameras. Our approach leverages deep-learning-based floor-segmentation to extract ground points from the depth channel of RGB-D images. Subsequently, the normal vector to these points is estimated. The known orientation of the normal to the floor segment and the gravity vector sensed in the accelerometer’s frame are utilized in a robust estimation approach to estimate the extrinsic calibration parameters. We illustrate that the developed method outperforms MATLAB’s Toolboxes and exhibits similar performance to Kalibr without the use of specialized checkerboard targets.

Body cleanliness is considered an important indicator for evaluating cow welfare. At present, assessing the cleanliness of different cow body parts is considered as a subjective and labor-intensive task. Automatic body cleanliness scoring needs to start with body parts segmentation. Despite the fact that pattern recognition methods for human body detection and analysis have flourished in the last decade, computer vision analysis of cow body-part is scarce in literature, and most of cow body detection in video images recognizes and segments the cow as a whole, not using all of the body part information. This study presents a computer vision method that automatically identifies nine cow’s body parts, i.e. head, torso, udder, belly (or rear), left foreleg, right foreleg, left hindleg right hindleg and tail. We used a method for body parts detection using an improved version of template matching. The entire image process included video recording, image restoration and pre-processing, skeleton extraction, templates matching, recognition of right or left leg, calibration and body parts identification. 1421 side-view images of 113 cows and 859 back-view images of 75 cows were collected on a Chinese research dairy farm using an RGB-depth camera installed in the barn for Holstein lactating cows. The average body parts segmentation accuracies for side-view and back-view were 96% and 91%, respectively. The results indicate that it is possible to automatically detect and extract body parts from RGB-depth images without any manual interference.

The thesis addresses the following challenging problems of detecting and tracking humans in the presence of occlusions in typical surveillance videos: (1) adaptation of semantic-part-based human detectors to new surveillance video sequence when trained detectors using other video data not performing well on the new video data; (2) tracking of humans with person identification minimizing identification errors over longer tracking periods; and (3) hierarchical spatial and temporal analysis for discriminative tracking of human targets. The thesis aims to improve the state-of-the-art performance in human detection and tracking by studying the human detectors, extended tracking of track segments (tracklets) generated from short term tracking of detection responses. For the adaptation of semantic-part-based human detectors to new surveillance video sequence, a uni ed deep CNN model for joint learning of features, semantic pedestrian part detectors and a transfer learning model is developed. The components within this deep CNN model interact with each other in the learning process, which facilitates the optimization of the learned components during the co-operative learning. In particular, an adaptation layer is proposed to embed the capability of knowledge transfer into the CNN model. As a result, the proposed transferred CNN (T-CNN) model is able to transfer the visual knowledge of the semantic pedestrian parts from the source data to target data. Extensive experimental evaluations show that the proposed method is better than other deep learning based methods in terms of detection performance. Moreover, the adaptive deep features can be complementary to the pre-defined features used by other state-of-the-art methods. For tracking of humans with person identification minimizing identification errors over longer tracking periods, a novel method, based on online target-specific metric learning and coherent dynamics estimation, for tracklet association by network flow optimization is developed. The proposed framework aims to exploit appearance and motion cues to prevent identity switches during tracking and also to recover missed detections. The target-specific metrics (appearance cue) and motion dynamics (motion cue) are proposed to be learned and estimated online, i.e. during the tracking process. Furthermore, a learning algorithm to learn the weights of motion and appearance tracking cues for tracklet affinity models is proposed to handle some difficult situations. Extensive evaluations following state-of-the-art practices have been conducted and the results from these evaluations show the improvements by the proposed method over some existing state-of-the-art methods. In hierarchical spatial and temporal analysis for discriminative tracking of human targets, inspired by recent advances in convolutional neural network (CNN) architectures, a novel uni ed deep model for tracklet association, which can jointly learn the CNNs and temporally constrained metrics, is developed. Furthermore, a novel loss function incorporating temporally constrained multi-task learning mechanism is developed to make the deep model more effective in solving the tracklet association problem. Extensive experimental results comparing with the state-of-the-art methods demonstrate the effectiveness and superiority of the proposed unified deep model.

All video surveillance system uses some important steps to detach moving object from background and thus differentiate all the objects in video image frames. The main endeavor of this paper is to use an object bounding box concept to detect the human blob and human activity detection using a system. The images are captured by using single static camera to make the system cost effective. The bounding box that bounds each object in the image frames can be utilized to track and detect activities of the moving objects in further frames. These bounding boxes can be utilized to detect human and human activities like crowd and the estimation of crowd. This paper gives the implementation results of bounding box for activity detection.KeywordsVideo surveillanceHuman blobBounding BoxesHuman activity detectionStatic cameraCrowdCrowd estimationTracking

Real-time three-dimensional (3D) reconstruction of real-world environments has many significant applications in various fields, including telepresence technology. When depth sensors, such as those from Microsoft's Kinect series, are introduced simultaneously and become widely available, a new generation of telepresence systems can be developed by combining a real-time 3D reconstruction method with these new technologies. This combination enables users to engage with a remote person while remaining in their local area, as well as control remote devices while viewing their 3D virtual representation. There are numerous applications in which having a telepresence experience could be beneficial, including remote collaboration and entertainment, as well as education, advertising, and rehabilitation. The purpose of this systematic literature review is to analyze the recent advances in 3D reconstruction methods for telepresence systems and the significant related work in this field. Next, we determine the input data and the technological device employed to acquire the input data, which will be utilized in the 3D reconstruction process. The methods of 3D reconstruction implemented in the telepresence system as well as the evaluation of the system, have been extracted and assessed from the included studies. Through the analysis and summarization of many dimensions, we discussed the input data used for the 3D reconstruction method, the real-time 3D reconstruction methods implemented in the telepresence system, and how to evaluate the system. We conclude that real-time 3D reconstruction methods for telepresence systems have progressively improved over the years in conjunction with the advancement of machines and devices such as Red Green Blue-Depth (RGB-D) cameras and Graphics Processing Unit (GPU).

Human position tracking and detection of human being is used as an initial step in biometric recognition. Continuous track a person will allow identifying people at any time. Its system consists of two parts person detection and another one is tracking. Active contours are a catch-all name for methods that found the curve that best separates objects in an image. This is known as segmentation. Segmentation is extraordinarily related to tracking. In 3D - it discusses about active surfaces. One way to divide segmentation methods - single object vs. multiple objects. Another way to divide segmentation methods: Edge-based - the object to be segmented should accept its boundary visible in the image, as some sort of outstanding edge. Region-based - the region of the object in the image should have a different actuarial in some sort of feature space, estimate to its surroundings. These functional are integrating with information on the shape of the area being extracted.