- Research Article
15
- 10.1016/j.image.2020.115876
Segmentation mask guided end-to-end person search
- May 08, 2020
- Signal Processing: Image Communication
- Dingyuan Zheng + 3 more +3
Segmentation mask guided end-to-end person search
Person re-identification (re-id) aims at matching person images across multiple surveillance cameras. Currently, most re-id systems highly rely on color cues, which are only effective in good illumination conditions, but fail in low lighting conditions. However, for security issues, it is very important to conduct surveillance in low lighting conditions. To remedy this problem, we propose using depth cameras to perform surveillance in dark places, while using traditional RGB cameras in bright places. Such a heterogeneous camera network brings a challenge to match images across depth and RGB cameras. In this paper, we mine the correlation between two heterogeneous cues (depth and RGB) on both feature-level and transformation-level. As such, depth-based features and RGB-based features are transformed to the same space, which alleviates the problem of cross-modality matching between depth and RGB cameras. Experimental results on two benchmark heterogeneous person re-id datasets show the effectiveness of our method.
Segmentation mask guided end-to-end person search
Segmentation mask guided end-to-end person search
Single-shot extrinsic calibration of a generically configured RGB-D camera rig from scene constraints
With the increasing use of commodity RGB-D cameras for computer vision, robotics, mixed and augmented reality and other areas, it is of significant practical interest to calibrate the relative pose between a depth (D) camera and an RGB camera in these typ
Read moreAdding Object Manipulation Capabilities to Social Robots by using 3D and RGB Cameras Data
This paper outlines the design and implementation of novel object manipulation for a social robot, here Pepper by SoftBank Robotics. It is primarily designed for verbal interaction and has therefore not been equipped with object manipulation capabilities. The proposed routine exploits the built-in RGB and 3D cameras. First, semantic segmentation based on the Mini-YOLOv3 neural network is run on the RGB image. Next, 3D sensor data are used to position the hand over the object, implementing a novel routine to grab the object and to scan it for recognition purposes. To preserve patient and location sensitive data, the here-proposed architecture operates automatically and offline, running on the robot’s operating system. Experimental results on 370 grabbing processes showed how the manipulation routine achieves a grabbing success rate of up to 96%. They also proved that the success rate remains unaltered if the target object is positioned in a rectangular area of ± 6 cm × ± 3 cm centered in the nominal position provided by an initial positioning grid. The grabbing success rate remains above 80% even if the object to be grabbed is stored with an angle that ranges between 10° and 45° within the above-reported area.
Read morePerson Re-Identification with RGB-D Camera in Top-View Configuration through Multiple Nearest Neighbor Classifiers and Neighborhood Component Features Selection
Person re-identification is an important topic in retail, scene monitoring, human-computer interaction, people counting, ambient assisted living and many other application fields. A dataset for person re-identification TVPR (Top View Person Re-Identification) based on a number of significant features derived from both depth and color images has been previously built. This dataset uses an RGB-D camera in a top-view configuration to extract anthropometric features for the recognition of people in view of the camera, reducing the problem of occlusions while being privacy preserving. In this paper, we introduce a machine learning method for person re-identification using the TVPR dataset. In particular, we propose the combination of multiple k-nearest neighbor classifiers based on different distance functions and feature subsets derived from depth and color images. Moreover, the neighborhood component feature selection is used to learn the depth features’ weighting vector by minimizing the leave-one-out regularized training error. The classification process is performed by selecting the first passage under the camera for training and using the others as the testing set. Experimental results show that the proposed methodology outperforms standard supervised classifiers widely used for the re-identification task. This improvement encourages the application of this approach in the retail context in order to improve retail analytics, customer service and shopping space management.
Read moreTop-Push Constrained Modality-Adaptive Dictionary Learning for Cross-Modality Person Re-Identification
Person re-identification aims to match person captured by multiple non-overlapping cameras that mainly mean standard RGB cameras. In contemporary surveillance, cameras of different modalities such as infrared cameras and depth cameras are introduced because of their unique advantages in poor illumination scenarios. However, re-identifying the persons across such cameras of different modalities is extremely difficult and, unfortunately, seldom discussed. It is mainly caused by extremely different appearances of the person shown under such different camera modalities. In this paper, we tackle this challenging cross-modality people re-identification through a top-push constrained modality-adaptive dictionary learning. The proposed model asymmetrically projects the heterogeneous features from dissimilar modalities onto a common space. In this way, the modality-specific bias is mitigated. Thus, the heterogeneous data can be simultaneously enforced by a shared dictionary in a canonical space. Moreover, a top-push ranking graph regularization is embedded in the proposed model to improve the discriminability, which efficiently further boosts the matching accuracy. In order to implement the proposed model, an iterative process is developed in this paper to optimize these two processes jointly. Extensive experiments on the benchmark SYSU-MM01 and BIWI RGBD-ID person re-identification datasets show promising results which outperform state-of-the-art methods.
Read moreAdvanced Scene Sensing for Virtual Teleconference
Over the last year the need for video conferences has risen significantly due to the ongoing global pandemic. The goal of this project is to improve user experience from having access to only voice and plain 2D image by adding a third spatial dimension, creating a more immersive setting. Azure Kinect Development Kit utilizes multiple cameras, namely the RGB camera and depth camera. Depth camera is based on ToF principle, which uses near-IR to cast modulated illumination onto the scene. The setup uses multiple Azure Kinect devices in sync and offset in space to obtain non-static 3D capture of a person. Unity engine with Azure Kinect SDK is used to process the data gathered by all devices. Firstly, a depth spatial map is created by combining overlaid outputs from each device. Secondly, RGB pixels are mapped onto depth spatial points to provide a final texture to the 3D model. Taking into account the need to export a continuous capture of raw data to a server, body tracking and image processing algorithms are used. Finally, the processed data can be exported and utilized in AR, VR or any other 3D capable interface. This 3D projection aims to enhance sensory experience by utilising non-verbal communication along with classical speech in video conferences.Keywords3D sensingVolumetric alignmentPoint cloud
Read morePerson re-identification from depth cameras using skeleton and 3D face data
In the typical approach, person re-identification is performed using appearance in 2D still images or videos, thus invalidating any application in which a person may change dress across subsequent acquisitions. For example, this is a relevant scenario for home patient monitoring. Depth cameras enable person re-identification exploiting 3D information that captures biometric cues such as face and characteristic dimensions of the body. Unfortunately, face and skeleton quality is not always enough to grant a correct recognition from depth data. Both features are affected by the pose of the subject and the distance from the camera. In this paper, we propose a model to incorporate a robust skeleton representation with a highly discriminative face feature, weighting samples by their quality. Our method combining face and skeleton data improves rank-1 accuracy compared to individual cues especially on short realistic sequences.
Read moreImplementation of perspective-n-point techniques and YOLOv5 algorithm based on surveillance camera for localization
The technology of processors has advanced significantly, resulting in smaller and more powerful devices with much processing capability. Particularly, camera technology has witnessed extensive research in utilizing images for various applications. Currently, surveillance cameras are widely used for security purposes when abnormal events occur. In this research, the benefits of utilizing data from surveillance cameras are explored to assist in determining the position of a moving robot using the perspective-n-point (PnP) technique. the scale factor, which varies, has been improved by integrating checks with the YOLOv5 algorithm. This algorithm employs a custom model to specifically detect the robot of interest, enabling the determination of its real-world position using multiple surveillance cameras. These cameras have different perspectives within the same area. Considering the deviation caused by determining the position from a single viewpoint, multiple cameras are employed to mitigate this issue.
Read moreSign Language Recognition Using Microsoft Kinect
Sign language is a language used by people with hearing impairments to communicate using gestures. Hand motion is one of the strategies utilized in gesture-based communication. Through the years, many researchers have put forth their models for automatic sign language recognition. The use of an RGB camera for recognition was a development, some of the models required signers to wear cyber gloves, etc., for ease in recognition. Despite the fact that the utilization of gesture-based communication is mainstream among the hard of hearing individuals yet different groups do not attempt to learn it, this hinders communication and thus turns into a reason for the disconnection of certain groups in society. There is a necessity of a framework that can encourage a method for communication between these two groups. This paper provides a model that overcomes issues with using just an RGB camera. The framework makes use of Kinect, an RGB camera and a depth sensor. The model was developed using a convolutional neural network.KeywordsSign languageRGBKinectConvolutional neural network
Read moreSingle-modal Incremental Terrain Clustering from Self-Supervised Audio-Visual Feature Learning
The key to an accurate understanding of terrain is to extract the informative features from the multi-modal data obtained from different devices. Sensors, such as RGB cameras, depth sensors, vibration sensors, and microphones, are used as the multi-modal data. Many studies have explored ways to use them, especially in the robotics field. Some papers have successfully introduced single-modal or multi-modal methods. However, in practice, robots can be faced with extreme conditions; microphones do not work well in crowded scenes, and an RGB camera cannot capture terrains well in the dark. In this paper, we present a novel framework using the multi-modal variational autoencoder and the Gaussian mixture model clustering algorithm on image data and audio data for terrain type clustering. Our method enables the terrain type clustering even if one of the modalities (either image or audio) is missing at the test-time. We evaluated the clustering accuracy with a conventional multi-modal terrain type clustering method and we conducted ablation studies to show the effectiveness of our approach.
Read moreHuman activity recognition: A review
Human Activity Recognition is one of the active research areas in computer vision for various contexts like security surveillance, healthcare and human computer interaction. In this paper, a total of thirty-two recent research papers on sensing technologies used in HAR are reviewed. The review covers three area of sensing technologies namely RGB cameras, depth sensors and wearable devices. It also discusses on the pros and cons of the mentioned sensing technologies. The findings showed that RGB cameras have lower popularity when compared to depth sensors and wearable devices in HAR research.
Read moreRGBDTCalibNet: End-to-end Online Extrinsic Calibration between a 3D LiDAR, an RGB Camera and a Thermal Camera
For applications in robotics and autonomous vehicles, fusing 3D LiDAR, RGB camera, and thermal camera can improve the perception in day and night environments. To fuse the sensors, accurate extrinsic calibration is important. However, most existing methods use offline calibration, which is tedious, as it requires the use of special targets and human intervention. Meanwhile, among the online calibration methods, little attention has been paid to LiDAR and thermal camera online calibration. Thus, in this paper, an online extrinsic calibration method is proposed to solve the problem, called RGBDTCalibNet. It can achieve the extrinsic calibration between a 3D LiDAR and an RGB camera, as well as a 3D LiDAR and a thermal camera. It leverages CNN-based deep learning methods for feature extraction and feature matching. The proposed network only takes a pair of images and point cloud as the input and outputs the extrinsic parameters in a single shot. It is trained and tested on two datasets: KITTI360 and our own dataset collected in the NTU campus, using two types of LiDAR: Velodyne HDL-64 and Livox Horizon. Both demonstrate the effectiveness, robustness and accuracy of RGBDTCalibNet. The code will be open-sourced at https://github.com/sanatmharolkar/RGBDTCalibNet.git
Read moreHigh Quality Photometric Reconstruction Using a Depth Camera
In this paper we present a depth-guided photometric 3D reconstruction method that works solely with a depth camera like the Kinect. Existing methods that fuse depth with normal estimates use an external RGB camera to obtain photometric information and treat the depth camera as a black box that provides a low quality depth estimate. Our contribution to such methods are two fold. Firstly, instead of using an extra RGB camera, we use the infra-red (IR) camera of the depth camera system itself to directly obtain high resolution photometric information. We believe that ours is the first method to use an IR depth camera system in this manner. Secondly, photometric methods applied to complex objects result in numerous holes in the reconstructed surface due to shadows and self-occlusions. To mitigate this problem, we develop a simple and effective multiview reconstruction approach that fuses depth and normal information from multiple viewpoints to build a complete, consistent and accurate 3D surface representation. We demonstrate the efficacy of our method to generate high quality 3D surface reconstructions for some complex 3D figurines.
Read moreClustering and Identification of key body extremities through topological analysis of multi-sensors 3D data
This paper presents a framework of a marker-less human pose recognition system by identifying key body extremity parts through a network of calibrated low-cost depth sensors. The usage of depth sensors overcomes challenges related to low illuminations which usually compromises the information from the RGB cameras. Furthermore, the addition of multiple depth sensors complements the existing information with more visibility and less self-occlusion. A simple algorithm was applied which finds the connections between aligned and updated meshes produced from multiple sensors. These connections help to fuse the meshes into one large geodesic graph network. On this graph, a novel algorithm is applied to identify key body extremities such as head, hands, and feet of a human subject. A geodesic mapping is applied to the fused point cloud to produce a set of distinct topological clusters of 3D points. These clusters generate a hierarchical skeleton tree graph (Reeb graph) and produce a set of features for semantic identification of key body extremities. The combination of both the shape model and semantic classification finally leads to pose recognition. The paper presents the assessment of the proposed framework and its comparison with another available technique in a succession of experimental configurations.
Read moreEV-LFV: Synthesizing Light Field Event Streams From an Event Camera and Multiple RGB Cameras.
Light field videos captured in RGB frames (RGB-LFV) can provide users with a 6 degree-of-freedom immersive video experience by capturing dense multi-subview video. Despite its potential benefits, the processing of dense multi-subview video is extremely resource-intensive, which currently limits the frame rate of RGB-LFV (i.e., lower than 30 fps) and results in blurred frames when capturing fast motion. To address this issue, we propose leveraging event cameras, which provide high temporal resolution for capturing fast motion. However, the cost of current event camera models makes it prohibitive to use multiple event cameras for RGB-LFV platforms. Therefore, we propose EV-LFV, an event synthesis framework that generates full multi-subview event-based RGB-LFV with only one event camera and multiple traditional RGB cameras. EV-LFV utilizes spatial-angular convolution, ConvLSTM, and Transformer to model RGB-LFV's angular features, temporal features, and long-range dependency, respectively, to effectively synthesize event streams for RGB-LFV. To train EV-LFV, we construct the first event-to-LFV dataset consisting of 200 RGB-LFV sequences with ground-truth event streams. Experimental results demonstrate that EV-LFV outperforms state-of-the-art event synthesis methods for generating event-based RGB-LFV, effectively alleviating motion blur in the reconstructed RGB-LFV.
Read more