An Efficiently 3D Face Recognizing Method Using Range Image and Sparse Representation

Recent researches in three-dimensional (3D) face recognition area have proved that, 3D face recognition methods, achieve better accuracy than its 2D counterpart. One of the main advantages of 3D face recognition methods is that it measures the geometry of rigid features on the human face, due to which it becomes an invariant to illumination, expressions and rotations of head (pose variation). But the main drawback of 3D face recognition methods is the acquisition of 3D face images which generally needs a range camera. It is where the research community is facing difficulty. Research in 3D face recognition also depends on available databases. In this paper, we present a qualitative review of all the available 3D face databases. A comparative table of all the databases is also given at the end with all the necessary parameters.

In this thesis we present two applications where 3D surface modeling and registration play a role of paramount importance. The first application is in archaeology, where our proposed method assists the archaeologist in the tedious process of reconstructing ceramic vessels from the excavated fragments (or shards). The method takes advantage of the surface intrinsic features coupled with a library of generic models constructed by the archaeologists to produce a virtual reconstruction of what the original vessel may have looked like. Generic models are generated based on the expert historical knowledge of the period, provenance of the artifact, and location of the digging site. The generic models need not to be identical to the original vessel, but must be within a geometric transformation of it in most of its parts. The methods are suited for ceramic vessels with distinctive color patterns or reliefs (i.e., surface with molding, carving, or stamping). We introduce a novel curvature B-spline fitting method that makes the extraction of robust features possible. The alignment of the shards against the generic model uses a novel set of 3D weighted curve moments and absolute affine invariants derived from them. The transformation is computed from corresponding parabolic contours on the shard and the generic model. When the shards have no color or relief on them, they are virtually mended to abutting shards using intrinsic differential anchor points computed on the shards break curves and a set of absolute invariants. For axially symmetric vessels, a global constraint induced by the surface of revolution is applied to guarantee global mending consistency. In our experiment involving around 300 shards, more than 85% of them are successfully mended. The mended shards consist of over 90% surface areas of the ceramic vessels. The second application is in 3D human face synthesis and recognition, where our proposed method assists the national security agencies in constructing a 3D face database to be used for spotting missing personnel or identify possible suspects when close up images of that person are flagged up. A gender/race generic 3D face model is morphed into a personalized unique 3D face of a given person from using a limited number of 2D images of the given person taken from different viewpoints. The 2D images need not to be of the same resolution, or taken under similar brightness or same place, or even same time period. The method applies also to drawing sketches of the person in question. The reconstructed 3D personalized model is an accurate rendition of the actual human face, and the reconstruction errors are smaller than a 3D scanner resolution that is used to scan the person's face. Our method renders a difficult 2D face recognition problem into a much simpler 3D surface registration (alignment) problem. Given a set of images of a test person, a personalized 3D face model is constructed and "best" aligned with one of the 3D faces stored in the database. When the minimum alignment errors are below an accepted threshold (lower than the scanner resolution), the identity of the test person is matched with the one in the database. Experimental results are shown in controlled environment with high resolution images, under ideal illumination condition, as well as for images taken in uncontrolled environment with low resolution, arbitrary illumination. In both conditions, the recognition rate is over 95%. It is also proved that better face recognition rate is achieved by using a gender/race based generic 3D face model which allows for accurate pose estimation as well as the face identification in 3D.

Foetal alcohol syndrome (FAS) is a preventable condition caused by maternal alcohol consumption during pregnancy. The FAS facial phenotype is an important factor for diagnosis, alongside central nervous system impairments and growth abnormalities. Current methods for analysing the FAS facial phenotype rely on 3D facial image data, obtained from costly and complex surface scanning devices. An alternative is to use 2D images, which are easy to acquire with a digital camera or smart phone. However, 2D images lack the geometric accuracy required for accurate facial shape analysis. Our research offers a solution through the reconstruction of 3D human faces from single or multiple 2D images. We have developed a framework for evaluating 3D human face reconstruction from a single-input 2D image using a 3D face model for potential use in FAS assessment. We first built a generative morphable model of the face from a database of registered 3D face scans with diverse skin tones. Then we applied this model to reconstruct 3D face surfaces from single frontal images using a model-driven sampling algorithm. The accuracy of the predicted 3D face shapes was evaluated in terms of surface reconstruction error and the accuracy of FAS-relevant landmark locations and distances. Results show an average root mean square error of 2.62 mm. Our framework has the potential to estimate 3D landmark positions for parts of the face associated with the FAS facial phenotype. Future work aims to improve on the accuracy and adapt the approach for use in clinical settings. Significance: Our study presents a framework for constructing and evaluating a 3D face model from 2D face scans and evaluating the accuracy of 3D face shape predictions from single images. The results indicate low generalisation error and comparability to other studies. The reconstructions also provide insight into specific regions of the face relevant to FAS diagnosis. The proposed approach presents a potential cost-effective and easily accessible imaging tool for FAS screening, yet its clinical application needs further research.

The use of three-dimensional (3D) facial scans for facial analysis is increasing in maxillofacial treatment. The aim of this study was to investigate the consistency of two-dimensional (2D) and 3D facial analyses performed by multiple raters. Six men and four women (25–36-year-old) participated in this study. The 2D images of the smiling and resting faces in the frontal and sagittal planes were obtained. The 3D facial and intraoral scans were merged to generate virtual 3D faces. Ten clinicians performed facial analyses by investigating 14 indices of 2D and 3D faces. Intra- and interrater agreements of the results of 2D and 3D facial analyses within and among the participants were evaluated. The intrarater agreement between the 2D and 3D facial analyses varied according to the indices. The highest and lowest agreements were found for the dental crowding index (0.94) and smile line curvature index (0.56) in the frontal plane, and Angle's classification (canine) index (0.98) and occlusal plane angle index (0.55) in the profile plane. In the frontal plane, the interrater agreements were generally higher for the 3D images than for the 2D images, while in the profile plane, the interrater agreements were high in the Angle's classification (canine) index however low in the other indices. Several occlusion-related indices were missing in the 2D images because the posterior teeth were not observed. Esthetic analysis results between 2D and 3D face images can differ according to the evaluation indices. The use of 3D faces is recommended over 2D images to increase the reliability of facial analyses, as it can fully assess both esthetic and occlusion-related indices.

The first handicap in 3D faces recognizing under unconstrained problem is the largest variability of the visual aspect when we use various sources. This great variability complicates the task of identifying persons from their 3D facial scans and it is the most reason that bring to face detection and recognition of the major problems in pattern recognition fields, biometrics and computer vision. We propose a new 3D face identification and recognition method based on Gappy Wavelet Neural Network (GWNN) that is able to provide better accuracy in the presence of facial occlusions. The proposed approach consists of three steps: the first step is face detection. The second step is to identify and remove occlusions. Occluded regions detection is done by considering that occlusions can be defined as local face deformations. These deformations are detected by a comparison between the input facial test wavelet coefficients and wavelet coefficients of generic face model formed by the mean data base faces. They are beneficial for neighborhood relationships between pixels rotation, dilation and translation invariant. Then, occluded regions are refined by removing wavelet coefficient above a certain threshold. Finally, the last stage of processing and retrieving is made based on wavelet neural network to recognize and to restore 3D occluded regions that gathers the most. The experimental results on this challenging database demonstrate that the proposed approach improves recognition rate performance from 93.57 to 99.45 % which represents a competitive result compared to the state of the art.

Proceedings of the 2011 joint ACM workshop on Human gesture and behavior understanding

Three-dimensional object construction has seen a great deal of activity in the past decade, as has been pointed out in recent surveys, with the advancement of technology and easy availability of the depth sensors the 3D scanning technology has taken off. A wide range of commercial sensors such as Intel RealSense, Microsoft Kinect are being widely used for real time 3D capturing. Efficient 3D face scanning is one of the important areas of 3D scanning. 3D printer compatible texture supported scanning has a wide range of commercial applications. Such methods are also being used for 3D avatars and characterizations for games. Even though several commercial grade applications are available, most of the techniques suffer from background and light variations. Therefore an efficient face scanning technique is of extreme importance. In this paper we propose a 3D face scanning method based on Intel RealSense technology that combines 3D face detection, background segmentation and 3D mesh mapping to produce realistic 3D face model along with texture export. Further the models then can be manipulated using any 3D editing software along with texture and wireframe manipulation.

3D face identification based on the detection and comparison of keypoints of the face is a promising solution to extend face recognition approaches to the case of 3D scans with occlusions and missing parts. In fact, approaches that perform sparse keypoints matching can naturally allow for partial face comparison. However, such methods typically use a large number of keypoints, locally described by high-dimensional feature vectors: This, combined with the combinatorial number of keypoint comparisons required to match two face scans, results in a high computational cost that does not scale well with large datasets. Motivated by these considerations, in this paper, we present a 3D face recognition approach based on the meshDOG keypoints detector and local GH descriptor, and propose original solutions to improve keypoints stability and select the most effective features from the local descriptors. Experiments have been performed to assess the validity of the proposed optimizations for stable keypoints detection and feature selection. Recognition accuracy has been evaluated on the Bosphorus database, showing competitive results with respect to existing 3D face identification solutions based on 3D keypoints.

Creating and animating realistic 3D human faces is an important element of virtual reality, video games, and other areas that involve interactive 3D graphics. In this paper, we propose a system to generate photorealistic 3D blendshape-based face models automatically using only a single consumer RGB-D sensor. The capture and processing requires no artistic expertise to operate, takes 15 seconds to capture and generate a single facial expression, and approximately 1 minute of processing time per expression to transform it into a blendshape model. Our main contributions include a complete end-to-end pipeline for capturing and generating photorealistic blendshape models automatically and a registration method that solves dense correspondences between two face scans by utilizing facial landmarks detection and optical flows. We demonstrate the effectiveness of the proposed method by capturing different human subjects with a variety of sensors and puppeteering their 3D faces with real-time facial performance retargeting. The rapid nature of our method allows for just-in-time construction of a digital face. To that end, we also integrated our pipeline with a virtual reality facial performance capture system that allows dynamic embodiment of the generated faces despite partial occlusion of the user's real face by the head-mounted display.

We propose and evaluate a three-dimensional (3D) face recognition approach that applies the speeded up robust feature (SURF) algorithm to the depth representation of shape index map, under real-world conditions, using only a single gallery sample for each subject. First, the 3D scans are preprocessed, then SURF is applied on the shape index map to find interest points and their descriptors. Each 3D face scan is represented by keypoints descriptors, and a large dictionary is built from all the gallery descriptors. At the recognition step, descriptors of a probe face scan are sparsely represented by the dictionary. A multitask sparse representation classification is used to determine the identity of each probe face. The feasibility of the approach that uses the SURF algorithm on the shape index map for face identification/authentication is checked through an experimental investigation conducted on Bosphorus, University of Milano Bicocca, and CASIA 3D datasets. It achieves an overall rank one recognition rate of 97.75%, 80.85%, and 95.12%, respectively, on these datasets.

In this paper, we present an automatic point matching method to overcome the dense point alignment of 3D face scans. We adopt TPS (thin plate spline) transformation to model the deformation of different 3D faces, because TPS is a type of non-rigid transformation with good smooth property and suitable for formulating the complexities of human facial morphology. Generally, TPS is derived from the interpolation between two sets of aligned controlling points. To obtain fully automatic point matching method, a random point selecting method is proposed to get the controlling points of 3D faces. Integrating the point generating method with an iterative closest point searching strategy, we achieve a point-to-point alignment solution for dense 3D face scans. The point matching tests on BJUT 3D face database reveal that the TPS based method has flexible shape deformation ability. Comparing with the hand-selecting controlling point method our method has better point matching accuracy and stability. It is proved that the presented method is efficient for dense point registration with non-rigid deformation.

This paper considers the segmentation of range image measurements into surface patches which are either plane or curved and which are described formally by a function. After a formal description of the segmentation, we present and compare three methods suited for plane and curved patch segmentation and show the results of experiments conducted for testing their practical behaviour. The two first methods use the classical approach of region growing whereas the third method is based on a relaxation process. This original and last method exhibits simplicity and low computational complexity. Thanks to its parallel nature, it can be considered as a good candidate for range image segmentation in real-time applications.

A technique to generate a photorealistic three-dimensional (3D) image and color-textured model using a dedicated optical digitizer is presented. The proposed technique is started with the range and texture image acquisition from different viewpoints, followed by the registration and integration of multiple range images to get a complete and nonredundant point cloud that represents a real-life object. The accuracy of the range image and the precision of correspondence between the range image and texture image are guaranteed by sensor system calibration. Based on the point cloud, a geometric model is established by considering the connectivity of adjacent range image points. In order to enhance the photorealistic effect, we suggest a texture blending technique that utilizes a composite-weight strategy to blend the texture images within the overlapped region. This technique allows more efficient removal of the artifacts existing in the registered texture image, leading to a 3D image with photorealistic quality and color-texture modeling. Experimental results are also presented to testify to the validity of the proposed method.

Multi-view range image registration is a significant and challenging problem for 3D modeling. This paper presents a reference shape based multi-view range image registration algorithm. First, a set of Rotational Projection Statistics (RoPS) features are extracted from the input range images. Next, the reference shape is initialized by selecting a range image from the input. The reference shape is then iteratively updated by registering itself with the remaining range images. The registration between the reference shape and any range image is completed by RoPS feature matching. Finally, all input range images are registered according to their corresponding reference shapes. A number of experiments were performed to test the performance of our algorithm. The experimental results show that the reference shape based algorithm can perform multi-view registration on a mixed set of unordered range images corresponding to several different objects. It is also very accurate and efficient. It outperformed the state-of-the-arts including the spanning tree based and connected graph based algorithms.

Research on 3D face recognition has been intensified in recent years. However, most research has focused on the 3D static data analysis. In this paper, we investigate the face recognition problem using dynamic 3D face model sequences. Based on our newly created 3D dynamic face database, we propose to use a spatio-temporal hidden Markov model (HMM) which incorporates 3D surface feature characterization to learn the spatial and temporal information of faces. The advantage of using 3D dynamic data for face recognition has been evaluated by comparing our approach to three conventional approaches: 2D video based temporal HMM model, conventional 2D-texture based approach (e.g., Gabor wavelet based approach), and static 3D-model-based approaches.