Effects on facial expression in 3D face recognition

Face recognition research mainly focuses on traditional 2D color images, which is extremely susceptible to be affected by external factors such as various viewpoints and has limited recognition accuracy. In order to achieve improved recognition performance, as well as the 3D face holds more abundant information than 2D, we present a 3D human face recognition algorithm using the Microsoft's Kinect. The proposed approach integrates the depth data with the RGB data to generate 3D face raw data and then extracts feature points, identifies the target via a two-level cascade classifier. Also, we build a 3D-face database including 16 individuals captured exclusively using Kinect. The experimental results indicate that the introduced algorithm can not only achieve better recognition accuracy in comparison to existing 2D and 3D face recognition algorithms when the probe face is exactly in front of Kinect sensor, but also can increase 9.3% of recognition accuracy compared to the PCA-3D algorithm when it is not confronting the camera.

Face recognition based on 2D images is a difficult problem, because the illumination, pose and expression changes in the images create great statistical differences and the identity of the face itself becomes shadowed by these factors. 3D face recognition has the potential to overcome feature localization, pose and illumination problems, and it can be used in conjunction with 2D systems. We review the relevant work on 3D face recognition here, and discuss merits and demerits of different representations and recognition algorithms. This survey focuses on 3D face data acquisition and 3D face recognition using histogram features. Future works involved in developing more accurate 3D face recognition are also discussed. These include the need for improved 3D face data acquisition, 3D face recognition algorithms and 3D face experimental methodology.

In this paper, a noble nonintrusive three-dimensional (3D) face modeling system for random-profile-based 3D face recognition is presented. Although recent two-dimensional (2D) face recognition systems can achieve a reliable recognition rate under certain conditions, their performance is limited by internal and external changes, such as illumination and pose variation. To address these issues, 3D face recognition, which uses 3D face data, has recently received much attention. However, the performance of 3D face recognition highly depends on the precision of acquired 3D face data, while also requiring more computational power and storage capacity than 2D face recognition systems. In this paper, we present a developed nonintrusive 3D face modeling system composed of a stereo vision system and an invisible near-infrared line laser, which can be directly applied to profile-based 3D face recognition. We further propose a novel random-profile-based 3D face recognition method that is memory-efficient and pose-invariant. The experimental results demonstrate that the reconstructed 3D face data consists of more than 50 k 3D point clouds and a reliable recognition rate against pose variation.

3D face recognition has lately been attracting ever increasing attention. In this paper we review the full spectrum of 3D face processing technology, from sensing to recognition. The review covers 3D face modelling, 3D to 3D and 3D to 2D registration, 3D based recognition and 3D assisted 2D based recognition. The fusion of 2D and 3D modalities is also addressed. The paper complements other reviews in the face biometrics area by focusing on the sensor technology, and by detailing the efforts in 3D face modelling and 3D assisted 2D face matching.

We present an algorithm for 3D face modeling from a frontal image and a profile image of a person's face. The algorithm starts by computing the 3D coordinates of automatically extracted facial feature points. The coordinates of the selected feature points are then used to deform a 3D generic face model to obtain a 3D face model for that person. Procrustes analysis is used to minimize globally the distances between the facial feature vertices in the model and the corresponding 3D points obtained from the images. Then, local deformation is performed on the facial feature vertices to obtain a more realistic 3D model for the person. Preliminary experiments to asses the applicability of the models for face recognition show encouraging results.

In this paper, a novel framework for 3D face reconstruction from a single 2D face image was proposed. We focus on generating 3D face model without expensive devices and complicated calculation. First, we preprocess 2D face image, including illumination compensation, face detection and feature point extraction. Then, the method is based on a 3D morphable face model that encodes shape and texture in terms of model parameters. The prior 3D face model is a linear combination of “eigenheads” obtained by applying PCA to a training set of laser-scanned 3D faces. To account for pose and illumination variations, the algorithm simulates the process of image formation in 3D face and it estimates 3D face and texture of faces from a single image. As opposed to previous literature, our method can locate facial feature points automatically in a 2D facial image. Moreover, we also show that our proposed method for pose estimation can be successfully applied to 3D face reconstruction. In our experiment results, the method proposed has a satisfied performance regarding calculating time and the reconstructing photorealistic 3D faces.

This paper presents an efficient 3D face recognition method to handle facial expression. The proposed method uses the Surfaces Empirical Mode Decomposition (SEMD), facial curves and local shape descriptor in a matching process to overcome the distortions caused by expressions in faces. The basic idea is that, the face is presented at different scales by SEMD. Then the isometric-invariant features on each scale are extracted. After that, the geometric information is obtained on the 3D surface in terms of radial and level facial curves. Finally, the feature vectors on each scale are associated with their corresponding geometric information. The presented method is validated on GavabDB database resulting a rank 1 recognition rate (RR) of 98.9% for all faces with neutral and non-neutral expressions. This result outperforms other 3D expression-invariant face recognition methods on the same database.

We present a new algorithm for 3D face recognition, and compare its performance to that of previous approaches. We focus especially on the case of facial expression change between gallery and probe images. We first establish performance comparisons using a PCA (eigenface) algorithm and an ICP (iterative closest point) algorithm similar to ones reported in the literature. Experimental results show that the performance of either approach degrades substantially in the case Then we introduce a new algorithm, Adaptive Rigid Multi-region Selection, is introduced to independently matches multiple facial regions and creates a fused result. This algorithm is fully automated and used no manually selected landmark points. Experimental results show that our new algorithm substantially improves performance in the case of varying facial expression. Our experimental results are based on the largest 3D face dataset to date, with 449 persons, over 4,000 3D images, and substantial lapse between gallery and probe images.

We present an approach to identify non-cooperative individuals at a distance from a sequence of images using 3D face models. Most biometric features (such as fingerprints, hand shape, iris or retinal scans) require cooperative subjects in close proximity to the biometric system. Even top 2D face recognition systems today are only reliable at a distance when presented with frontal images. Our approach is accomplished by processing images acquired with an ultra-high resolution video camera, inferring the location of the subjects' head, using this information to crop the region of interest, building a 3D face model, and using this 3D model to perform biometric identification. To build the 3D model, we use an image sequence, as natural head and body motion provides enough viewpoint variation to perform stereo-motion for 3D face reconstruction. The 3D face model allows the use of true shape invariants for recognition against 3D galleries. Also the 3D model can assist in matching non-frontal faces against 2D galleries by rotating and rendering the 3D reconstruction into a 2D frontal image when only a 2D gallery is available. This approach constitutes a considerable step forward in solving the challenging problem of biometric identification at a distance.

Recent researches in three-dimensional (3D) face recognition area have proved that, 3D face recognition methods, achieve better accuracy than its 2D counterpart. One of the main advantages of 3D face recognition methods is that it measures the geometry of rigid features on the human face, due to which it becomes an invariant to illumination, expressions and rotations of head (pose variation). But the main drawback of 3D face recognition methods is the acquisition of 3D face images which generally needs a range camera. It is where the research community is facing difficulty. Research in 3D face recognition also depends on available databases. In this paper, we present a qualitative review of all the available 3D face databases. A comparative table of all the databases is also given at the end with all the necessary parameters.

Three-dimensional human face modeling is a very popular topic with many applications,such as facial animation, face recognition and model-based facial video communication. Therefore, how to model the facial geometry, texture intensity and illumination variation is important in computer vision and graphics. Previous works on 3D head modeling from a single face image utilized prior information on 3D head models. However, it is difficult to accurately reconstruct the 3D face model from a single face image with expression since the facial expression induces 3D face model deformation in a complex manner. The main challenge is the coupling of the neutral 3D face model and the 3D deformation due to expression, thus making the 3D model estimation from a single face image with expression very challenging. On the other hand, the illumination condition also makes the problem more difficult. In this thesis,we focus on developing a 3D face model reconstruction system including surface registration and training of 3D face models with expressional deformations as well as the estimation of the 3D neutral shape and the 3D expressional deformation from a single face image. The proposed reconstruction algorithm integrates the linear and non-linear subspace representations for a prior 3D neutral morphable model and the probabilistic manifold-based 3D expressional deformation. We incorporate the face geometry, expression deformation, texture and illumination information into the problem so that it is well constrained. The reconstructed 3D face models can also be further extended and applied to many real-world applications.

3D face modeling from 2D face images is of significant importance for face analysis, animation and recognition. Previous research on this topic mainly focused on 3D face modeling from a single 2D face image; however, a single face image can only provide a limited description of a 3D face. In many applications, for example, law enforcement, multi-view face images are usually captured for a subject during enrollment, which makes it desirable to build a 3D face texture model, given a pair of frontal and profile face images. We first determine the correspondence between un-calibrated frontal and profile face images through facial landmark alignment. An initial 3D face shape is then reconstructed from the frontal face image, followed by shape refinement utilizing the depth information provided by the profile image. Finally, face texture is extracted by mapping the frontal face image on the recovered 3D face shape. The proposed method is utilized for 2D face recognition in two scenarios: (i) normalization of probe image, and (ii) enhancing the representation capability of gallery set. Experimental results comparing the proposed method with a state-of-the-art commercial face matcher and densely sampled LBP on a subset of the FERET database show the effectiveness of the proposed 3D face texture model.

In this thesis we present two applications where 3D surface modeling and registration play a role of paramount importance. The first application is in archaeology, where our proposed method assists the archaeologist in the tedious process of reconstructing ceramic vessels from the excavated fragments (or shards). The method takes advantage of the surface intrinsic features coupled with a library of generic models constructed by the archaeologists to produce a virtual reconstruction of what the original vessel may have looked like. Generic models are generated based on the expert historical knowledge of the period, provenance of the artifact, and location of the digging site. The generic models need not to be identical to the original vessel, but must be within a geometric transformation of it in most of its parts. The methods are suited for ceramic vessels with distinctive color patterns or reliefs (i.e., surface with molding, carving, or stamping). We introduce a novel curvature B-spline fitting method that makes the extraction of robust features possible. The alignment of the shards against the generic model uses a novel set of 3D weighted curve moments and absolute affine invariants derived from them. The transformation is computed from corresponding parabolic contours on the shard and the generic model. When the shards have no color or relief on them, they are virtually mended to abutting shards using intrinsic differential anchor points computed on the shards break curves and a set of absolute invariants. For axially symmetric vessels, a global constraint induced by the surface of revolution is applied to guarantee global mending consistency. In our experiment involving around 300 shards, more than 85% of them are successfully mended. The mended shards consist of over 90% surface areas of the ceramic vessels. The second application is in 3D human face synthesis and recognition, where our proposed method assists the national security agencies in constructing a 3D face database to be used for spotting missing personnel or identify possible suspects when close up images of that person are flagged up. A gender/race generic 3D face model is morphed into a personalized unique 3D face of a given person from using a limited number of 2D images of the given person taken from different viewpoints. The 2D images need not to be of the same resolution, or taken under similar brightness or same place, or even same time period. The method applies also to drawing sketches of the person in question. The reconstructed 3D personalized model is an accurate rendition of the actual human face, and the reconstruction errors are smaller than a 3D scanner resolution that is used to scan the person's face. Our method renders a difficult 2D face recognition problem into a much simpler 3D surface registration (alignment) problem. Given a set of images of a test person, a personalized 3D face model is constructed and "best" aligned with one of the 3D faces stored in the database. When the minimum alignment errors are below an accepted threshold (lower than the scanner resolution), the identity of the test person is matched with the one in the database. Experimental results are shown in controlled environment with high resolution images, under ideal illumination condition, as well as for images taken in uncontrolled environment with low resolution, arbitrary illumination. In both conditions, the recognition rate is over 95%. It is also proved that better face recognition rate is achieved by using a gender/race based generic 3D face model which allows for accurate pose estimation as well as the face identification in 3D.

2D Face recognition has been extensively studied for decades and has reached remarkable results in recent years. However, 2D Face recognition is sensitive to variations in poses, facial expressions and illuminations. Depth images provide valuable information to help model facial boundaries and understand the global facial layout and provide low frequency patterns. Intuitively, RGB-D images are more robust to external environments than RGB images. Unfortunately, RGB-D datasets are orders of magnitude smaller than 2D datasets and insufficient to train a deep CNN model as effective as RGB-based models. To tackle these challenges, we present an RGB-D ResNet50 model which can be transferred from a pretrained RGB model and takes RGB-D images as input. We achieved an accuracy of 94.64% and won the \(1^{\text {st}}\) place on 3D Face Recognition Algorithm Challenge, 2019 BenchCouncil International Artificial Intelligence System Challenges.

Abstract 3D face recognition has received a lot of attention in the last decade, leading to improved sensors and algorithms that promise to enable large-scale deployment of biometric systems that rely on this modality. This chapter discusses advances in 3D face recognition with respect to current research and technology trends, together with its open challenges. Five real-world scenarios are described for application of 3D face biometrics. Then we provide a comparative overview of the currently available commercial sensors, and point out to research databases acquired with each technology. The algorithmic aspects of 3D face recognition are broadly covered; we inspect automatic landmarking and automatic registration as sine qua non parts of a complete 3D facial biometric system. We summarize major coordinated actions in evaluating 3D face recognition algorithms, and conclude with a case study on a recent and challenging database.KeywordsFace RecognitionGesture RecognitionIterative Close PointBiometric TemplateAutomatic FaceThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.