Improving RGB-D Face Recognition via Transfer Learning from a Pretrained 2D Network

In this paper, a noble nonintrusive three-dimensional (3D) face modeling system for random-profile-based 3D face recognition is presented. Although recent two-dimensional (2D) face recognition systems can achieve a reliable recognition rate under certain conditions, their performance is limited by internal and external changes, such as illumination and pose variation. To address these issues, 3D face recognition, which uses 3D face data, has recently received much attention. However, the performance of 3D face recognition highly depends on the precision of acquired 3D face data, while also requiring more computational power and storage capacity than 2D face recognition systems. In this paper, we present a developed nonintrusive 3D face modeling system composed of a stereo vision system and an invisible near-infrared line laser, which can be directly applied to profile-based 3D face recognition. We further propose a novel random-profile-based 3D face recognition method that is memory-efficient and pose-invariant. The experimental results demonstrate that the reconstructed 3D face data consists of more than 50 k 3D point clouds and a reliable recognition rate against pose variation.

After decades of extensive research, the field of 2D face recognition is fruitful. However, 2D face recognition is sensitive to changes in pose, facial expression and illumination. 3D face recognition has great potential due to the inherent invariance of pose and illumination changes, but these 3D face images are still difficult to acquire due to several issues such as cost and accessibility. In contrast, RGB-D images captured by low-cost depth sensors such as Kinect are relatively easy to acquire, and research on such low-quality inputs is still limited. In this thesis, an end-to-end multimodal fusion method based on spatial and channel attention is proposed to effectively fuse two image modalities, RGB and depth, to enhance RGB-D face recognition. Our work is specified as follows: firstly, three branches combined with the spatial and channel attention modules are established on the basis of ResNet18, so that the features of three modalities (RGB, depth map, and their fusion modalities, respectively) are obtained. The features under these three modalities are then fused, fed into a shared layer, and the fused features are further learned for deeper discriminative features and passed through a spatial attention vectorization module at the end. The method achieves good results on the Intellifusion RGBD dataset, while we perform ablation trials, which well reveal the contribution of each component of our method to the final performance.

An efficient face recognition system should recognize faces in different views and poses. The efficiency of a human face recognition system depends on the capability of face recognition in presence of changes in the appearance of face due to expression, pose and illumination. A novel algorithm which utilizes the combination of texture and depth information based on Modular PCA to overcome the problem of pose variation and illumination change for face recognition is proposed. The system has combined 2D and 3D systems in the feature level which presents higher performance in contrast with methods which utilizes either 2D or 3D system separately. A multimodal face recognition based on Modular PCA when compared with conventional PCA algorithm has an improved recognition rate for face images with large variations in illumination and facial expression. The proposed algorithm is tested with FRAV3D database that has faces with pose variation and illumination changes. Recognition rates from experimental results show the superiority of Modular PCA over conventional PCA methods in tackling face images with different pose variations and changes in illuminations. The proposed algorithm shows a recognition rate of 86% that is achieved in fusion experiment.

Automatic Face recognition of people is a challenging problem which has received much attention during the recent years due to its potential multimedia applications in different fields such as 3D videoconference, security applications or video indexing. However, there is no technique that provides a robust solution to all situations and different applications, yet. Face recognition includes a set of challenges like expression variations, occlusions of facial parts, similar identities, resolution of the acquired images, aging of the subjects and many others. Among all these challenges, most of the face recognition techniques have evolved in order to overcome two main problems: illumination and pose variation. Either of these influences can cause serious performance degradation in a 2D face recognition system.Some of the new face recognition strategies tend to overcome both research topics from a 3D perspective. The 3D data points corresponding to the surface of the face may be acquired using different alternatives: A multi camera system (stereoscopy), structured light, range cameras or 3D laser and scanner devices The main advantage of using 3D data is that geometry information does not depend on pose and illumination and therefore the representation of the object does not change with these parameters, making the whole system more robust. However, the main drawback of the majority of 3D face recognition approaches is that they need all the elements of the system to be well calibrated and synchronized to acquire accurate 3D data (texture and depth maps). Moreover, most of them also require the cooperation or collaboration of the subject during a certain period of time. All these requirements can be available during the training stage of many applications. When enrolling a new person in the database, it could be performed off-line, with the help o human interaction and with the cooperation of the subject to be enrolled. On the contrary, the previous conditions are not always available during the test stage. The recognition will be in most of the cases in a semicontrolled or uncontrolled scenario, where the only input of the system will probably consists of a 2D intensity image acquired from a single camera. This leads to a new paradigm where 2D-3D mixed face recognition approaches are used. The idea behind this kind of approaches is that these take profit of the 3D data during the training stage but then they can use either 3D data (when available) or 2D data during the recognition stage. Belonging to this category, some of 2D statistical approaches like Eigenfaces of Fisherfaces have been extended to fit in this new paradigm leading to the Partial Principal Component Analysis (P2CA) approach. This algorithm intends to cope with big pose variations (±90 ∘) by using 180∘ cylindrical texture maps for training the system but then only images acquired from a single, normal camera are used for the recognition. These training images provide pose information from different views (2.5D data). Nevertheless they can also be extended to a complete 3D multimodal system where depth and texture information is used. This chapter is structured as follows: First, a brief overview of the state-of-the-art in face recognition is introduced. The most relevant methods are grouped by multimedia scenarios and concrete applications. Afterwards, novel 2D-3D mixed face recognition approaches will be introduced.KeywordsFace RecognitionLocal Binary PatternMultimedia ApplicationActive Appearance ModelFace SpaceThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Facial expression recognition has been widely used in human computer interaction (HCI) systems. Over the years, researchers have proposed different feature descriptors, implemented different classification methods, and carried out a number of experiments on various datasets for automatic facial expression recognition. However, most of them used 2D static images or 2D video sequences for the recognition task. The main limitations of 2D-based analysis are problems associated with variations in pose and illumination, which reduce the recognition accuracy. Therefore, an alternative way is to incorporate depth information acquired by 3D sensor, because it is invariant in both pose and illumination. In this paper, we present a two-stream convolutional neural network (CNN)-based facial expression recognition system and test it on our own RGB-D facial expression dataset collected by Microsoft Kinect for XBOX in unspontaneous scenarios since Kinect is an inexpensive and portable device to capture both RGB and depth information. Our fully annotated dataset includes seven expressions (i.e., neutral, sadness, disgust, fear, happiness, anger, and surprise) for 15 subjects (9 males and 6 females) aged from 20 to 25. The two individual CNNs are identical in architecture but do not share parameters. To combine the detection results produced by these two CNNs, we propose the late fusion approach. The experimental results demonstrate that the proposed two-stream network using RGB-D images is superior to that of using only RGB images or depth images.

Face recognition with Symmetric Local Graph Structure (SLGS)

High performance for face recognition systems occurs in controlled environments and degrades with variations in illumination, facial expression, and pose. Efforts have been made to explore alternate face modalities such as infrared (IR) and 3-D for face recognition. Studies also demonstrate that fusion of multiple face modalities improve performance as compared with singlemodal face recognition. This paper categorizes these algorithms into singlemodal and multimodal face recognition and evaluates methods within each category via detailed descriptions of representative work and summarizations in tables. Advantages and disadvantages of each modality for face recognition are analyzed. In addition, face databases and system evaluations are also covered.

Over the years, face recognition has been the research topic that has attracted many researchers around the world. One of the most significant applications of face recognition is the access control system. The access control system allows authorized persons to enter or exit certain or restricted areas. As a result, it will increase the security situation without over-investment in staff security. The access information can be the identification, time, and location, etc. It can be used to carry out human resource management tasks such as attendance and inspection of employees in a more fair and transparent manner. Although face recognition has been widely used in access control systems because of its better accuracy and convenience without requiring too much user cooperation, the 2D-based face recognition systems also retain many limitations due to the variations in pose and illumination. By analyzing facial geometries, 3D facial recognition systems can theoretically overcome the disadvantages of prior 2D methods and improve robustness in different working conditions. In this paper, we propose the 3D facial recognition algorithm for use in an access control system. The proposed algorithm includes the preprocessing, feature extraction, and classification stages. The application of the proposed access control system is the automatic sliding door, the controller of the system, the web-based monitoring, control, and storage of data.

Recent researches in three-dimensional (3D) face recognition area have proved that, 3D face recognition methods, achieve better accuracy than its 2D counterpart. One of the main advantages of 3D face recognition methods is that it measures the geometry of rigid features on the human face, due to which it becomes an invariant to illumination, expressions and rotations of head (pose variation). But the main drawback of 3D face recognition methods is the acquisition of 3D face images which generally needs a range camera. It is where the research community is facing difficulty. Research in 3D face recognition also depends on available databases. In this paper, we present a qualitative review of all the available 3D face databases. A comparative table of all the databases is also given at the end with all the necessary parameters.

This research aims to develop artificial intelligence (AI) algorithms in the context of facial recognition with a focus on increasing accuracy in difficult environmental conditions. Although facial recognition technology has made great progress, challenges such as poor lighting, variations in facial expressions, and head rotation are still problems that must be overcome. The research methodology involved collecting a wide dataset covering a wide variety of faces under various environmental conditions. This data is then processed and its features are extracted using computer image processing techniques. Furthermore, several deep neural network architectures, such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), were developed, trained, and evaluated for face recognition tasks. The expected result is the development of an AI algorithm that is able to overcome challenges in facial recognition with higher accuracy than existing methods. In particular, significant improvements in facial recognition accuracy are expected especially under low lighting conditions and variations in facial expressions. This research has a major impact in a variety of security applications, such as border surveillance, building access control, and corporate security. With higher facial recognition accuracy, security risks can be significantly reduced, resulting in safer and more efficient security solutions. In conclusion, this research aims to bring innovation in facial recognition technology through advanced AI approaches, with the potential to improve security in various contexts.

This paper presents an efficient 3D face recognition method to handle facial expression. The proposed method uses the Surfaces Empirical Mode Decomposition (SEMD), facial curves and local shape descriptor in a matching process to overcome the distortions caused by expressions in faces. The basic idea is that, the face is presented at different scales by SEMD. Then the isometric-invariant features on each scale are extracted. After that, the geometric information is obtained on the 3D surface in terms of radial and level facial curves. Finally, the feature vectors on each scale are associated with their corresponding geometric information. The presented method is validated on GavabDB database resulting a rank 1 recognition rate (RR) of 98.9% for all faces with neutral and non-neutral expressions. This result outperforms other 3D expression-invariant face recognition methods on the same database.

This is the first study to compare the PCA and ICP approaches to 3D face recognition, and to propose a local region approach coping with expression variation in 3D face recognition. A new algorithm for 3D face recognition is proposed for handling expression variation. It uses a surface registration-based technique for 3D face recognition. The proposed method uses a fully automatic approach to use to initialize the 3D matching. Results are presented for gallery and probe datasets of 355 subjects imaged in 3D, with significant time lapse between gallery and probe images of a given subject yielding 3,205 3D models. We find that an ICP-based method performs better than a PCA-based method. The evaluation results show that our proposed new algorithm substantially improves performance in the case of varying facial expression. We also examined subject factors in the proposed method on 3D face models by age and gender.

Face recognition algorithms generally utilize 2D images for feature extraction and matching. To achieve higher resilience toward covariates, such as expression, illumination, and pose, 3D face recognition algorithms are developed. While it is challenging to use specialized 3D sensors due to high cost, RGB-D images can be captured by low-cost sensors such as Kinect. This research introduces a novel face recognition algorithm using RGB-D images. The proposed algorithm computes a descriptor based on the entropy of RGB-D faces along with the saliency feature obtained from a 2D face. Geometric facial attributes are also extracted from the depth image and face recognition is performed by fusing both the descriptor and attribute match scores. The experimental results indicate that the proposed algorithm achieves high face recognition accuracy on RGB-D images obtained using Kinect compared with existing 2D and 3D approaches.

Face recognition (FR) is a practical application of pattern recognition (PR) and remains a compelling topic in the study of computer vision. However, in real-world FR systems, interferences in images, including illumination condition, occlusion, facial expression and pose variation, make the recognition task challenging. This study explored the impact of those interferences on FR performance and attempted to alleviate it by taking face symmetry into account. A novel and robust FR method was proposed by combining multi-mirror symmetry with local binary pattern (LBP), namely multi-mirror local binary pattern (MMLBP). To enhance FR performance with various interferences, the MMLBP can 1) adaptively compensate lighting under heterogeneous lighting conditions, and 2) generate extracted image features that are much closer to those under well-controlled conditions (i.e., frontal facial images without expression). Therefore, in contrast with the later variations of LBP, the symmetrical singular value decomposition representation (SSVDR) algorithm utilizing the facial symmetry and a state-of-art non-LBP method, the MMLBP method is shown to successfully handle various image interferences that are common in FR applications without preprocessing operation and a large number of training images. The proposed method was validated with four public data sets. According to our analysis, the MMLBP method was demonstrated to achieve robust performance regardless of image interferences.

In this paper performance of elastic bunch graph matching (EBGM) for face recognition under variation in facial expression, variation in lighting condition and variation in poses are given. In this approach faces are represented by labelled graphs. Experimental results of EBGM on ORL, Yale B and FERET datasets are provided. Strong and weak features of EBGM algorithm are discussed.