Illumination Analysis of Deep Face Recognition

We present an algorithm for 3D face modeling from a frontal image and a profile image of a person's face. The algorithm starts by computing the 3D coordinates of automatically extracted facial feature points. The coordinates of the selected feature points are then used to deform a 3D generic face model to obtain a 3D face model for that person. Procrustes analysis is used to minimize globally the distances between the facial feature vertices in the model and the corresponding 3D points obtained from the images. Then, local deformation is performed on the facial feature vertices to obtain a more realistic 3D model for the person. Preliminary experiments to asses the applicability of the models for face recognition show encouraging results.

Face recognition has attracted enormous interest from researchers due to its widescale adaptation. However, most of the work in the literature is based on face recognition in controlled environments i.e., high-resolution face images with slight variation in pose, illumination and expression. CNN is used as backbone architecture in the face recognition models. Low-resolution face images having large variations in pose, illumination and expressions make a face recognition problem more challenging. It is difficult to extract discriminative features from low-resolution images. That's why the performance of face recognition models trained on high-resolution images deteriorate on low-resolution images. In this work, quality of low-resolution images is enhanced through multiple super-resolution GANs in order to improve the identification accuracy. Experiments are performed on SCface dataset and the improvement in the identification accuracy shows the effectiveness of using enhanced images generated through super-resolution GANs.

A comprehensive account of almost all the feature extraction methods used in current face recognition systems is presented. Specifically we have made distinction in the holistic and local feature extraction and differentiate them qualitatively as opposed to quantitatively. It is argued that a global feature representation should be preferred over a bag-of-feature approach. The problems in current feature extraction techniques and their reliance on a strict alignment is discussed. Finally we have introduced to use face-GLOH signatures that are invariant with respect to scale, translation and rotation and therefore do not require properly aligned images. The resulting dimensionality of the vector is also low as compared to other commonly used local features such as Gabor, LBP etc. and therefore learning based methods can also benefit from it. In a typical multi-view face recognition task, where it is assumed to have several examples of a subject available for training, we have shown in an extensive experimental setting the advantages and weaknesses of commonly used feature descriptions. Our results show that under more realistic assumptions, most of the classifiers failed on conventional features. While using the introduced face-GLOH-signature representation is relatively less affected by large in-class variations. This has been demonstrated by providing a fair performance comparison of several classifiers under more practical conditions such as misalignments, large number of subjects and large pose variations. An important conclusion is to be drawn from the results on FERET is that conventional multi-view face recognition cannot cope well with regards to large pose variations. Even using a large number of training examples in different poses for a subject do not suffice for a satisfactory recognition. In order to solve the problem where only one training example per subject is available, many recent methods propose to use image synthesis to generate a given subject at all other views and then perform a conventional multi-view recognition (Beymer and Poggio, 1995; Gross et al, 2004). Besides the fact that such synthesis techniques cause severe artefacts and thus cannot preserve the identity of an individual, a conventional classification cannot yield good recognition results, as has been shown in an extensive experimental setting. More sophisticated methods are therefore needed in order to address pose invariant face recognition. Large pose differences cause significant appearance variations that in general are larger than the appearance variation due to identity. One possible way of addressing this is to learn these variations across each pose, more specifically by fixing the pose and establishing a correspondence on how a person’s appearance changes under this pose one could reduce the in-class appearance variation significantly. In our very recent work (Sarfraz and Hellwich, 2009), we demonstrate the usefulness of face-GLOH signature in this direction.

Face recognition systems extract embedding vectors from face images and use these embeddings to verify or identify individuals. Face reconstruction attack (also known as template inversion) refers to reconstructing face images from embeddings and using the reconstructed image to enter a face recognition system. In this paper, we propose to use a face foundation model to reconstruct face images from the embeddings of a blackbox face recognition model. The foundation model is trained with 42M images to generate face images from the facial embeddings of a fixed face recognition model. We propose to use an adapter (called Face Adapter) to translate target embeddings into the embedding space of the foundation model. The generated images are evaluated on different face recognition models and different datasets, demonstrating the effectiveness of our method to translate embeddings of different face recognition models. We also evaluate the transferability of reconstructed face images when attacking different face recognition models. Our experimental results show that our reconstructed face images outperform previous reconstruction attacks against face recognition models. Project Page

Latent face model for across-media face recognition

Most traditional face recognition classifiers attempt to minimize recognition error rate rather than misclassification costs, which is unreasonable in many real world applications. On the other hand, many facial images are usually unlabeled, and the label process may result in high costs. Considering imbalanced misclassification costs and the hardship of gathering sufficient labeled images, an incremental sequential three-way decisions (3WD) model for cost-sensitive face recognition is proposed, in which a deep stacked autoencoder (DSAE) is used to extract an efficient deep feature set. The model takes full account of the costs of obtaining labeled data in real life. In addition, the model incorporates the boundary decision into the process of making decision, leading to a delayed decision with insufficient labeled images, which simulates the decision-making process from a small amount to a large amount of data. In summary, the model aims to select an optimal decision step so as to gain the desirable recognition results with the least amount of data. This strategy is applied to two facial image databases, which validate the effectiveness of the proposed methods.

Three-dimensional human face modeling is a very popular topic with many applications,such as facial animation, face recognition and model-based facial video communication. Therefore, how to model the facial geometry, texture intensity and illumination variation is important in computer vision and graphics. Previous works on 3D head modeling from a single face image utilized prior information on 3D head models. However, it is difficult to accurately reconstruct the 3D face model from a single face image with expression since the facial expression induces 3D face model deformation in a complex manner. The main challenge is the coupling of the neutral 3D face model and the 3D deformation due to expression, thus making the 3D model estimation from a single face image with expression very challenging. On the other hand, the illumination condition also makes the problem more difficult. In this thesis,we focus on developing a 3D face model reconstruction system including surface registration and training of 3D face models with expressional deformations as well as the estimation of the 3D neutral shape and the 3D expressional deformation from a single face image. The proposed reconstruction algorithm integrates the linear and non-linear subspace representations for a prior 3D neutral morphable model and the probabilistic manifold-based 3D expressional deformation. We incorporate the face geometry, expression deformation, texture and illumination information into the problem so that it is well constrained. The reconstructed 3D face models can also be further extended and applied to many real-world applications.

Face quality evaluation can filter out low quality face image to save computational resources and improve the system performance, labeling the face image quality score by manual consume too much manpower. To solve this problem, an unsupervised face image evaluation based on face recognition is proposed. We use the face recognition model to calculate the features of faces and label the images quality score. The face recognition model is compressed by knowledge distillation method to obtain efficient quality assessment model. Experimental results show that this method can effectively evaluate the quality of face image and improve the performance of face recognition.

Attending classes by students is associated with the assimilation of educational material by students and the ability to plan and organize activities. However, at present in educational institutions, as a rule, student attendance is recorded manually. Activities are performed frequently and repeatedly, thus wasting instructors' study time. Additionally, the face is one of the most widely used biometric characteristics for personal identification so an automated attendance system using face recognition has been proposed. In recent years, convolutional neural networks (CNN) have become the dominant deep le11arning method for face recognition. In this article, the features of building an automated student attendance system by biometric face recognition using the convolution neural network modelhas been discussed. Analyzed and solved the main tasks that arise when building an automated student attendance monitoring system: creating a dataset of students' face images; building and training a biometric face recognition model; face recognition from the camera and registration in the database; extension to the face image dataset. The use of the capabilities of the Python and OpenCV libraries is shown. The conducted testing of the accuracy of the developed CNN model of biometric face recognition showed good results – the overall accuracy score is not less than 0.75. The developed automated student attendance monitoring system in classrooms can be used to determine student attendance in different forms of the educational process. Its implementation will significantly reduce the monitoring time and reduce the number of errors in maintaining attendance logs. The introduction of an automated attendance monitoring system will significantly improve the organization of the educational process to ensure its quality.

Deep face recognition has achieved rapid development but still suffers from occlusions, illumination and pose variations, especially for face identification. The success of deep learning models in face recognition lies in large-scale high quality face data with accurate labels. However, in real-world applications, the collected data may be mixed with severe label noise, which significantly degrades the generalization ability of deep models. To alleviate the impact of label noise on face recognition, inspired by curriculum learning, we propose a self-paced deep learning model (SPDL) by introducing a negative \(l_1\)-norm regularizer for face recognition with label noise. During training, SPDL automatically evaluates the cleanness of samples in each batch and chooses cleaner samples for training while abandons the noisy samples. To demonstrate the effectiveness of SPDL, we use deep convolution neural network architectures for the task of robust face recognition. Experimental results show that our SPDL achieves superior performance on LFW, MegaFace and YTF when there are different levels of label noise.

The impacts of plastic surgery on face recognition systems have been investigated in the past decade by many researchers. Diverse well-known face recognition approaches, e.g. based on PCA or LBP, have been bench-marked mostly on the web-collected IIITD plastic surgery face database. Generally, significant performance drops were reported when comparing facial images taken before and after plastic surgeries. On the one side, some researchers reported problems with said plastic surgery database, i.e. the presence of low quality images. On the other side, the applied methods no longer reflect the state-of-the-art in face recognition. This calls for evaluating the impact of plastic surgery on state-of-the-art deep face recognition systems anew considering high quality imagery of most relevant plastic surgeries.This work introduces the new Hochschule Darmstadt (HDA) plastic surgery database of facial images taken before and after surgery. This database vastly complies with the quality requirements defined by the International Civil Aviation Organization (ICAO) for electronic travel documents and comprises face images of the five most frequently applied facial plastic surgeries. The HDA plastic surgery database, the IIITD plastic surgery database, and a non-surgery database, i.e. ICAO-compliant subsets of the FRGCv2 and FERET datasets, are used for comparative verification and identification evaluations which are conducted using the commercial Cognitec FaceVACS system and the open-source ArcFace system. The obtained results suggest that the impact of plastic surgery on deep face recognition systems is less significant than that observed for previously benchmarked methods.

3D face modeling from 2D face images is of significant importance for face analysis, animation and recognition. Previous research on this topic mainly focused on 3D face modeling from a single 2D face image; however, a single face image can only provide a limited description of a 3D face. In many applications, for example, law enforcement, multi-view face images are usually captured for a subject during enrollment, which makes it desirable to build a 3D face texture model, given a pair of frontal and profile face images. We first determine the correspondence between un-calibrated frontal and profile face images through facial landmark alignment. An initial 3D face shape is then reconstructed from the frontal face image, followed by shape refinement utilizing the depth information provided by the profile image. Finally, face texture is extracted by mapping the frontal face image on the recovered 3D face shape. The proposed method is utilized for 2D face recognition in two scenarios: (i) normalization of probe image, and (ii) enhancing the representation capability of gallery set. Experimental results comparing the proposed method with a state-of-the-art commercial face matcher and densely sampled LBP on a subset of the FERET database show the effectiveness of the proposed 3D face texture model.

Mental health assessments that combine patients' facial expressions and behaviors have been proven effective, but screening large-scale student populations for mental health problems is time-consuming and labor-intensive. This study aims to provide an efficient and accurate intelligent method for further psychological diagnosis and treatment, which combines artificial intelligence technologies to assist in evaluating the mental health problems of college students. We propose a mixed-method study of mental health assessment that combines psychological questionnaires with facial emotion analysis to comprehensively evaluate the mental health of students on a large scale. The Depression Anxiety and Stress Scale-21(DASS-21) is used for the psychological questionnaire. The facial emotion recognition model is implemented by transfer learning based on neural networks, and the model is pre-trained using FER2013 and CFEE datasets. Among them, the FER2013 dataset consists of 48 × 48-pixel face gray images, a total of 35,887 face images. The CFEE dataset contains 950,000 facial images with annotated action units (au). Using a random sampling strategy, we sent online questionnaires to 400 college students and received 374 responses, and the response rate was 93.5%. After pre-processing, 350 results were available, including 187 male and 153 female students. First, the facial emotion data of students were collected in an online questionnaire test. Then, a pre-trained model was used for emotion recognition. Finally, the online psychological questionnaire scores and the facial emotion recognition model scores were collated to give a comprehensive psychological evaluation score. The experimental results of the facial emotion recognition model proposed to show that its classification results are broadly consistent with the mental health survey results. This model can be used to improve efficiency. In particular, the accuracy of the facial emotion recognition model proposed in this paper is higher than that of the general mental health model, which only uses the traditional single questionnaire. Furthermore, the absolute errors of this study in the three symptoms of depression, anxiety, and stress are lower than other mental health survey results and are only 0.8%, 8.1%, 3.5%, and 1.8%, respectively. The mixed method combining intelligent methods and scales for mental health assessment has high recognition accuracy. Therefore, it can support efficient large-scale screening of students' psychological problems.

Face recognition is one of the biometric identification methods with the highest potential. The existing face recognition algorithms relying on the texture information of face images are affected greatly by the variation of expression, scale and illumination. Whereas the algorithms based on the shape topology weaken the influence of illumination to some extent, but the impact of expression, scale and illumination on face recognition is still unsolved. To this end, we propose a new method for face recognition by integrating texture information with shape information, called biview face recognition algorithm. The texture models are constructed by using subspace learning methods and shape topologies are formed by building graphs for face images. The proposed biview face recognition method is compared with recognition algorithms merely based on texture or shape information. Experimental results of recognizing faces under the variation of illumination, expression and scale demonstrate that the performance of the proposed biview face recognition outperforms texture-based and shape-based algorithms.

Deep neural networks (DNNs) have become increasingly effective in difficult machine learning tasks, such as image classification, speech recognition, and natural language processing. Face recognition (FR) using DNNs shows high performance and is widely used in various domains such as payment systems and immigration inspection. However, DNNs are vulnerable to adversarial examples generated by adding a small amount of noise to an original sample, resulting in misclassification by the DNNs. In this study, we attempt to deceive state-of-the-art FR by attaching noise markers on a face in the real world. To deceive an FR model in the real world, we address challenges in the attack process, including selection of locations of noise markers, the differences between colors of digital noise markers and those of noise markers after printing, the differences between the colors of noise markers that are attached to the face and those of noise markers after a picture is taken, and the differences between the locations of digital noise markers and those of noise markers that are attached to the face. In experiments, we generate noise markers considering these challenges and show that state-of-the-art FR can be deceived by attaching a maximum of 10 noise markers to a face. This can cause a security risk for FR models using DNNs.