Towards a Face Recognition Model Analyzer

Face recognition systems extract embedding vectors from face images and use these embeddings to verify or identify individuals. Face reconstruction attack (also known as template inversion) refers to reconstructing face images from embeddings and using the reconstructed image to enter a face recognition system. In this paper, we propose to use a face foundation model to reconstruct face images from the embeddings of a blackbox face recognition model. The foundation model is trained with 42M images to generate face images from the facial embeddings of a fixed face recognition model. We propose to use an adapter (called Face Adapter) to translate target embeddings into the embedding space of the foundation model. The generated images are evaluated on different face recognition models and different datasets, demonstrating the effectiveness of our method to translate embeddings of different face recognition models. We also evaluate the transferability of reconstructed face images when attacking different face recognition models. Our experimental results show that our reconstructed face images outperform previous reconstruction attacks against face recognition models. Project Page

Mental health assessments that combine patients' facial expressions and behaviors have been proven effective, but screening large-scale student populations for mental health problems is time-consuming and labor-intensive. This study aims to provide an efficient and accurate intelligent method for further psychological diagnosis and treatment, which combines artificial intelligence technologies to assist in evaluating the mental health problems of college students. We propose a mixed-method study of mental health assessment that combines psychological questionnaires with facial emotion analysis to comprehensively evaluate the mental health of students on a large scale. The Depression Anxiety and Stress Scale-21(DASS-21) is used for the psychological questionnaire. The facial emotion recognition model is implemented by transfer learning based on neural networks, and the model is pre-trained using FER2013 and CFEE datasets. Among them, the FER2013 dataset consists of 48 × 48-pixel face gray images, a total of 35,887 face images. The CFEE dataset contains 950,000 facial images with annotated action units (au). Using a random sampling strategy, we sent online questionnaires to 400 college students and received 374 responses, and the response rate was 93.5%. After pre-processing, 350 results were available, including 187 male and 153 female students. First, the facial emotion data of students were collected in an online questionnaire test. Then, a pre-trained model was used for emotion recognition. Finally, the online psychological questionnaire scores and the facial emotion recognition model scores were collated to give a comprehensive psychological evaluation score. The experimental results of the facial emotion recognition model proposed to show that its classification results are broadly consistent with the mental health survey results. This model can be used to improve efficiency. In particular, the accuracy of the facial emotion recognition model proposed in this paper is higher than that of the general mental health model, which only uses the traditional single questionnaire. Furthermore, the absolute errors of this study in the three symptoms of depression, anxiety, and stress are lower than other mental health survey results and are only 0.8%, 8.1%, 3.5%, and 1.8%, respectively. The mixed method combining intelligent methods and scales for mental health assessment has high recognition accuracy. Therefore, it can support efficient large-scale screening of students' psychological problems.

This paper presents a face and posture recognition model using convolutional neural network for adapting different resolution images. The performance of face recognition has dramatically improved by applying deep learning. However, it is known that the accuracy of face recognition would be decreased according to the resolution of face image. This problem is not negligible for some applications such as the real-time face recognition by surveillance camera, since the camera cannot always capture a face in a suitable position and distance. In order to solve this problem, our model adapts the resolution of an input image by selecting a proper model according to the resolution. In the experiment, we confirm the relationships between the recognition accuracy of face and posture and the resolution of captured image.

In the biometrics community, face and speaker recognition are mature fields in which several systems have been proposed over the past twenty years. While existing systems perform well under controlled recording conditions, mismatch caused by the use of different sensors or a lack of cooperation from the subject still significantly affects performance, especially in challenging scenarios such as in forensics. Furthermore, existing methods suffer from scalability issues, which prevents them from taking advantage of increasingly large amounts of training data. This is otherwise a promising approach to improve accuracy in such challenging scenarios. In this thesis we address these problems of mismatch and complexity by developing scalable probabilistic models that we apply to face, speaker and bimodal recognition. Our contributions are four-fold. First, we propose a unified framework for session variability modeling techniques based on Gaussian mixture models (GMM), that encompasses inter-session variability (ISV) modeling, joint factor analysis (JFA) and total variability (TV) modeling. Second, we propose a novel exact and scalable formulation of probabilistic linear discriminant analysis (PLDA), which is a probabilistic and generative framework that models between-class and within-class variations. This formulation solves a major scalability issue, by improving both the time complexity of the training procedure from cubic to linear with respect to the number of samples per class, and the complexity of the scoring procedure. Furthermore, the implementations of all the proposed techniques are integrated into a novel collaborative open source software library called Bob 1 that enforces fair evaluations and encourages reproducible research. Fourth and finally, large-scale experiments are conducted with all of the above machine learning algorithms on several databases such as FRGC for face recognition, NIST SRE12 for speaker recognition and MOBIO for bimodal recognition, showing competitive performances.

Deep CNN models have become state-of-the-art techniques in many application, e.g., face recognition, speaker recognition, and image classification. Although many studies address on speedup or compression of individual models, very few studies focus on co-compressing and unifying models from different modalities. In this work, to joint and compress face and speaker recognition models, a shared-codebook approach is adopted to reduce the redundancy of the combined model. Despite the modality of the inputs of these two CNN models are quite different, the shared codebook can support two CNN models of sound and image for speaker and face recognition. Experiments show the promising results of unified and co-compressing heterogeneous models for efficient inference.

With the aim of improving security measures in public and government buildings, this study presents a system to protect the House of Representatives using Vicon sensors and facial recognition technology to detect threats as soon as they occur and alert accordingly to take appropriate measures. A face recognition model was realized, which took advantage of a DenseNet169-based feature extractor and a dense layer-based classifier. The machine learning models and Vicon sensors used by the physical motion capture system allowed for highly accurate analysis of real-world movements. Using Decision Tree (DT) and K-Nearest Neighbors (KNN) algorithms, the system achieved optimal accuracy using a dataset that included ten categories of behaviors performed by ten employees. Thanks to the combination of motion capture and facial recognition technology, allowing for precise threat classification, the House of Representatives will have a robust security system. This research highlights the importance of technical improvements in defending public employees and facilities.

Abstract: Every day, children, young people, disabled people, underage children, elderly people with Alzheimer's disease, etc. Many people went missing, including many, and many are still missing. According to the latest data from the National Crime Records Bureau (NCRB), 59,262 children went missing in India in 2020. We propose a method that can assist police and the public by using facial recognition to speed up the search process. When a person goes missing, police can send that person's photo and keep it on file. Whenever anyone comes across a complaint, they can take a photo of that person and upload it on our portal. The facial recognition model in our system will try to find matches in the data using facial coding. The face coding of the uploaded image is done by comparing it with the face coding of the image in the database. If a match is found, the police will be notified. The app will keep track of all missing persons and when a match is found the relevant authorities in the area will be notified. In addition to the photo, the user's location at the time the video was recorded may also be useful to the police. Also children, runaway teenagers, mentally ill, Alzheimer's patients, criminals, etc. Information on all missing persons will also be classified. Many families and even governments offer rewards to people who provide information leading to disappearances. The missing person was found. We can provide information about this gift through the application. When a new photo of a missing person is uploaded to the database, all users of the app will receive a notification with the person's name, appearance, rewards offered and more. We may also combine our data with government Aadhar data, which includes biometric data that can be used in our system, if permitted. Since many mobile phones are now equipped with fingerprint sensors, fingerprint matching is also possible. In addition to all the features mentioned above, we also plan to create a child labor section where users can upload images of all child labor practices they may be engaging in and authorities will then be notified about these practices so that action can be taken.Every day, children, young people, disabled people, underage children, elderly people with Alzheimer's disease, etc. Many people went missing, including many, and many are still missing. According to the latest data from the National Crime Records Bureau (NCRB), 59,262 children went missing in India in 2020. We propose a method that can assist police and the public by using facial recognition to speed up the search process. When a person goes missing, police can send that person's photo and keep it on file. Whenever anyone comes across a complaint, they can take a photo of that person and upload it on our portal. The facial recognition model in our system will try to find matches in the data using facial coding. The face coding of the uploaded image is done by comparing it with the face coding of the image in the database. If a match is found, the police will be notified. The app will keep track of all missing persons and when a match is found the relevant authorities in the area will be notified. In addition to the photo, the user's location at the time the video was recorded may also be useful to the police. Also children, runaway teenagers, mentally ill, Alzheimer's patients, criminals, etc. Information on all missing persons will also be classified. Many families and even governments offer rewards to people who provide information leading to disappearances. The missing person was found. We can provide information about this gift through the application. When a new photo of a missing person is uploaded to the database, all users of the app will receive a notification with the person's name, appearance, rewards offered and more. We may also combine our data with government Aadhar data, which includes biometric data that can be used in our system, if permitted. Since many mobile phones are now equipped with fingerprint sensors, fingerprint matching is also possible. In addition to all the features mentioned above, we also plan to create a child labor section where users can upload images of all child labor practices they may be engaging in and authorities will then be notified about these practices so that action can be taken.

In recent years, deep neural networks (DNNs) have made significant progress on face recognition (FR). However, DNNs have been found to be vulnerable to adversarial examples, leading to fatal consequences in real-world applications. This article focuses on improving the transferability of adversarial examples against FR models. We propose gradient eroding (GE) to make the gradient of the residual blocks more diverse, by eroding the back-propagation dynamically. We also propose a novel black-box adversarial attack named corrasion attack based on GE. Extensive experiments demonstrate that our approach can effectively improve the transferability of adversarial attacks against FR models. Our approach overperforms 29.35% in fooling rate than state-of-the-art black-box attacks. Leveraging adversarial training with adversarial examples generated by us, the robustness of models can be improved by up to 43.2%. Besides, corrasion attack successfully breaks two online FR systems, achieving a highest fooling rate of 89.8%.

Face Recognition (FR) technology has made significant strides with the emergence of deep learning. Typically, most existing FR models are built upon Convolutional Neural Networks (CNN) and take RGB face images as the model's input. In this work, we take a closer look at existing FR paradigms from high-efficiency, security, and precision perspectives, and identify the following three problems: (i) CNN frameworks are vulnerable in capturing global facial features and modeling the correlations between local facial features. (ii) Selecting RGB face images as the model's input greatly degrades the model's inference efficiency, increasing the extra computation costs. (iii) In the real-world FR system that operates on RGB face images, the integrity of user privacy may be compromised if hackers successfully penetrate and gain access to the input of this model. To solve these three issues, we propose two novel FR frameworks, i.e., TransFace and TransFace++, which successfully explore the feasibility of applying ViTs and image bytes to FR tasks, respectively. Firstly, as revealed from our observations, we find that ViTs perform vulnerably when applied to FR scenarios with extremely large datasets. We investigate the reasons for this phenomenon and discover that the existing data augmentation approaches and hard sample mining strategies are incompatible with ViTs-based FR backbone due to the lack of tailored consideration on preserving face structural information and leveraging each local token information. To remedy these problems, we first propose a superior FR model called TransFace, which contains a patch-level data augmentation strategy named Dominant Patch Amplitude Perturbation (DPAP) and a hard sample mining strategy named Entropy-guided Hard Sample Mining (EHSM). Furthermore, to improve inference efficiency and user privacy protection, we investigate the intrinsic property of image bytes and propose a superior FR model termed TransFace++. The proposed model is trained directly on image bytes, presenting a novel approach to address the aforementioned issues. Specifically, considering the importance of local correlations in bytes, an image bytes compression strategy named Topology-based Image Bytes Compression (TIBC) is introduced to extract prominent features from the raw bytes and integrate these features with byte embeddings, effectively mitigating information loss during the bytes mapping process. Moreover, to strengthen the model's perception on geometric information encoded in image bytes, a novel cross-attention module named Structure Information-guided Cross-Attention (SICA) is designed to inject structure information into byte tokens for information interaction, significantly improving the model's generalization ability. Experiments on popular face benchmarks demonstrate the superiority of our TransFace and TransFace++.

Text-guided face recognition (TGFR) aims to improve the performance of state-of-the-art face recognition (FR) algorithms by incorporating auxiliary information, such as distinct facial marks and attributes, provided as natural language descriptions. Current TGFR algorithms have been proven to be highly effective in addressing performance drops in state-of-the-art FR models, particularly in scenarios involving sensor noise, low resolution, and turbulence effects. Although existing methods explore various algorithms using different cross-modal alignment and fusion techniques, they encounter practical limitations in real-world applications. For example, during inference, textual descriptions associated with face images may be missing, lacking crucial details, or incorrect. Furthermore, the presence of inherent modality heterogeneity poses a significant challenge in achieving effective cross-modal alignment. To address these challenges, we introduce CaptionFace, a TGFR framework that integrates GPTFace, a face image captioning model designed to generate context-rich natural language descriptions from low-resolution facial images. By leveraging GPTFace, we overcome the issue of missing textual descriptions, expanding the applicability of CaptionFace to single-modal FR datasets. Additionally, we introduce a multi-scale feature alignment (MSFA) module to ensure semantic alignment between face-caption pairs at different granularities. Furthermore, we introduce an attribute-aware loss and perform knowledge adaptation to specifically adapt textual knowledge from facial features. Extensive experiments on three face-caption datasets and various unconstrained single-modal benchmark datasets demonstrate that CaptionFace significantly outperforms state-of-the-art FR models and existing TGFR approaches.

Illumination variation significantly affects the performance of face recognition algorithms. However, it is difficult to construct a database that meets the demand of illumination analysis with current face image database construction technology. The influence of illumination variation on deep face recognition based on 3D face models is investigated. First a new method is presented to produce face images under arbitrary illuminations by using a 3D face model based on face basis image representation theory proposed. Then the database used for illumination analysis is constructed. Afterwards, an optimal way is explored to select illuminations used for constructing a high quality face image database by analyzing the impacts of different illumination sampling strategies on the performances of face recognition models. With the advantage of accurate illumination labeling of virtual data, the performances of different illumination labeling methods on the training of face recognition network are tested on a multi-task learning framework. Through experiments on the effects of different lighting sampling schemes and labeling methods on the performances of face recognition models carried out on virtual data and real data, it is concluded that using an appropriate amount of lightings of basis images to build a database is an effective lighting sampling scheme, and accurate lighting annotation can help further improve the face recognition rate, the corresponding recognition models have face recognition accuracy rates of more than 98% on virtual and real monitoring datasets which contain face images captured under very challenging lighting conditions. Proposed work improve the performance of deep face recognition models, and facilitate the construction of high quality face image databases with various illuminations.

Com tecnologias como Internet das Coisas em constante evolução, surge a oportunidade para que outras áreas, como reconhecimento facial, incorporem novos conceitos ao seu funcionamento e conquistem novas aplicações nas cidades inteligentes. Segurança pública é um dos aspectos que compõem as cidades, com maneiras criativas de aprimorar surgindo a todo momento, elas também desfrutam dos benefícios que a implementação de sistemas de reconhecimento facial trazem. Esses sistemas são capazes de captar imagens de indivíduos e detectar faces através de algoritmos, que são submetidos a modelos capazes de realizar o reconhecimento dessas pessoas comparando com uma base de dados. Esse trabalho realiza um estudo acerca do processo de reconhecimento facial, visando a análise e avaliação de modelos de detecção e reconhecimento facial no contexto de segurança pública.

In the present study, we examined whether children and older adults exhibit an own-age face recognition bias. Participants studied photographs of children, younger adults, middle-aged adults, and older adults and were administered a recognition test. Results showed that both children and older adults more accurately recognized own-age faces than other-age faces. These data suggest that individuals may acquire expertise for identifying faces from their own age group and are discussed in terms of Sporer's (2001) in-group/out-group model of face recognition.

Face Recognition is a challenging task present in different applications and systems. An existing challenge is to recognize faces when imaging conditions are adverse, for example when images come from low-quality cameras or when the subject and the camera are far apart, thus impacting the accuracy of these recognizing systems. Super-Resolution techniques can be used to improve both image resolution and quality, hopefully improving the accuracy of the face recognition task. Among these techniques, the actual state-of-the-art uses Generative Adversarial Networks. One promising option is to train Super-Resolution and Face Recognition as one single network, conducting the network to learn super resolution features that will improve its capability when recognizing faces. In the present work, we trained a super resolution face recognition model using a jointly-learn approach, combining a generative network for super resolution and a ResNet50 for Face Recognition. The model was trained with a discriminator network, following the generative adversarial training. The images generated by the network were convincing, but we could not converge the face recognition model. We hope that our contributions could help future works on this topic. Code is publicly available at https://github.com/OliRafa/SRFR-GAN.KeywordsSuper-resolutionFace RecognitionGenerative Adversarial NetworksMachine learning

Face occlusion has been a long standing challenging issue in face recognition. In the state-of-the-art deep Convolutional Neural Network (CNN) face recognition models, occluded facial parts are generally embedded into the learned features together with the non-occluded parts in an equivalent manner. As such, the discriminative power of the generated face representation may be weakened for occluded face images. To address this problem, we propose the MaskNet, a trainable module which can be included in existing CNN architectures. With end-to-end training supervised by only the personal identity labels, MaskNet learns a proper way of adaptively generating different feature map masks for different occluded face images. Intuitively, MaskNet automatically assigns higher weights to the hidden units activated by the non-occluded facial parts and lower weights to those that are activated by the occluded facial parts. Experiments on datasets consisting of real-life and synthetic occluded faces demonstrate that MaskNet can effectively improve the robustness of CNN models towards occlusions in face recognition.