Face Reconstruction from Face Embeddings Using Adapter to a Face Foundation Model

Recent studies have shown that machine learning as a service is susceptible to reconstruction attacks that would expose users' privacy. When it comes to face recognition systems, privacy leakage could incur unpredictable effects on confidential or financial matters. This paper focuses on black-box reconstruction attacks on face recognition systems, including face verification and face identification applications. Considering a face recognition system in the real world, the adversary only has access to the similarity score of a query compared to a certain ID. We build our method based on a pretrained StyleGAN face generator and optimize its latent code to obtain a face image that maximizes the similarity score. Given the similarity score alone, we adopt zeroth-order optimization to estimate the gradients faithfully. Our method can attack 1:1 verification system with efficient and high-quality. To validate the effectiveness of the proposed method, we perform extensive experiments on face recognition systems, and reveal their risks of privacy leakage.

Mental health assessments that combine patients' facial expressions and behaviors have been proven effective, but screening large-scale student populations for mental health problems is time-consuming and labor-intensive. This study aims to provide an efficient and accurate intelligent method for further psychological diagnosis and treatment, which combines artificial intelligence technologies to assist in evaluating the mental health problems of college students. We propose a mixed-method study of mental health assessment that combines psychological questionnaires with facial emotion analysis to comprehensively evaluate the mental health of students on a large scale. The Depression Anxiety and Stress Scale-21(DASS-21) is used for the psychological questionnaire. The facial emotion recognition model is implemented by transfer learning based on neural networks, and the model is pre-trained using FER2013 and CFEE datasets. Among them, the FER2013 dataset consists of 48 × 48-pixel face gray images, a total of 35,887 face images. The CFEE dataset contains 950,000 facial images with annotated action units (au). Using a random sampling strategy, we sent online questionnaires to 400 college students and received 374 responses, and the response rate was 93.5%. After pre-processing, 350 results were available, including 187 male and 153 female students. First, the facial emotion data of students were collected in an online questionnaire test. Then, a pre-trained model was used for emotion recognition. Finally, the online psychological questionnaire scores and the facial emotion recognition model scores were collated to give a comprehensive psychological evaluation score. The experimental results of the facial emotion recognition model proposed to show that its classification results are broadly consistent with the mental health survey results. This model can be used to improve efficiency. In particular, the accuracy of the facial emotion recognition model proposed in this paper is higher than that of the general mental health model, which only uses the traditional single questionnaire. Furthermore, the absolute errors of this study in the three symptoms of depression, anxiety, and stress are lower than other mental health survey results and are only 0.8%, 8.1%, 3.5%, and 1.8%, respectively. The mixed method combining intelligent methods and scales for mental health assessment has high recognition accuracy. Therefore, it can support efficient large-scale screening of students' psychological problems.

Inverse biometrics methods are a major privacy concern for users of biometric recognition systems. Affine‐based reconstruction attack is an inverse biometrics method that models the biometric recognition algorithm by an affine approximation. This type of attack reconstructs targeted biometric references using the modelled biometric recognition algorithm and the comparison scores issued by the system. Although this reconstruction method has only been successfully applied to reconstruct face images, the common consensus is that any biometric system that issues comparison scores could be vulnerable to such an attack since this method is sufficiently general to be applied to other biometric templates. Here it is shown that the attack fails to regenerate sparse vascular feature point templates. The reconstruction attack on feature point patterns extracted from retina and hand vascular images is tested. The inverse attack match rate for reconstructed reference templates was 0.3% in one experiment using retinal vasculature and 0% for all others. These results show that the reconstruction attack is not as catastrophic as it is widely accepted to be, and that vascular biometric template protection schemes that store sparse templates as references and reveal comparison scores are not susceptible to affine‐based reconstruction attacks.

Face quality evaluation can filter out low quality face image to save computational resources and improve the system performance, labeling the face image quality score by manual consume too much manpower. To solve this problem, an unsupervised face image evaluation based on face recognition is proposed. We use the face recognition model to calculate the features of faces and label the images quality score. The face recognition model is compressed by knowledge distillation method to obtain efficient quality assessment model. Experimental results show that this method can effectively evaluate the quality of face image and improve the performance of face recognition.

Text-guided face recognition (TGFR) aims to improve the performance of state-of-the-art face recognition (FR) algorithms by incorporating auxiliary information, such as distinct facial marks and attributes, provided as natural language descriptions. Current TGFR algorithms have been proven to be highly effective in addressing performance drops in state-of-the-art FR models, particularly in scenarios involving sensor noise, low resolution, and turbulence effects. Although existing methods explore various algorithms using different cross-modal alignment and fusion techniques, they encounter practical limitations in real-world applications. For example, during inference, textual descriptions associated with face images may be missing, lacking crucial details, or incorrect. Furthermore, the presence of inherent modality heterogeneity poses a significant challenge in achieving effective cross-modal alignment. To address these challenges, we introduce CaptionFace, a TGFR framework that integrates GPTFace, a face image captioning model designed to generate context-rich natural language descriptions from low-resolution facial images. By leveraging GPTFace, we overcome the issue of missing textual descriptions, expanding the applicability of CaptionFace to single-modal FR datasets. Additionally, we introduce a multi-scale feature alignment (MSFA) module to ensure semantic alignment between face-caption pairs at different granularities. Furthermore, we introduce an attribute-aware loss and perform knowledge adaptation to specifically adapt textual knowledge from facial features. Extensive experiments on three face-caption datasets and various unconstrained single-modal benchmark datasets demonstrate that CaptionFace significantly outperforms state-of-the-art FR models and existing TGFR approaches.

Illumination variation significantly affects the performance of face recognition algorithms. However, it is difficult to construct a database that meets the demand of illumination analysis with current face image database construction technology. The influence of illumination variation on deep face recognition based on 3D face models is investigated. First a new method is presented to produce face images under arbitrary illuminations by using a 3D face model based on face basis image representation theory proposed. Then the database used for illumination analysis is constructed. Afterwards, an optimal way is explored to select illuminations used for constructing a high quality face image database by analyzing the impacts of different illumination sampling strategies on the performances of face recognition models. With the advantage of accurate illumination labeling of virtual data, the performances of different illumination labeling methods on the training of face recognition network are tested on a multi-task learning framework. Through experiments on the effects of different lighting sampling schemes and labeling methods on the performances of face recognition models carried out on virtual data and real data, it is concluded that using an appropriate amount of lightings of basis images to build a database is an effective lighting sampling scheme, and accurate lighting annotation can help further improve the face recognition rate, the corresponding recognition models have face recognition accuracy rates of more than 98% on virtual and real monitoring datasets which contain face images captured under very challenging lighting conditions. Proposed work improve the performance of deep face recognition models, and facilitate the construction of high quality face image databases with various illuminations.

Transportation accidents, are significantly affected by human factors, which account for a substantial proportion of incidents and fatalities. Factors such as fatigue, stress, illness, medication, and substance use impair pilot performance, leading to compromised decision-making, reduced situational awareness, and increased risk-taking behavior (Wingelaar-Jagt et al., 2021). While regulatory guidelines and medical evaluations exist to address these challenges, current measures often rely on self-reporting and subjective assessments that can be prone to bias. Artificial Intelligence (AI) driven facial recognition model has been used in other industries to assess human subjects’ health status (Chan et al., 2024) and cognitive workload (Iarlori et al., 2024). This research aims to develop an AI-driven facial recognition model to objectively assess pilot fitness to fly by analyzing micro expressions, facial symmetry, eye movement, and other biomarkers that reflect fatigue, stress, and impairment. The AI model will be trained using publicly available datasets containing facial images of individuals in varying conditions such as fatigue, drowsiness, stress, sadness, and under the influence of alcohol, drugs, or medication. Data preprocessing will employ facial landmark detection and attention-based image segmentation to isolate key facial regions, including the eyes (tracking movement and redness), mouth (symmetry, dryness, or tremor), and skin tone (color changes indicative of intoxication or stress)(Chan et al., 2024). Model training will leverage deep convolutional neural networks (CNNs), utilizing transfer learning techniques to enhance performance with smaller datasets. There are three tasks in this research. Task 1 focuses on model building using secondary data from publicly available facial image datasets in different conditions. Task 2 involves a laboratory-based experiment with healthy individuals to validate and refine the AI algorithm’s accuracy in detecting cognitive performance changes under stress. Participants will perform cognitive tasks under high-stress conditions, and facial images will be captured to fine-tune the algorithm. Task 3 includes a pilot simulation-based experiment to fine-tune the AI algorithm for aviation-specific applications. Licensed pilots will perform flight simulation tasks under high-workload or stressful conditions, such as emergency scenarios and adverse weather conditions. Data from facial images and simulator metrics like decision-making speed, navigation accuracy, and task prioritization will be analyzed to adapt the AI algorithm for real-time, aviation-specific assessments. The integration of this technology into preflight screening process will provide real-time, non-invasive assessments, complementing existing protocols and enhancing aviation safety by offering early warnings of performance degradation, thereby reducing accident risks and improving operational efficiency. Such AI facial recognition technology can also be utilized in-flight to detect subtle cues informing the pilot of their assessed condition. The authors would like to acknowledge Embry-Riddle Aeronautical University – FIRST Program for the funding provided. The authors would also like to acknowledge the consistent support from College of Aviation - School of Graduate Studies, and College of Engineering - Mechanical Engineering department.

Machine learning allows computers to learn a model for a given task, such as face recognition, with a high degree of accuracy, using data. However, after these models are generated, they are often treated as black boxes by developers and the limitations of a model are often unknown to end-users. To address these issues, this paper introduces the Face Recognition Model Analyzer (FRMA) ontology and a semantically enabled Resultset viewer. Together these resources describe image features relevant to face recognition and allow users to explore how well a face recognition model does at classifying images that contain an image feature. We evaluated the ontology and Resultset viewer by loading in the Labeled Faces in the Wild [1] dataset, enriching the images with image tags [2], and exploring two popular face recognition models, Facenet [3] and DLib [4]. Using the FRMA ontology and the Result-set viewer, we discovered several classic face recognition model limitations, such as trouble classifying images with occlusions. This evaluation shows that these resources can discover model limitations which can make face recognition model reuse easier for future users.

Face recognition has attracted enormous interest from researchers due to its widescale adaptation. However, most of the work in the literature is based on face recognition in controlled environments i.e., high-resolution face images with slight variation in pose, illumination and expression. CNN is used as backbone architecture in the face recognition models. Low-resolution face images having large variations in pose, illumination and expressions make a face recognition problem more challenging. It is difficult to extract discriminative features from low-resolution images. That's why the performance of face recognition models trained on high-resolution images deteriorate on low-resolution images. In this work, quality of low-resolution images is enhanced through multiple super-resolution GANs in order to improve the identification accuracy. Experiments are performed on SCface dataset and the improvement in the identification accuracy shows the effectiveness of using enhanced images generated through super-resolution GANs.

This paper presents a face and posture recognition model using convolutional neural network for adapting different resolution images. The performance of face recognition has dramatically improved by applying deep learning. However, it is known that the accuracy of face recognition would be decreased according to the resolution of face image. This problem is not negligible for some applications such as the real-time face recognition by surveillance camera, since the camera cannot always capture a face in a suitable position and distance. In order to solve this problem, our model adapts the resolution of an input image by selecting a proper model according to the resolution. In the experiment, we confirm the relationships between the recognition accuracy of face and posture and the resolution of captured image.

Attending classes by students is associated with the assimilation of educational material by students and the ability to plan and organize activities. However, at present in educational institutions, as a rule, student attendance is recorded manually. Activities are performed frequently and repeatedly, thus wasting instructors' study time. Additionally, the face is one of the most widely used biometric characteristics for personal identification so an automated attendance system using face recognition has been proposed. In recent years, convolutional neural networks (CNN) have become the dominant deep le11arning method for face recognition. In this article, the features of building an automated student attendance system by biometric face recognition using the convolution neural network modelhas been discussed. Analyzed and solved the main tasks that arise when building an automated student attendance monitoring system: creating a dataset of students' face images; building and training a biometric face recognition model; face recognition from the camera and registration in the database; extension to the face image dataset. The use of the capabilities of the Python and OpenCV libraries is shown. The conducted testing of the accuracy of the developed CNN model of biometric face recognition showed good results – the overall accuracy score is not less than 0.75. The developed automated student attendance monitoring system in classrooms can be used to determine student attendance in different forms of the educational process. Its implementation will significantly reduce the monitoring time and reduce the number of errors in maintaining attendance logs. The introduction of an automated attendance monitoring system will significantly improve the organization of the educational process to ensure its quality.

Face recognition is the most stable and robust biometric technique for identifying and authenticating human faces. However, training face recognition models using a deep learning architecture requires large training images. In addition, labeling face image collections manually is a time-consuming and costly process. Augmenting face images using the HyperStyle architecture can solve these issues. Experimental results obtained using the ResNet-50 architecture trained on a dataset of 44,000 face images (the original FaceScrub dataset combined with the HyperStyle age and smile augmented images) demonstrate that the model achieved an F1-score of 79%, therefore outperforming the model trained using the original FaceScrub dataset without HyperStyle augmentation, i.e., F1-score of 63%.

Based on sub-image feature extraction method of face images and least square support vector machine (LS-SVM) classifier, a face recognition model is proposed. In this model, face images are divided into sub-images firstly, then the discrete cosine transform is operated on each sub-image and the optimal coefficients are selected to denote the sub-image, and combine them into a single feature vector of the whole face image, finally LS-SVM is employed as a classifier with its optimized parameters. On ORL face database, experimental results show the proposed method is effective and has fast recognition speed.

A Human Brain may translate a person's voice to its corresponding face image even if never seen before. Training adeep learning network to do the same can be used in detecting human faces based on their voice, which may be used in findinga criminal that we only have a voice recording for. The goal in this paper is to build a Conditional Generative Adversarial Network that produces face images from human speeches which can then be recognized by a face recognition model to identifythe owner of the speech. The model was trained, and the face recognition model gave an accuracy of 80.08% in training and 56.2% in testing. Compared to the basic GAN model, this model has improved the results by about 30%. Key Word: Face image synthesis, Generative adversarial network, Face Recognition

Face Recognition (FR) technology has made significant strides with the emergence of deep learning. Typically, most existing FR models are built upon Convolutional Neural Networks (CNN) and take RGB face images as the model's input. In this work, we take a closer look at existing FR paradigms from high-efficiency, security, and precision perspectives, and identify the following three problems: (i) CNN frameworks are vulnerable in capturing global facial features and modeling the correlations between local facial features. (ii) Selecting RGB face images as the model's input greatly degrades the model's inference efficiency, increasing the extra computation costs. (iii) In the real-world FR system that operates on RGB face images, the integrity of user privacy may be compromised if hackers successfully penetrate and gain access to the input of this model. To solve these three issues, we propose two novel FR frameworks, i.e., TransFace and TransFace++, which successfully explore the feasibility of applying ViTs and image bytes to FR tasks, respectively. Firstly, as revealed from our observations, we find that ViTs perform vulnerably when applied to FR scenarios with extremely large datasets. We investigate the reasons for this phenomenon and discover that the existing data augmentation approaches and hard sample mining strategies are incompatible with ViTs-based FR backbone due to the lack of tailored consideration on preserving face structural information and leveraging each local token information. To remedy these problems, we first propose a superior FR model called TransFace, which contains a patch-level data augmentation strategy named Dominant Patch Amplitude Perturbation (DPAP) and a hard sample mining strategy named Entropy-guided Hard Sample Mining (EHSM). Furthermore, to improve inference efficiency and user privacy protection, we investigate the intrinsic property of image bytes and propose a superior FR model termed TransFace++. The proposed model is trained directly on image bytes, presenting a novel approach to address the aforementioned issues. Specifically, considering the importance of local correlations in bytes, an image bytes compression strategy named Topology-based Image Bytes Compression (TIBC) is introduced to extract prominent features from the raw bytes and integrate these features with byte embeddings, effectively mitigating information loss during the bytes mapping process. Moreover, to strengthen the model's perception on geometric information encoded in image bytes, a novel cross-attention module named Structure Information-guided Cross-Attention (SICA) is designed to inject structure information into byte tokens for information interaction, significantly improving the model's generalization ability. Experiments on popular face benchmarks demonstrate the superiority of our TransFace and TransFace++.