Occlusion robust face recognition based on mask learning

Evaluation of the benchmark datasets for testing the efficacy of deep convolutional neural networks

Smart Library Automation System is increasingly attractive as an effective digital library system. Adapting automation in the library helps reduce the duplication of work, time-saving, and boosts work efficiency. One such significant feature is face recognition integrated. With this application, the system can use face recognition to enter and get the details of an end user. In recent years, face recognition has achieved many prodigious accomplishments based on Deep Convolutional Neural Networks (DCNN). In addition to constructing large face datasets, designing new effective DCNN architectures and loss functions are two trends to improve the performance of face recognition systems. Therefore, many studies have been published in state-of-the-art methods, making high-accuracy face recognition systems more possible than days in the past. However, it is still difficult for all research communities to train robust face recognition models because it depends heavily on their resources. This paper investigates and analyzes the effect of several effective loss functions based on softmax. Moreover, we also evaluate how hyper-parameter settings can impact the optimization process as well as the final recognition performance of the model trained by re-implementing these methods. The results of our experiments achieve state-of-the-art figures, which show the proposed method’s massive potential in improving face recognition performance. KeywordsFace recognitionDCNNEffective loss function

Face Recognition (FR) technology has made significant strides with the emergence of deep learning. Typically, most existing FR models are built upon Convolutional Neural Networks (CNN) and take RGB face images as the model's input. In this work, we take a closer look at existing FR paradigms from high-efficiency, security, and precision perspectives, and identify the following three problems: (i) CNN frameworks are vulnerable in capturing global facial features and modeling the correlations between local facial features. (ii) Selecting RGB face images as the model's input greatly degrades the model's inference efficiency, increasing the extra computation costs. (iii) In the real-world FR system that operates on RGB face images, the integrity of user privacy may be compromised if hackers successfully penetrate and gain access to the input of this model. To solve these three issues, we propose two novel FR frameworks, i.e., TransFace and TransFace++, which successfully explore the feasibility of applying ViTs and image bytes to FR tasks, respectively. Firstly, as revealed from our observations, we find that ViTs perform vulnerably when applied to FR scenarios with extremely large datasets. We investigate the reasons for this phenomenon and discover that the existing data augmentation approaches and hard sample mining strategies are incompatible with ViTs-based FR backbone due to the lack of tailored consideration on preserving face structural information and leveraging each local token information. To remedy these problems, we first propose a superior FR model called TransFace, which contains a patch-level data augmentation strategy named Dominant Patch Amplitude Perturbation (DPAP) and a hard sample mining strategy named Entropy-guided Hard Sample Mining (EHSM). Furthermore, to improve inference efficiency and user privacy protection, we investigate the intrinsic property of image bytes and propose a superior FR model termed TransFace++. The proposed model is trained directly on image bytes, presenting a novel approach to address the aforementioned issues. Specifically, considering the importance of local correlations in bytes, an image bytes compression strategy named Topology-based Image Bytes Compression (TIBC) is introduced to extract prominent features from the raw bytes and integrate these features with byte embeddings, effectively mitigating information loss during the bytes mapping process. Moreover, to strengthen the model's perception on geometric information encoded in image bytes, a novel cross-attention module named Structure Information-guided Cross-Attention (SICA) is designed to inject structure information into byte tokens for information interaction, significantly improving the model's generalization ability. Experiments on popular face benchmarks demonstrate the superiority of our TransFace and TransFace++.

In recent years, pattern recognition has garnered widespread attention, especially in the domain of face recognition. Traditional face recognition methods have certain limitations in unconstrained environments due to factors such as lighting, facial expressions, and poses. Deep learning can be used to address these challenges. This paper proposes a comprehensive approach to face occlusion recognition based on a two-stage Convolutional Neural Network (CNN). Face verification aims at verifying whether two face images belong to the same individual, and it is a more fundamental task compared to face recognition. The process of face recognition essentially involves multiple instances of face verification, sequentially validating different individuals to ultimately determine the corresponding individual for each face. The primary steps in this research include facial detection, image preprocessing, facial landmark localization, facial landmark extraction, feature matching recognition, and 2D image-assisted 3D face reconstruction. A novel two-stage CNN was designed for facial detection and alignment. The first stage of the network is dedicated to the search for facial windows and regressing vector boundaries. The second stage utilizes 2D images to assist in 3D face reconstruction and perform secondary recognition for cases not identified in the first stage. This method demonstrated excellent performance in handling facial occlusions, achieving high accuracy on datasets such as AFW and FDDB. On the test dataset, face recognition accuracy reached 97.3%, surpassing the original network accuracy of 89.1%. This method outperforms traditional algorithms and general CNN approaches. This study achieved efficient face validation and further handling of unrecognized situations, contributing to the enhancement of face recognition system performance.

An empirical study of the impact of masks on face recognition

Transportation accidents, are significantly affected by human factors, which account for a substantial proportion of incidents and fatalities. Factors such as fatigue, stress, illness, medication, and substance use impair pilot performance, leading to compromised decision-making, reduced situational awareness, and increased risk-taking behavior (Wingelaar-Jagt et al., 2021). While regulatory guidelines and medical evaluations exist to address these challenges, current measures often rely on self-reporting and subjective assessments that can be prone to bias. Artificial Intelligence (AI) driven facial recognition model has been used in other industries to assess human subjects’ health status (Chan et al., 2024) and cognitive workload (Iarlori et al., 2024). This research aims to develop an AI-driven facial recognition model to objectively assess pilot fitness to fly by analyzing micro expressions, facial symmetry, eye movement, and other biomarkers that reflect fatigue, stress, and impairment. The AI model will be trained using publicly available datasets containing facial images of individuals in varying conditions such as fatigue, drowsiness, stress, sadness, and under the influence of alcohol, drugs, or medication. Data preprocessing will employ facial landmark detection and attention-based image segmentation to isolate key facial regions, including the eyes (tracking movement and redness), mouth (symmetry, dryness, or tremor), and skin tone (color changes indicative of intoxication or stress)(Chan et al., 2024). Model training will leverage deep convolutional neural networks (CNNs), utilizing transfer learning techniques to enhance performance with smaller datasets. There are three tasks in this research. Task 1 focuses on model building using secondary data from publicly available facial image datasets in different conditions. Task 2 involves a laboratory-based experiment with healthy individuals to validate and refine the AI algorithm’s accuracy in detecting cognitive performance changes under stress. Participants will perform cognitive tasks under high-stress conditions, and facial images will be captured to fine-tune the algorithm. Task 3 includes a pilot simulation-based experiment to fine-tune the AI algorithm for aviation-specific applications. Licensed pilots will perform flight simulation tasks under high-workload or stressful conditions, such as emergency scenarios and adverse weather conditions. Data from facial images and simulator metrics like decision-making speed, navigation accuracy, and task prioritization will be analyzed to adapt the AI algorithm for real-time, aviation-specific assessments. The integration of this technology into preflight screening process will provide real-time, non-invasive assessments, complementing existing protocols and enhancing aviation safety by offering early warnings of performance degradation, thereby reducing accident risks and improving operational efficiency. Such AI facial recognition technology can also be utilized in-flight to detect subtle cues informing the pilot of their assessed condition. The authors would like to acknowledge Embry-Riddle Aeronautical University – FIRST Program for the funding provided. The authors would also like to acknowledge the consistent support from College of Aviation - School of Graduate Studies, and College of Engineering - Mechanical Engineering department.

The COVID-19 pandemic started in 2019, from this situation people learned that the use of face masks is one of the most effective ways to protect themselves from Coronavirus. A problem has arisen from this situation. Face recognition systems are widely used nowadays but all those systems are trained to detect perfectly exposed faces, not masked or occluded faces. As most people wear masks recently, it has become challenging for the existing face recognition systems to recognise faces. To suppress this problem, a feasible method for masked face recognition is proposed in the paper. For extracting the facial features of the non-occluded part of the face, VGG Face model is used. After extracting the facial features, those would be included in the dataset along with zoomed and rotated facial images for training. After that CS classifier is used for the classification and determines if the masked face is recognised or not. We have created Masked and Non-masked Face Dataset for the experiments.

Deep learning, in particular Convolutional Neural Network (CNN), has achieved promising results in face recognition recently. However, it remains an open question: why CNNs work well and how to design a 'good' architecture. The existing works tend to focus on reporting CNN architectures that work well for face recognition rather than investigate the reason. In this work, we conduct an extensive evaluation of CNN-based face recognition systems (CNN-FRS) on a common ground to make our work easily reproducible. Specifically, we use public database LFW (Labeled Faces in the Wild) to train CNNs, unlike most existing CNNs trained on private databases. We propose three CNN architectures which are the first reported architectures trained using LFW data. This paper quantitatively compares the architectures of CNNs and evaluate the effect of different implementation choices. We identify several useful properties of CNN-FRS. For instance, the dimensionality of the learned features can be significantly reduced without adverse effect on face recognition accuracy. In addition, traditional metric learning method exploiting CNN-learned features is evaluated. Experiments show two crucial factors to good CNN-FRS performance are the fusion of multiple CNNs and metric learning. To make our work reproducible, source code and models will be made publicly available.

Deep learning based approaches proved to be dramatically effective to address many computer vision applications, including recognition in the wild. It has been extensively demonstrated that methods exploiting Deep Convolutional Neural Networks (DCNN) are powerful enough to overcome to a great extent many problems that negatively affected computer vision algorithms based on hand-crafted features. These problems include variations in illumination, pose, expression and occlusion, to mention some. The DCNNs excellent discriminative power comes from the fact that they learn low-and high-level representations directly from the raw image data. Considering this, it can be assumed that the performance of a DCNN are influenced by the characteristics of the raw image data that are fed to the network. In this work, we evaluate the effect of different bounding box dimensions, alignment, positioning and data source on face recognition using DCNNs, and present a thorough evaluation on two well known, public DCNN architectures.

Seeing through disguise: Getting to know you with a deep convolutional neural network

Application of Face Recognition Method Under Deep Learning Algorithm in Embedded Systems

Problem statement: Face occlusion is a very challenging problem in fac e recognition. The performance of face recognition system decreases dr astically due to the presence of partial occlusion on the face. Extracting discriminative features to achieve accurate detection versus computational overhead in extracting the features, which affects the classification speed, would be a sustained problem. The objective of this study is to segment the human face into non-occluded and occluded part of the occluded human face image. In General, for f ace detection special facial features are extracted . In the proposed study a simplified algorithm to ext ract the features is developed. Approach: An algorithm which enables the automatic detection of the presence of occlusions on the face would be a useful tool to increase the performances of the sys tem. The face image was preprocessed to enhance the input face images in order to reduce the loss o f classification performance due to changes in faci al appearance. The experiment also balances both illumination and facial expression changes. Results: In this study, a Mean Based Weight Matrix (MBWM) algorithm has been proposed to enhance the performance by 4.25% than the LBP method. Conclusion: The proposed model has been tested on occluded face images with a dataset obtained from t he MIT face database.

Following the advance of style transfer with Convolutional Neural Networks (CNNs), the role of styles in CNNs has drawn growing attention from a broader perspective. In this paper, we aim to fully leverage the potential of styles to improve the performance of CNNs in general vision tasks. We propose a Style-based Recalibration Module (SRM), a simple yet effective architectural unit, which adaptively recalibrates intermediate feature maps by exploiting their styles. SRM first extracts the style information from each channel of the feature maps by style pooling, then estimates per-channel recalibration weight via channel-independent style integration. By incorporating the relative importance of individual styles into feature maps, SRM effectively enhances the representational ability of a CNN. The proposed module is directly fed into existing CNN architectures with negligible overhead. We conduct comprehensive experiments on general image recognition as well as tasks related to styles, which verify the benefit of SRM over recent approaches such as Squeeze-and-Excitation (SE). To explain the inherent difference between SRM and SE, we provide an in-depth comparison of their representational properties.

Face recognition has been widely used in people’s daily lives due to its contactless process and high accuracy. Existing works can be divided into two categories: global and local approaches. The mainstream global approaches usually extract features on whole faces. However, global faces tend to suffer from dramatic appearance changes under the scenarios of large pose variations, heavy occlusions, and so on. On the other hand, since some local patches may remain similar, they can play an important role in such scenarios. Existing local approaches mainly rely on cropping local patches around facial landmarks and then extracting corresponding local representations. However, facial landmark detection may be inaccurate or even fail, which would limit their applications. To address this issue, attention mechanisms are applied to automatically locate discriminative facial parts, while suppressing noisy parts. Following this motivation, several models are proposed, including: the Local multi-Scale Convolutional Neural Networks (LS-CNN), Hierarchical Pyramid Diverse Attention (HPDA) networks, Contrastive Quality-aware Attentions (CQA-Face), Diverse and Sparse Attentions (DSA-Face), and Attention Augmented Networks (AAN-Face). Firstly, a novel spatial attention (local aggregation networks, LANet) is proposed to adaptively locate useful facial parts. Meanwhile, different facial parts may appear at different scales due to pose variations and expression changes. In order to solve this issue, LS-CNN are proposed to extract discriminative local information at different scales. Secondly, it is observed that some important facial parts may be neglected, if without a proper guidance. Besides, hierarchical features from different layers are not fully exploited which can contain rich low-level and high-level information. To overcome these two issues, HPDA are proposed. Specifically, a diverse learning is proposed to enlarge the Euclidean distances between each two spatial attention maps, locating diverse facial parts. Besides, hierarchical bilinear pooling is adopted to effectively combine features from different layers. Thirdly, despite the decent performance of the HPDA, the Euclidean distance may not be flexible enough to control the distances between each two attention maps. Further, it is also important to assign different quality scores for various local patches because various facial parts contain information with various importance, especially for faces with heavy occlusions, large pose variations, or quality changes. The CQA-Face is proposed which mainly consists of the contrastive attention learning and quality-aware networks where the former proposes a better distance function to enlarge the distances between each two attention maps and the latter applies a graph convolutional network to effectively learn the relations

Recently, face recognition became a key element in social cognition which is used in various applications including human–robot interaction (HRI), pedestrian identification, and surveillance systems. Deep convolutional neural networks (CNNs) have achieved notable progress in recognizing faces. However, achieving accurate and real-time face recognition is still a challenging problem, especially in unconstrained environments due to occlusion, lighting conditions, and the diversity in head poses. In this paper, we present a robust face recognition and tracking framework in unconstrained settings. We developed our framework based on lightweight CNNs for all face recognition stages, including face detection, alignment and feature extraction, to achieve higher accuracies in these challenging circumstances while maintaining the real-time capabilities required for HRI systems. To maintain the accuracy, a single-shot multi-level face localization in the wild (RetinaFace) is utilized for face detection, and additive angular margin loss (ArcFace) is employed for recognition. For further enhancement, we introduce a face tracking algorithm that combines the information from tracked faces with the recognized identity to use in the further frames. This tracking algorithm improves the overall processing time and accuracy. The proposed system performance is tested in real-time experiments applied in an HRI study. Our proposed framework achieves real-time capabilities with an average of 99%, 95%, and 97% precision, recall, and F-score respectively. In addition, we implemented our system as a modular ROS package that makes it straightforward for integration in different real-world HRI systems.