Deep Learning Face Attributes for Detection and Alignment

Facial attribute analysis is an important step in many biometric algorithms, including face recognition based human authentication. Detecting the state of face attributes is becoming even more popular in mobile based applications, where a biometric authentication application is used to quickly and accurately verify the claimed identity of the owners device, and also, keep the device secure if an intruder attempts to gain unauthorized access to it. While there is a large number of facial attributes that can be automatically detected to support a face recognition system, in this paper we focus on detecting three specific ones that are useful during both an enrollment and authentication process: (1) determining whether the eyes of a subject are open or closed, (2) determining whether a subject is wearing glasses or not, and, finally, (3) detecting whether a subjects facial pose is either frontal or non-frontal. These attributes are associated with face image quality control, which is a very useful component of modern FR systems under the following context: during live authentication, by limiting low quality face image data, we can enhance the face-based authentication accuracy, while at the same time improve user satisfaction via improved system efficiency (i.e. less false match attempts during authentication). Thus, to automatically and efficiently detect all of the aforementioned facial attributes, we developed both conventional and deep learning based models. These models are trained and tested on diverse and challenging face image datasets, using data captured from traditional cameras and mobile devices, when operating at multiple standoff distances, in either indoor or outdoor conditions. Our proposed attribute-specific detection models are robust, yielding up to 100% accuracy (in terms of F1 score) depending on the attribute tested, as well as the model and dataset(s) used for training and testing.

Recovering a photorealistic face from an artistic portrait is a challenging task since crucial facial details are often distorted or completely lost in artistic compositions. To handle this loss, we propose an Attribute-guided Face Recovery from Portraits (AFRP) that utilizes a Face Recovery Network (FRN) and a Discriminative Network (DN). FRN consists of an autoencoder with residual block-embedded skip-connections and incorporates facial attribute vectors into the feature maps of input portraits at the bottleneck of the autoencoder. DN has multiple convolutional and fully-connected layers, and its role is to enforce FRN to generate authentic face images with corresponding facial attributes dictated by the input attribute vectors. %Leveraging on the spatial transformer networks, FRN automatically compensates for misalignments of portraits. % and generates aligned face images. For the preservation of identities, we impose the recovered and ground-truth faces to share similar visual features. Specifically, DN determines whether the recovered image looks like a real face and checks if the facial attributes extracted from the recovered image are consistent with given attributes. %Our method can recover high-quality photorealistic faces from unaligned portraits while preserving the identity of the face images as well as it can reconstruct a photorealistic face image with a desired set of attributes. Our method can recover photorealistic identity-preserving faces with desired attributes from unseen stylized portraits, artistic paintings, and hand-drawn sketches. On large-scale synthesized and sketch datasets, we demonstrate that our face recovery method achieves state-of-the-art results.

In computer vision, classifying facial attributes has attracted deep interest from researchers and corporations. Deep Neural Network based approaches are now widely spread for such tasks and have reached higher detection accuracies than previously manually-designed approaches. Our paper reports how preprocessing and face image alignment influence accuracy scores when detecting face attributes. More importantly it demonstrates how the combination of a representation of the shape of a face and its appearance, organized as a sequence of convolutional neural networks, improves classification scores of facial attributes when compared with previous work on the FER+ dataset. While most studies in the field have tried to improve detection accuracy by averaging multiple very deep networks, exposed work concentrates on building efficient models while maintaining high accuracy scores. By taking advantage of the face shape component and relying on an efficient shallow CNN architecture, we unveil the first available, highly accurate real-time implementation on mobile browsers.

We address the challenging large-scale content-based face image retrieval problem, intended as searching images based on the presence of specific subject, given one face image of him/her. To this end, one natural demand is a supervised binary code learning method. While the learned codes might be discriminating, people often have a further expectation that whether some semantic message (e.g., visual attributes) can be read from the human-incomprehensible codes. For this purpose, we propose a novel binary code learning framework by jointly encoding identity discriminability and a number of facial attributes into unified binary code. In this way, the learned binary codes can be applied to not only fine-grained face image retrieval, but also facial attributes prediction, which is the very innovation of this work, just like killing two birds with one stone. To evaluate the effectiveness of the proposed method, extensive experiments are conducted on a new purified large-scale web celebrity database, named CFW 60K, with abundant manual identity and attributes annotation, and experimental results exhibit the superiority of our method over state-of-the-art.

The tremendous progress in deep learning has enabled to extract soft-biometric attributes from faces, which raises privacy concerns over images collected for face recognition. Advances toward attribute privacy have been able to conceal multiple attributes while preserving identity information but suffer from limitations: they 1) only consider a few soft-biometric attributes and 2) fail to support reversibility for attribute privacy preservation. To break these limitations, we design a reversible privacy-preserving scheme for various face attributes, called reversible attribute privacy preservation (RAPP). RAPP benefits from two modules: 1) The attribute obfuscator introduces a stream cipher to determine that special attributes have to be concealed with the user-defined password, which also supports recovering original attributes. 2) The attribute adversarial network is proposed to generate perturbed images that conceal various attributes while retaining the utility of face verification. In addition, when a wrong password is provided, the returned image with wrong attribute classification results still keeps realistic, which confuses an attacker to know whether the recovery is correct. Extensive experiments demonstrate that RAPP enables to conceal various attributes and recover original images while facilitating face verification.

A crucial part of the face recognition process, along with facial attribute analysis and face verification, is facial landmark detection. In order to prepare face images for feature extraction in facial analysis jobs, face alignment is an essential step. Alignment is frequently used in training and inference to standardize the locations of important facial landmarks for applications including face recognition, facial emotion detection, and facial attribute classification. The effectiveness of facial analysis models is known to be greatly impacted by the approach and application of face alignment. However, there has not been enough research done on how alignment affects the quality of facial images. Although face alignment techniques based on convolutional neural networks have advanced significantly recently, occlusion remains a fundamental obstacle to achieving high accuracy. In order to enhance performance, we present in this study the attentioned distillation module from our earlier work on the Occlusion-Adaptive Deep Network model. A distillation module in this model infers the occlusion probability of every place in high-level features. When measuring the relationship between facial shape and look, it can be automatically learned. However, because of the missing semantic features, the holistic face cannot be represented by the clean feature representation. We must use a low-rank learning module to recover lost features in order to achieve exhaustive and complete feature representation.

Cascaded regression has recently become the method of choice for solving non-linear least squares problems such as deformable image alignment. Given a sizeable training set, cascaded regression method learns a set of generic rules that are sequentially applied to solve the least squares problem. Despite the success of cascaded regression for problems such as face alignment and head pose estimation, there are several shortcomings arising in the strategies proposed thus far. For example, the result is sensitive to the initial shape and the storage of the model is too large, and no robustness to self-occlusion. This paper proposes an approach to solve that problem. First, we reconstruct the 3D face from a 2D face image with arbitrary poses and expressions. Second, we project the 3D face shape model to the 2D plane, then we can get a 2D face shape. Third, we use the projected 2D face shape as the initial shape for the cascaded regression instead of the generic mean shape. Comparing to the existing methods the proposed method can increase the accuracy of the face alignment under confirming appropriate performance.

This paper describes a vector representation of 3D shapes acquired in the form of 2D range images produced using a range finder (NEC Danae-R). The vectorization was performed by first extracting feature points that correspond to each other as landmarks among different 3D face shapes, and then re-sampling each surface region of the face divided by the landmarks into other regions to obtain corresponding elements of the 3D shape vector. A linear description of the 3D shape spaces obtained by principal component analysis allows a smooth morphable modeling of a human face in terms of a smaller number of projection parameters. We scientifically solved the relations between shapes and impressions of faces and attempted face-image manipulations that transform impressions on facial attributes.

This paper presents a novel method for face alignment under unknown head poses and nonrigid warp, within the framework of markov random field. The proposed method learns a 3D face shape model comprised of 31 facial features and a texture model for each facial feature from a 3D face database. The models are combined to serve as the unary, pairwise and high order constraints of the markov random field. The facial features are located by minimizing the potential function of markov random field, which is solved with dual decomposition. The main contribution of this paper is composed of three aspects. First, Random Project Tree is utilized to learn the manifold structure of the facial feature appearance under different view points. Second, a 3D face shape model is learned to capture the linear part of face shape distribution due to the change of identity and expression, which is served as the high order constraints. Third, Markov random field is introduced to model the nonlinear part of the face shape distribution, and also to deal with occlusion of facial features due to head pose variation or ornaments. Experiments was taken on the Texas 3D face database and face images downloaded from the Internet, results shows capability of adapting large head pose variations of the method.KeywordsFace ImageFacial FeatureMarkov Random FieldFace ShapeTexture ModelThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Breast cancer is the most prevalent cancer globally and presents a significant threat to human health. The low survival rate for advanced stages highlights the importance of early and accurate diagnosis. Pathology images, providing insights into cancer's cellular aggressiveness, are crucial for breast cancer diagnosis. Advances in deep learning have transformed automated medical image diagnosis. However, traditional deep learning methods, reliant on large labeled datasets, face challenges with the typically small size of medical image datasets. To overcome this, our study combines deep transfer learning (DTL) with convolutional neural networks(CNN). We utilize and enhance a ResNet50 model pretrained on a large dataset. Additionally, we apply four different deep transfer learning methods to feature extraction and classification of the BreakHis dataset, aiming to determine if breast cancer is benign or malignant. Ultimately, our system achieved the best average accuracy rates under four different magnifications: 98.57%, 96.83%, 97.76%, and 97.16%. These results demonstrate the accuracy and potential of deep transfer learning in classifying breast cancer histopathological images and its promising application in tumor prediction.

Notwithstanding the enhancement obtained in the last decade researches, the recognition of facial attributes is still today a trend. Besides the mere face recognition, the singular face features, like mouth, nose and hair, are considered as soft biometrics; these can be useful for human identification in cases the face is partially occluded, and only some regions are visible. In this paper we propose a model generated by transfer learning approach for the recognition of the face attributes. Also, an unsupervised clustering model is described, which is in charge of dividing and grouping faces based on their characteristics. Furthermore, we show how clusters can be evaluated by a compact summary of them, and how Deep Learning models should be properly trained for attribute prediction tasks.

Waste management handles all kinds of waste, including household, industrial, municipal, organic, biomedical, biological, and radioactive wastes. People still face challenges in proper disposal methods for different types of waste, including landfill-bound items, recyclable materials, and biodegradable waste. Inadequate waste management poses a significant and multifaceted global challenge. The conventional method of segregating waste is a time-consuming and ineffective method that wastes human power and money. To address this issue in real time, sophisticated and sustainable waste management systems need to be implemented. The latest advancements in computer vision and deep learning offer efficient solutions for effective recycling and waste management. Existing deep learning models exhibited various limitations, such as detection accuracy and computational inefficiency, particularly when dealing with objects of varying sizes and exhibiting high degrees of visual similarity. These limitations generate various challenges in effectively capturing and representing the nuanced features of visually similar objects. To address this problem, we proposed the stacking of an enhanced Swin Transformer, improved ConvNeXt, and a spatial attention mechanism. The enhanced Swin transformers incorporate two key components- hierarchical feature extraction and shifting window mechanism to extract the global features from the garbage images effectively. The shifting window mechanism extracts the most important features from various regions of the images to identify the objects. In contrast, the hierarchical feature extraction captures long-range dependencies within the image to effectively identify different types of garbage. The improved ConvNext block with optimized parameterization extracts the local features of the image. This enhanced feature extraction capability enables the model to effectively discern fine-grained details of individual garbage particles, such as shape, texture, and subtle variations in color and appearance, leading to more accurate classification results. When we evaluated the performance of the proposed model using the publicly available Garbage Classification dataset, it attained 98.97% accuracy, 98.42% Precision, and 98.61% Recall. Due to its lightweight and low computational time and power, the proposed model surpasses the existing state-of-the-art deep learning models.

Quantitative analysis of facial soft tissue using weighted cascade regression model applicable for facial plastic surgery

Deep Learning is a subfield of machine learning methods based on artificial neural networks. Thanks to the increased data availability and computational power, such as Graphic Process Units (GPU), training deep networks - a time-consuming process - became possible. Cloud computing is an excellent option to acquire the computational power to train these models since it provides elastic products with a pay-per-use model. Amazon Web Services (AWS), for instance, has GPU-based virtual machine instances in its catalog, which differentiates themselves by the GPU type, number of GPUs, and price per hour. The challenge consists in determining which instance is better for a specific deep learning problem. This paper presents the implications, in terms of runtime and cost, of running two different deep learning problems on AWS GPU-based instances, and it proposes a methodology, based on the previous study cases, that analyzes instances for deep learning algorithms by using the information provided by the Keras framework. Our experimental results indicate that, despite having a higher price per hour, the instances that contain the NVIDIA V100 GPUs (p3) are faster and usually less expensive to use than the instances that contain the NVIDIA K80 GPUs (p2) for the problems we analyzed. Also, the results indicate that the performance of both applications did not scale well with the number of GPUs and that increasing the batch size to improve scalability may affect the final model accuracy. Finally, the proposed methodology provides accurate cost and estimated runtime for the tested applications on different AWS instances with a small cost.

Robust face alignment and tracking by combining local search and global fitting