Constrained Joint Cascade Regression Framework for Simultaneous Facial Action Unit Recognition and Facial Landmark Detection

A crucial part of the face recognition process, along with facial attribute analysis and face verification, is facial landmark detection. In order to prepare face images for feature extraction in facial analysis jobs, face alignment is an essential step. Alignment is frequently used in training and inference to standardize the locations of important facial landmarks for applications including face recognition, facial emotion detection, and facial attribute classification. The effectiveness of facial analysis models is known to be greatly impacted by the approach and application of face alignment. However, there has not been enough research done on how alignment affects the quality of facial images. Although face alignment techniques based on convolutional neural networks have advanced significantly recently, occlusion remains a fundamental obstacle to achieving high accuracy. In order to enhance performance, we present in this study the attentioned distillation module from our earlier work on the Occlusion-Adaptive Deep Network model. A distillation module in this model infers the occlusion probability of every place in high-level features. When measuring the relationship between facial shape and look, it can be automatically learned. However, because of the missing semantic features, the holistic face cannot be represented by the clean feature representation. We must use a low-rank learning module to recover lost features in order to achieve exhaustive and complete feature representation.

Facial action unit (AU) recognition is an important task for facial expression analysis. Traditional AU recognition methods typically include a supervised training, where the AU annotated training images are needed. AU annotation is a time consuming, expensive, and error prone process. While AU is hard to annotate, facial expression is relatively easy to label. To take advantage of this, we introduce a new learning method that trains an AU classifier using images with incomplete AU annotation but with complete expression labels. The goal is to use expression labels as hidden knowledge to complement the missing AU labels. Towards this goal, we propose to construct a Bayesian Network (BN) to capture the relationships between facial expression and AUs. Structural Expectation Maximum is used to learn the structure and parameters of the BN when the AU labels are missing. Given the learned BNs and measurements of AUs and expression, we can then perform AU recognition within the BN through a probabilistic inference. Experimental results on the CK+ and ISL databases demonstrate the effectiveness of our method.

Facial action unit (AU) recognition has attracted great attention because of the applications in a wide range of fields. Missing labels and class imbalance (CIB) are both challenges for facial action unit recognition. Missing labels means that there are only apart label assignments for training samples. CIB is observed from two perspectives: firstly, the number of positive AUs is much smaller than that of negative AUs for each expressional image; secondly, the rate of positive samples of different AUs are significantly different. Both missing labels and CIB lead to performance degradation in AU recognition. In this work, we propose to handle these two challenges in AU recognition simultaneously. Specifically, we formulate AU recognition with missing labels as a multi label learning with missing labels (MLML) problem, which handles the missing label challenge naturally. However, different from most existing MLML approaches which usually employ same features from whole image for all classes, we select the most related features for each AU. To handle the CIB challenge, we further introduce class cardinality bounds which constrain the number of positive AUs for each data instance, as well as the number of positive labels for each AU in the overall dataset. The class cardinality bounds serve as linear constraints for the objective function, which turns the optimization NP-hard. Thus we present convex approximation based on the Lovasz extension, which leads to a linear program that can be efficiently solved by the alternative direction method of multipliers (ADMM). Experimental results on both posed and spontaneous facial expression datasets demonstrate the superiority of the proposed method compared to state-of-the-art.

Facial action unit (AU) recognition is essential to facial expression analysis. Since there are highly positive or negative correlations between AUs, some existing AU recognition works have focused on modeling AU relations. However, previous relationship-based approaches typically embed predefined rules into their models and ignore the impact of various AU relations in different crowds. In this paper, we propose a novel Attention Based Relation Network (ABRNet) for AU recognition, which can automatically capture AU relations without unnecessary or even disturbing predefined rules. ABRNet uses several relation learning layers to automatically capture different AU relations. The learned AU relation features are then fed into a self-attention fusion module, which aims to refine individual AU features with attention weights to enhance the feature robustness. Furthermore, we propose an AU relation dropout strategy and AU relation loss (AUR-Loss) to better model AU relations, which can further improve AU recognition. Extensive experiments show that our approach achieves state-of-the-art performance on the DISFA and DISFA+ datasets.

Efficient facial landmark localization using spatial–contextual AdaBoost algorithm

Facial action units (AUs) represent facial muscular activities, and our emotions can be expressed through their combinations. Thus, AU recognition is often used in many different applications, including marketing, healthcare, and education. Numerous studies have been conducted on recognizing AUs through several network architectures; however, their performances remain unsatisfactory. One of the difficulties comes from the lack of information regarding a neutral state (i.e., no facial muscular activities) of each person owing to the individuality of a neutral state. This lack of information degrades the recognition performance because the intensities of AUs are derived from a neutral state. In this paper, we propose a novel method using Pseudo-INtensities and their Transformation (PINT) to tackle this problem. To exclude the individuality of a neutral state and accurately capture the changes in facial appearance regarding AUs, we first calculate pseudo-intensities based only on the differences among the intensity states of the same person. We utilize a siamese network architecture and the facial image pairs of the same person to calculate the pseudo-intensities. These pseudo-intensities are then transformed into the actual intensities based on the low pseudo-intensities of the same person, which are considered to correspond to neutral states. We carried out evaluation experiments using two public datasets and found that our method, PINT, achieved a state-of-art performance. The improvements in the average intra-class correlation coefficient score over existing methods were 7.1% on DISFA dataset and 3.1% on FERA2017 dataset.

The inherent dependencies among facial action units (AU) caused by the underlying anatomic mechanism are essential for the proper recognition of AUs and estimation of intensity levels, but they have not been exploited to their full potential. We are proposing novel methods to recognize AUs and estimate intensity via hybrid Bayesian networks. The upper two layers are latent regression Bayesian networks (LRBNs), and the lower layers are Bayesian networks (BNs). The visible nodes of the LRBN layers are representations of ground-truth AU occurrences or AU intensities. Through the directed connections from latent layer and visible layer, an LRBN can successfully represent relationships between multiple AUs or AU intensities. The lower layers include Bayesian networks with two nodes for AU recognition, and Bayesian networks with three nodes for AU intensity estimation. The bottom layers incorporate measurements from facial images with AU dependencies for intensity estimation and AU recognition. Efficient learning algorithms of the hybrid Bayesian networks are proposed for AU recognition as well as intensity estimation. Furthermore, the proposed hybrid Bayesian network models are extended for facial expression-assisted AU recognition and intensity estimation, as AU relationships are closely related to facial expressions. We test our methods on three benchmark databases for AU recognition and two benchmark databases for intensity estimation. The results demonstrate that the proposed approaches faithfully model the complex and global inherent AU dependencies, and the expression labels available only during training can boost the estimation of AU dependencies for both AU recognition and intensity estimation.

Facial action unit (AU) recognition has attracted increasing attention due to its indispensable role in affective computing, especially in the field of affective human-computer interaction. Due to the subtle and transient nature of AU, it is challenging to capture the delicate and ambiguous motions in local facial regions among consecutive frames. Considering that context is essential to resolve ambiguity in human visual system, modeling context within or among facial images emerges as a promising approach for AU recognition task. To this end, we propose CaFGraph, a novel context-aware facial multi-graph that can model both morphological & muscular-based region-level local context and region-level temporal context. CaFGraph is the first work to construct a universal facial multi-graph structure that is independent of both task settings and dataset statistics for almost all fine-grained facial behavior analysis tasks, including but not limited to AU recognition. To make full use of the context, we then present CaFNet that learns context-aware facial graph representations via CaFGraph from facial images for multi-label AU recognition. Experiments on two widely used benchmark datasets, BP4D and DISFA, demonstrate the superiority of our CaFNet over the state-of-the-art methods.

Facial action unit (AU) detection and face alignment are two highly correlated tasks since facial landmarks can provide precise AU locations to facilitate the extraction of meaningful local features for AU detection. Most existing AU detection works often treat face alignment as a preprocessing and handle the two tasks independently. In this paper, we propose a novel end-to-end deep learning framework for joint AU detection and face alignment, which has not been explored before. In particular, multi-scale shared features are learned firstly, and high-level features of face alignment are fed into AU detection. Moreover, to extract precise local features, we propose an adaptive attention learning module to refine the attention map of each AU adaptively. Finally, the assembled local features are integrated with face alignment features and global features for AU detection. Experiments on BP4D and DISFA benchmarks demonstrate that our framework significantly outperforms the state-of-the-art methods for AU detection.

In recent years, human face recognition technology has made remarkable progress. However, research on animal face recognition has not progressed as much as that on human face recognition. In this study, we focus on cats. The detection of facial landmarks is considered to be a prerequisite for face recognition. In this paper, we show that we succeeded in detecting facial landmarks in cat images using a model trained on HRNetV2.

Facial landmark detection is an important yet challenging task for real-world\ncomputer vision applications. This paper proposes an effective and robust\napproach for facial landmark detection by combining data- and model-driven\nmethods. Firstly, a Fully Convolutional Network (FCN) is trained to compute\nresponse maps of all facial landmark points. Such a data-driven method could\nmake full use of holistic information in a facial image for global estimation\nof facial landmarks. After that, the maximum points in the response maps are\nfitted with a pre-trained Point Distribution Model (PDM) to generate the\ninitial facial shape. This model-driven method is able to correct the\ninaccurate locations of outliers by considering the shape prior information.\nFinally, a weighted version of Regularized Landmark Mean-Shift (RLMS) is\nemployed to fine-tune the facial shape iteratively. This\nEstimation-Correction-Tuning process perfectly combines the advantages of the\nglobal robustness of data-driven method (FCN), outlier correction capability of\nmodel-driven method (PDM) and non-parametric optimization of RLMS. Results of\nextensive experiments demonstrate that our approach achieves state-of-the-art\nperformances on challenging datasets including 300W, AFLW, AFW and COFW. The\nproposed method is able to produce satisfying detection results on face images\nwith exaggerated expressions, large head poses, and partial occlusions.\n

Facial landmark localization is well known as one of the bottlenecks in face recognition. This paper proposes a novel facial landmark localization method, which introduces facial context constrains into cascaded AdaBoost framework. The motivation of our method lies in the basic human physiology observation that not only the local texture information but also the global context information is used together for human to realize the landmark location task. Therefore, in our solution, a novel type of Haar-like feature, called discontinuous Haar-like feature, is proposed to characterize the facial context, i.e. the cooccurrence relationship between target facial landmark and other local texture patterns within face region (including other landmarks, facial organs and also smoothing regions). For the locating task, traditional Haar-like features (characterizing local texture information) and discontinuous Haar-like features (characterizing context constrains in global sense) are combined together to form more powerful representations. Through Real AdaBoost learning, distinctive features are selected automatically and used for facial landmark detection. Our experiments on BioID and Cohn-Kanade databases have validated the proposed method by comparing with other state-of-the-art results.

Quantitative analysis of facial soft tissue using weighted cascade regression model applicable for facial plastic surgery

Feature and label relation modeling for multiple-facial action unit classification and intensity estimation

In this cross-sectional observational study, we developed a computer vision-based machine-learning model to estimate headache pain intensity from facial action units (AUs) and evaluated its association with self-reported pain. Facial videos were obtained from 80 adults aged 19-80 years receiving headache care at a single institution (outpatient, inpatient, or emergency; IRB No. 2024-08-003). A multitask cascaded convolutional network was used to detect and align faces, and convolutional neural networks estimated facial landmark locations and AU intensities. Recordings were synchronized and denoised. We extracted an APEX frame representing the peak facial expression and computed a headache pain intensity index (HPII) by combining pain-relevant AUs. The HPII showed a moderate positive Pearson correlation coefficient (r) with visual analog scale scores (r = 0.413-0.522, where higher r values indicate a stronger linear association) among participants with moderate-to-severe pain. This AU-based apex-frame approach may serve as a practical nonverbal indicator for monitoring headache pain intensity.