Domain adaptive representation learning for facial action unit recognition

Facial action unit (AU) recognition has attracted great attention because of the applications in a wide range of fields. Missing labels and class imbalance (CIB) are both challenges for facial action unit recognition. Missing labels means that there are only apart label assignments for training samples. CIB is observed from two perspectives: firstly, the number of positive AUs is much smaller than that of negative AUs for each expressional image; secondly, the rate of positive samples of different AUs are significantly different. Both missing labels and CIB lead to performance degradation in AU recognition. In this work, we propose to handle these two challenges in AU recognition simultaneously. Specifically, we formulate AU recognition with missing labels as a multi label learning with missing labels (MLML) problem, which handles the missing label challenge naturally. However, different from most existing MLML approaches which usually employ same features from whole image for all classes, we select the most related features for each AU. To handle the CIB challenge, we further introduce class cardinality bounds which constrain the number of positive AUs for each data instance, as well as the number of positive labels for each AU in the overall dataset. The class cardinality bounds serve as linear constraints for the objective function, which turns the optimization NP-hard. Thus we present convex approximation based on the Lovasz extension, which leads to a linear program that can be efficiently solved by the alternative direction method of multipliers (ADMM). Experimental results on both posed and spontaneous facial expression datasets demonstrate the superiority of the proposed method compared to state-of-the-art.

Facial action unit (AU) recognition is essential to facial expression analysis. Since there are highly positive or negative correlations between AUs, some existing AU recognition works have focused on modeling AU relations. However, previous relationship-based approaches typically embed predefined rules into their models and ignore the impact of various AU relations in different crowds. In this paper, we propose a novel Attention Based Relation Network (ABRNet) for AU recognition, which can automatically capture AU relations without unnecessary or even disturbing predefined rules. ABRNet uses several relation learning layers to automatically capture different AU relations. The learned AU relation features are then fed into a self-attention fusion module, which aims to refine individual AU features with attention weights to enhance the feature robustness. Furthermore, we propose an AU relation dropout strategy and AU relation loss (AUR-Loss) to better model AU relations, which can further improve AU recognition. Extensive experiments show that our approach achieves state-of-the-art performance on the DISFA and DISFA+ datasets.

Facial action unit (AU) recognition has attracted increasing attention due to its indispensable role in affective computing, especially in the field of affective human-computer interaction. Due to the subtle and transient nature of AU, it is challenging to capture the delicate and ambiguous motions in local facial regions among consecutive frames. Considering that context is essential to resolve ambiguity in human visual system, modeling context within or among facial images emerges as a promising approach for AU recognition task. To this end, we propose CaFGraph, a novel context-aware facial multi-graph that can model both morphological & muscular-based region-level local context and region-level temporal context. CaFGraph is the first work to construct a universal facial multi-graph structure that is independent of both task settings and dataset statistics for almost all fine-grained facial behavior analysis tasks, including but not limited to AU recognition. To make full use of the context, we then present CaFNet that learns context-aware facial graph representations via CaFGraph from facial images for multi-label AU recognition. Experiments on two widely used benchmark datasets, BP4D and DISFA, demonstrate the superiority of our CaFNet over the state-of-the-art methods.

Facial action unit (AU) recognition is an important task for facial expression analysis. Traditional AU recognition methods typically include a supervised training, where the AU annotated training images are needed. AU annotation is a time consuming, expensive, and error prone process. While AU is hard to annotate, facial expression is relatively easy to label. To take advantage of this, we introduce a new learning method that trains an AU classifier using images with incomplete AU annotation but with complete expression labels. The goal is to use expression labels as hidden knowledge to complement the missing AU labels. Towards this goal, we propose to construct a Bayesian Network (BN) to capture the relationships between facial expression and AUs. Structural Expectation Maximum is used to learn the structure and parameters of the BN when the AU labels are missing. Given the learned BNs and measurements of AUs and expression, we can then perform AU recognition within the BN through a probabilistic inference. Experimental results on the CK+ and ISL databases demonstrate the effectiveness of our method.

Facial action units (AUs) represent facial muscular activities, and our emotions can be expressed through their combinations. Thus, AU recognition is often used in many different applications, including marketing, healthcare, and education. Numerous studies have been conducted on recognizing AUs through several network architectures; however, their performances remain unsatisfactory. One of the difficulties comes from the lack of information regarding a neutral state (i.e., no facial muscular activities) of each person owing to the individuality of a neutral state. This lack of information degrades the recognition performance because the intensities of AUs are derived from a neutral state. In this paper, we propose a novel method using Pseudo-INtensities and their Transformation (PINT) to tackle this problem. To exclude the individuality of a neutral state and accurately capture the changes in facial appearance regarding AUs, we first calculate pseudo-intensities based only on the differences among the intensity states of the same person. We utilize a siamese network architecture and the facial image pairs of the same person to calculate the pseudo-intensities. These pseudo-intensities are then transformed into the actual intensities based on the low pseudo-intensities of the same person, which are considered to correspond to neutral states. We carried out evaluation experiments using two public datasets and found that our method, PINT, achieved a state-of-art performance. The improvements in the average intra-class correlation coefficient score over existing methods were 7.1% on DISFA dataset and 3.1% on FERA2017 dataset.

The inherent dependencies among facial action units (AU) caused by the underlying anatomic mechanism are essential for the proper recognition of AUs and estimation of intensity levels, but they have not been exploited to their full potential. We are proposing novel methods to recognize AUs and estimate intensity via hybrid Bayesian networks. The upper two layers are latent regression Bayesian networks (LRBNs), and the lower layers are Bayesian networks (BNs). The visible nodes of the LRBN layers are representations of ground-truth AU occurrences or AU intensities. Through the directed connections from latent layer and visible layer, an LRBN can successfully represent relationships between multiple AUs or AU intensities. The lower layers include Bayesian networks with two nodes for AU recognition, and Bayesian networks with three nodes for AU intensity estimation. The bottom layers incorporate measurements from facial images with AU dependencies for intensity estimation and AU recognition. Efficient learning algorithms of the hybrid Bayesian networks are proposed for AU recognition as well as intensity estimation. Furthermore, the proposed hybrid Bayesian network models are extended for facial expression-assisted AU recognition and intensity estimation, as AU relationships are closely related to facial expressions. We test our methods on three benchmark databases for AU recognition and two benchmark databases for intensity estimation. The results demonstrate that the proposed approaches faithfully model the complex and global inherent AU dependencies, and the expression labels available only during training can boost the estimation of AU dependencies for both AU recognition and intensity estimation.

Cascade regression framework has been shown to be effective for facial landmark detection. It starts from an initial face shape and gradually predicts the face shape update from the local appearance features to generate the facial landmark locations in the next iteration until convergence. In this paper, we improve upon the cascade regression framework and propose the Constrained Joint Cascade Regression Framework (CJCRF) for simultaneous facial action unit recognition and facial landmark detection, which are two related face analysis tasks, but are seldomly exploited together. In particular, we first learn the relationships among facial action units and face shapes as a constraint. Then, in the proposed constrained joint cascade regression framework, with the help from the constraint, we iteratively update the facial landmark locations and the action unit activation probabilities until convergence. Experimental results demonstrate that the intertwined relationships of facial action units and face shapes boost the performances of both facial action unit recognition and facial landmark detection. The experimental results also demonstrate the effectiveness of the proposed method comparing to the state-of-the-art works.

In the facial action unit (Facial Action Unit, AU) recognition process, due to the low occurrence probability of some AUs, the sample imbalance is serious, which severely limits the model recognition performance. Generative adversarial network GAN is an unsupervised learning method. Compared with the autoencoder and autoregressive model in the unsupervised learning method, its advantages are sufficient data fitting, higher efficiency and better generated samples. The original GAN model uses the minimum and maximum (minmax) to continuously optimize the training of the model; the conditional generation adversarial network CGAN adds conditional constraints to the model input to make the generated results controllable and prevent collapse in the model training process. GAN has been widely used in research fields such as image processing, natural language processing NLP, and real-time color correction of underwater images. This paper designs a model based on a conditional generation adversarial network to supplement the minority samples of a specific AU and improve the sample distribution space of the action unit.

Feature and label relation modeling for multiple-facial action unit classification and intensity estimation

Emotional stress detection can be performed analyzing different facial parameters. This paper focuses on the automated identification of facial Action Units (AU) as quantitative indices in order to discriminate between neutral and stress/anxiety state. Thus, a model for automatic recognition of facial action units is proposed being trained in two available annotated facial datasets, the UNBC and the BOSPHORUS datasets. Facial features, both geometric (non-rigid deformations of 3D shape of AAM landmarks) and appearance (Histograms of Oriented Gradients) were extracted. The intensity of each AU was regressed using Support Vector Regression (SVR). The corresponding models of each dataset were fused to a combined model. This combined model was applied to the experimental dataset (SRD’15) containing neutral states and inducing stressful states related to four types of stress. The results indicate that there are specific AU relevant to stress and the AU intensity are significant increased during stress leading to a more expressive human face.

Automated analysis of facial expressions is a crucial challenge in the emerging field of animal affective computing. One of the most promising approaches in this context is facial landmarks, which are well-studied for humans and are now being adopted for many non-human species. The scarcity of high-quality, comprehensive datasets is a significant challenge in the field. This paper is the first to present a novel Dog Facial Landmarks in the Wild (DogFLW) dataset containing 3732 images of dogs annotated with facial landmarks and bounding boxes. Our facial landmark scheme has 46 landmarks grounded in canine facial anatomy, the Dog Facial Action Coding System (DogFACS), and informed by existing cross-species landmarking methods. We additionally provide a benchmark for dog facial landmarks detection and demonstrate two case studies for landmark detection models trained on the DogFLW. The first is a pipeline using landmarks for emotion classification from dog facial expressions from video, and the second is the recognition of DogFACS facial action units (variables), which can enhance the DogFACS coding process by reducing the time needed for manual annotation. The DogFLW dataset aims to advance the field of animal affective computing by facilitating the development of more accurate, interpretable, and scalable tools for analysing facial expressions in dogs with broader potential applications in behavioural science, veterinary practice, and animal-human interaction research.

This paper describes an Imprecise Dirichlet Model and the maximum entropy criterion to learn Bayesian network parameters under insufficient and incomplete data. The method is applied to two distinct recognition problems, namely, a facial action unit recognition and an activity recognition in video surveillance sequences. The model treats a wide range of constraints that can be specified by experts, and deals with incomplete data using an ad-hoc expectation-maximization procedure. It is also described how the same idea can be used to learn dynamic Bayesian networks. With synthetic data, we show that our proposal and widely used methods, such as the Bayesian maximum a posteriori, achieve similar accuracy. However, when real data come in place, our method performs better than the others, because it does not rely on a single prior distribution, which might be far from the best one.

This paper focuses on the preprocessing of facial feature points and conducts research. Firstly, based on expert prior knowledge, ROI (Region of Interest) partitions were designed to lay the foundation for subsequent facial-related analysis; then, the design of the flexible AU (Action Unit) recognition network model was focused on, and a complete research framework including the system model and problem description, specific solutions, simulation results and discussion, and conclusion was constructed. The paper first elaborates on the research motivation, summarizes the related work background, and clarifies the research direction; then, it introduces the structure of the system model in detail, clearly expounds the essence of the problem and the corresponding solutions; subsequently, through simulation experiments, results are obtained and in-depth discussions are carried out to verify the effectiveness of the scheme; finally, the research results are summarized, forming a complete research system for flexible AU recognition network model, providing a valuable technical solution and research idea for the field of facial action unit recognition.

Facial action units (AUs) play an important role in human emotion understanding. One big challenge for data-driven AU recognition approaches is the lack of enough AU annotations, since AU annotation requires strong domain expertise. To alleviate this issue, we propose a knowledge-driven method for jointly learning multiple AU classifiers without any AU annotation by leveraging prior probabilities on AUs, including expression-independent and expression-dependent AU probabilities. These prior probabilities are drawn from facial anatomy and emotion studies, and are independent of datasets. We incorporate the prior probabilities on AUs as the constraints into the objective function of multiple AU classifiers, and develop an efficient learning algorithm to solve the formulated problem. Experimental results on five benchmark expression databases demonstrate the effectiveness of the proposed method, especially its generalization ability, and the power of the prior probabilities.

To overcome the barriers of multimodality and scarcity of annotations in medical image segmentation, many unsupervised domain adaptation (UDA) methods have been proposed, especially in cardiac segmentation. However, these methods may not completely avoid the interference of domain-specific information. To tackle this problem, we propose a novel Attention-enhanced Disentangled Representation (ADR) learning model for UDA in cardiac segmentation. To sufficiently remove domain shift and mine more precise domain-invariant features, we first put forward a strategy from image-level coarse alignment to fine removal of remaining domain shift. Unlike previous dual path disentanglement methods, we present channel-wise disentangled representation learning to promote mutual guidance between domain-invariant and domain-specific features. Meanwhile, Hilbert-Schmidt independence criterion (HSIC) is adopted to establish the independence between the disentangled features. Furthermore, we propose an attention bias for adversarial learning in the output space to enhance the learning of task-relevant domain-invariant features. To obtain more accurate predictions during inference, an information fusion calibration (IFC) is also proposed. Extensive experiments on the MMWHS 2017 dataset demonstrate the superiority of our method. Code is available at https://github.com/Sunxy11/ADR.KeywordsUnsupervised domain adaptationDisentanglementCardiac segmentation