Action unit detection by exploiting spatial-temporal and label-wise attention with transformer

Recently, an effective method to solve the problem of complex facial expression recognition is to encode individual facial expressions through the action units (AUs) encoded by the Facial Action Coding System (FACS). A large number of methods have been proposed for AU detection, but the problem of AU detection is still very challenging due to the different sizes and shapes of facial AU. Therefore, to solve the problem of locating AU, many methods first detect the landmark of the face and then detect AU according to the location of the landmark. However, in this way, the accuracy of landmark detection greatly affects the detection result of AU, making this method less robust. In this paper, we use a multi-scale structure (MTL) to solve the problem of facial AU region distribution size varies while being independent of the facial landmark information. Besides, due to the strong correlation between facial AU regions, we propose the structure of region relationship learning (LRT) using rich local information to learn the relationship between facial local regions. The end-to-end Multi-Scale Region with Local Relationship Learning (MTLRT-Net) we proposed is a lite network with low hardware requirements, and extensive experiments on the BP4D and DISFA demonstrate that our network framework is better than the state-of-the-art methods.

Facial action unit (AU) detection and face alignment are two highly correlated tasks since facial landmarks can provide precise AU locations to facilitate the extraction of meaningful local features for AU detection. Most existing AU detection works often treat face alignment as a preprocessing and handle the two tasks independently. In this paper, we propose a novel end-to-end deep learning framework for joint AU detection and face alignment, which has not been explored before. In particular, multi-scale shared features are learned firstly, and high-level features of face alignment are fed into AU detection. Moreover, to extract precise local features, we propose an adaptive attention learning module to refine the attention map of each AU adaptively. Finally, the assembled local features are integrated with face alignment features and global features for AU detection. Experiments on BP4D and DISFA benchmarks demonstrate that our framework significantly outperforms the state-of-the-art methods for AU detection.

Despite the success of deep neural networks on facial action unit (AU) detection, better performance depends on a large number of training images with accurate AU annotations. However, labeling AU is time-consuming, expensive, and error-prone. Considering AU detection and facial expression recognition (FER) are two highly correlated tasks, and facial expression (FE) is relatively easy to annotate, we consider learning AU detection and FER in a multi-task manner. However, the performance of the AU detection task cannot be always enhanced due to the negative transfer in the multi-task scenario. To alleviate this issue, we propose a Meta Auxiliary Learning method (MAL) that automatically selects highly related FE samples by learning adaptative weights for the training FE samples in a meta learning manner. The learned sample weights alleviate the negative transfer from two aspects: 1) balance the loss of each task automatically, and 2) suppress the weights of FE samples that have large uncertainties. Experimental results on several popular AU datasets demonstrate MAL consistently improves the AU detection performance compared with the state-of-the-art multi-task and auxiliary learning methods. MAL automatically estimates adaptive weights for the auxiliary FE samples according to their semantic relevance with the primary AU detection task.

Facial action unit (AU) detection is a hot topic in computer vision, but it remains challenging due to individual characteristics. Facial action features are a vital informative factor to explain facial anatomical variations but are often entangled with other facial attribution information leading to representation inconsistency within one category. We propose a novel Contrastive Disentangled Representation Autoencoder (CDAE) to learn discriminative identity-invariant representation for AU detection by factorizing face images into temporally varying action parts and stationary facial attributions components. Facial image space is mapped onto the facial action subspace and action-independent identity subspace to disentangle facial action information from identity information. In addition, we design a contrastive learning scheme to obtain a semantic-aware AU manifold by mapping the facial action features onto the continuous space of the latent variables, thus minimizing the misalignment between subjects and reducing the dimension of the facial action features. Experiment results show that CDAE outperforms or is comparable to previous AU detection methods on the challenging BP4D and DISFA benchmarks, demonstrating that the learned facial action representation is discriminative for AU detection.

Measuring the intensity of spontaneous facial action units with dynamic Bayesian network

We investigated adults' voluntary control of 20 facial action units theoretically associated with 6 basic emotions (happiness, fear, anger, surprise, sadness, and disgust). Twenty young adults were shown video excerpts of facial action units and asked to reproduce them as accurately as possible. Facial Action Coding System (FACS; Ekman & Friesen, 1978a) coding of the facial productions showed that young adults succeeded in activating 18 of the 20 target actions units, although they often coactivated other action units. Voluntary control was clearly better for some action units than for others, with a pattern of differences between action units consistent with previous work in children and adolescents.

The inherent dependencies among facial action units (AU) caused by the underlying anatomic mechanism are essential for the proper recognition of AUs and estimation of intensity levels, but they have not been exploited to their full potential. We are proposing novel methods to recognize AUs and estimate intensity via hybrid Bayesian networks. The upper two layers are latent regression Bayesian networks (LRBNs), and the lower layers are Bayesian networks (BNs). The visible nodes of the LRBN layers are representations of ground-truth AU occurrences or AU intensities. Through the directed connections from latent layer and visible layer, an LRBN can successfully represent relationships between multiple AUs or AU intensities. The lower layers include Bayesian networks with two nodes for AU recognition, and Bayesian networks with three nodes for AU intensity estimation. The bottom layers incorporate measurements from facial images with AU dependencies for intensity estimation and AU recognition. Efficient learning algorithms of the hybrid Bayesian networks are proposed for AU recognition as well as intensity estimation. Furthermore, the proposed hybrid Bayesian network models are extended for facial expression-assisted AU recognition and intensity estimation, as AU relationships are closely related to facial expressions. We test our methods on three benchmark databases for AU recognition and two benchmark databases for intensity estimation. The results demonstrate that the proposed approaches faithfully model the complex and global inherent AU dependencies, and the expression labels available only during training can boost the estimation of AU dependencies for both AU recognition and intensity estimation.

This study aims to explore a method capable of generating high-naturalness expressions for animated characters, thereby enhancing the audience’s emotional resonance. This study proposes a 3D face reconstruction system named Deep Convolutional Neural Network-Faces Learned with an Articulated Model and Expressions (DCNN-FLAME). DCNN-FLAME consists of an identity encoder, a mapping network, a Facial Landmark Embedding Model (FLAME) geometric decoding module, and a detail reconstruction network, forming an end-to-end processing pipeline from input images to 3D meshes and appearance textures. A style transfer module is constructed based on Deep Convolutional Neural Network (DCNN). It uses a pre-trained convolutional network to achieve effective separation of content and style, providing high-level semantic constraints for texture detail modeling of animated character expressions. On this basis, a dual-branch feature supervision mechanism composed of expression classification features and facial Action Units (AU) detection features is designed. Expression classification features provide global emotional semantic constraints to ensure macro-expression consistency. AU detection features guide local muscle movements from an anatomical perspective to enhance the realism of expression details. Experiments are conducted based on the large-scale face dataset VGGFace2. Systematic comparisons are performed with four 3D face reconstruction algorithms: 3D Morphable Model Fitting (3DMM-Fitting), RingNet, Detailed Expression Capture and Animation (DECA), and Fourier Analysis Networks-3D (FAN-3D). The proposed DCNN-FLAME model achieves a mean value of 1.29 in non-metric evaluation and 1.72 in metric evaluation. Both indicators are lower than those of all baseline methods, demonstrating higher geometric reconstruction accuracy and facial alignment quality. In the overall expression restoration evaluation, the F1 score of the proposed method reaches 0.564, reflecting comprehensive advantages in complex expression modeling. When both the expression classification branch and facial AU detection branch are enabled, the expression classification accuracy rate is 0.571 and the F1 score is 0.563, which are significantly better than the configuration using only a single feature for supervision. This verifies the key role of the dual-branch feature supervision mechanism in improving the naturalness and controllability of animated character expressions. This study provides an effective technical path integrating geometric reconstruction and texture enhancement for 3D animated character expression generation, and also offers new ideas and practical basis for the field of unsupervised 3D face reconstruction.

Facial action unit (AU) recognition is an important task for facial expression analysis. Traditional AU recognition methods typically include a supervised training, where the AU annotated training images are needed. AU annotation is a time consuming, expensive, and error prone process. While AU is hard to annotate, facial expression is relatively easy to label. To take advantage of this, we introduce a new learning method that trains an AU classifier using images with incomplete AU annotation but with complete expression labels. The goal is to use expression labels as hidden knowledge to complement the missing AU labels. Towards this goal, we propose to construct a Bayesian Network (BN) to capture the relationships between facial expression and AUs. Structural Expectation Maximum is used to learn the structure and parameters of the BN when the AU labels are missing. Given the learned BNs and measurements of AUs and expression, we can then perform AU recognition within the BN through a probabilistic inference. Experimental results on the CK+ and ISL databases demonstrate the effectiveness of our method.

In this paper, we aim to learn discriminative representation for facial action unit (AU) detection from large amount of videos without manual annotations. Inspired by the fact that facial actions are the movements of facial muscles, we depict the movements as the transformation between two face images in different frames and use it as the self-supervisory signal to learn the representations. However, under the uncontrolled condition, the transformation is caused by both facial actions and head motions. To remove the influence by head motions, we propose a Twin-Cycle Autoencoder (TCAE) that can disentangle the facial action related movements and the head motion related ones. Specifically, TCAE is trained to respectively change the facial actions and head poses of the source face to those of the target face. Our experiments validate TCAE's capability of decoupling the movements. Experimental results also demonstrate that the learned representation is discriminative for AU detection, where TCAE outperforms or is comparable with the state-of-the-art self-supervised learning methods and supervised AU detection methods.

Methods to assess individual facial actions have potential to shed light on important behavioral phenomena ranging from emotion and social interaction to psychological disorders and health. However, manual coding of such actions is labor intensive and requires extensive training. To date, establishing reliable automated coding of unscripted facial actions has been a daunting challenge impeding development of psychological theories and applications requiring facial expression assessment. It is therefore essential that automated coding systems be developed with enough precision and robustness to ease the burden of manual coding in challenging data involving variation in participant gender, ethnicity, head pose, speech, and occlusion. We report a major advance in automated coding of spontaneous facial actions during an unscripted social interaction involving three strangers. For each participant (n = 80, 47 % women, 15 % Nonwhite), 25 facial action units (AUs) were manually coded from video using the Facial Action Coding System. Twelve AUs occurred more than 3 % of the time and were processed using automated FACS coding. Automated coding showed very strong reliability for the proportion of time that each AU occurred (mean intraclass correlation = 0.89), and the more stringent criterion of frame-by-frame reliability was moderate to strong (mean Matthew's correlation = 0.61). With few exceptions, differences in AU detection related to gender, ethnicity, pose, and average pixel intensity were small. Fewer than 6 % of frames could be coded manually but not automatically. These findings suggest automated FACS coding has progressed sufficiently to be applied to observational research in emotion and related areas of study.

Facial Action Coding System (FACS) provides the description of all visual facial changes in terms of Action Units (AU), which express the movements of facial muscles. In this study we introduce a sparse learning method for AU detection based on facial landmark patches. The proposed method learns the active landmarks for each AU while finding their best representative patch sizes in a unified framework. The experiments performed on the publicly available CK+ dataset show that the proposed approach outperforms the state-of-the-art studies by gaining the highest F 1 scores for almost all of the lower face Action Units. The experiments also show that the subtle muscle movements belonging to the upper face require smaller landmark patches while the lower face AUs are detected better in larger patches.

Automatic facial action unit (AU) detection from video is a long-standing problem in computer vision. Two main approaches have been pursued: (1) static modeling - typically posed as a discriminative classification problem in which each video frame is evaluated independently; (2) temporal modeling - frames are segmented into sequences and typically modeled with a variant of dynamic Bayesian networks. We propose a segment-based approach, kSeg-SVM, that incorporates benefits of both approaches and avoids their limitations. kSeg-SVM is a temporal extension of the spatial bag-of-words. kSeg-SVM is trained within a structured output SVM framework that formulates AU detection as a problem of detecting temporal events in a time series of visual features. Each segment is modeled by a variant of the BoW representation with soft assignment of the words based on similarity. Our framework has several benefits for AU detection: (1) both dependencies between features and the length of action units are modeled; (2) all possible segments of the video may be used for training; and (3) no assumptions are required about the underlying structure of the action unit events (e.g., i.i.d.). Our algorithm finds the best k-or-fewer segments that maximize the SVM score. Experimental results suggest that the proposed method outperforms state-of-the-art static methods for AU detection.

Emotion analysis of students plays an important role in teaching effect evaluation. To develop robust algorithms for emotion analysis of students, a database from real classrooms is required. However, most existing databases were collected from adults and constructed in laboratory settings. In this article, we present a manually-annotated facial action unit database from juveniles in real classrooms. Our database has three main characteristics: (1) it provides numerous education-related action units data from primary and high schools, complementing the vacancy of the publicly available educational action unit databases; (2) it contains 256,220 manually-annotated facial images of 1,796 juveniles, frame-by-frame annotated with 12 action units and 6-level intensities for each action unit; (3) it covers many challenges in the wild, including various head poses, low facial resolution, illuminations, and occlusions, supplementing action unit databases in the wild for research. The baselines for action unit detection and action unit intensity estimation are provided for future references. Especially, we apply the weighted balance loss to solve imbalances <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">within</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">between</i> labels. Our database will be available to the research community: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">http://www.dlc.sjtu.edu.cn/rfau</uri> .

In this cross-sectional observational study, we developed a computer vision-based machine-learning model to estimate headache pain intensity from facial action units (AUs) and evaluated its association with self-reported pain. Facial videos were obtained from 80 adults aged 19-80 years receiving headache care at a single institution (outpatient, inpatient, or emergency; IRB No. 2024-08-003). A multitask cascaded convolutional network was used to detect and align faces, and convolutional neural networks estimated facial landmark locations and AU intensities. Recordings were synchronized and denoised. We extracted an APEX frame representing the peak facial expression and computed a headache pain intensity index (HPII) by combining pain-relevant AUs. The HPII showed a moderate positive Pearson correlation coefficient (r) with visual analog scale scores (r = 0.413-0.522, where higher r values indicate a stronger linear association) among participants with moderate-to-severe pain. This AU-based apex-frame approach may serve as a practical nonverbal indicator for monitoring headache pain intensity.