An empirical comparison of graph-based dimensionality reduction algorithms on facial expression recognition tasks

With the work and study pressure on people increasing and the importance of mental health problems on the rise, facial expression analysis plays an important role in mental health auxiliary diagnosis and treatment (MHADT). This study proposes a facial expression localization and recognition model that integrates Main Directional Mean Optical Flow (MDMO) with the Transformer architecture. It addresses the problems of insufficient generalization ability and limited temporal modeling ability of traditional methods in facial expression recognition. The study is based on the authoritative Audio/Visual Emotion Challenge 2019 Depression Detection Sub-challenge (AVEC2019 DDS) dataset in the mental health field, which contains 163 training samples, 56 validation samples, and 56 test samples. With Explicit Shape Regression, Local Binary Patterns, Mnemonic Descent Method, Convolutional Neural Network, and benchmark models as comparison objects, it systematically evaluates the model's performance in error, accuracy, and processing speed. The results show that the model achieves the best performance in both facial expression localization and recognition tasks. The validation set errors are 7.26 and 6.85, the localization accuracy reaches 89.6%, and the recognition accuracy reaches 88.9%, which is significantly better than other methods. At the same time, its image processing time is 55 milliseconds (ms) and 44ms, balancing high precision and real-time performance. The study indicates that the fusion of MDMO and Transformer can effectively capture the spatial and temporal features of facial expressions. Thus, this fusion method provides an efficient, stable, and scalable technical solution for emotion recognition in MHADT. This improves both the FER and localization accuracy and effect. Besides, it provides a novel reference in both approaches and application level for the intelligent development of mental health evaluation.

BackgroundAlcohol abuse and dependence can cause a wide variety of cognitive, psychomotor, and visual-spatial deficits. It is questionable whether this condition is associated with impairments in the recognition of affective and/or emotional information. Such impairments may promote deficits in social cognition and, consequently, in the adaptation and interaction of alcohol abusers with their social environment. The aim of this systematic review was to systematize the literature on alcoholics’ recognition of basic facial expressions in terms of the following outcome variables: accuracy, emotional intensity, and latency time.MethodsA systematic literature search in the PsycINFO, PubMed, and SciELO electronic databases, with no restrictions regarding publication year, was employed as the study methodology.ResultsThe findings of some studies indicate that alcoholics have greater impairment in facial expression recognition tasks, while others could not differentiate the clinical group from controls. However, there was a trend toward greater deficits in alcoholics. Alcoholics displayed less accuracy in recognition of sadness and disgust and required greater emotional intensity to judge facial expressions corresponding to fear and anger.ConclusionThe current study was only able to identify trends in the chosen outcome variables. Future studies that aim to provide more precise evidence for the potential influence of alcohol on social cognition are needed.

Facial expression recognition (FER) is utilized in various fields that analyze facial expressions. FER is attracting increasing attention for its role in improving the convenience in human life. It is widely applied in human–computer interaction tasks. However, recently, FER tasks have encountered certain data and training issues. To address these issues in FER, few-shot learning (FSL) has been researched as a new approach. In this paper, we focus on analyzing FER techniques based on FSL and consider the computational complexity and processing time in these models. FSL has been researched as it can solve the problems of training with few datasets and generalizing in a wild-environmental condition. Based on our analysis, we describe certain existing challenges in the use of FSL in FER systems and suggest research directions to resolve these issues. FER using FSL can be time efficient and reduce the complexity in many other real-time processing tasks and is an important area for further research.

Recent deep neural networks based methods have achieved state-of-the-art performance on various facial expression recognition tasks. Despite such progress, previous researches for facial expression recognition have mainly focused on analyzing color recording videos only. However, the complex emotions expressed by people with different skin colors under different lighting conditions through dynamic facial expressions can be fully understandable by integrating information from multi-modal videos. We present a novel method to estimate dimensional emotion states, where color, depth, and thermal recording videos are used as a multi-modal input. Our networks, called multi-modal recurrent attention networks (MRAN), learn spatiotemporal attention volumes to robustly recognize the facial expression based on attention-boosted feature volumes. We leverage the depth and thermal sequences as guidance priors for color sequence to selectively focus on emotional discriminative regions. We also introduce a novel benchmark for multi-modal facial expression recognition, termed as multi-modal arousal-valence facial expression recognition (MAVFER), which consists of color, depth, and thermal recording videos with corresponding continuous arousal-valence scores. The experimental results show that our method can achieve the state-of-the-art results in dimensional facial expression recognition on color recording datasets including RECOLA, SEWA and AFEW, and a multi-modal recording dataset including MAVFER.

<p>This thesis addresses three key challenges in gaze estimation and facial expression recognition datasets. Firstly, we tackle the issue of annotation inconsistency in gaze datasets by introducing the Gaze Adaptor Module (GAM) to ensure consistent gaze labels across different datasets. Additionally, we propose the Two-stage Transformer-based Gaze-feature Fusion (TTGF) architecture to enhance the fusion of face and eye features, resulting in improved gaze estimation accuracy. Secondly, we address the problem of imbalanced distribution in facial expression recognition datasets. To mitigate bias during training, we propose a recall-based weighting (RBW) strategy, which adjusts the weighting for expressions and demographic categories (such as gender) using the weighted cross-entropy loss. We also introduce a code-mixing synthesis approach to generate a balanced-synthesized dataset, and companion training with the synthesized dataset (CTSD) to reduce model bias. These techniques significantly improve the fairness and accuracy of facial expression recognition models. Lastly, we tackle the challenge of emotional frame sparsity in video datasets. We propose a novel model that incorporates an extra regression token to integrate features from all frames within each video. This enables effective handling of videos with varying lengths and addresses the issue of sparse emotional reaction frames. Experimental results demonstrate the superiority of our architecture in accurately estimating emotional reaction intensity in videos. By addressing these challenges, this thesis contributes to the advancement of gaze estimation and facial expression recognition.</p>

This paper designs a new facial Expression recognition network called a multi‐scale feature Fusion Convolutional neural Network (EFCN). This network is proposed to solve two problems in the facial expression recognition task. First, there are many commonalities between faces of different expression categories, and the recognition task cannot be precisely performed when the commonality is greater than the individuality. Secondly, facial detail features have a significant impact on the final results of expression recognition, while the image detail features extracted by traditional convolutional neural networks are not sufficient. In order to address the above issues, the feature enhancement network (FEN) and the detail information enhancement module (DEM) are designed. The FEN fuses deep and shallow features. Accordingly, the feature map contains richer information, making it easy to identify the samples. The DEM extracts and fuses the features passed by the backbone network with multi‐scale features to enhance the network's ability to extract features from small regions of the face. We validated the proposed method on three datasets, RAF‐DB, CK+, and JAFFE, and achieved 84.50%, 97.86%, and 91.05% accuracy, respectively, and the experimental results showed the effectiveness of the proposed method in this paper. For example, on the JAFFE dataset, the recognition accuracy of this method surpasses the MLT method by 1.87%.

Objective: This study investigated the relation between genetic variations in the dopamine system and facial expression recognition. Methods: A sample of Chinese college students (n = 478) was given a facial expression recognition task. Subjects were genotyped for 98 loci [96 single-nucleotide polymorphisms (SNPs) and 2 variable number tandem repeats] in 16 genes involved in the dopamine neurotransmitter system, including its 4 subsystems: synthesis (TH, DDC, and DBH), degradation/transport (COMT,MAOA,MAOB, and SLC6A3), receptors (DRD1,DRD2,DRD3,DRD4, and DRD5), and modulation (NTS,NTSR1,NTSR2, and NLN). To quantify the total contributions of the dopamine system to emotion recognition, we used a series of multiple regression models. Permutation analyses were performed to assess the posterior probabilities of obtaining such results. Results: Among the 78 loci that were included in the final analyses (after excluding 12 SNPs that were in high linkage disequilibrium and 8 that were not in Hardy-Weinberg equilibrium), 1 (for fear), 3 (for sadness), 5 (for anger), 13 (for surprise), and 15 (for disgust) loci exhibited main effects on the recognition of facial expressions. Genetic variations in the dopamine system accounted for 3% for fear, 6% for sadness, 7% for anger, 10% for surprise, and 18% for disgust, with the latter surviving a stringent permutation test. Conclusions: Genetic variations in the dopamine system (especially the dopamine synthesis and modulation subsystems) made significant contributions to individual differences in the recognition of disgust faces.

Facial Expression Recognition (FER) seeks to classify affective states from facial images, which remains a challenging problem due to variations in real-world conditions. FER task becomes particularly complex when handling unconstrained environments characterized by partial occlusions, different head poses, and so on. To address the above problems, current approaches rely on extensive learnable parameters and complex model architectures, which inevitably lead to overfitting and cause the FER model to focus on non-discriminative facial regions. In this work, we propose an HKAFER model that can adaptively enhance visual expression representations through efficiently fine-tuning the image encoder in large Visual Foundation Models (VFMs) and Vision-Language Models (VLMs). Specifically, we establish Heterogeneous Kronecker Adaptation (HeKA), which consists of multi-scale adapters based on Kronecker product in a parallel manner, offering significantly diverse subspaces to learn the incremental matrices. Besides, we also propose Dual-Branch Interactive Router (DBIR) to dynamically assign the weights of adapters, which promotes collaboration and information flow among them. In this way, our HKAFER can effectively capture robust spatial features and the regional associations. Experimental results demonstrate that our proposed model not only outperforms state-of-the-art methods on several FER benchmarks but also uses significantly fewer trainable parameters.

Facial expression recognition (FER) in the wild is a challenging task due to some uncontrolled factors such as occlusion, illumination, and pose variation. The current methods perform well in controlled conditions. However, there are still two issues with the in-the-wild FER task: (i) insufficient descriptions of long-range dependency of expression features in the facial information space and (ii) not finely refining subtle inter-classes distinction from multiple expressions in the wild. To overcome the above issues, an end-to-end model for FER, named attention-modulated contextual spatial information network (ACSI-Net), is presented in this paper, with the manner of embedding coordinate attention (CA) modules into a contextual convolutional residual network (CoResNet). Firstly, CoResNet is constituted by arranging contextual convolution (CoConv) blocks of different levels to integrate facial expression features with long-range dependency, which generates a holistic representation of spatial information on facial expression. Then, the CA modules are inserted into different stages of CoResNet, at each of which the subtle information about facial expression acquired from CoConv blocks is first modulated by the corresponding CA module across channels and spatial locations and then flows into the next layer. Finally, to highlight facial regions related to expression, a CA module located at the end of the whole network, which produces attentional masks to multiply by input feature maps, is utilized to focus on salient regions. Different from other models, the ACSI-Net is capable of exploring intrinsic dependencies between features and yielding a discriminative representation for facial expression classification. Extensive experimental results on AffectNet and RAF_DB datasets demonstrate its effectiveness and competitiveness compared to other FER methods.

Background: Individuals with child maltreatment (CM) experiences show alterations in emotion recognition (ER). However, previous research has mainly focused on populations with specific mental disorders, which makes it unclear whether alterations in the recognition of facial expressions are related to CM, to the presence of mental disorders or to the combination of CM and mental disorders, and on ER of emotional, rather than neutral facial expressions. Moreover, commonly, recognition of static stimulus material was researched. Objective: We assessed recognition of dynamic (closer to real life) negative, positive and neutral facial expressions in individuals characterised by CM, rather than a specific mental disorder. Moreover, we assessed whether they show a negativity bias for neutral facial expressions and whether the presence of one or more mental disorders affects recognition. Methods: Ninety-eight adults with CM experiences (CM+) and 60 non-maltreated (CM−) adult controls watched 200 non-manipulated coloured video sequences, showing 20 neutral and 180 emotional facial expressions, and indicated whether they interpreted each expression as neutral or as one of eight emotions. Results: The CM+ showed significantly lower scores in the recognition of positive, negative and neutral facial expressions than the CM− group (p < .050). Furthermore, the CM+ group showed a negativity bias for neutral facial expressions (p < .001). When accounting for mental disorders, significant effects stayed consistent, except for the recognition of positive facial expressions: individuals from the CM+ group with but not without mental disorder scored lower than controls without mental disorder. Conclusions: CM might have long-lasting influences on the ER abilities of those affected. Future research should explore possible effects of ER alterations on everyday life, including implications of the negativity bias for neutral facial expressions on emotional wellbeing and relationship satisfaction, providing a basis for interventions that improve social functioning.

We propose a novel set of features extracted using Hybrid Discriminant Analysis (HDA) for use in a facial expression recognition task. HDA is formulated by combining the optimization problems which are solved in Linear Discriminant Analysis (LDA) and Principal Component Analysis (PCA) methods. By tuning weighting parameters, new sets of discriminant directions are obtained. Using these directions for feature extraction, we show that we can obtain excellent recognition rates of 95.9% on a database of 6 expressions plus neutral which is much better than that obtained using PCA or LDA. The performance of HDA is comparable to that of Gabor wavelet based methods. However, the computation time required to form a HDA feature vector is significantly lower than the time required to form a Gabor feature vector making this method very attractive for real-time applications.

The nature of facial expression recognition deficits following orbitofrontal cortex damage.

Facial expression recognition faces great challenges due to factors such as face similarity, image quality, and age variation. Although various existing end-to-end Convolutional Neural Network (CNN) architectures have achieved good classification results in facial expression recognition tasks, these network architectures share a common drawback that the convolutional kernel can only compute the correlation between elements of a localized region when extracting expression features from an image. This leads to difficulties for the network to explore the relationship between all the elements that make up a complete expression. In response to this issue, this article proposes a facial expression recognition network called HFE-Net. In order to capture the subtle changes of expression features and the whole facial expression information at the same time, HFE-Net proposed a Hybrid Feature Extraction Block. Specifically, Hybrid Feature Extraction Block consists of parallel Feature Fusion Device and Multi-head Self-attention. Among them, Feature Fusion Device not only extracts the local information in expression features, but also measures the correlation between distant elements in expression features, which helps the network to focus more on the target region while realizing the information interaction between distant features. And Multi-head Self-attention can calculate the correlation between the overall elements in the feature map, which helps the network to extract the overall information of the expression features. We conducted a lot of experiments on four publicly available facial expression datasets and verified that the Hybrid Feature Extraction Block constructed in this paper can improve the network's recognition ability for facial expressions.

Local Binary Patterns (LBP) have been well exploited for facial image analysis recently. In the existing work, the LBP histograms are extracted from local facial regions, and used as a whole for the regional description. However, not all bins in the LBP histogram are necessary to be useful for facial representation. In this paper, we propose to learn discriminative LBP-Histogram (LBPH) bins for the task of facial expression recognition. Our experiments illustrate that the selected LBPH bins provide a compact and discriminative facial representation. We experimentally illustrate that it is necessary to consider multiscale LBP for representing faces, and most discriminative information is contained in uniform patterns. By adopting SVM with the selected multiscale LBPH bins, we obtain the best recognition performance of 93.1% on the Cohn-Kanade database.

Deep convolutional neural networks (CNNs) have established their feet in the ground of computer vision and machine learning, used in various applications. In this work, an attempt is made to learn a CNN for a task of facial expression recognition (FER). Our network has convolutional layers linked with an FC layer with a skip-connection to the classification layer. Motivation behind this design is that lower layers of a CNN are responsible for lower level features, and facial expressions can be mainly encoded in low-to-mid level features. Hence, in order to leverage the responses from lower layers, all convo-lutional layers are integrated via FC layers. Moreover, a network with shared parameters is used to extract landmark motion trajectory features. These visual and landmark features are fused to improve the performance. Our method is evaluated on the CK+ and Oulu-CASIA facial expression datasets.