CHTMAE: Cross-Modal Hierarchical Temporal-Spatial Masked Autoencoder Model for Micro-Expression Recognition

Micro-expressions (MEs) are rapid, involuntary facial expressions which reveal emotions that people do not intend to show. Studying MEs is valuable as recognizing them has many important applications, particularly in forensic science and psychotherapy. However, analyzing spontaneous MEs is very challenging due to their short duration and low intensity. Automatic ME analysis includes two tasks: ME spotting and ME recognition. For ME spotting, previous studies have focused on posed rather than spontaneous videos. For ME recognition, the performance of previous studies is low. To address these challenges, we make the following contributions: (i) We propose the first method for spotting spontaneous MEs in long videos (by exploiting feature difference contrast). This method is training free and works on arbitrary unseen videos. (ii) We present an advanced ME recognition framework, which outperforms previous work by a large margin on two challenging spontaneous ME databases (SMIC and CASMEII). (iii) We propose the first automatic ME analysis system (MESR), which can spot and recognize MEs from spontaneous video data. Finally, we show our method outperforms humans in the ME recognition task by a large margin, and achieves comparable performance to humans at the very challenging task of spotting and then recognizing spontaneous MEs.

Micro-expressions (MEs) are important clues for reflecting the real feelings of humans, and micro-expression recognition (MER) can thus be applied in various real-world applications. However, it is difficult to perceive and interpret MEs correctly. With the advance of deep learning technologies, the accuracy of micro-expression recognition is improved but still limited by the lack of large-scale datasets. In this paper, we propose a novel micro-expression recognition approach by combining Action Units (AUs) and emotion category labels. Specifically, based on facial muscle movements, we model different AUs based on relational information and integrate the AUs recognition task with MER. Besides, to overcome the shortcomings of limited and imbalanced training samples, we propose a data augmentation method that can generate nearly indistinguishable image sequences with AU intensity of real-world micro-expression images, which effectively improve the performance and are compatible with other micro-expression recognition methods. Experimental results on three mainstream micro-expression datasets, i.e., CASME II, SAMM, and SMIC, manifest that our approach outperforms other state-of-the-art methods on both single database and cross-database micro-expression recognition.

Micro-Expression (ME) recognition is a hot topic in computer vision as it presents a gateway to capture and understand daily human emotions. It is nonetheless a challenging problem due to ME typically being transient (lasting less than 200 ms) and subtle. Recent advances in machine learning enable new and effective methods to be adopted for solving diverse computer vision tasks. In particular, the use of deep learning techniques on large datasets outperforms classical approaches based on classical machine learning which rely on hand-crafted features. Even though available datasets for spontaneous ME are scarce and much smaller, using off-the-shelf Convolutional Neural Networks (CNNs) still demonstrates satisfactory classification results. However, these networks are intense in terms of memory consumption and computational resources. This poses great challenges when deploying CNN-based solutions in many applications, such as driver monitoring and comprehension recognition in virtual classrooms, which demand fast and accurate recognition. As these networks were initially designed for tasks of different domains, they are over-parameterized and need to be optimized for ME recognition. In this paper, we propose a new network based on the well-known ResNet18 which we optimized for ME classification in two ways. Firstly, we reduced the depth of the network by removing residual layers. Secondly, we introduced a more compact representation of optical flow used as input to the network. We present extensive experiments and demonstrate that the proposed network obtains accuracies comparable to the state-of-the-art methods while significantly reducing the necessary memory space. Our best classification accuracy was 60.17% on the challenging composite dataset containing five objectives classes. Our method takes only 24.6 ms for classifying a ME video clip (less than the occurrence time of the shortest ME which lasts 40 ms). Our CNN design is suitable for real-time embedded applications with limited memory and computing resources.

Micro-expression (ME) is an involuntary, fleeting, and subtle facial expression. It may occur in high-stake situations when people attempt to conceal or suppress their true feelings. Therefore, MEs can provide essential clues to people’s true feelings and have plenty of potential applications, such as national security, clinical diagnosis, and interrogations. In recent years, ME analysis has gained much attention in various fields due to its practical importance, especially automatic ME analysis in computer vision as MEs are difficult to process by naked eyes. In this survey, we provide a comprehensive review of ME development in the field of computer vision, from the ME studies in psychology and early attempts in computer vision to various computational ME analysis methods and future directions. Four main tasks in ME analysis are specifically discussed, including ME spotting, ME recognition, ME action unit detection, and ME generation in terms of the approaches, advance developments, and challenges. Through this survey, readers can understand MEs in both aspects of psychology and computer vision, and apprehend the future research direction in ME analysis.

This study aimed to explore the influence of various mask attributes on the recognition of micro-expressions (happy, neutral, and fear) and facial favorability under different background emotional conditions (happy, neutral, and fear). The participants were asked to complete an ME (micro-expression) recognition task, and the corresponding accuracy (ACC), reaction time (RT), and facial favorability were analyzed. Results: (1) Background emotions significantly impacted the RT and ACC in micro-expression recognition, with fear backgrounds hindering performance. (2) Mask wearing, particularly opaque ones, prolonged the RT but had little effect on the ACC. Transparent masks and non-patterned masks increased facial favorability. (3) There was a significant interaction between background emotions and mask attributes; negative backgrounds amplified the negative effects of masks on recognition speed and favorability, while positive backgrounds mitigated these effects. This study provides insights into how masks influence micro-expression recognition, crucial for future research in this area.

Micro-Expression Recognition (MER) plays a pivotal role in comprehending concealed emotions which aids in identifying mental health issues. Recent advancements in MER use surface electromyography (sEMG) for detecting Micro-expressions (MEs), but is challenging due to low amplitude and its transient properties. In this study, an attempt has been made to classify micro-expressions using nonlinear features of sEMG and random forest. For this purpose, myoelectric signals and facial expressions are collected from eight participants while they are exposed to emotion eliciting videos, which focuses on happiness, sadness, fear, and anger. The signals are obtained from, four muscle regions: Mentalis (MT), Zygomaticus (ZM), Frontalis (FT), and Orbital (OT). These signals are pre-processed and segmented to micro-expressions using Facial Action Unit Coding (FACS) of facial expression. Nonlinear features, namely, Shannon, approximate, sample and permutation entropies are extracted and used to design random forest model. There are 2084 MEs obtained from our measurements: 94 happiness, 396 sadness, 915 fear, and 679 anger. All features except Shannon entropy show significant difference among the emotions. Synthetic Minority Over-sampling Technique is used to address the issue of data imbalance. The proposed model yields a sensitivity of 82%, specificity of 93%, accuracy of 81% and F1 score of 82%. It appears that nonlinear characteristics of sEMG could be useful in spotting micro-expressions.

Micro-expressions (MEs) are brief, involuntary facial expressions that reveal genuine emotions, making their accurate detection crucial in various applications, such as security, psychology, and human-computer interaction. Due to its small intensity and short duration, how accurately capturing the subtle movements of micro-expression is a challenging problem. This paper presents a novel AU prototype-based method for micro-expression spotting, which offers high accuracy and robustness. Action Units (AUs) are basic facial actions, such as brow lower and lip corner puller, that are widely used for micro-expression analysis, and an expression can be encoded as a sequence of AUs. Our approach involves designing AU prototypes that record representative dynamic information of AUs. We then calculate the prototype matching index between AU prototypes and the image sequence to construct time-domain prototype matching curves for ME spotting. In the experimental section, AU prototypes derived from CASMEII dataset enable a more intuitive analysis of AU within micro-expressions. Results on the CAS(ME)2 dataset demonstrate that our ME spotting method significantly outperforms existing approaches. This makes our method highly valuable for various application scenarios, potentially enhancing emotion recognition and analysis in real-world settings.

Encoding facial expressions via Action Units (AUs) has been found effective for resolving the ambiguity issue among different expressions. In the literature, AU detection has extensive researches in macro-expressions. However, there is limited research about AU analysis for micro-expressions (MEs). Micro-expression Action Unit (MEAU) detection becomes a challenging problem because of the subtle facial motion. To alleviate this problem, in this paper, we study the contrastive learning for modeling subtle AUs and propose a novel MEAU detection method by learning the intra- and inter-contrastive information among MEs. Through the intra-contrastive learning module, the difference between the onset and apex frames is enlarged and utilized to obtain the discriminative representation for low-intensity AU detection. In addition, considering the subtle difference between MEAUs, the inter-contrastive learning is designed to automatically explore and enlarge the difference between different AUs to enhance the MEAU detection robustness. Intensive experiments on two widely used ME databases have demonstrated the effectiveness and generalization ability of our proposed method.

When an emotional state is involuntarily and spontaneously delivered with low intensity and short duration, micro expression (ME) occurs. Developing from psychological perspectives to computer vision standpoints, ME has obtained huge advancements and breakthroughs. A variety of feature extraction techniques have been introduced based on different outlooks. With exclusive characteristics of ME, geometric feature learning is involved in approaching the problem in this paper. Specifically, we propose a method where dominant facial regions of interest are extracted and used to further learn spatio-temporal features with space-time auto-correlation of gradients (STACOG) technique. The facial motion features are fed into a multi-layer perceptron network for emotion classification. The combination of ROIs and STACOG captures ME with respect to geometrical property along side with the temporal aspect in three dimensional space. This puts a stress on presumably suboptimal features, making them more salient and resilient for the classification stage. The framework is experimented on three well-known, state-of-the-art spontaneous ME databases CASMEII, SMIC and SAM.

Spot-then-Recognize: A Micro-Expression Analysis Network for Seamless Evaluation of Long Videos

Micro-expressions are brief facial movements characterized by short duration, involuntariness and low intensity. Recognition of spontaneous facial micro-expressions is a great challenge. In this paper, we propose a simple yet effective Main Directional Mean Optical-flow (MDMO) feature for micro-expression recognition. We apply a robust optical flow method on micro-expression video clips and partition the facial area into regions of interest (ROIs) based partially on action units. The MDMO is a ROI-based, normalized statistic feature that considers both local statistic motion information and its spatial location. One of the significant characteristics of MDMO is that its feature dimension is small. The length of a MDMO feature vector is $36 \times 2=72$ , where $36$ is the number of ROIs. Furthermore, to reduce the influence of noise due to head movements, we propose an optical-flow-driven method to align all frames of a micro-expression video clip. Finally, a SVM classifier with the proposed MDMO feature is adopted for micro-expression recognition. Experimental results on three spontaneous micro-expression databases, namely SMIC, CASME and CASME II, show that the MDMO can achieve better performance than two state-of-the-art baseline features, i.e., LBP-TOP and HOOF.

Feature Elimination through Data Complexity for Error-Correcting Output Codes based micro-expression recognition

Facial micro-expressions (MEs) are involuntary facial motions revealing people’s real feelings and play an important role in the early intervention of mental illness, the national security, and many human-computer interaction systems. However, existing micro-expression datasets are limited and usually pose some challenges for training good classifiers. To model the subtle facial muscle motions, we propose a robust micro-expression recognition (MER) framework, namely muscle motion-guided network (MMNet). Specifically, a continuous attention (CA) block is introduced to focus on modeling local subtle muscle motion patterns with little identity information, which is different from most previous methods that directly extract features from complete video frames with much identity information. Besides, we design a position calibration (PC) module based on the vision transformer. By adding the position embeddings of the face generated by the PC module at the end of the two branches, the PC module can help to add position information to facial muscle motion-pattern features for the MER. Extensive experiments on three public micro-expression datasets demonstrate that our approach outperforms state-of-the-art methods by a large margin. Code is available at https://github.com/muse1998/MMNet.

Micro-expressions are rapid and subtle facial movements. Different from ordinary facial expressions in our daily life, micro-expressions are very difficult to detect and recognize. In recent years, due to a wide range of potential applications in many domains, micro-expression recognition has aroused extensive attention from computer vision. Because available micro-expression datasets are very small, deep neural network models with a huge number of parameters are prone to over-fitting. In this article, we propose an OF-PCANet+ method for micro-expression recognition, in which we design a spatiotemporal feature learning strategy based on shallow PCANet+ model, and we incorporate optical flow sequence stacking with the PCANet+ network to learn discriminative spatiotemporal features. We conduct comprehensive experiments on publicly available SMIC and CASME2 datasets. The results show that our lightweight model obviously outperforms popular hand-crafted methods and also achieves comparable performances with deep learning based methods, such as 3D-FCNN and ELRCN.

Micro-expression can reveal underlying genuine emotions, but those rapid and subtle changes are hard to be captured by humans. Most existing research focuses on frontal face micro-expression recognition, which largely prevents the developed methods from the real applications and ignores the underlying geometry information. In this paper, we propose a multiview geometry consistency framework to enable the same emotion to be recognized under different perspectives, which is difficult for existing systems. Based on the developed 3D face reconstruction network, the multi-view micro-expression recognition framework empowers the emotion recognition capability to learn from multiple perspectives of the 3D reconstructed faces based on view-consistency, and a spiking neural network is further applied to capture omitted tiny and detailed changes. With a sequence of images, we explore the subtle changes across frames through optical flow, which, as a clue, enhances the performance of our designated network for micro-expression recognition. Extensive experiments on benchmark micro-expression datasets CAS(ME) 2 and SMIC demonstrate the proposed method achieves promising results on novel-view micro-expression recognition where existing methods mainly fail.