An adaptive training based on classification system for patterns in facial expressions using SURF descriptor templates

We proposed a facial motion tracking and expression recognition system based on video data. By a 3D deformable facial model, the online statistical model (OSM) and cylinder head model (CHM) were combined to track 3D facial motion in the framework of particle filtering. For facial expression recognition, a fast and efficient algorithm and a robust and precise algorithm were developed. With the first, facial animation and facial expression were retrieved sequentially. After that facial animation was obtained, facial expression was recognized by static facial expression knowledge learned from anatomical analysis. With the second, facial animation and facial expression were simultaneously retrieved to increase the reliability and robustness with noisy input data. Facial expression was recognized by fusing static and dynamic facial expression knowledge, the latter of which was learned by training a multi-class expressional Markov process using a video database. The experiments showed that facial motion tracking by OSM+CHM is more pose robust than that by OSM, and the facial expression score of the robust and precise algorithm is higher than those of other state-of-the-art facial expression recognition methods.

Automatic recognition of facial expression remains a challenging task in computer vision, because of large amount of image data and subtle difference in facial expressions. In this paper, we present a customizable system for facial expression recognition using non-negative matrix factorization (NMF).Our system can be described in two major phases: the first phase uses preprocessing and the NMF for decomposing facial expression images into a set of basis images; the second phase incorporates dimensionality reduction and similarity to classify if an input image depicts one of the five facial expressions (i.e., normal, smile, sad, surprise, and angry) given known facial expressions as template images. Our results clearly demonstrate that the system was able to provide an effective basis for characterizing facial feature regions and reasonable classification in various facial expressions.

Human neuroimaging and behavioural studies suggest that somatomotor ‘mirroring’ of seen facial expressions may support their recognition. Here we show that viewing specific facial expressions triggers the representation corresponding to that expression in the observer’s brain. Twelve healthy female volunteers underwent two separate fMRI sessions: one where they observed and another where they displayed three types of facial expressions (joy, anger and disgust). Pattern classifier based on Bayesian logistic regression was trained to classify facial expressions (i) within modality (trained and tested with data recorded while observing or displaying expressions) and (ii) between modalities (trained with data recorded while displaying expressions and tested with data recorded while observing the expressions). Cross-modal classification was performed in two ways: with and without functional realignment of the data across observing/displaying conditions. All expressions could be accurately classified within and also across modalities. Brain regions contributing most to cross-modal classification accuracy included primary motor and somatosensory cortices. Functional realignment led to only minor increases in cross-modal classification accuracy for most of the examined ROIs. Substantial improvement was observed in the occipito-ventral components of the core system for facial expression recognition. Altogether these results support the embodied emotion recognition model and show that expression-specific somatomotor neural signatures could support facial expression recognition.

Most Facial Expression Recognition (FER) systems rely on machine learning approaches that require large databases for an effective training. As these are not easily available, a good solution is to augment the databases with appropriate data augmentation (DA) techniques, which are typically based on either geometric transformation or oversampling augmentations (e.g., generative adversarial networks (GANs)). However, it is not always easy to understand which DA technique may be more convenient for FER systems because most state-of-the-art experiments use different settings which makes the impact of DA techniques not comparable. To advance in this respect, in this paper, we evaluate and compare the impact of using well-established DA techniques on the emotion recognition accuracy of a FER system based on the well-known VGG16 convolutional neural network (CNN). In particular, we consider both geometric transformations and GAN to increase the amount of training images. We performed cross-database evaluations: training with the "augmented" KDEF database and testing with two different databases (CK+ and ExpW). The best results were obtained combining horizontal reflection, translation and GAN, bringing an accuracy increase of approximately 30%. This outperforms alternative approaches, except for the one technique that could however rely on a quite bigger database.

The digital animation process is a complex endeavour requiring professional animators to acquire substantial expertise and technique through years of study and practice. Particularly, facial animation, where virtual humans express specific mental states or emotions with a desire for realism, is further complicated by the “Uncanny Valley” phenomenon. In this context, it is posited that pre-validated facial expressions for certain emotions could serve as references for the novice or inexperienced animators during the facial animation and posing process of their virtual humans using morph targets, also known as blend shapes or shape keys. This research presents a comparative study between two Facial Expression Recognition (FER) systems that employ pre-trained models for facial recognition applied to emotion recognition in virtual humans. Given that these systems were not designed or trained for this particular purpose but for facial recognition in real humans, this study aims to investigate their level of applicability in scenarios where virtual humans are used instead of real humans. This assessment is a critical step towards evaluating the feasibility of integrating FER models as part of a support tool for facial animation and the posing of virtual humans. Through this investigation, this research provides evidence of the reliability of applying the FER library and Deepface systems for emotion recognition in virtual humans, contributing to investigating new ways to enhance the digital animation process and overcoming the inherent complexities of facial animation.

An automatic system for facial expression recognition should be able to recognize on-line multiple facial expressions (i.e. “emotional segments”) without interruption. The current paper proposes a new method for the automatic segmentation of “emotional segments” and the dynamic recognition of the corresponding facial expressions in video sequences. First, a new spatial filtering method based on Log-Normal filters is introduced for the analysis of the whole face towards the automatic segmentation of the “emotional segments”. Secondly, a similar filtering-based method is applied to the automatic and precise segmentation of the transient facial features (such as nasal root wrinkles and nasolabial furrows) and the estimation of their orientation. Finally, a dynamic and progressive fusion process of the permanent and transient facial feature deformations is made inside each “emotional segment” for a temporal recognition of the corresponding facial expression. When tested for automatic detection of “emotional segment” in 96 sequences from the MMI and Hammal-Caplier facial expression databases, the proposed method achieved an accuracy of 89%. Tested on 1655 images the automatic detection of transient features achieved a mean precision of 70 % with an error of 2.5 for the estimation of the corresponding orientation. Finally compared to the original model for static facial expression classification, the introduction of transient features and the temporal information increases the precision of the classification of facial expression by 12% and compare favorably to human observers’ performances.

Facial expressions can be seen as a form of non-verbal communication as well as a primary means of conveying social information among humans.Automatic facial expression recognition (FER) can be applied to a wide range of scenarios in human-computer interaction, facial animation, entertainment, and psychology studies. For feature representation in a FER system, various texture descriptors have been employed to derive an effective solution for this system. However, these individual texture descriptor-based FER systems have often failed to achieve effective performance in the recognition of facial expressions. In this sense, it is necessary to further improve the general performance of a facial expression recognition system, evaluating different feature representations. In this paper, a novel local descriptor for a facial expression recognition system is proposed, designated the level of difference descriptor (LOD). The main goal is to use this descriptor as a supplement to state-of-the-art local descriptors to further improve the performance of a FER system in terms of classification accuracy. Furthermore, the fusion of various texture features for devising a robust feature representation for multi-view facial expression recognition is presented.

Facial expressions convey nonverbal cues which play an important role in interpersonal relations, and are widely used in behavior interpretation of emotions, cognitive science, and social interactions. In this paper we analyze different ways of representing geometric feature and present a fully automatic facial expression recognition (FER) system using salient geometric features. In geometric feature-based FER approach, the first important step is to initialize and track dense set of facial points as the expression evolves over time in consecutive frames. In the proposed system, facial points are initialized using elastic bunch graph matching (EBGM) algorithm and tracking is performed using Kanade-Lucas-Tomaci (KLT) tracker. We extract geometric features from point, line and triangle composed of tracking results of facial points. The most discriminative line and triangle features are extracted using feature selective multi-class AdaBoost with the help of extreme learning machine (ELM) classification. Finally the geometric features for FER are extracted from the boosted line, and triangles composed of facial points. The recognition accuracy using features from point, line and triangle are analyzed independently. The performance of the proposed FER system is evaluated on three different data sets: namely CK+, MMI and MUG facial expression data sets.

A human face has a distinct and unique characteristics which make it play a very critical role in recognizing facial expression in a “facial expression recognition system.” Identifying or as we say it detection of expressions plays a big and significant role in a facial expression recognition system. If we talk about a human being it becomes an easy task recognize expression in any particular image sequence, but at the same time if we talk about fully automated systems not many are currently available or capable to do so. The field of facial expression recognition do have many different applications and its importance, it might be used to have an interaction between a human being and a computer, here a user, without using his hands, can give commands or instruct to the computer system with the help of facial expression recognition system. Quite a few options are available to identify a face in an image in an efficient and accurate manner, although similar cannot be said for features detection in a video sequence frame. Most systems are still dependent on manual operations for same. Here in this paper we have emphasizes on color normalization and facial feature extraction which uses LBP (Local Binary Pattern) as an effective feature detection approach, where the existing algorithms have been modified to improve the facial expression recognition accuracy. The recognition accuracy on the Indian database is observe to be 94.7\%.

In this paper, Facial Expression Recognition (FER) system, aid in modeling facial expression space for facial expression recognition by facial characteristic points and Gini index. Facial expression information are mostly concentrated on facial expression information regions, so the, eyes, eyebrows, and mouth are extracted from the input image. When a face image is input the feature extraction is performed which help in detecting facial characteristic points. Facial animation parameters are calculated in order to recognize one of the six basic facial expressions. The proposed technique is applied to JAFFE database consisting 30 images, each having six basic facial expression images (neutral, happy, surprise, fear, sad and angry). For certain expressions like, surprise, fear and happy it gives the promising results with best recognition rate, however for expressions like, neutral, sad and angry the recognition rate is lower. The proposed technique is implemented in MATLAB version 7.8.0.347 (R2009a).

Recently, facial expression recognition has attracted much attention in machine vision research because of its various applications. Accordingly, many facial expression recognition systems have been proposed. However, the majority of existing systems suffer from a critical problem: geometric variability. It directly affects the performance of geometric feature-based facial expression recognition approaches. Furthermore, it is a crucial challenge in appearance feature-based techniques. This variability appears in both neutral faces and facial expressions. Appropriate face geometry normalization can improve the accuracy of each facial expression recognition system. Therefore, this paper proposes different geometric models or shapes for normalization. Face geometry normalization removes geometric variability of facial images and consequently, appearance feature extraction methods can be accurately utilized to represent facial images. Thus, some expression-based geometric models are proposed for facial image normalization. Next, local binary patterns and local phase quantization are used for appearance feature extraction. A combination of an effective geometric normalization with accurate appearance representations results in more than a 4% accuracy improvement compared to several state-of-the-arts in facial expression recognition. Moreover, utilizing the model of facial expressions which have larger mouth and eye region sizes gives higher accuracy due to the importance of these regions in facial expression.

Single‐mode sensors suffer from poor robustness and insufficient data features in facial expression recognition, so fusing multi‐sensor signals is the key to improving the accuracy of expression recognition systems. Here, a biocompatible capacitive‐electromyographic dual‐mode sensor (CEDS) is presented, consisting of a capacitive pressure sensing unit and dry electrodes for electrophysiological signal monitoring, assembled in a 3D stacking fashion. A double‐coupled microstructure is prepared and the electrical double‐layer effect is realized by doping ionic liquid, which significantly improves the capacitive performance of the sensor. The application of dry electrodes effectively solves the problems of hydrogel electrodes that are prone to water loss and skin irritation. Besides, the good biocompatibility and antimicrobial properties of CEDS are verified through cytotoxicity and bacteriostatic tests. Based on the sensing of a single signal, a fatigue driving monitoring system and a manipulator control system are constructed respectively. By further integrating the capacitive and electrophysiological signal monitoring functions of CEDS, a 1D convolutional neural network‐assisted facial expression recognition system is constructed, which effectively improves the accuracy of expression recognition and demonstrates the great potential of facial expression monitoring systems based on flexible sensor technology in practical applications.

Facial expression recognition (FER) systems must ultimately work on real data in uncontrolled environments although most research studies have been conducted on lab-based data with posed or evoked facial expressions obtained in pre-set laboratory environments. It is very difficult to obtain data in real-world situations because privacy laws prevent unauthorized capture and use of video from events such as funerals, birthday parties, marriages etc. It is a challenge to acquire such data on a scale large enough for benchmarking algorithms. Although video obtained from TV or movies or postings on the World Wide Web may also contain ‘acted’ emotions and facial expressions, they may be more ‘realistic’ than lab-based data currently used by most researchers. Or is it? One way of testing this is to compare feature distributions and FER performance. This paper describes a database that has been collected from television broadcasts and the World Wide Web containing a range of environmental and facial variations expected in real conditions and uses it to answer this question. A fully automatic system that uses a fusion based approach for FER on such data is introduced for performance evaluation. Performance improvements arising from the fusion of point-based texture and geometry features, and the robustness to image scale variations are experimentally evaluated on this image and video dataset. Differences in FER performance between lab-based and realistic data, between different feature sets, and between different train-test data splits are investigated.

This research paper proposes a novel method for estimating food acceptance from real-time food consumption video streaming under partially occluded facial expressions. The facial expressions are evaluated based on the Facial Expression Recognition (FER) system. Here, the occlusion is identified as a spoon, fork, or hand. And the Object Detection API from TensorFlow is used as an occlusion detection tool. The combination of the Exponential Decay and the Bayes' theorem (called EB algorithm) is used to estimate the probabilities of food acceptance from the occluded facial expressions. The EB algorithm also calculates the facial reactions towards food tastes such as bitterness, sourness, sweetness, umami, and saltiness to predict the likelihood of food acceptance. The simulations and the customer comparison results indicate that the presented food acceptance system is one of the accurate ways to signify food acceptance in a real-time food environment.

Facial expression recognition is a very important research field to understand human emotions. Many facial expression recognition systems have been proposed in the literature over the years. Some of these methods use neural network approaches with deep architectures to address the problem. Although it seems that the facial expression recognition problem has been solved, there is a large difference between the results achieved using the same database to train and test the network and the cross-database protocol. In this paper, we extensively investigate the performance influence of fine-tuning with cross-database approach. In order to perform the study, the VGG-Face Deep Convolutional Network model (pre-trained for face recognition) was fine-tuned to recognize facial expressions considering different well-established databases in the literature: CK+, JAFFE, MMI, RaFD, KDEF, BU3DFE, and AR Face. The cross-database experiments were organized so that one of the databases was separated as test set and the others as training, and each experiment was ran multiple times to ensure the results. Our results show a significant improvement on the use of pre-trained models against randomly initialized Convolutional Neural Networks on the facial expression recognition problem, for example achieving 88.58%, 67.03%, 85.97%, and 72.55% average accuracy testing in the CK+, MMI, RaFD, and KDEF, respectively. Additionally, in absolute terms, the results show an improvement in the literature for cross-database facial expression recognition with the use of pre-trained models.