Parallel Scale Invariant Feature Transform Based Approach for Facial Expression Recognition

Purpose Facial expression recognition has gained significance as a means of providing artificial agents with the capacity to discern the emotional states of users. Notable performance has been achieved by deep learning approaches; however, its direct use for recognizing facial expressions in individuals with intellectual disabilities has not yet been studied in the literature, to the best of our knowledge. The aim of this work is to thoroughly explore the applicability of these systems to people with intellectual disabilities. Methods To address this objective, we work with a set of 12 convolutional neural networks (CNNs) in different approaches, including an ensemble of datasets without individuals with intellectual disabilities and a dataset featuring such individuals. Further, we apply the explainable artificial intelligence technique LIME that allows understanding the important regions considered by the CNN. Results Our examination of the outcomes, both the performance and the important image regions for the models, reveals significant distinctions in facial expressions between individuals with and without intellectual disabilities, as well as among individuals with intellectual disabilities. Conclusions Our findings show the need of facial expression recognition within this population through tailored user-specific training methodologies, which enable the models to effectively address the unique expressions of each user.

Facial Expression Recognition (FER) is an important area in human computer interaction. FER has different applications such as analysis of student behaviour in virtual class room, driver mood detection, security systems, and medicine. The analysis of facial expressions is an interesting and exciting problem. Feature extraction plays important role in any FER system. Local Binary Pattern (LBP) and its variants are popular for feature extraction due to simplicity in computation and monotonic illumination invariant property. However, the performance of LBP is poor in the presence of noise. This work proposes a novel approach for feature extraction to improve the performance of the FER. In this approach, the LBP is calculated considering 4-neighbors and diagonal neighbours separately. Further, for affective feature description, the concept of adaptive window and averaging in radial directions is introduced. This approach reduces the length of the feature vector as well as immune to noise. Support Vector Machine (SVM) is considered for classification. Recognition rate and confusion matrix are used to assess the performance of the proposed algorithm. Extensive experimental results on JAFFE, CK, FERG and FEI face databases show significant improvement in recognition rate compared to the available techniques both in noise free and noisy conditions.

In this paper, we consider facial expression recognition using an unsupervised learning framework. Specifically, given a data set composed of a number of facial images of the same subject with different facial expressions, the algorithm segments the data set into groups corresponding to different facial expressions. Each facial image can be regarded as a point in a high-dimensional space, and the collection of images of the same subject resides on a manifold within this space. We show that different facial expressions reside on distinct subspaces if the manifold is unfolded. In particular, semi-definite embedding is used to reduce the dimensionality and unfold the manifold of facial images. Next, generalized principal component analysis is used to fit a series of subspaces to the data points and associate each data point to a subspace. Data points that belong to the same subspace are shown to belong to the same facial expression.

Facial expression is an important form of human nonverbal communication. Recognition of this nonverbal sign may enable developers to understand the feedback of the servers of smart devices or advertising. Existing approaches for facial expression recognition are mainly based on cameras or on-body sensors, which are either sensitive to lighting conditions or require users to wear cumbersome devices on their faces. In this paper, we propose a new facial expression recognition system based on Wi-Fi signals, named WiFace. Our key intuition is that facial muscle movements in different expressions will induce distinctive waveform patterns in the time-series of Channel State Information (CSI) in Wi-Fi signals. We develop a series of algorithms to process the CSI signals and extract the most representative waveform patterns for facial expression classification. We build a fully-functional prototype of WiFace using commercial off-the-shelf devices, which can recognize 6 typical facial expressions. We conduct extensive experiments to evaluate the performance of WiFace, and the experimental results show that the average recognition accuracy is 94.0%.

In this article, we present a local global graph (LGG) method for recognizing facial expressions from static images irrespective of different illumination conditions, shadows and cluttered backgrounds. First, a neural color constancy based skin detection procedure to detect skin in complex real world images is presented. Second, the LGG method for detecting faces and facial expressions with a maximum confidence from skin segmented images is presented. The LGG approach presented here emulates the human visual perception for face and expression detection. In general, humans first extract the most important facial features such as eyes, nose, mouth, etc. and then inter-relate them for face and facial expression representations. The LG Graph embeds both the local information (the shape of facial feature is stored within the local graph at each node) and the global information (the topology of the face). Facial expression recognition from the detected face images is obtained by comparing the LG Expression Graphs with the existing the LG expression models present in the LGG database. Experimental results on the AR database and real-world images suggest the robustness of the proposed approach for facial expression recognition

Facial expressions are important in people’s daily communications. Recognising facial expressions also has many important applications in the areas such as healthcare and e-learning. Existing facial expression recognition systems have problems such as background interference. Furthermore, systems using traditional approaches like SVM (Support Vector Machine) have weakness in dealing with unseen images. Systems using deep neural network have problems such as requirement for GPU, longer training time and requirement for large memory. To overcome the shortcomings of pure deep neural network and traditional facial recognition approaches, this paper presents a new facial expression recognition approach which has image pre-processing techniques to remove unnecessary background information and combines deep neural network ResNet50 and a traditional classifier-- the multiclass model for Support Vector Machine to recognise facial expressions. The proposed approach has better recognition accuracy than traditional approaches like Support Vector Machine and doesn’t need GPU. We have compared 3 proposed frameworks with a traditional SVM approach against the Karolinska Directed Emotional Faces (KDEF) Database, the Japanese Female Facial Expression (JAFFE) Database and the extended Cohn-Kanade dataset (CK+), respectively. The experiment results show that the features extracted from the layer 49Relu have the best performance for these three datasets.

Facial expression has an important role in human interaction and non-verbal communication. Hence more and more applications, which automatically detect facial expressions, start to be pervasive in various fields, such as education, entertainment, psychology, human-computer interaction, behavior monitoring, just to cite a few. In this paper, we present a new approach for facial expression recognition using a so-called deep conspicuous neural network. The proposed method builds a conspicuous map of region faces, training it via a deep network. Experimental results achieved an average accuracy of 90% over the extended Cohn-Kanade data set for seven basic expressions, demonstrating the best performance against four state-of-the-art methods.

Facial feature extraction and recognition plays a prominent role in human non-verbal interaction and it is one of the crucial factors among pose, speech, facial expression, behaviour and actions which are used in conveying information about the intentions and emotions of a human being. In this article an extended local binary pattern is used for the feature extraction process and a principal component analysis (PCA) is used for dimensionality reduction. The projections of the sample and model images are calculated and compared by Euclidean distance method. The combination of extended local binary pattern and PCA (ELBP+PCA) improves the accuracy of the recognition rate and also diminishes the evaluation complexity. The evaluation of proposed facial expression recognition approach will focus on the performance of the recognition rate. A series of tests are performed for the validation of algorithms and to compare the accuracy of the methods on the JAFFE, Extended Cohn-Kanade images database.

Real Time Facial Expression Recognition using Deep Learning

For the communication, there are two modes of communication i.e. verbal and non-verbal. The speech is the medium of verbal communication whenever the gestures and the facial expressions are the medium of non-verbal communication. In our daily social life, the facial expressions play an important role to convey some information between the speaker and the listener. The facial expressions can not only change the flow of conversation but also provide some contextual information to the listener, therefore, the listener can easily sense the actual state of the convener. Facial expression express human's inner emotional states. In this paper, an approach has been proposed to detect the specific region of the face which consist of very high information for emotion recognition. In this work, the approach is to detect some particular region of the face instead to detect all facial parts. Therefore this approach is faster than the existing approaches for facial expression recognition. For the classification, a fusion approach has been used and achieved a good recognition rate with the combination of HOG and LBP features.

We introduce a system that processes a sequence of images of a front-facing human face and recognises a set of facial expressions. We use an efficient appearance-based face tracker to locate the face in the image sequence and estimate the deformation of its non-rigid components. The tracker works in real time. It is robust to strong illumination changes and factors out changes in appearance caused by illumination from changes due to face deformation. We adopt a model-based approach for facial expression recognition. In our model, an image of a face is represented by a point in a deformation space. The variability of the classes of images associated with facial expressions is represented by a set of samples which model a low-dimensional manifold in the space of deformations. We introduce a probabilistic procedure based on a nearest-neighbour approach to combine the information provided by the incoming image sequence with the prior information stored in the expression manifold to compute a posterior probability associated with a facial expression. In the experiments conducted we show that this system is able to work in an unconstrained environment with strong changes in illumination and face location. It achieves an 89% recognition rate in a set of 333 sequences from the Cohn–Kanade database.

Creating innovative applications of ambient intelligence for smart environments became a real challenge. Indeed, the existing systems are considered as being obtrusive. Therefore, the ongoing trends consist in finding new methods which are less intrusive while being accurate and computer efficient. To that end, we introduce an ingenious way to assist the elderly and people with cognitive impairment in their daily life using emotions through facial expressions. Our hypothesis assumes the correlation between expressed emotions and the resulting errors in daily activities. In order to verify it, we introduce an experimentation protocol with neurotypical subjects. In this paper, we mainly focus on designing an approach for facial expression recognition using Convolutional Neural Network. The carried experiments with benchmark datasets provided promising results. In fact, the proposed approach outperforms the existing methods in terms of accuracy. Moreover, our approach has been implemented in an embedded system with limited resources. The main objective of this paper is to provide an effective approach recognizing facial expressions that will be used for our further experimentations.

We propose a self-supervised contrastive learning approach for facial expression recognition (FER) in videos. We propose a novel temporal sampling-based augmentation scheme to be utilized in addition to standard spatial augmentations used for contrastive learning. Our proposed temporal augmentation scheme randomly picks from one of three temporal sampling techniques: (1) pure random sampling, (2) uniform sampling, and (3) sequential sampling. This is followed by a combination of up to three standard spatial augmentations. We then use a deep R(2+1)D network for FER, which we train in a self-supervised fashion based on the augmentations and subsequently fine-tune. Experiments are performed on the Oulu-CASIA dataset and the performance is compared to other works in FER. The results indicate that our method achieves an accuracy of 89.4%, setting a new state-of-the-art by outperforming other works. Additional experiments and analysis confirm the considerable contribution of the proposed temporal augmentation versus the existing spatial ones.

An efficient CNN approach for facial expression recognition with some measures of overfitting

Facial expression recognition plays a vital role in various domains, including human-computer interaction, affective computing, and psychological research. However, achieving robust performance across different datasets remains a significant challenge due to variations in image quality, subject demographics, and annotation protocols. This research paper presents a novel technique for facial expression recognition based on Convolutional Neural Networks (CNN) that achieves significant results on cross datasets. By employing transfer learning and database-independent training strategies, the proposed method is trained on the FER-2013 dataset and demonstrates good performance on FERG-DB, CK+, FER-2013, and JAFFE datasets, respectively, of 92.05%, 89.93%, 72.16% and 85.71%. Experimental results show the effectiveness of the technique in achieving database independence and satisfactory performance on diverse datasets.