Recognising facial expressions in video sequences

In this paper, a person-specific facial expression recognition method which is based on Personal Facial Expression Space (PFES) is presented. The multidimensional scaling maps facial images as points in lower dimensions in PFES. It reflects personality of facial expressions as it is based on the peak instant of facial expression images of a specific person. In constructing PFES for a person, his/her whole normalized facial image is considered as a single pattern without block segmentation and differences of 2-D DCT coefficients from neutral facial image of the same person are used as features. Therefore, in the early part of the paper, separation characteristics of facial expressions in the frequency domain are analyzed using a still facial image database which consists of neutral, smile, anger, surprise and sadness facial images for each of 60 Japanese males (300 facial images). Results show that facial expression categories are well separated in the low frequency domain. PFES is constructed using multidimensional scaling by taking these low frequency domain of differences of 2-D DCT coefficients as features. On the PFES, trajectory of a facial image sequence of a person can be calculated in real time. Based on this trajectory, facial expressions can be recognized. Experimental results show the effectiveness of this method.© (2000) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.

In this paper, we consider facial expression recognition using an unsupervised learning framework. Specifically, given a data set composed of a number of facial images of the same subject with different facial expressions, the algorithm segments the data set into groups corresponding to different facial expressions. Each facial image can be regarded as a point in a high-dimensional space, and the collection of images of the same subject resides on a manifold within this space. We show that different facial expressions reside on distinct subspaces if the manifold is unfolded. In particular, semi-definite embedding is used to reduce the dimensionality and unfold the manifold of facial images. Next, generalized principal component analysis is used to fit a series of subspaces to the data points and associate each data point to a subspace. Data points that belong to the same subspace are shown to belong to the same facial expression.

In order to change the defects of the similar traditional Retinex algorithm that it is easy to appear the phenomenon in the illumination image acquisition process, we proposed a study on the face images recognition under the illumination condition based on the improved Retinex Algorithm, which is based on the Retinex Theory Algorithm defects and analyzes the brightness, contrast rate of the face images as well as the design of the illumination background in the improved Retinex, and draw the conclusion that this algorithm can overcome the Halo phenomenon to a certain extent when occurs the illumination mutation. In order to verify the effectiveness of the algorithm, we did a simulation to the face images in the Yale B face database finally and studied the comparison test of similar algorithms in different images. The result shows that the improved Retinex algorithm can be a good solution to the Halo problem, reduce the time complexity and can be applied to the face recognition under the condition of strong illumination changes, the recognition rate of which is higher, and the illumination robustness is stronger.

Problem statement: Face occlusion is a very challenging problem in fac e recognition. The performance of face recognition system decreases dr astically due to the presence of partial occlusion on the face. Extracting discriminative features to achieve accurate detection versus computational overhead in extracting the features, which affects the classification speed, would be a sustained problem. The objective of this study is to segment the human face into non-occluded and occluded part of the occluded human face image. In General, for f ace detection special facial features are extracted . In the proposed study a simplified algorithm to ext ract the features is developed. Approach: An algorithm which enables the automatic detection of the presence of occlusions on the face would be a useful tool to increase the performances of the sys tem. The face image was preprocessed to enhance the input face images in order to reduce the loss o f classification performance due to changes in faci al appearance. The experiment also balances both illumination and facial expression changes. Results: In this study, a Mean Based Weight Matrix (MBWM) algorithm has been proposed to enhance the performance by 4.25% than the LBP method. Conclusion: The proposed model has been tested on occluded face images with a dataset obtained from t he MIT face database.

In this paper, we demonstrate the effects of the affect display of anger by virtual facial image synthesis with facial color enhancement and expression on the basis of the analysis of facial color under the emotional change of anger. First, we examine the relationship between the change in the facial color and the physiological index of facial skin temperature, which indicates the change in the blood flux under the emotional change of anger. Second, on the basis of this result, we synthesize 11 enhanced facial color images by using an average facial image of 68 females with their eyes and mouth masked out; we evaluate the effectiveness of these images through a sensory evaluation by performing a paired comparison. Further, we synthesize enhanced facial color images by using the average facial image in which only the eyes were masked out and evaluate the correspondence between the lip color and facial color after performing the enhanced facial color synthesis for an affect display of facial color through a sensory evaluation. Finally, we synthesize the facial expression of anger with enhanced facial color by using the average facial image and evaluate the effectiveness for the affect display of the image through the sensory evaluation of seven-point bipolar ratings.

In this article, we present a local global graph (LGG) method for recognizing facial expressions from static images irrespective of different illumination conditions, shadows and cluttered backgrounds. First, a neural color constancy based skin detection procedure to detect skin in complex real world images is presented. Second, the LGG method for detecting faces and facial expressions with a maximum confidence from skin segmented images is presented. The LGG approach presented here emulates the human visual perception for face and expression detection. In general, humans first extract the most important facial features such as eyes, nose, mouth, etc. and then inter-relate them for face and facial expression representations. The LG Graph embeds both the local information (the shape of facial feature is stored within the local graph at each node) and the global information (the topology of the face). Facial expression recognition from the detected face images is obtained by comparing the LG Expression Graphs with the existing the LG expression models present in the LGG database. Experimental results on the AR database and real-world images suggest the robustness of the proposed approach for facial expression recognition

Facial expression recognition by human observers is affected by subjective components. Indeed there is no ground truth. We have developed Discrete Choice Models (DCM) to capture the human perception of facial expressions. In a first step, the static case is treated, that is modelling perception of facial images. Image information is extracted using a computer vision tool called Active Appearance Model (AAM). DCMs attributes are based on the Facial Action Coding System (FACS), Expression Descriptive Units (EDUs) and outputs of AAM. Some behavioural data have been collected using an Internet survey, where respondents are asked to label facial images from the Cohn– Kanade database with expressions. Different models were estimated by likelihood maximization using the obtained data. In a second step, the proposed static discrete choice framework is extended to the dynamic case, which considers facial video instead of images. The model theory is described and another Internet survey is currently conducted in order to obtain expressions labels on videos. In this second Internet survey, videos come from the Cohn–Kanade database and the Facial Expressions and Emotions Database (FEED).

Over the course of the last decade, infrared (IR) and particularly thermal IR imaging based face recognition has emerged as a promising complement to conventional, visible spectrum based approaches which continue to struggle when applied in the real world. While inherently insensitive to visible spectrum illumination changes, IR images introduce specific challenges of their own, most notably sensitivity to factors which affect facial heat emission patterns, e.g. emotional state, ambient temperature, and alcohol intake. In addition, facial expression and pose changes are more difficult to correct in IR images because they are less rich in high frequency detail which is an important cue for fitting any deformable model. In this paper we describe a novel method which addresses these major challenges. Specifically, to normalize for pose and facial expression changes we generate a synthetic frontal image of a face in a canonical, neutral facial expression from an image of the face in an arbitrary pose and facial expression. This is achieved by piecewise affine warping which follows active appearance model (AAM) fitting. This is the first publication which explores the use of an AAM on thermal IR images; we propose a pre-processing step which enhances detail in thermal images, making AAM convergence faster and more accurate. To overcome the problem of thermal IR image sensitivity to the exact pattern of facial temperature emissions we describe a representation based on reliable anatomical features. In contrast to previous approaches, our representation is not binary; rather, our method accounts for the reliability of the extracted features. This makes the proposed representation much more robust both to pose and scale changes. The effectiveness of the proposed approach is demonstrated on the largest public database of thermal IR images of faces on which it achieved 100% identification rate, significantly outperforming previously described methods.

In this paper, a three-dimensional anisotropic diffusion equation is used to conduct an in-depth study and analysis of students’ concentration in video recognition in English teaching classrooms. A multifeature fusion face live detection method based on diffusion model extracts Diffusion Kernel (DK) features and depth features from diffusion-processed face images, respectively. DK features provide a nonlinear description of the correlation between successive face images and express face image sequences in the temporal dimension; depth features are extracted by a pretrained depth neural network model that can express the complex nonlinear mapping relationships of images and reflect the more abstract implicit information inside face images. To improve the effectiveness of the face image features, the extracted DK features and depth features are fused using a multicore learning method to obtain the best combination and the corresponding weights. The two features complement each other, and the fused features are more discriminative, which provides a strong basis for the live determination of face images. Experiments show that the method has excellent performance and can effectively discriminate the live nature of faces in images and resist forged face attacks. Based on the above face detection and expression recognition algorithms, the classroom concentration analysis system based on expression recognition is designed to achieve real-time acquisition and processing of classroom images, complete student classroom attendance records using face detection and face recognition methods, and analyze students’ concentration from the face integrity and facial expression of students facing the blackboard by combining face detection and expression recognition to visualize and display students’ classroom data for teachers, students, and parents with more data support and help.

A novel facial emotion recognition method for stress inference of facial nerve paralysis patients

Parkinson's disease (PD) is a neurodegenerative disease which is prevalent among the elder population and severely affects the life quality of patients and their families. Therefore, it is important to conduct an early diagnosis for potential patients with PD, so as to promote prompt treatment and avoid the aggravation of the disease. Recently, the in-vitro PD diagnosis based on facial expressions has received increasing attention because of its distinguishability (i.e., PD patients always possess the characteristics of “masked face”) and affordability. However, the performance of the existing facial expression-based PD diagnosis approaches is limited by: 1) the small-scale training data on PD patients' facial expressions, and 2) the weak prediction model. To address these two problems, we propose a new facial expression guided PD diagnosis method based on high-quality training data augmentation and deep neural network prediction. Specifically, the proposed method consists of three stages: Firstly, we synthesize virtual facial expression images with 6 basic emotions (i.e., anger, disgust, fear, happiness, sadness, and surprise) based on multi-domain adversarial learning to approximate the premorbid expressions of PD patients. Secondly, we introduce three facial image quality assessment (FIQA) criteria to measure the quality of these synthesized facial expression images and design a fusion screening strategy that shortlists the high-quality ones to augment the training data. Finally, we train a deep neural network prediction model based on the original and synthesized high-quality facial expression images for PD diagnosis. To show real-world impacts and evaluate the proposed method under different facial expressions, we also create a (currently largest) multiple facial expressions-based PD face dataset in collaboration with a hospital. Extensive experiments are performed to demonstrate the effectiveness of the multi-domain adversarial learning-based facial expression synthesis and the fusion screening strategy, particularly the superior performance of the proposed method for PD diagnosis.

This study was carried out with the aim of exploring the full-convolution Siamese network (SiamFC) in the application of neonatal facial video image tracking, achieving accurate recognition of neonatal pain and helping doctors evaluate neonatal emotions in an automatic manner. The current technology shows low accuracy on facial image recognition of newborns, so the SiamFC algorithm under the deep learning was optimized in this study. Besides, a newborn facial video image tracking model (FVIT model) was constructed based on the SiamFC algorithm in combination with the attention mechanism with face tracking algorithm, and the facial features of newborns were tracked and recognized. In addition, a newborn face database was constructed based on the adult face database to evaluate performance of the FVIT model. It was found that the accuracy of the improved algorithm is 0.889, higher by 0.036 in contrast to other models; the area under the curve (AUC) of success rate reaches 0.748, higher by 0.075 compared with other algorithms. What’s more, the improved algorithm shows good performance in tracking the facial occlusion, facial expression changes, and scale conversion of newborns. Therefore, the improved algorithm shows higher accuracy and success rate and has good effect in capturing and tracking the facial images of newborns, thereby providing an experimental basis for facial recognition and pain assessment of newborns in the later stage.

The purpose of the paper is to study the holistic image transformation methods for person's identity authentication from a thermographic image of faces. Within this study the datasets of images of faces in the far infrared range (LWIR) have been collected. The novelty of the study consists in the features of the dataset of images which collected in real conditions that affect the quality of authentication, such as facial expressions, wearing glasses or a medical mask, applying makeup / cosmetics, different illumination and temperature conditions of the environment, head turns. The methods under the study are based on the construction and selection of image features while reducing the dimension and converting the image into another form of representation. These methods are used to solve the problem of distinguishing features in images, in authentication person identity by 2D image of his face and allows solving other computer vision problems. This paper discusses classical methods of integral image transformation: principal component analysis, principal component analysis using a kernel, linear descriptor analysis, independent component analysis, truncated singular value decomposition, discrete cosine transformation. As a measure of the proximity of images, the Euclidean distance between the vectors of image features is used. Testing of the methods was performed on a set of thermograms consisting of 632 thousand images of the faces of 158 people. The F-measure was used as a metric to assess the quality of comparison of the selected methods. As a result of the experiment, the independent component analysis method showed the highest values of the F-measure metric – 0.72. The results of the study can find applications in access control and control systems to increase the fault tolerance of authentication of persons. The use of the considered methods is effective in the tasks of processing thermographic images for authenticating a person by secondary signs, by the pattern of his veins and vessels on the face, in cases of changes in facial expressions and appearance by means of applying makeup and wearable accessories.

Online education has become increasingly common due to the Covid-19 pandemic.A key difference between online and face-to-face education is that instructors often cannot see their students' facial expressions online.This is problematic because facial expressions can help an instructor gauge engagement and understanding.Therefore, it would be useful to find a mechanism whereby students' facial expressions during an online lecture could be monitored.This information can be used as feedback for teachers to change a particular teaching method or maintain another method.This research presents a system that can automatically distinguish students' facial expressions.These comprise eight expressions (anger, attention, disgust, fear, happiness, neutrality, sadness, and surprise).The data for this research was collected from pictures of 70 university students' facial expressions.The data included 6720 images of students' faces distributed equally among the eight expressions mentioned above, that is, 840 images for each category.In this paper, pre-trained deep learning networks (AlexNet, MobileNetV2, GoogleNet, ResNet18, ResNet50, and VGG16) with transfer learning (TL) and K-fold validation (KFCV) were used for recognizing the facial expressions of students.The experiments were conducted using MATLAB 2021a and the best results were recorded by ResNet18 for F1-score and for AUC curve 99%, and 100% respectively.

An automatic system for facial expression recognition should be able to recognize on-line multiple facial expressions (i.e. “emotional segments”) without interruption. The current paper proposes a new method for the automatic segmentation of “emotional segments” and the dynamic recognition of the corresponding facial expressions in video sequences. First, a new spatial filtering method based on Log-Normal filters is introduced for the analysis of the whole face towards the automatic segmentation of the “emotional segments”. Secondly, a similar filtering-based method is applied to the automatic and precise segmentation of the transient facial features (such as nasal root wrinkles and nasolabial furrows) and the estimation of their orientation. Finally, a dynamic and progressive fusion process of the permanent and transient facial feature deformations is made inside each “emotional segment” for a temporal recognition of the corresponding facial expression. When tested for automatic detection of “emotional segment” in 96 sequences from the MMI and Hammal-Caplier facial expression databases, the proposed method achieved an accuracy of 89%. Tested on 1655 images the automatic detection of transient features achieved a mean precision of 70 % with an error of 2.5 for the estimation of the corresponding orientation. Finally compared to the original model for static facial expression classification, the introduction of transient features and the temporal information increases the precision of the classification of facial expression by 12% and compare favorably to human observers’ performances.